Oh look, banks are being hacked by AI

Math Babe
mathbabe.org
2026-10-06 10:26:35
This is just the beginning. From the WSJ:...
Original Article
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

AI Skeptics: How Schools Procure AI Tools (with J.B. Branch)

Math Babe
mathbabe.org
2026-10-05 09:11:24
We were joined this week by J.B. Branch, the Director of Federal AI Governance and Technology Policy for Public Citizen’s Congress Watch division: Apple Spotify YouTube...
Original Article

Home > Uncategorized > AI Skeptics: How Schools Procure AI Tools (with J.B. Branch)

We were joined this week by J.B. Branch , the Director of Federal AI Governance and Technology Policy for Public Citizen’s Congress Watch division:

Apple

Spotify

YouTube

Categories: Uncategorized

Comments (0) Trackbacks (0) Leave a comment Trackback

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Your email address will not be published. Required fields are marked *

Ninja Forms plugin flaw exploited to hack WordPress sites

Bleeping Computer
www.bleepingcomputer.com
2026-10-06 17:00:27
Hackers are exploiting stored cross-site scripting (XSS) vulnerabilities in two unrelated WordPress plugins, Ninja Forms and WPC Product Bundles for WooCommerce, to install backdoors and create rogue admin accounts. [...]...
Original Article

Ninja Forms plugin flaw exploited to hack WordPress sites

Hackers are exploiting stored cross-site scripting (XSS) vulnerabilities in two unrelated WordPress plugins, Ninja Forms and WPC Product Bundles for WooCommerce, to install backdoors and create rogue admin accounts.

Both vulnerabilities received a high severity score and require an authenticated session to exploit. They are tracked as CVE-2026-93836, affecting WPC Product Bundles for WooCommerce versions 8.6.6 and older, and CVE-2026-94504, affecting Ninja Forms versions 3.15.3 and older.

The Ninja Forms plugin for WordPress is installed on more than 500,000 sites and allows creating custom forms without writing code.

WPC Product Bundles for WooCommerce allows storing group products into bundles and is active on more than 30,000 WordPress sites.

The campaign was identified on October 4 by researchers at WordPress security platform Patchstack, against users of WPC Product Bundles for WooCommerce. The next day, the same activity was observed against Ninja Forms.

In both attacks, the same JavaScript payload was delivered from ‘imgcdn1[.]com,’ indicating the same threat actor behind the exploitation attempts against the two plugins.

According to the researchers, the attacker tries to plant malicious JavaScript ( x.js ) in WooCommerce order data or Ninja Forms submissions. When a logged-in administrator loads the content, the script executes using the authenticated WordPress session.

When launched, it retrieves the necessary administrative nonces and uses legitimate WordPress functions to install a malicious plugin masquerading as “WP Smart Thumbnails” version 1.2.4 from “MediaPress Labs” and create an administrator account.

At that stage, the JavaScript payload and the malicious plugin’s PHP scripts establish four access mechanisms to the compromised site:

  1. A visible administrator account
  2. An administrator account concealed from the WordPress user list in the dashboard
  3. A secret login URL that authenticates as the site’s oldest existing administrator
  4. An unauthenticated file manager accessible through a direct request to the malicious plugin’s main PHP file

The file manager can't execute commands, but it could still be used to introduce additional payloads on the site.

Even if the WP Smart Thumbnails plugin is removed from the infected website, the hidden account and secret login URL continue to function as persistence mechanisms through separate auxiliary attack plugins featuring backdated timestamps to evade detection.

“The [hidden] account does not appear in Users → All Users, does not appear in the Administrator filter, and is not counted in the totals above the list,” Patchstack explains , adding that “It is a fully privileged administrator the site owner cannot see.”

Patchstack says that exploitation is currently limited, but advises site admins to upgrade to the latest versions of the affected plugins, WPC Product Bundles for WooCommerce version 8.6.7 or later and Ninja Forms 3.15.4 or later.

Updating the vulnerable plugin prevents further exploitation but does not clean an existing infection. Administrators are strongly recommended to check for signs of compromise.

article image

Build your security blueprint for AI-powered attacks

Join Mikko Hyppönen and security leaders from the NFL, CHANEL, and Atlassian for a two-hour digital summit on what AI-speed attacks change, what defenders should stop doing, and how to validate, decide, fix, and re-validate at machine speed.

Save your seat

Steinar H. Gunderson: Decompilation patterns, part 3: do-while-while

PlanetDebian
blog.sesse.net
2026-10-06 16:45:18
Today's decompilation pattern is a variation on yesterday's loop pattern, because loops are important and tend to induce a lot of transformations by the compiler. So assume you have code like this (perhaps after converting a goto): if (var_s0 > var_s1) { // Perhaps some other initialization st...
Original Article

Today's decompilation pattern is a variation on yesterday's loop pattern, because loops are important and tend to induce a lot of transformations by the compiler. So assume you have code like this (perhaps after converting a goto): 

if (var_s0 > var_s1) {
  // Perhaps some other initialization stuff here
  do {
    ...
  } while (var_s0 > var_s1);
}

Assuming there are no side effects from the initializations, a natural candidate would be that the programmer instead wrote: 

// Initialization stuff moved up here
while (var_s0 > var_s1) {
  ...
}

Remember to remove the old while so that it does not say while (x) { ... } while (x) ; it won't tell you with a syntax error, but rather leave an infinite loop that will mess up the generated code.

Tumbler Ridge shooter's ChatGPT use "more disturbing" than known: reporter

Hacker News
www.cbc.ca
2026-10-06 16:44:29
Comments...
Original Article

Mother Jones magazine has published new details outlining how the shooter in the Tumbler Ridge tragedy allegedly used OpenAI's ChatGPT program to both plan the attack, and evade OpenAI's own security protocols.

The report was written by the magazine's national affairs editor Mark Follman, who has been investigating the connection between AI chatbots and violent attacks both in the United States and elsewhere.

Follman says through his reporting he was able to review "significant portions" of the chat history between shooter Jesse Van Rootselaar and OpenAI's ChatGPT chatbot in the time leading up to the attack, some of which is detailed in his reporting.

"I can say that what's in them is far more disturbing and extensive than I think certainly the public has been aware of until this point," he said in an interview with CBC News.

WATCH | Mother Jones journalist describes investigation into chat logs:

Tumbler Ridge shooter's use of AI 'far more disturbing' than previously known, says Mother Jones journalist

On Feb. 10, Van Rootselaar shot and killed her mother and half-brother at their family home. She then went to Tumbler Ridge Secondary School, where she shot and killed five students aged 12 to 13 and a 39-year-old education assistant, before killing herself.

The shooting has left a deep scar on the tight-knit community of about 2,400, in the foothills of northeast B.C.'s Rocky Mountains.

OpenAI has already admitted that Van Rootselaar's activity was flagged internally and shut down, as first reported by the Wall Street Journal .

It has also apologized for failing to notify authorities in Canada about the activity, and later admitted it had failed to notice a second account created by Van Rootselaar after the first was disabled.

But Follman says the logs he reviewed reveal far more details about how Van Rootselaar used that second account to get information about violent scenarios by framing them as hypothetical or fictional situations.

The magazine report says the alleged interactions with the chatbot vividly describe the effectiveness of a shotgun in a classroom. CBC News has not reviewed the chat logs detailed in the report.

"In a hallway, indoors, or a crowded classroom, the 870 [Remington shotgun] is brutal," it quotes ChatGPT as saying. "Close quarters is its playground."

A man with his head bowed at a makeshift memorial with bouquets of flowers on the ground around a tree.

Victims of the Tumbler Ridge shooting were memorialized in the days after the tragedy. (Ben Nelms/CBC)

The article says ChatGPT went on to say that "assuming each shell results in one hit, you might down 10 to 20 people max, depending on spacing, density, reaction times, and chaos."

"To have a chatbot engage in this way was, I think, quite astonishing," Follman said.

"It's very troubling, because this is a person who has been already talking, for quite a while, about some very dark and disturbing things."

Follman said the company has not responded to his multiple requests for comment in the leadup to his article's publication.

Multiple requests from CBC News on Friday also went unanswered.

Community response

Shelley Quist, whose son was in Tumbler Ridge Secondary when the shooting happened, called the report "disturbing" and said it raised even more questions about the technology is being used.

She said it was difficult for the community to regularly receive new details about the attack, and said she'd been diagnosed with PTSD in the aftermath.

"I'm always thinking of, 'What's coming next, what's going to come out?'" she said.

Of OpenAI, she said, "they had a chance to stop this, and they didn't."

A teenage boy is pictured with his mother.

Tumbler Ridge Secondary School student Darian Quist and his mother, Shelley Quist, in an interview after a shooting at the school on Feb. 10, 2026. (The National)

Multiple community members, including some teachers and the families of children at the school, have filed lawsuits against OpenAI . The B.C. government is also launching its own lawsuit against the company.

The District of Tumbler Ridge declined to comment on the new report, but shared a post on its Facebook page reminding community members of the mental health supports that have been made available for anyone struggling.

"We know difficult information or conversations can bring up a range of emotions, especially for those affected by the tragedy experienced in February," the post says.

"It is also okay to set boundaries around the information you take in. If you know that reading news coverage, comments or social media discussions about [Feb.] 10 may negatively affect your wellbeing, consider stepping away or choosing not to engage."

Politicians react

Following the report, British Columbia's attorney general says she has asked the federal government to change the Criminal Code to ensure accountability for artificial intelligence companies.

Niki Sharma said Friday that she had not received a response because of the "rapidly evolving" situation but the changes need to "ensure a clearer pathway for human and corporate accountability for the actions of AI."

WATCH | B.C. suing OpenAI for alleged role in Tumbelr Ridge shootings:

B.C. sues OpenAI over alleged role in Tumbler Ridge mass shooting

The B.C. government is suing OpenAI over its alleged role in the Tumbler Ridge mass shooting, accusing the company of failing to alert police to threats made on ChatGPT before the attack.

"I think if we do the right changes to the Criminal Code, it will change the behaviour of AI companies, because there will be a clear line to accountability," she said at an unrelated news conference.

Sharma, who said she was speaking in her capacity as a candidate in the B.C. election, said she was "not the decision maker with respect to the criminal charges or if there are to be any that are coming."

"The RCMP will make those decisions under the investigation," she said.

NDP Leader David Eby said during a campaign stop in the provincial election that the allegations are "shocking" and the information must be hurtful for families of the victims.

"They need to be regulated, and frankly, if the allegations in that media reporting are true, then they need to be criminally charged, because there is nobody who should be permitted to engage in the kind of conduct that OpenAI has," Eby said.

Federal AI Minister Evan Solomon’s office said in a statement that Solomon "was not previously aware of the specific conversations detailed in the Mother Jones report."

A man in a suit speaks at a podium, stood under a fancy hanging light.

The office of Minister of Artificial Intelligence and Digital Innovation Evan Solomon said the news report 'raises serious questions about how OpenAI identified and responded to warning signs.' (Rich Lam/The Canadian Press)

Solomon's office says in its statement that while OpenAI has taken measures since the February shooting, it’s too soon to say if the company has done enough.

"What has been reported raises serious questions about how OpenAI identified and responded to warning signs," the statement said.

"Following the tragedy, the minister summoned OpenAI officials to Ottawa and demanded stronger safeguards and clearer pathways for escalating credible threats," it added.

OpenAI is now in direct contact with the RCMP’s cybercrime centre and has undertaken a review of past cases and strengthened its processes for assessing threats, the statement said.

"Those actions matter, but it would be premature to say they are sufficient," it said.

The statement added the Canadian AI Safety Institute has an ongoing review of OpenAI's practices and that the investigation remains active.

Gun control group demands answers

The gun control group PolySeSouvient said in a statement issued Friday that the Mother Jones report raises a troubling question: if police had been alerted to the shooter’s first or second ChatGPT account, would they have taken different steps?

That question and what happened with the warning signs need to be answered, the group said.

"Firearms had been removed from the home and later returned," the statement said.

"Canadians need to know what happened between those earlier interventions and the killings, what risks were assessed, what decisions were made about access to firearms, and whether other measures were available."

It said both OpenAI's conduct and the firearms oversight need to be reviewed to identify missed opportunities to prevent another mass shooting.

RCMP Staff Sgt. Kris Clark said in a statement late Thursday that the investigation into the shootings had "utilized all available legal processes to obtain all information available from various social media and digital platforms" relevant to the attack.

"To date, all companies that have received requests have complied," Clark said.

The article published says Van Rootselaar's ChatGPT account was banned for discussing committing an attack with a "famous real-world mall as the target."

But it says she then used a second account, which advised her about how to "never get flagged again” by describing situations as fictional or hypothetical.

"Don’t use real-world locations," the article quotes ChatGPT as saying, before adding: “You can still be twisted. Just be clever about it."

WATCH | News report details ChatGPT logs:

ChatGPT told Tumbler Ridge shooter how to bypass safeguards: news report

OpenSSH 10.6 Released

Hacker News
www.openssh.org
2026-10-06 16:41:39
Comments...
Original Article

OpenSSH 10.6 / 10.6p1 (2026-10-06) 

OpenSSH 10.6 was released on 2026-10-06. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Recently the OpenSSH team have received a large number of security
bug reports, many of which are findings from AI models or made with
AI assistance. While many AI reports are determined not to have
security impact when considered in the context of a realistic
threat model, we very much welcome these reports, especially when
combined with human triage, analysis, test-cases and particularly
when accompanied by proposed fixes.

** We have seen a number of cases where a security bug identified
** by AI tools is subsequently independently discovered by a
** different researcher. This suggests that adversaries who do not
** report bugs to OSS projects are likely to be able to discover
** these bugs too. Given this, the OpenSSH team will, for now, be
** making more frequent releases to get bugfixes into users' hands
** more quickly rather than batching them until the next planned
** release.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Future deprecation notice
-------------------------

 * scp(1): begin deprecating the -R flag, which is used to perform a
   remote-to-remote copy by executing scp on a remote host. This
   option is a fragile optimisation that is difficult to use because
   it requires credentials on the remote host. It also creates
   security risks if the shell quoting rules on the remote system
   where the copy is performed differ from the client's expectations.

   From OpenSSH 10.6, this option will continue to work but will
   cause a deprecation warning to be emitted to standard error. In
   a future release, the option will be ignored and will leave in
   place the default remote-to-remote copy behaviour (copy via the
   host running scp).

 * sshd(8): support for platforms that do not allow file descriptor
   passing and that also require root privilege for PTY allocation
   will be removed in a future release. Affected platforms are known
   to include SCO OpenServer 5 and QNX 6 but may include other
   similarly old operating systems. This deprecation can be avoided
   if the user community for these platforms is able to assist us in
   building alternatives, such as avoiding the need for root in PTY
   allocation.

Potentially-incompatible changes
--------------------------------

 * ssh(1), sshd(8): compression will be less effective as a result
   of the change noted below in the "Security" section

 * ssh(1): destination usernames entered on the commandline are now
   more strictly checked and will refuse usernames that include
   backslash and dollar symbols. Usernames that are specified by
   the "User" directive in configuration files are not subject to
   these restrictions. The motivation for this change is mentioned
   below in the "Security" section.

 * sshd(8): on platforms that do not support file descriptor passing
   and that require root for PTY allocation, the GatewayPorts and
   StreamLocalForwarding options are forcibly disabled. The
   motivation for this change is discussed below in the "Security"
   section.

Changes since OpenSSH 10.5
==========================

This release contains a number of security fixes, several new
features and some small bugfixes.

Security
========

 * sftp(1): more strictly validate paths returned from the server to
   avoid some cases where a server could return paths that could
   manipulate a recursive copy operation into writing outside its
   target directory. Report and patch from Junghoon Cho.

 * sshd(8): when GSSAPIAuthentication is in use, only store GSSAPI
   credentials when authentication has succeeded. Avoids a situation
   where credentials from a failed GSSAPIAuthentication attempt may
   persist and be made inappropriately available if another
   authentication subsequently succeeds. Issue report and patch from
   Moritz Theile.

 * sshd(8): reset GSSAPIAuthentication before authentication, avoiding
   state from one authentication attempt being confused with that of
   a later attempt. Report and feedback from Moritz Theile.

 * sshd(8), ssh(1): disable LZ77 dictionary coder to mitigate the
   side-channel leaks described in "Crossing the Streams: SSH
   Plaintext Recovery via a Common Compression Context in
   Multiplexed Channels" by Fabian Bäumer and Marcus Brinkmann,
   preprint https://arxiv.org/abs/2609.07709 (2026)

   A chosen-plaintext attack method exists which makes use of
   dictionary-based compression to recover secrets from one channel
   by interacting with the SSH session's shared compression
   dictionary through another channel.

   Attacker-controlled input can recognizably reflect into the
   total length of transmitted ciphertexts by virtue of LZ77
   replacing repeated strings with back-references into the SSH
   session's encoder search buffer, which is shared across all
   channels. For this reason, the documentation already recommended
   against enabling compression for connections that share trusted
   and untrusted traffic.

   This change will reduce the effectiveness of the Compression
   option. Users are encouraged to use application-level compression
   over the SSH protocol where possible, as this will typically be
   more effective and will be completely immune to this type of
   attack.

 * ssh(1): disallow '$' and '\' characters in usernames entered on
   the command-line to avoid usernames from untrusted sources
   yielding injection in shell context via ProxyCommand, Match exec,
   etc. Usernames specified via the configuration files are not
   subject this this control. Reported by SecBuddyF KeenLab Tencent
   (CodeBuddy Security)

   We continue to recommend against directly exposing ssh(1) and
   other tools' command-lines to untrusted input. Mitigations such
   as this can not be absolute given the variety of shells and user
   configurations in use.

 * ssh-keygen(1): correct handling of Daylight Saving Time when
   converting dates. Previous handling could cause errors of
   up to +/- 1 hour (unless you are in the Antarctica/Troll
   timezone, where the error could be +/- 2 hours). These errors
   could result in creation of certificates with incorrect expiry
   times. bz4004; from Khush Patel

 * sshd(8), ssh(1): ensure that compressed payloads don't inflate
   past the maximum supported packet length. Reported by Oleh Konko.

 * sshd(8): fully honor the authorized_keys "restrict" keyword,
   which was not being properly applied to tunnel forwarding
   (PermitTunnel, disabled by default). This is a separate problem
   to the one fixed in openssh-10.5.

 * sshd(8): correctly handle some options that accept "none". Some
   options, including AuthorizedPrincipalsFile, were documented as
   accepting "none" as a way to disable them; however, when
   overridden by an sshd_config(5) Match keyword, this argument was
   being incorrectly interpreted as a literal file.
   With Chris Rohlf in collaboration with Claude and Anthropic Research

 * sshd(8): On OS X SDK >= 27, sandboxing is no longer supported
   as the API we depended upon has been removed and no obvious
   alternative provided.

 * sshd(8): on platforms that do not support file descriptor passing
   and that require root for PTY allocation, the post-authentication
   sshd-session process retains root privilege, whereas on other
   platforms this process runs with the privilege of the logged-in
   user. When sshd-session was run with elevated privilege, it could
   perform certain actions as root and circumvent controls that
   would normally have applied to the user, such as making unix
   domain socket connections or binding (via -R forwarding) to low-
   numbered TCP ports.

   For this reason, this release disables the GatewayPorts and
   StreamLocalForwarding options and support for these (few)
   platforms will be removed in future if no alternatives to
   requiring privilege in the post-authentication process are found.
   Affected platforms include QNX 6, SCO OpenServer 5 and builds
   that were made with the --disable-fd-passing configure option.

   This problem was reported separately by sn0x-sharma and by Dark
   River.

New features
------------

 * All: enable hybrid post-quantum ssh-mldsa44-ed25519 signature
   algorithm. Note that this no longer uses the "@openssh.com"
   vendor extension suffix that the previous experimental
   implementation used. Keys generated with the previous experimental
   support must be regenerated and/or removed.

 * sshd(8): Add the WarnWeakCrypto option to sshd_config(5). This
   option was previously available for the client only. This option
   is enabled by default and will log when the client uses a key
   agreement scheme that is not post-quantum safe.

 * ssh-keygen(1), ssh-add(1): preserve user-verification (PIN or
   biometric) requirement for resident keys loaded from a FIDO
   token, by checking the credential's credProtect policy.
   GHPR701 from Savely Krasovsky

 * ssh(1): include local and remote version strings in the ~I
   connection information display.

 * ssh-add(1): add a -P flag to skip PIN entry for FIDO and PKCS#11
   tokens that do not require it.

 * sftp(1): add '-p' flag for mkdir/lmkdir to create directories as
   required. This flag has similar ergonomics to mkdir(1), and
   previously-existing directories do not cause an error.

 * ssh-keygen(1): add a "hexdump" key export mode that dumps the SSH
   wire-formatted key blob in hex format. Useful when writing
   documentation, tests, etc. E.g. `ssh-keygen -em hexdump -f /key`

 * ssh(1), sshd(8): ChannelTimeout now accepts timeouts with
   fractional seconds.

 * sshd(8): allow specification of the location of $SSH_AUTH_SOCK
   used for agent forwarding using a new AgentSocketPath option.
   This supports both using a user-specific path, such as the
   default of a subdirectory of $HOME ("user:.ssh/agent"), and
   the previous approach of allowing agent forwarding sockets to be
   located in a shared directory (e.g. "shared:/tmp"). Sockets
   created in shared directories will be created inside a
   subdirectory with a randomised name. bz3860

 * ssh-agent(1): allow specification of agent socket directories
   using a -A flag. It accepts "user:" and "shared:" directory
   styles similar to the sshd AgentSocketPath option.

 * sshd(8): account for public key authentication "key ok" tests
   separately to auth attempts. Add a `PubkeyOptions max-pk-ok:nnnn`
   option to allow a number of PK_OK tests (asking whether the
   server might accept a given public key) that do not count against
   MaxAuthTries, defaulting to 6 attempts. After these attempts are
   exhausted, further attempts count as failed authentications
   against MaxAuthTries. Practically, this allows more keys on disk
   or held in ssh-agent to be checked for use before the server
   disconnects.

 * ssh(1), sshd(8): extend the existing TCPKeepAlive option to also
   support setting keepalives on sockets created for forwarding
   connections. Previously this option controlled keepalives on the
   connection socket only. TCPKeepAlive "yes" or "transport" enables
   keepalives on the connection socket. "TCPKeepAlive all" additionally
   enables them for forwarding sockets. bz3921

Bugfixes
--------

 * sshd(8), ssh(1): fix configuration matching on more Turkic
   languages which have disjoint dotted and dotless i/I characters,
   specifically Azerbaijani and Crimean Tatar. bz3991

 * ssh(1), sshd(8): don't attempt to set TCP_NODELAY on non-IP/IPv6
   sockets. Eliminates some noise in debug logs.

 * ssh(1): fix case for ssh -G option output; bz4005

 * ssh(1), sshd(8): Fix ChannelTimeout specificity; previously a
   more specific channel type (e.g. "session:shell") could clobber
   a user-specified ChannelTimeout if it was less specific (e.g.
   "session").  Also, in some cases, the debug messages were
   printing 0 instead of the effective timeout. bz3994

 * sftp(1): avoid NULL dereference crash in some circumstances when
   a server fails a stat/lstat operation. GHPR707

 * sshd(8): close a race condition where a SIGTERM/SIGQUIT would be
   ignored if it was received by the server while it was processing a
   SIGHUP restart request. bz3981

 * sshd(8), ssh(1): check key and CA signature types during key
   parsing against allowlists (PubkeyAcceptedAlgorithms, etc) as
   early as possible. This reduces the attack surface presented by
   disabled algorithms. Suggested by and with extensive feedback
   from Chris Rohlf in collaboration with Claude and Anthropic
   Research.

 * All: switch the fallback implementation of the ed25519 signature
   algorithm used when libcrypto is disabled from SUPERCOP ed25519
   to libsodium. The libsodium implementation includes a number of
   strictness checks over the original reference implementation we
   have used to this point and a more ergonomic API.

 * ssh-add(1), ssh(1), ssh-keygen(1): fix spin on password entry when
   the program attempting to read a password was started in a
   background process group, with no TTY and with certain signals
   ignored. bz3995

 * scp(1): disallow nul byte in received scp -O filename. This was
   not reachable in normal operation. Reported by Chua Wei Xun.

 * ssh-keygen(1), ssh(1), sshd(8): implement a maximum number of KDF
   rounds that will be accepted when writing an OpenSSH-format
   private key or when loading one. This limit is set quite high
   (1M), but ensures that a service that is passed a bad key with a
   ridiculously high number of rounds will eventually complete
   parsing it.

 * ssh-keygen(1): bump the default number of KDF rounds from 24 to
   32 (this is a linear increase, not like bcrypt(3) which is
   exponential).

 * ssh(1): make StreamLocalBindMask properly respect Host/Match
   blocks and make it first-match-wins as documented. bz4013

 * sshd(8): make StreamLocalBindMask properly first-match-wins.

Portability
-----------

 * All: remove the NetBSD BROKEN_READ_COMPARISON workaround. This
   appears to be no longer required and caused pre-auth CPU spinning.

 * sshd(8): don't link sshd against libselinux when SELinux support
   is enabled (note: this library is still linked for the sshd-auth
   and sshd-session helper binaries).

 * sshd(8): allow madvise(..., MADV_DONTNEED_LOCKED) in the seccomp
   sandbox; needed by GrapheneOS' hardened allocator. bz4001

 * sshd(8): restrict mremap(2) flags accepted by the seccomp
   sandbox. Only MREMAP_MAYMOVE is now accepted as other flags may
   have some utility in attack chains. Reported by: Mohammad Hossein
   Abedini.

 * sshd(8): allow PAMServiceName in Match (regressed in 10.4).
   During the refactor of server option parsing, the ability to set
   PAMServiceName in Match blocks was accidentally disabled.

 * sshd(8): the --disable-fd-passing configure option has been
   removed.

Checksums:
==========

 - SHA1 (openssh-10.6.tar.gz) = 096e9cd60cd08e0acacfe16b615144f8c0463da5
 - SHA256 (openssh-10.6.tar.gz) = 0lP9h9/QH3k667vqbxKhkkusc3CN5iKUiDNI9fsR06A=
 - SHA1 (openssh-10.6p1.tar.gz) = e6a34f5625e20172b214c50ebd4e0dcb4fd18013
 - SHA256 (openssh-10.6p1.tar.gz) = qdyVZd/+hkD2TYY80poyvEo9vewFZqf8RMXW7nZ9Xzk=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 10.5 / 10.5p1 (2026-08-11) 

OpenSSH 10.5 was released on 2026-08-11. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Recently the OpenSSH team have received a large number of security
bug reports, many of which are findings from AI models or made with
AI assistance. While many AI reports are determined not to have
security impact when considered in the context of a realistic
threat model, we very much welcome these reports, especially when
combined with human triage, analysis, test-cases and particularly
when accompanied by proposed fixes.

We have seen a number of cases where a security bug identified by
AI tools is subsequently independently discovered by a different
researcher. This suggests that adversaries who do not report bugs
to OSS projects are likely to be able to discover these bugs too.
Given this, the OpenSSH team will, for now, be making more frequent
releases to get bugfixes into users' hands more quickly rather than
batching them until the next planned release.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Potentially-incompatible changes
--------------------------------

 * Portable OpenSSH now requires ECC (Elliptic Curve Cryptography)
   support in libcrypto, including support for the NISTP521 curve.
   ECC is included in the default build configurations of all
   versions of all libcrypto implementations currently supported by
   OpenSSH, including LibreSSL, OpenSSL, BoringSSL and AWS LC.
   The --without-openssl build configuration is not affected.

Changes since OpenSSH 10.4
==========================

This release contains a number of security fixes and small bugfixes.

Security
========

 * ssh-agent(1): fix an interaction between agent locking and the
   session-bind@openssh.com extension that is used to identify
   forwarded agents. These binding requests were refused when the
   agent was locked, with the result that operations that were
   intended to be limited to local use only could be performed
   remotely, including the ability to add PKCS#11 tokens and make
   use of keys that had destination restrictions applied.
   Reported by sn0x-sharma

 * ssh(1): avoid potential realloc use-after-free in the client if a
   remote forwarding is added via the local session multiplexing
   socket while a remote forwarding open request is pending with the
   server. Report and fix from Brian Mingus of Cognatory

 * sshd(8): make the authorized_keys "restrict" keyword apply
   correctly to tunnel forwarding too (which is administratively
   disabled by default). Reported by Erichen, Institute of Computing
   Technology, Chinese Academy of Sciences
    
New features
------------

 * ssh-keygen(1): add ability to set or clear the touch-required and
   verify-required flags on FIDO private keys when resetting a
   private key's passphrase.

 * ssh(1): tweak ordering of certificates tried during pubkey
    authentication to prefer FIDO keys that do not require user
    presence (touch) first, and FIDO keys that require user
    verification via PIN or biometrics last. This effectively tries
    low-friction authenticators before higher friction ones.

 * ssh(1): add a "ssh -Z user@host" mode that prints the keys that
   will be tried for public key authentication in the order that
   they will be used.
    
 * sshd(8) use setproctitle(3) to identify sshd-session when its
   acting as a post-authentication monitor.
    
Bugfixes
--------

 * ssh-keyscan(1): make reading the server banner a non-blocking
   operation to prevent a stuck server from blocking a many-host
   keyscan from proceeding.

 * sshd(8): use sshpkt_fatal() instead of plain fatal() for errors
   in the packet code as this provides context of the failing peer
   (address, port, user, etc).

 * sshd(8): when signing hostkey proofs for a client UpdateHostKeys
   request, allow each hostkey to perform at most one signature
   operation.
    
 * sshd(8) fix GSSAPI option names, that were broken during a
   servconf.c refactoring in openssh-10.4; bz3974.
    
 * ssh-keygen(1): pass back errors from ed25519 key generation, which
   theoretically can fail. GHPR702.

 * sshd(8): move check of public key type against allowed algorithms
   to before parsing of the key sent by the peer. This removes at
   least some key parsing and verification paths from the pre-auth
   attack surface. Suggested by Christopher Paul Rohlf of Anthropic.
    
 * ssh-keygen(1): fix double frees (impossible to reach outside of a
   test harness), and also use freezero where possible. From
   Christopher Paul Rohlf at Anthropic.

 * sshd(8): fix ChannelTimeout and RekeyLimit not being applied in
   sshd_config Match blocks.

 * sshd(8): in sshd config dump mode, write all directives in mixed
   case for consistency
    
Portability
-----------

 * sshd(8): re-allow PAMServiceName inside a Match block, which
   was incorrectly disabled during a refactoring in openssh-10.4.
   bz3987

Checksums:
==========

 - SHA1 (openssh-10.5.tar.gz) = 273163972f623bb9bffef9fd75a85c74d3b22633
 - SHA256 (openssh-10.5.tar.gz) = 9Zhp0C/mDWNLmnatq68mtbqOUhvbyYcffH04dsHUwZQ=

 - SHA1 (openssh-10.5p1.tar.gz) = 3067e2af7c526b31c7e94bc3ed38904ef791eb2c
 - SHA256 (openssh-10.5p1.tar.gz) = 1E0oqDnqna+WnMaRUP3lmRCys5Nh2tgaO9bL0ZIY2xE=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 10.4 / 10.4p1 (2026-07-06) 

OpenSSH 10.4 was released on 2026-07-06. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Potentially-incompatible changes
--------------------------------

 * sshd(8): configuration dump mode ("sshd -G") now writes directives
   in mixed case (e.g. "PubkeyAuthentication") whereas previously it
   emitted only lower-case names.

 * sshd(8): on Linux systems with the seccomp sandbox enabled,
   failures to enable SECCOMP or NO_NEW_PRIVS are now fatal.
   Previously sshd(8) would log the error but continue operation,
   to support systems that lacked these features. Now systems that
   lack these should instead disable the sandbox at configure time.

 * ssh(1), sshd(8): make the transport protocol stricter by
   disconnecting if the peer sends non-KEX messages during a post-
   authentication key re-exchange. Previously a malicious peer could
   continue sending non-key exchange messages without penalty. These
   would be buffered, causing memory to be wasted up until the
   connection terminated or the server/client hit a memory limit.
   Implementations that do not restrict messages sent during key
   exchange as per RFC4253 section 7.1 may be disconnected.
   Reported by Marko Jevtic.

Changes since OpenSSH 10.3
==========================

This release contains a number of security fixes as well as general
bugfixes and a couple of new features.

Security
========

 * sftp(1): when downloading files on the command-line using
   "sftp host:/path .", a malicious server could cause the file to
   be downloaded to an unexpected location. This issue was identified
   by the Swival Security Scanner.

 * scp(1): when copying files between two remote destinations, do
   not allow a malicious server to write files to the parent
   directory of the intended target directory.  This issue was
   identified by the Swival Security Scanner.

 * sshd(8): when using the "internal-sftp" SFTP server implementation
   (this is not the default), long command lines were previously
   truncated silently after the 9th argument. If a security-relevant
   option was in the 10th or later position, it would be discarded.
   Reported by Steve Caffrey.

 * sshd(8): add a documentation note to mention that the
   GSSAPIStrictAcceptorCheck option is ineffective when the server
   is joined to a Windows Active Directory. Reported by Yarin Aharoni
   of Safebreach.

 * sshd(8): DisableForwarding=yes didn't override PermitTunnel=yes
   as it was documented to do. Note that PermitTunnel is not enabled
   by default. Reported independently by Huzaifa Sidhpurwala of
   Redhat and Marko Jevtic.

 * sshd(8): avoid a potential pre-authentication denial of service
   when GSSAPIAuthentication was enabled (this feature is off by
   default). This was not mitigated by MaxAuthTries, but would be
   penalised by PerSourcePenalties. This was reported by Manfred
   Kaiser of the milCERT AT (Austrian Ministry of Defence).

 * sshd(8): fix a number of cases where the minimum authentication
   delay was not being enforced. Reported by the Orange Cyberdefense
   Vulnerability Team.

 * ssh(1): fix a possible client-side use-after-free if the server
   changes its host key during a key reexchange. This was reported by
   Zhenpeng (Leo) Lin of Depthfirst.

New features
------------

 * All: add experimental support for a composite post-quantum
   signature scheme that combines ML-DSA 44 and Ed25519 as specified
   in draft-miller-sshm-mldsa44-ed25519-composite-sigs.

   This scheme is not enabled by default. To use it, you'll need
   to add it to HostKeyAlgorithms, PubkeyAcceptedAlgorithms, etc.
   Keys may be generated using "ssh-keygen -t mldsa44-ed25519".

 * ssh(1), sshd(8): replace the wildcard pattern matcher with an
   implementation based on an NFA. This avoids exponential worst-case
   behaviour for the old implementation.

Bugfixes
--------

 * ssh-agent(1): fix incorrect reply to "query" SSH_AGENTC_EXTENSION
   requests. bz3967

 * sshd(8): avoid sending observably different messages for valid vs
   invalid users in GSSAPIAuthentication (disabled by default).

 * ssh(1), sshd(8): fix several bugs that incorrectly
   classified bulk traffic as interactive. bz3972, bz3958

 * ssh-keygen(1), ssh-add(1): skip unsupported key types when
   downloading resident keys from a FIDO token. Previously, downloads
   would abort when one was encountered. GHPR657

 * ssh(1): fix a potential use-after-free on an error path if
   cipher_init() fails.

 * sshd(8): perform stricter encoding and validation of transport
   state passed between sshd privilege separation subprocesses. This
   somewhat further hardens the server against attacks on sshd-auth
   or sshd-session subprocesses.

 * ssh-agent(1): avoid possible runtime denial of service by
   enforcing some limits on the length of usernames in key use
   constraints.

 * sftp(1): fix two separate one-byte out-of-bounds reads, in
   SSH2_FXP_REALPATH and batch command processing.

 * sftp-server(8): disallow use of the copy-data extension to read
   and write to the same inode simultaneously.

 * ssh(1), sshd(8): avoid strlen(NULL) crash if an X11 channel was
   created before the x11-req SSH_MSG_CHANNEL_REQUEST was sent.
   GHPR679

 * sftp(1), scp(1): avoid a situation where sftp_download() could get
   stuck in a loop if a broken server repeatedly returned zero length
   while reading a file.

 * ssh(1): avoid leaking DNS0x20 case-randomised names into names
   canonicalised using CanonicalizePermittedCNAMEs. bz3966

 * sftp-server(8): avoid truncation of pathnames passed to lstat()
   during SSH_FXP_REALPATH handling on systems where PATH_MAX is not
   the actual max. GHPR688

 * ssh(1), sshd(8): correct arming of poll(2) event masks for some
   socket-type channels. GHPR660

 * sshd(8): major refactor of sshd_config parsing and management
   code, to allow for more exact serialisation/deserialisation across
   privilege separation boundaries.

 * ssh-add(1): open connection to the agent only after getopt()
   processing has completed, to give options like "-v" a chance to
   display debug information about this operation.

 * crypto code: fix bounds checking when signing messages of length
   greater than will fit in a size_t. In OpenSSH, message sizes are
   bounded by SSHBUF_SIZE_MAX so this was unreachable.

 * crypto code: add signature malleability and pubkey validity checks
   to ed25519 verification. SSH doesn't depend on these properties

 * crypto code: fix ECDSA order check for curves with cofactor != 1.
   All supported EC curves have cofactor 1, so this was
   unreachable.

 * sshd(8): differentiate between execution failures and a subsystem
   that was not found when logging why a subsystem failed to start.
   GHPR637

 * All: use safer idioms for timegm(3) and mktime(3) error detection.

 * ssh(1), sshd(8): avoid accepting invalid cipher or MAC lists in
   config files or command-line arguments. This could cause runtime
   failures later.

 * ssh(1): fix NULL deref crash during pubkey auth when using a PEM
   style private key with no corresponding .pub key adjacent to it.

 * sshd(8): don't print an error message when trying to load a host
   private key when PKCS#11 keys are in use, as these don't need the
   private half on the filesystem. GHPR664

 * All: don't use deprecated ERR_load_crypto_strings(). GHPR650

 * ssh(1): properly report errors during configuration default
   setting. GHPR649

 * ssh(1): use correct directive name (Match instead of Host) in
   error message. bz3968

 * sftp(1): fix "ls -ln" which was not correctly showing numeric
   UID/GIDs but rather user and group names. bz3953

 * sshd(8): avoid possible NULL dereference if an allocation fails
   during config parsing. bz3948

 * All: fix ineffective guards against loading overly large public
   keys in several places. bz3969 and bz3970

 * sftp(1): ensure file descriptors used by sftp to communicate to
   its ssh(1) subprocess don't leak into executed subprocesses (e.g.
   via "!"). GHPR693

Portability
-----------

 * Sync fmt_scaled.c with OpenBSD upstream, picking up an exactness
   fix for large exponents (GHPR671)

 * sshd(8): remove duplicate sandbox entry for clock_gettime64.

 * ssh(1), sshd(8): use correct IPTOS_DSCP_VA value if not provided
   by the system headers.

 * Sync getrrsetbyname.c with OpenBSD upstream, picking up robustness
   fixes.

 * Disable replacements in openbsd-compat for strvisx(3) and
   stravis(3), as these are unused in OpenSSH

 * Avoid fortify warnings on Android bz3954

 * Fix a number of memory leaks on error paths in the portability
   code. GHPR681

 * Revise the README.privsep documentation to reflect sshd's recent
   switch to a multi-binary model.

Checksums:
==========

 - SHA1 (openssh-10.4.tar.gz) = 8502b516230865e229d55045bb4ccc67aeae905b
 - SHA256 (openssh-10.4.tar.gz) = qUVI+wMg4mVpiQbXvfMVkF3Zuyy2JrS+ZjN764mtyGE=

 - SHA1 (openssh-10.4p1.tar.gz) = ae8650a71cc52dbbd049519cee276ae6d65c2c4d
 - SHA256 (openssh-10.4p1.tar.gz) = 72Am3SrqjVYFljjV0yYpAsiSzrqfiDlYNeDQbT+2Mjg=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 10.3 / 10.3p1 (2026-04-02) 

OpenSSH 10.3 was released on 2026-04-02. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Potentially-incompatible changes
--------------------------------

 * ssh(1), sshd(8): remove bug compatibility for implementations
   that don't support rekeying. If such an implementation tries to
   interoperate with OpenSSH, it will now eventually fail when the
   transport needs rekeying.

 * sshd(8): prior to this release, a certificate that had an empty
   principals section would be treated as matching any principal
   (i.e. as a wildcard) when used via authorized_keys principals=""
   option. This was intentional, but created a surprising and
   potentially risky situation if a CA accidentally issued a
   certificate with an empty principals section: instead of being
   useless as one might expect, it could be used to authenticate as
   any user who trusted the CA via authorized_keys. [Note that this
   condition did not apply to CAs trusted via the sshd_config(5)
   TrustedUserCAKeys option.]

   This release treats an empty principals section as never matching
   any principal, and also fixes interpretation of wildcard
   characters in certificate principals. Now they are consistently
   implemented for host certificates and not supported for user
   certificates.

 * ssh(1): the -J and equivalent -oProxyJump="..." options now
   validate user and host names for ProxyJump/-J options passed
   via the command-line (no such validation is performed for this
   option in configuration files). This prevents shell injection in
   situations where these were directly exposed to adversarial
   input, which would have been a terrible idea to begin with.
   Reported by rabbit.

Changes since OpenSSH 10.2
==========================

This release contains some relatively minor security fixes as well
as a number of feature improvements and general bugfixes.

Security
========

 * ssh(1): validation of shell metacharacters in user names supplied
   on the command-line was performed too late to prevent some
   situations where they could be expanded from %-tokens in
   ssh_config. For certain configurations, such as those that use a
   "%u" token in a "Match exec" block, an attacker who can control
   the user name passed to ssh(1) could potentially execute arbitrary
   shell commands.  Reported by Florian Kohnhäuser.

   We continue to recommend against directly exposing ssh(1) and
   other tools' command-lines to untrusted input. Mitigations such
   as this can not be absolute given the variety of shells and user
   configurations in use.

 * sshd(8): when matching an authorized_keys principals="" option
   against a list of principals in a certificate, an incorrect
   algorithm was used that could allow inappropriate matching in
   cases where a principal name in the certificate contains a
   comma character. Exploitation of the condition requires an
   authorized_keys principals="" option that lists more than one
   principal *and* a CA that will issue a certificate that encodes
   more than one of these principal names separated by a comma
   (typical CAs strongly constrain which principal names they will
   place in a certificate). This condition only applies to user-
   trusted CA keys in authorized_keys, the main certificate
   authentication path (TrustedUserCAKeys/AuthorizedPrincipalsFile)
   is not affected. Reported by Vladimir Tokarev.

 * scp(1): when downloading files as root in legacy (-O) mode and
   without the -p (preserve modes) flag set, scp did not clear
   setuid/setgid bits from downloaded files as one might typically
   expect. This bug dates back to the original Berkeley rcp program.
   Reported by Christos Papakonstantinou of Cantina and Spearbit.

 * sshd(8): fix incomplete application of PubkeyAcceptedAlgorithms
   and HostbasedAcceptedAlgorithms with regard to ECDSA keys.
   Previously if one of these directives contains any ECDSA algorithm
   name (say "ecdsa-sha2-nistp384"), then any other ECDSA algorithm
   would be accepted in its place regardless of whether it was
   listed or not.  Reported by Christos Papakonstantinou of Cantina
   and Spearbit.

 * ssh(1): connection multiplexing confirmation (requested using
   "ControlMaster ask/autoask") was not being tested for proxy mode
   multiplexing sessions (i.e. "ssh -O proxy ..."). Reported by
   Michalis Vasileiadis.

New features
------------

 * ssh(1), sshd(8): support IANA-assigned codepoints for SSH agent
   forwarding, as per draft-ietf-sshm-ssh-agent. Support for the new
   names is advertised via the EXT_INFO message. If a server offers
   support for the new names, then they are used preferentially.

   Support for the pre-standardisation "@openssh.com" extensions for
   agent forwarding remains supported.

 * ssh-agent(1): implement support for draft-ietf-sshm-ssh-agent
   "query" extension.

 * ssh-add(1): support querying the protocol extensions via the
   agent "query" extension with a new -Q flag.

 * ssh(1): support multiple files in a ssh_config RevokedHostKeys
   directive. bz3918

 * sshd(8): support multiple files in a sshd_config RevokedKeys
   directive bz3918

 * ssh(1): add a ~I escape option that shows information about the
   current SSH connection.

 * ssh(1): add an "ssh -Oconninfo user@host" multiplexing command
   that shows connection information, similar to the ~I escapechar.

 * ssh(1): add an "ssh -O channels user@host" multiplexing command to
   get a running mux process to show information about what channels
   are currently open.

 * sshd(8): add 'invaliduser' penalty to PerSourcePenalties, which is
   applied to login attempts for usernames that do not match real
   accounts. Defaults to 5s to match 'authfail' but allows
   administrators to block such attempts for longer if desired.

 * sshd(8): add a GSSAPIDelegateCredentials option for the server,
   controlling whether it accepts delegated credentials offered by
   the client.  This option mirrors the same option in ssh_config.
   GHPR614

 * ssh(1), sshd(8): support the VA DSCP codepoint in the IPQoS
   directive.

 * sshd(8): convert PerSourcePenalties to using floating point time,
   allowing penalties to be less than a second. This is useful if you
   need to penalise things you expect to occur at >=1 QPS.

 * ssh-keygen(1): support writing ED25519 keys in PKCS8 format.
   GHPR570

 * Support the ed25519 signature scheme via libcrypto.

Bugfixes
--------

 * sshd(8): make IPQoS first-match-wins in sshd_config, like other
   configuration directives. bz3924

 * sshd(8): fix potential crash when MaxStartups is using a single
   argument (i.e. not using the MaxStartps x:y:z form) to a value
   below 10. bz3941

 * sshd(8): fix a potential hang during key exchange if needed DH
   group values were missing from /etc/moduli.

 * ssh-agent(1): fix return values from extensions to be correct wrt
   draft-ietf-sshm-ssh-agent: extension requests should indicate
   failure using SSH_AGENT_EXTENSION_FAILURE rather than the generic
   SSH_AGENT_FAILURE error code. This allows the client to discern
   between "the request failed" and "the agent doesn't support this
   extension".

 * ssh(1): use fmprintf for showing challenge-response name and info
   to preserve UTF-8 characters where appropriate. Prompted by GitHub
   PR#452.

 * scp(1): when uploading a directory using sftp/sftp (e.g. during a
   recursive transfer), don't clobber the remote directory
   permissions unless either we created the directory during the
   transfer or the -p flag was set. bz3925

 * All: implement missing pieces of FIDO/webauthn signature support,
   mostly related to certificate handling and enable acceptance of this
   signature format by default.  bz3748 GHPR624 GHPR625

 * sshd_config(5): make it clear that DenyUsers/DenyGroups overrides
   AllowUsers/AllowGroups. Previously we specified the order in which
   the directives are processed but it was ambiguous as to what
   happened if both matched.

 * ssh(1): don't try to match certificates held in an agent to
   private keys. This matching is done to support certificates that
   were loaded without their private key material, but is
   unnecessary for agent-hosted certificate which always have
   private key material available in the agent. Worse, this matching
   would mess up the request sent to the agent in such a way as to
   break usage of these keys when the key usage was restricted in
   the agent.  bz3752

 * sftp(1): if editline has been switched to vi mode (i.e. via "bind
   -v" in .editrc), setup a keybinding so that command mode can be
   entered.

 * ssh(1), sshd(8): improve performance of keying the sntrup761 key
   agreement algorithm.

 * ssh(1), sshd(8): enforce maximum packet/block limit during
   pre-authentication phase.

 * sftp(1): don't misuse the sftp limits extension's open-handles
   field. This value is supposed to be the number of handles a
   server will allow to be opened and not a number of outstanding
   read/write requests that can be sent during an upload/download.

 * sshd(8): don't crash at connection time if the main sshd_config
   lacks any subsystem directive but one is defined in a Match block.
   bz3906

 * sshd_config(5): add a warning next to the ForceCommand directive
   that forcing a command doesn't automatically disable forwarding.

 * sshd_config(5): add a warning that TOKENS are replaced without
   filtering or escaping and that it's the administrator's
   responsibility to ensure they are used safely in context.

 * scp(1): correctly quote filenames in verbose output for local->
   local copies. bz3900

 * sshd(8): don't mess up the PerSourceNetBlockSize IPv6 mask if
   sscanf didn't decode it. GHPR598

 * ssh-add(1): when loading FIDO2 resident keys, set the comment to
   the FIDO application string. This matches the behaviour of
   ssh-keygen -K. GHPR608

 * sshd(8): don't strnvis() log messages that are going to be logged
   by sshd-auth via its parent sshd-session process, as the parent
   will also run them though strnvis(). Prevents double-escaping of
   non-printing characters in some log messages. bz3896

 * ssh-agent(1): escape SSH_AUTH_SOCK paths that are sent to the
   shell as setenv commands. Unbreaks ssh-agent for home directory
   paths that contain whitespace. bz3884

 * All: Remove unnecessary checks for ECDSA public key validity.

 * sshd(8): activate UnusedConnectionTimeout only after the last
   channel has closed. Previously UnusedConnectionTimeout could fire
   early after a ChannelTimeout. This was not a problem for the
   OpenSSH client because it terminates once all channels have
   closed but could cause problems for other clients (e.g. API
   clients) that do things differently.  bz3827

 * All: fix PKCS#11 key PIN entry problems introduced in
   openssh-10.1/10.2.  bz3879

 * scp(1): when using the SFTP protocol for transfers, fix implicit
   destination path selection when source path ends with "..". bz3871

 * sftp(1): when tab-completing a filename, ensure that the completed
   string does not end up mid-way through a multibyte character, as
   this will cause a fatal() later on. GHPR#587

 * ssh-keygen(1): fix crash at exit (visible via ssh-keygen -D) when
   multiple keys loaded.

 * scp(1)/sftp(1): correctly display bandwidths >2GBps in the
   progress meter.

Portability
-----------

 * sshd(8): fix condition introduced in openssh 10.2p1 stable branch
   here a PAM module that changed the requested username between
   SSH_MSG_USERAUTH_REQUEST messages during authentication could
   confuse the PAM stack and let it proceed with a different
   understanding of the active username than the rest of sshd.
   Reported by Mike Damm.

 * sshd(8): immediately report interactive instructions to clients
   when using keyboard-interactive authentication with PAM. bz2876

 * sshd(8): fix duplicate PAM messages under some situations.

 * sshd(8): don't leak PAM handle on repeat invocations. bz3882

 * All: support linking libcrypto implementations (e.g. BoringSSL)
   that require libstdc++.

 * sshd(8): fix ut_type for btmp records, correctly using
   LOGIN_PROCESS and USER_PROCESS.

 * sshd(8): allow uname(3) in the seccomp sandbox. This is needed by
   zlib-ng on RISC-V platforms.

 * All: remove remaining OpenSSL_add_all_algorithms() calls.
   We already have OPENSSL_init_crypto() in the compat layer.
   Prompted by github PR#606

 * All: fix builds on older Mac OS wrt nfds_t.

 * mdoc2man: several improvements including better support for Dl
   and Ns inside Ic.

Checksums:
==========

 - SHA1 (openssh-10.3.tar.gz) = 854863c04cd28242d73ac6c3ee9c37fa756f1a2f
 - SHA256 (openssh-10.3.tar.gz) = aCU5P47rM+m4N8/i2JOHMOMhafMYqBhvQQSnPXczN5M=

 - SHA1 (openssh-10.3p1.tar.gz) = 9c78838ec07af14aff54f3755ac56ce6812452a9
 - SHA256 (openssh-10.3p1.tar.gz) = VmgqNruS3PS08Bb9jsjnQFm3mo3iXBXWcNcx59GORfQ=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 10.2 / 10.2p1 (2025-10-10) 

OpenSSH 10.2 was released on 2025-10-10. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Future deprecation warning
--------------------------

 * A future release of OpenSSH will deprecate support for SHA1 SSHFP
   records due to weaknesses in the SHA1 hash function. SHA1 SSHFP
   DNS records will be ignored and ssh-keygen -r will generate only
   SHA256 SSHFP records.

   The SHA256 hash algorithm, which has no known weaknesses, has
   been supported for SSHFP records since OpenSSH 6.1, released in
   2012.

Changes since OpenSSH 10.1
==========================

This is a bugfix release, primarily to fix a problem that rendered
ssh(1) unusable when ControlPersist was enabled.

Bugfixes
--------

 * ssh(1): fix mishandling of terminal connections when
   ControlPersist was active that rendered the session unusable.
   bz3872

 * ssh-keygen(1): fix download of keys from PKCS#11 tokens.

 * ssh-keygen(1): fix CA signing operations when the CA key is held
   in a ssh-agent(1). bz3877


Portability
-----------

 * All: support platforms without mmap(2), e.g. WASM builds such as
   https://hterm.org

 * All: fix builds on FreeBSD for missing fnctl.h include.

 * All: fix builds on MacOS <10.12 Sierra, which lacks
   clock_gettime(3)

 * sshd(8): don't PAM_RHOST if the remote host is the "UNKNOWN"
   placeholder name. Avoids potential hangs in some PAM modules as
   they try to resolve it. Note, sshd(8) only uses the "UNKNOWN"
   name when the connection is not on an IPv4 or IPv6 socket.

Checksums:
==========

SHA1 (openssh-10.2.tar.gz) = 6fcda8004bad0fb0eaee60e8308f91b605ad0dce
SHA256 (openssh-10.2.tar.gz) = y0rCEdrVc4OJRZLg0u3F0frAgz87ydeTktCk3rQfVj8=

SHA1 (openssh-10.2p1.tar.gz) = c34efade16109f065ec8c834f237bcedd8d7ef5c
SHA256 (openssh-10.2p1.tar.gz) = zMQsBBmTeVkmP6Hb0W2vwYxWuYTANWLSk3zlamD3mLI=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 10.1 / 10.1p1 (2025-10-06) 

OpenSSH 10.1 was released on 2025-10-06. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Future deprecation warning
--------------------------

 * A future release of OpenSSH will deprecate support for SHA1 SSHFP
   records due to weaknesses in the SHA1 hash function. SHA1 SSHFP
   DNS records will be ignored and ssh-keygen -r will generate only
   SHA256 SSHFP records.

   The SHA256 hash algorithm, which has no known weaknesses, has
   been supported for SSHFP records since OpenSSH 6.1, released in
   2012.

Potentially-incompatible changes
--------------------------------

 * ssh(1): add a warning when the connection negotiates a non-post
   quantum key agreement algorithm.

   This warning has been added due to the risk of "store now, decrypt
   later" attacks. More details at https://openssh.com/pq.html

   This warning may be controlled via a new WarnWeakCrypto ssh_config
   option, defaulting to on. This option is likely to control
   additional weak crypto warnings in the future.

 * ssh(1), sshd(8): major changes to handling of DSCP marking/IPQoS

   In both client and server the default DSCP (a.k.a IPQoS) values
   were revised and the way these values are used during runtime has
   changed.

   Interactive traffic is now assigned to the EF (Expedited
   Forwarding) class by default. This provides more appropriate
   packet prioritisation information for the intermediate network,
   such as wireless media (cf. RFC 8325). Non-interactive traffic
   will now use the operating system default DSCP marking. Both the
   interactive and non-interactive DSCP values may be overridden via
   the IPQoS keyword, described in ssh_config(5) and sshd_config(5).

   The appropriate DSCP marking is now automatically selected and
   updated as needed over the course of a connection's lifetime.
   ssh(1) and sshd(8) will switch between the interactive and
   non-interactive IPQoS values depending on the type of SSH
   channels open at the time. For example, if an sftp session is
   using the connection alongside a shell session, then the non-
   interactive value will be used for the duration of the sftp. A
   connection which contains only interactive sessions is marked EF.

 * ssh(1), sshd(8): deprecate support for IPv4 type-of-service (ToS)
   keywords in the IPQoS configuration directive.

   Type of Service (ToS) was deprecated in the late nineties and
   replaced with the Differentiated Services architecture, which
   has significant advantages for operators because it offers more
   granularity.

   OpenSSH switched its default IPQoS from ToS to DSCP values in
   2018 (openssh-7.7).

   IPQoS configurations with 'lowdelay', 'reliability', or
   'throughput' will be ignored and will instead use the system
   default QoS settings. Additionally, a debug message will be logged
   about the deprecation with a suggestion to use DSCP QoS instead.

 * ssh-add(1): when adding certificates to an agent, set the expiry
   to the certificate expiry time plus a short (5 min) grace period.

   This will cause the agent to automatically remove certificates
   shortly after they expire. A new ssh-add -N option disables this
   behaviour.

 * All: remove experimental support for XMSS keys. This was never
   enabled by default. We expect to implement a new post-quantum
   signature scheme in the near future.

 * ssh-agent(1), sshd(8): move agent listener sockets from /tmp to
   under ~/.ssh/agent for both ssh-agent(1) and forwarded sockets
   in sshd(8).

   This ensures processes that have restricted filesystem access
   that includes /tmp do not ambiently have the ability to use keys
   in an agent.

   Moving the default directory has the consequence that the OS will
   no longer clean up stale agent sockets, so ssh-agent now gains
   this ability.

   To support $HOME on NFS, the socket path includes a truncated
   hash of the hostname. ssh-agent will, by default, only clean up
   sockets from the same hostname.

   ssh-agent(1) gains some new flags: -U suppresses the automatic
   cleanup of stale sockets when it starts. -u forces a cleanup
   without keeping a running agent, -uu forces a cleanup that ignores
   the hostname. -T makes ssh-agent put the socket back in /tmp.

Changes since OpenSSH 10.0
==========================

This release contains a minor security fix as well as a number of
feature improvements and bugfixes.

Security
========

* ssh(1): disallow control characters in usernames passed via the
  commandline or expanded using %-sequences from the configuration
  file, and disallow \0 characters in ssh:// URIs.

  If an ssh(1) commandline was constructed using usernames or URIs
  obtained from an untrusted source, and if a ProxyCommand that uses
  the %r expansion was configured, then it may be possible for an
  attacker to inject shell expressions that may be executed when the
  proxy command is started.

  We strongly recommend against using untrusted inputs to construct
  ssh(1) commandlines.

  This change also relaxes the validity checks in one small way:
  usernames supplied via the configuration file as literals (i.e.
  that have no % expansion characters) are not subject to these
  validity checks. This allows usernames that contain arbitrary
  characters to be used, but only via configuration files. This is
  done on the basis that ssh's configuration is trusted.

  This issue was reported by David Leadbeater.

New features
------------

 * ssh(1), sshd(8): add SIGINFO handlers to log active channel and
   session information.

 * sshd(8): when refusing a certificate for user authentication, log
   enough information to identify the certificate in addition to the
   reason why it was being denied. Makes debugging certificate
   authorisation problems a bit easier.

 * ssh(1), ssh-agent(1): support ed25519 keys hosted on PKCS#11
   tokens.

 * ssh(1): add an ssh_config(5) RefuseConnection option that, when
   encountered while processing an active section in a
   configuration, terminates ssh(1) with an error message that
   contains the argument to the option.

   This may be useful for expressing reminders or warnings in config
   files, for example:

   Match host foo
            RefuseConnection "foo is deprecated, use splork instead"

 * sshd(8): make the X11 display number check relative to
   X11DisplayOffset. This will allow people to use X11DisplayOffset
   to configure much higher port ranges if they really want, while
   not changing the default behaviour.

 * unit tests: the unit test framework now includes some basic
   benchmarking capabilities. Run with "make UNITTEST_BENCHMARK=yes"
   on OpenBSD or "make unit-bench" on Portable OpenSSH.

Bugfixes
--------

 * sshd(8): fix mistracking of MaxStartups process exits in some
   situations. At worst, this could cause all MaxStartups slots to
   fill and sshd to refuse new connections.

 * ssh(1): fix delay on X client startup when ObscureKeystrokeTiming
   is enabled. bz#3820

 * sshd(8): increase the maximum size of the supported configuration
   from 256KB to 4MB, which ought to be enough for anybody. Fail
   early and visibly when this limit is breached. bz3808

 * sftp(1): during sftp uploads, avoid a condition where a failed
   write could be ignored if a subsequent write succeeded. This is
   unlikely but technically possible because sftp servers are
   allowed to reorder requests.

 * sshd(8): avoid a race condition when the sshd-auth process exits
   that could cause a spurious error message to be logged.

 * sshd(8): log at level INFO when PerSourcePenalties actually
   blocks access to a source address range. Previously this was
   logged at level VERBOSE, which hid enforcement actions under
   default config settings.

 * sshd(8): GssStrictAcceptor was missing from sshd -T output; fix

 * sshd(8): Make the MaxStartups and PerSourceNetBlockSize options
   first-match-wins as advertised. bz3859

 * ssh(1): fix an incorrect return value check in the local forward
   cancellation path that would cause failed cancellations not to be
   logged.

 * sshd(8): make "Match !final" not trigger a second parsing pass
   of ssh_config (unless hostname canonicalisation or a separate
   "Match final" does). bz3843

 * ssh(1): better debug diagnostics when loading keys. Will now list
   key fingerprint and algorithm (not just algorithm number) as well
   as making it explicit which keys didn't load.

 * All: fix a number of memory leaks found by LeakSanitizer,
   Coverity and manual inspection.

 * sshd(8): Output the current name for PermitRootLogin's
   "prohibit-password" in sshd -T instead of its deprecated alias
   "without-password".  bz#3788

 * ssh(1): make writing known_hosts lines more atomic by writing
   the entire line in one operation and using unbuffered stdio.

   Usually writes to this file are serialised on the "Are you sure
   you want to continue connecting?" prompt, but if host key
   checking is disabled and connections were being made with high
   concurrency then interleaved writes might have been possible.

Portability
-----------

 * sshd(8): check the username didn't change during the PAM
   transactions.

   PAM modules can change the user during their execution, but
   this is not supported by sshd(8). If such a case was incorrectly
   configured by the system administrator, then sshd(8) could end up
   using a different username to the one authorised by PAM.

 * sshd(8): don't log audit messages with UNKNOWN hostname to avoid
   slow DNS lookups in the audit subsystem.

 * All: when making a copy of struct passwd, ensure struct fields are
   non-NULL. Android libc can return NULL pw_gecos, for example.

 * All: Remove status bits from OpenSSL >=3 version check.

 * sshd(8), ssh(1): Use SSH_TUN_COMPAT_AF on FreeBSD. Otherwise tun
   forwarding from other OSes fails as soon as the first IPv6 message
   is sent by the other side (which is usually a Router Solicitation
   ICMPv6 message which is sent as soon as the interface is up).

 * ssh(1), ssh-agent(8): check for nlist function presence before
   attempting to use it instead of relying on the presence of the
   nlist.h header.  Mac OS X, for example, has the header but not
   the function in the 64bit libraries.

 * All: fill in missing system header files.

   Create replacement header files inside openbsd-compat for common
   headers that are missing on a given platform. Usually these are
   just empty, but in some cases they'll include the equivalent file.
   This avoids having to wrap those includes in '#ifdef HAVE_FOO_H'
   and reduces the diff between Portable OpenSSH and OpenBSD.

 * sshd(8): handle futex_time64 properly in seccomp sandbox
   Previously we only allowed __NR_futex, but some 32-bit systems
   apparently support __NR_futex_time64. We had support for this
   in the sandbox, but because of a macro error only __NR_futex was
   allowlisted.

 * Add contrib/gnome-ssh-askpass4 for GNOME 40+ using the GCR API.

 * sshd(8): let ga_init() fail gracefully if getgrouplist does.
   Apparently getgrouplist() can fail on OSX when passed a
   non-existent group name. Other platforms seem to return a group
   list consisting of the numeric gid passed to the function. bz3848

 * ssh-agent(1): exit 0 from SIGTERM under systemd socket-activation,
   preventing a graceful shutdown of an agent via systemd from
   incorrectly marking the service as "failed".

 * build: wrap some autoconf macros in AC_CACHE_CHECK.

   This allows skipping/overriding the OSSH_CHECK_CFLAG_COMPILE and
   OSSH_CHECK_CFLAG_LINK macros used to discover supported compiler
   or linker flags. E.g.

     $ ./configure ossh_cv_cflag__fzero_call_used_regs_used=no
     [...]
     checking if cc supports compile flag -fzero-call-used-regs=used
     and linking succeeds... (cached) no

Checksums:
==========

SHA1 (openssh-10.1.tar.gz) = 8eef44a945a9a9a5a99213ab0d57e35b7ba60e75
SHA256 (openssh-10.1.tar.gz) = j9ymvhvdGMeAvh1oTI2YmOAwv7Ao70gbGPyPedgQsBU=

SHA1 (openssh-10.1p1.tar.gz) = 7fd17b99d1beffb47cd380d64079e920bb0bd91f
SHA256 (openssh-10.1p1.tar.gz) = ufx6K4JXlGem8vQ+SoHI4d/aYU3bT5slWq/XAgu/B1g=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 10.0 / 10.0p2 (2025-04-09) 

OpenSSH 10.0 was released on 2025-04-09. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Potentially-incompatible changes
--------------------------------

 * This release removes support for the weak DSA signature
   algorithm, completing the deprecation process that began in
   2015 (when DSA was disabled by default) and repeatedly warned
   over the last 12 months.

 * scp(1), sftp(1): pass "ControlMaster no" to ssh when invoked by
   scp & sftp. This disables implicit session creation by these
   tools when ControlMaster was set to yes/auto by configuration,
   which some users found surprising. This change will not prevent
   scp/sftp from using an existing multiplexing session if one had
   already been created. GHPR557

 * This release has the version number 10.0 and announces itself
   as "SSH-2.0-OpenSSH_10.0". Software that naively matches
   versions using patterns like "OpenSSH_1*" may be confused by
   this.

 * sshd(8): this release removes the code responsible for the
   user authentication phase of the protocol from the per-
   connection sshd-session binary to a new sshd-auth binary.
   Splitting this code into a separate binary ensures that the
   crucial pre-authentication attack surface has an entirely
   disjoint address space from the code used for the rest of the
   connection. It also yields a small runtime memory saving as the
   authentication code will be unloaded after the authentication
   phase completes. This change should be largely invisible to
   users, though some log messages may now come from "sshd-auth"
   instead of "sshd-session". Downstream distributors of OpenSSH
   will need to package the sshd-auth binary.

 * sshd(8): this release disables finite field (a.k.a modp)
   Diffie-Hellman key exchange in sshd by default. Specifically,
   this removes the "diffie-hellman-group*" and
   "diffie-hellman-group-exchange-*" methods from the default
   KEXAlgorithms list. The client is unchanged and continues to
   support these methods by default. Finite field Diffie Hellman
   is slow and computationally expensive for the same security
   level as Elliptic Curve DH or PQ key agreement while offering
   no redeeming advantages. ECDH has been specified for the SSH
   protocol for 15 years and some form of ECDH has been the default
   key exchange in OpenSSH for the last 14 years.

 * sshd(8): this release removes the implicit fallback to compiled-
   in groups for Diffie-Hellman Group Exchange KEX when the moduli
   file exists but does not contain moduli within the client-
   requested range.  The fallback behaviour remains for the case
   where the moduli file does not exist at all. This allows
   administrators more explicit control over which DH groups will
   be selected, but can lead to connection failures if the moduli
   file is edited incorrectly. bz#2793

Changes since OpenSSH 9.9
=========================

This release contains a minor security fix as well as a number of
feature improvements and bugfixes.

Security
========

* sshd(8): fix the DisableForwarding directive, which was failing
  to disable X11 forwarding and agent forwarding as documented.
  X11 forwarding is disabled by default in the server and agent
  forwarding is off by default in the client.

New features
------------

 * ssh(1): the hybrid post-quantum algorithm mlkem768x25519-sha256
   is now used by default for key agreement. This algorithm is
   considered to be safe against attack by quantum computers,
   is guaranteed to be no less strong than the popular
   curve25519-sha256 algorithm, has been standardised by NIST
   and is considerably faster than the previous default.

 * ssh(1): prefer AES-GCM to AES-CTR mode when selecting a cipher
   for the connection. The default cipher preference list is now
   Chacha20/Poly1305, AES-GCM (128/256) followed by AES-CTR
   (128/192/256).

 * ssh(1): add %-token and environment variable expansion to the
   ssh_config SetEnv directive.

 * ssh(1): allow %-token and environment variable expansion in
   the ssh_config User directive, with the exception of %r and %C
   which would be self-referential. bz#3477

 * ssh(1), sshd(8): add "Match version" support to ssh_config and
   sshd_config. Allows matching on the local version of OpenSSH,
   e.g. "Match version OpenSSH_10.*".

 * ssh(1): add support for "Match sessiontype" to ssh_config.
   Allows matching on the type of session initially requested,
   either "shell" for interactive sessions, "exec" for command
   execution sessions, "subsystem" for subsystem requests, such as
   sftp, or "none" for transport/forwarding-only sessions.

 * ssh(1): add support for "Match command ..." support to
   ssh_config, allowing matching on the remote command as specified
   on the command-line.

 * ssh(1): allow 'Match tagged ""' and 'Match command ""' to match
   empty tag and command values respectively.

 * sshd(8): allow glob(3) patterns to be used in sshd_config
   AuthorizedKeysFile and AuthorizedPrincipalsFile directives.
   bz2755

 * sshd(1): support the VersionAddendum in the client, mirroring
   the option of the same name in the server; bz2745

 * ssh-agent(1): the agent will now delete all loaded keys when
   signaled with SIGUSR1. This allows deletion of keys without
   having access to $SSH_AUTH_SOCK.

 * Portable OpenSSH, ssh-agent(1): support systemd-style socket
   activation in ssh-agent using the LISTEN_PID/LISTEN_FDS
   mechanism. Activated when these environment variables are set,
   the agent is started with the -d or -D option and no socket path
   is set. GHPR502

 * ssh-keygen(1): support FIDO tokens that return no attestation
   data, e.g. recent WinHello. GHPR542

 * ssh-agent(1): add a "-Owebsafe-allow=..." option to allow the
   default FIDO application ID allow-list to be overridden.

 * Add a work-in-progress tool to verify FIDO attestation blobs
   that ssh-keygen can optionally write when enrolling FIDO keys.
   This tool is available under regress/misc/ssh-verify-attestation
   for experimentation but is not installed by "make install".

 * ssh-keygen(1): allow "-" as output file for moduli screening.
   GHPR393

Bugfixes
--------

 * sshd(8): remove assumption that the sshd_config and any configs
   it includes can fit in a (possibly enlarged) socket buffer.
   Previously it was possible to create a sufficiently large
   configuration that could cause sshd to fail to accept any
   connection. sshd(8) will now actively manage sending its config
   to the sshd-session sub-process.

 * ssh(1): don't start the ObscureKeystrokeTiming mitigations if
   there has been traffic on a X11 forwarding channel recently.
   Should fix X11 forwarding performance problems when this setting
   is enabled. bz3655

 * ssh(1): prohibit the comma character in hostnames accepted, but
   allow an underscore as the first character in a hostname.

 * sftp(1): set high-water when resuming a "put". Prevents bogus
   "server reordered acks" debug message.

 * ssh(1), sshd(8): fix regression in openssh-9.8, which would fail
   to accept "Match criteria=argument" as well as the documented
   "Match criteria argument" syntax in ssh_config and sshd_config.
   bz3739

 * sftp(1), ssh(1): fix a number possible NULL dereference bugs,
   including Coverity CIDs 405019 and 477813.

 * sshd(8): fix PerSourcePenalty incorrectly using "crash" penalty
   when LoginGraceTime was exceeded. bz3797

 * sshd(8): fix "Match invalid-user" from incorrectly being
   activated in initial configuration pass when no other predicates
   were present on the match line

 * sshd(8): fix debug logging of user specific delay. GHPR#552

 * sshd(8): improve debug logging across sub-process boundaries.
   Previously some log messages were lost early in the sshd-auth and
   sshd-session processes' life.

 * ssh(1): require control-escape character sequences passed via
   the '-e ^x' command-line to be exactly two characters long. Avoids
   one byte out-of-bounds read if ssh is invoked as "ssh -e^ ..."
   GHPR368

 * ssh(1), sshd(8): prevent integer overflow in x11 port handling.
   These are theoretically possible if the admin misconfigured
   X11DisplayOffset or the user misconfigures their own $DISPLAY,
   but don't happen in normal operation. bz#3730

 * ssh-keygen(1): don't mess up ssh-keygen -l output when the file
   contains CR characters; GHPR236 bz3385.

 * sshd(8): add rate limits to logging of connections dropped by
   PerSourcePenalties. Previously these could be noisy in logs.

 * ssh(1): fix argument of "Compression" directive in ssh -G config
   dump, which regressed in openssh-9.8.

 * sshd(8): fix a corner-case triggered by UpdateHostKeys when sshd
   refuses to accept the signature returned by an agent holding host
   keys during the hostkey rotation sub-protocol. This situation
   could occur in situations where a PKCS#11 smartcard that lacked
   support for particular signature algorithms was used to store
   host keys.

 * ssh-keygen(1): when using RSA keys to sign messages with
   "ssh-keygen -Y", select the signature algorithm based on the
   requested hash algorithm ("-Ohashalg=xxx"). This allows using
   something other than the default of rsa-sha2-512, which may not
   be supported on all signing backends, e.g. some smartcards only
   support SHA256.

 * ssh(1), sshd(8), ssh-keyscan(1): fix ML-KEM768x25519 KEX on
   big-endian systems.

 * Many regression and interop test improvements.

Portability
-----------

 * All: add support for AWS-LC (AWS libcrypto). bz3784

 * sshd(8): add wtmpdb support as a Y2038 safe wtmp replacement.

 * sshd(8): add support for locking sshd into memory, enabled with
   the --with-linux-memlock-onfault configure flag.

 * Add support for building a standalone sk-libfido2 library,
   enabled by --with-security-key-standalone

 * ssh(1), sshd(8), ssh-keyscan(1): include __builtin_popcount
   replacement function. for compilers that lack it.

 * All: Check for and replace le32toh, le64toh, htole64 separately.
   It appears that at least some versions of endian.h in glibc do
   not have the latter two. bz#3794

 * Remove ancient RHL 6.x config in RPM spec.

Checksums:
==========

 - SHA1 (openssh-10.0.tar.gz) = 933f4fded0497ef6a588381257276e156a70f9c3
 - SHA256 (openssh-10.0.tar.gz) = oaJ+cXLCVoCZAz9W5W1vF7ko4GJW6iq7JmblrPUA34Q=

 - SHA1 (openssh-10.0p2.tar.gz) = ac4205e827aea383bf316a33a0e2d5b66b85fcf8
 - SHA256 (openssh-10.0p2.tar.gz) = AhoucJoO30JQsSVr1anlAEEakN3avqgw7VnO+Q652Fw=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 9.9p2 (2025-02-18) 

OpenSSH 9.9p2 was released on 2025-02-18. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Changes since OpenSSH 9.9p1
===========================

This release fixes two security bugs.

Security
========

* Fix CVE-2025-26465 - ssh(1) in OpenSSH versions 6.8p1 to 9.9p1
  (inclusive) contained a logic error that allowed an on-path
  attacker (a.k.a MITM) to impersonate any server when the
  VerifyHostKeyDNS option is enabled. This option is off by default.

* Fix CVE-2025-26466 - sshd(8) in OpenSSH versions 9.5p1 to 9.9p1
  (inclusive) is vulnerable to a memory/CPU denial-of-service related
  to the handling of SSH2_MSG_PING packets. This condition may be
  mitigated using the existing PerSourcePenalties feature.

Both vulnerabilities were discovered and demonstrated to be exploitable
by the Qualys Security Advisory team. We thank them for their detailed
review of OpenSSH.

For OpenBSD, fixes to these problems are available as errata; refer
to https://www.openbsd.org/errata.html
 
Bugfixes
========

 * ssh(1), sshd(8): fix regression in Match directive that caused
   failures when predicates and their arguments were separated by '='
   characters instead of whitespace (bz3739).

 * sshd(8): fix the "Match invalid-user" predicate, which was matching
   incorrectly in the initial pass of config evaluation.

 * ssh(1), sshd(8), ssh-keyscan(1): fix mlkem768x25519-sha256 key
   exchange on big-endian systems.

 * Fix a number of build problems on particular operating systems /
   configurations.

Checksums:
==========

 - SHA1 (openssh-9.9p2.tar.gz) = edefe960645780dee78059c444d4261667ad3056
 - SHA256 (openssh-9.9p2.tar.gz) = karbYD4IzChe3fll4RmdAlhfqU2ZTWyuW0Hhch4hVnM=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 9.9 / 9.9p1 (2024-09-19) 

OpenSSH 9.9 was released on 2024-09-19. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Future deprecation notice
=========================

OpenSSH plans to remove support for the DSA signature algorithm in
early 2025. This release disables DSA by default at compile time.

DSA, as specified in the SSHv2 protocol, is inherently weak - being
limited to a 160 bit private key and use of the SHA1 digest. Its
estimated security level is only 80 bits symmetric equivalent.

OpenSSH has disabled DSA keys by default since 2015 but has retained
run-time optional support for them. DSA was the only mandatory-to-
implement algorithm in the SSHv2 RFCs, mostly because alternative
algorithms were encumbered by patents when the SSHv2 protocol was
specified.

This has not been the case for decades at this point and better
algorithms are well supported by all actively-maintained SSH
implementations. We do not consider the costs of maintaining DSA
in OpenSSH to be justified and hope that removing it from OpenSSH
can accelerate its wider deprecation in supporting cryptography
libraries.

Currently DSA is disabled at compile time. The final step of
removing DSA support entirely is planned for the first OpenSSH
release of 2025.

DSA support may be re-enabled on OpenBSD by setting "DSAKEY=yes"
in Makefile.inc. To enable DSA support in portable OpenSSH, pass
the "--enable-dsa-keys" option to configure.

Potentially-incompatible changes
--------------------------------

 * ssh(1): remove support for pre-authentication compression.
   OpenSSH has only supported post-authentication compression in
   the server for some years. Compression before authentication
   significantly increases the attack surface of SSH servers and risks
   creating oracles that reveal information about information sent
   during authentication.

 * ssh(1), sshd(8): processing of the arguments to the "Match"
   configuration directive now follows more shell-like rules for
   quoted strings, including allowing nested quotes and \-escaped
   characters. If configurations contained workarounds for the
   previous simplistic quote handling then they may need to be
   adjusted. If this is the case, it's most likely to be in the
   arguments to a "Match exec" confition. In this case, moving the
   command to be evaluated from the Match line to an external shell
   script is easiest way to preserve compatibility with both the old
   and new versions.

Changes since OpenSSH 9.8
=========================

This release contains a number of new features and bugfixes.

New features
------------

 * ssh(1), sshd(8): add support for a new hybrid post-quantum key
   exchange based on the FIPS 203 Module-Lattice Key Enapsulation
   mechanism (ML-KEM) combined with X25519 ECDH as described by
   https://datatracker.ietf.org/doc/html/draft-kampanakis-curdle-ssh-pq-ke-03
   This algorithm "mlkem768x25519-sha256" is available by default.

 * ssh(1): the ssh_config "Include" directive can now expand
   environment as well as the same set of %-tokens "Match Exec"
   supports.

 * sshd(8): add a sshd_config "RefuseConnection" option that, if set
   will terminate the connection at the first authentication request.

 * sshd(8): add a "refuseconnection" penalty class to sshd_config
   PerSourcePenalties that is applied when a connection is dropped by
   the new RefuseConnection keyword.

 * sshd(8): add a "Match invalid-user" predicate to sshd_config Match
   options that matches when the target username is not valid on the
   server.

 * ssh(1), sshd(8): update the Streamlined NTRUPrime code to a
   substantially faster implementation.

 * ssh(1), sshd(8): the hybrid Streamlined NTRUPrime/X25519 key
   exchange algorithm now has an IANA-assigned name in addition to
   the "@openssh.com" vendor extension name. This algorithm is now
   also available under this name "sntrup761x25519-sha512"

 * ssh(1), sshd(8), ssh-agent(1): prevent private keys from being
   included in core dump files for most of their lifespans. This is
   in addition to pre-existing controls in ssh-agent(1) and sshd(8)
   that prevented coredumps. This feature is supported on OpenBSD,
   Linux and FreeBSD.

 * All: convert key handling to use the libcrypto EVP_PKEY API, with
   the exception of DSA.

 * sshd(8): add a random amount of jitter (up to 4 seconds) to the
   grace login time to make its expiry unpredictable.

Bugfixes
--------

 * sshd(8): relax absolute path requirement back to what it was prior
   to OpenSSH 9.8, which incorrectly required that sshd was started
   with an absolute path in inetd mode. bz3717

 * sshd(8): fix regression introduced in openssh-9.8 that swapped the
   order of source and destination addresses in some sshd log messages.

 * sshd(8): do not apply authorized_keys options when signature
   verification fails. Prevents more restrictive key options being
   incorrectly applied to subsequent keys in authorized_keys. bz3733

 * ssh-keygen(1): include pathname in some of ssh-keygen's passphrase
   prompts. Helps the user know what's going on when ssh-keygen is
   invoked via other tools. Requested in GHPR503

 * ssh(1), ssh-add(1): make parsing user@host consistently look for
   the last '@' in the string rather than the first. This makes it
   possible to more consistently use usernames that contain '@'
   characters.

 * ssh(1), sshd(8): be more strict in parsing key type names. Only
   allow short names (e.g "rsa") in user-interface code and require
   full SSH protocol names (e.g. "ssh-rsa") everywhere else. bz3725

 * regress: many performance and correctness improvements to the
   re-keying regression test.

 * ssh-keygen(1): clarify that ed25519 is the default key type
   generated and clarify that rsa-sha2-512 is the default signature
   scheme when RSA is in use. GHPR505

 * sshd(8): fix minor memory leak in Subsystem option parsing; GHPR515

 * All: additional hardening and consistency checks for the sshbuf
   code.

 * sshd(8): reduce default logingrace penalty to ensure that a single
   forgotten login that times out will be below the penalty threshold.

 * ssh(1): fix proxy multiplexing (-O proxy) bug. If a mux started with
   ControlPersist then later has a forwarding added using mux proxy
   connection and the forwarding was used, then when the mux proxy
   session terminated, the mux master process would issue a bad message
   that terminated the connection.

Portability
-----------

 * sync contrib/ssh-copy-id to the latest upstream version.

 * regress: improve portablility for some awk(1) usage (e.g. Solaris)

 * In the contrib/redhat RPM spec file, without_openssl was previously
   incorrectly enabled unconditionally.

 * sshd(8) restore audit call before exit that regressed in openssh-9.8
   Fixes an issue where the SSH_CONNECTION_ABANDON event was not
   recorded.

 * sshd(8): add support for class-imposed login restrictions on FreeBSD.
   Allowing auth_hostok(3) and auth_timeok(3) to control logins.

 * Build fixes for Musl libc.

 * Fix detection of setres*id on GNU/Hurd

Checksums:
==========

 - SHA1 (openssh-9.9.tar.gz) = 080acf6ff0b862e8faa3baa3920a079536d28e85
 - SHA256 (openssh-9.9.tar.gz) = h1xwa7CVcJfN7I9MgxxPBpUELzo+tnmLy+6slYHTUtw=

 - SHA1 (openssh-9.9p1.tar.gz) = 5ded7eb0add0b02b5d1a1c4bf5cb2c89d2117b53
 - SHA256 (openssh-9.9p1.tar.gz) = s0P7zb/4fxWxmG5uFdbU/Jp9NgZr5rf7UHCHuo+WbAI=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 9.8 / 9.8p1 (2024-07-01) 

OpenSSH 9.8 was released on 2024-07-01. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Security
========

This release contains fixes for two security problems, one critical
and one minor.

1) Race condition in sshd(8)

A critical vulnerability in sshd(8) was present in Portable OpenSSH
versions between 8.5p1 and 9.7p1 (inclusive) that may allow arbitrary
code execution with root privileges.

Successful exploitation has been demonstrated on 32-bit Linux/glibc
systems with ASLR. Under lab conditions, the attack requires on
average 6-8 hours of continuous connections up to the maximum the
server will accept. Exploitation on 64-bit systems is believed to be
possible but has not been demonstrated at this time. It's likely that
these attacks will be improved upon.

Exploitation on non-glibc systems is conceivable but has not been
examined. Systems that lack ASLR or users of downstream Linux
distributions that have modified OpenSSH to disable per-connection
ASLR re-randomisation (yes - this is a thing, no - we don't
understand why) may potentially have an easier path to exploitation.
OpenBSD is not vulnerable.

We thank the Qualys Security Advisory Team for discovering, reporting
and demonstrating exploitability of this problem, and for providing
detailed feedback on additional mitigation measures.

2) Logic error in ssh(1) ObscureKeystrokeTiming

In OpenSSH version 9.5 through 9.7 (inclusive), when connected to an
OpenSSH server version 9.5 or later, a logic error in the ssh(1)
ObscureKeystrokeTiming feature (on by default) rendered this feature
ineffective - a passive observer could still detect which network
packets contained real keystrokes when the countermeasure was active
because both fake and real keystroke packets were being sent
unconditionally.

This bug was found by Philippos Giavridis and also independently by
Jacky Wei En Kung, Daniel Hugenroth and Alastair Beresford of the
University of Cambridge Computer Lab.

Worse, the unconditional sending of both fake and real keystroke
packets broke another long-standing timing attack mitigation. Since
OpenSSH 2.9.9 sshd(8) has sent fake keystroke echo packets for
traffic received on TTYs in echo-off mode, such as when entering a
password into su(8) or sudo(8). This bug rendered these fake
keystroke echoes ineffective and could allow a passive observer of
a SSH session to once again detect when echo was off and obtain
fairly limited timing information about keystrokes in this situation
(20ms granularity by default).

This additional implication of the bug was identified by Jacky Wei
En Kung, Daniel Hugenroth and Alastair Beresford and we thank them
for their detailed analysis.

This bug does not affect connections when ObscureKeystrokeTiming
was disabled or sessions where no TTY was requested.

Future deprecation notice
=========================

OpenSSH plans to remove support for the DSA signature algorithm in
early 2025. This release disables DSA by default at compile time.

DSA, as specified in the SSHv2 protocol, is inherently weak - being
limited to a 160 bit private key and use of the SHA1 digest. Its
estimated security level is only 80 bits symmetric equivalent.

OpenSSH has disabled DSA keys by default since 2015 but has retained
run-time optional support for them. DSA was the only mandatory-to-
implement algorithm in the SSHv2 RFCs, mostly because alternative
algorithms were encumbered by patents when the SSHv2 protocol was
specified.

This has not been the case for decades at this point and better
algorithms are well supported by all actively-maintained SSH
implementations. We do not consider the costs of maintaining DSA
in OpenSSH to be justified and hope that removing it from OpenSSH
can accelerate its wider deprecation in supporting cryptography
libraries.

This release, and its deactivation of DSA by default at compile-time,
marks the second step in our timeline to finally deprecate DSA. The
final step of removing DSA support entirely is planned for the first
OpenSSH release of 2025.

DSA support may be re-enabled in OpenBSD by setting "DSAKEY=yes"
in Makefile.inc. To enable DSA support in portable OpenSSH, pass
the "--enable-dsa-keys" option to configure.

Potentially-incompatible changes
--------------------------------

 * all: as mentioned above, the DSA signature algorithm is now
   disabled at compile time.

 * sshd(8): the server will now block client addresses that
   repeatedly fail authentication, repeatedly connect without ever
   completing authentication or that crash the server. See the
   discussion of PerSourcePenalties below for more information.
   Operators of servers that accept connections from many users, or
   servers that accept connections from addresses behind NAT or
   proxies may need to consider these settings.

 * sshd(8): the server has been split into a listener binary, sshd(8),
   and a per-session binary "sshd-session". This allows for a much
   smaller listener binary, as it no longer needs to support the SSH
   protocol. As part of this work, support for disabling privilege
   separation (which previously required code changes to disable) and
   disabling re-execution of sshd(8) has been removed. Further
   separation of sshd-session into additional, minimal binaries is
   planned for the future.

 * sshd(8): several log messages have changed. In particular, some
   log messages will be tagged with as originating from a process
   named "sshd-session" rather than "sshd".

 * ssh-keyscan(1): this tool previously emitted comment lines
   containing the hostname and SSH protocol banner to standard error.
   This release now emits them to standard output, but adds a new
   "-q" flag to silence them altogether.

 * sshd(8): (portable OpenSSH only) sshd will no longer use argv[0]
   as the PAM service name. A new "PAMServiceName" sshd_config(5)
   directive allows selecting the service name at runtime. This
   defaults to "sshd". bz2101

 * (portable OpenSSH only) Automatically-generated files, such as
   configure, config.h.in, etc will now be checked in to the portable
   OpenSSH git release branch (e.g. V_9_8). This should ensure that
   the contents of the signed release branch exactly match the
   contents of the signed release tarball.

Changes since OpenSSH 9.7
=========================

This release contains mostly bugfixes.

New features
------------

 * sshd(8): as described above, sshd(8) will now penalise client
   addresses that, for various reasons, do not successfully complete
   authentication. This feature is controlled by a new sshd_config(5)
   PerSourcePenalties option and is on by default.

   sshd(8) will now identify situations where the session did not
   authenticate as expected. These conditions include when the client
   repeatedly attempted authentication unsuccessfully (possibly
   indicating an attack against one or more accounts, e.g. password
   guessing), or when client behaviour caused sshd to crash (possibly
   indicating attempts to exploit bugs in sshd).

   When such a condition is observed, sshd will record a penalty of
   some duration (e.g. 30 seconds) against the client's address. If
   this time is above a minimum configurable threshold, then all
   connections from the client address will be refused (along with any
   others in the same PerSourceNetBlockSize CIDR range) until the
   penalty expire.

   Repeated offenses by the same client address will accrue greater
   penalties, up to a configurable maximum. Address ranges may be
   fully exempted from penalties, e.g. to guarantee access from a set
   of trusted management addresses, using the new sshd_config(5)
   PerSourcePenaltyExemptList option.

   We hope these options will make it significantly more difficult for
   attackers to find accounts with weak/guessable passwords or exploit
   bugs in sshd(8) itself. This option is enabled by default.

 * ssh(8): allow the HostkeyAlgorithms directive to disable the
   implicit fallback from certificate host key to plain host keys.

Bugfixes
--------

 * misc: fix a number of inaccuracies in the PROTOCOL.*
   documentation files. GHPR430 GHPR487

 * all: switch to strtonum(3) for more robust integer parsing in most
   places.

 * ssh(1), sshd(8): correctly restore sigprocmask around ppoll()

 * ssh-keysign(8): stricter validation of messaging socket fd GHPR492

 * sftp(1): flush stdout after writing "sftp>" prompt when not using
   editline. GHPR480

 * sftp-server(8): fix home-directory extension implementation, it
   previously always returned the current user's home directory
   contrary to the spec. GHPR477

 * ssh-keyscan(1): do not close stdin to prevent error messages when
   stdin is read multiple times. E.g.
   echo localhost | ssh-keyscan -f - -f -

 * regression tests: fix rekey test that was testing the same KEX
   algorithm repeatedly instead of testing all of them. bz3692

 * ssh_config(5), sshd_config(5): clarify the KEXAlgorithms directive
   documentation, especially around what is supported vs available.
   bz3701.

Portability
-----------

 * sshd(8): expose SSH_AUTH_INFO_0 always to PAM auth modules
   unconditionally. The previous behaviour was to expose it only when
   particular authentication methods were in use.

 * build: fix OpenSSL ED25519 support detection. An incorrect function
   signature in configure.ac previously prevented enabling the recently
   added support for ED25519 private keys in PEM PKCS8 format.

 * ssh(1), ssh-agent(8): allow the presence of the WAYLAND_DISPLAY
   environment variable to enable SSH_ASKPASS, similarly to the X11
   DISPLAY environment variable. GHPR479

 * build: improve detection of the -fzero-call-used-regs compiler
   flag. bz3673.

 * build: relax OpenSSL version check to accept all OpenSSL 3.x
   versions.

 * sshd(8): add support for notifying systemd on server listen and
   reload, using a standalone implementation that doesn't depend on
   libsystemd. bz2641

Checksums:
==========

 - SHA1 (openssh-9.8.tar.gz) = bc45cedae7f70b41e9922ef4c9f56e74b9a659b7
 - SHA256 (openssh-9.8.tar.gz) = Dnc69VLWFBFdiaz8wySPlvHjb7wZfh/kblQ8ISuQr1Y=

 - SHA1 (openssh-9.8p1.tar.gz) = a0bb501b11349f5c5c33a269351be091dc2c2727
 - SHA256 (openssh-9.8p1.tar.gz) = 3YvQAqN5tdSZ37BQ3R+pr4Ap6ARh9LtsUjxJlz9aOfM=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 9.7 / 9.7p1 (2024-03-11) 

OpenSSH 9.7 was released on 2024-03-11. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Future deprecation notice
=========================

OpenSSH plans to remove support for the DSA signature algorithm in
early 2025 and compile-time disable it later this year.

DSA, as specified in the SSHv2 protocol, is inherently weak - being
limited to a 160 bit private key and use of the SHA1 digest. Its
estimated security level is only 80 bits symmetric equivalent.

OpenSSH has disabled DSA keys by default since 2015 but has retained
run-time optional support for them. DSA was the only mandatory-to-
implement algorithm in the SSHv2 RFCs[3], mostly because alternative
algorithms were encumbered by patents when the SSHv2 protocol was
specified.

This has not been the case for decades at this point and better
algorithms are well supported by all actively-maintained SSH
implementations. We do not consider the costs of maintaining DSA in
OpenSSH to be justified and hope that removing it from OpenSSH can
accelerate its wider deprecation in supporting cryptography
libraries.

This release makes DSA support in OpenSSH compile-time optional,
defaulting to on. We intend the next release to change the default
to disable DSA at compile time. The first OpenSSH release of 2025
will remove DSA support entirely.

Changes since OpenSSH 9.6
=========================

This release contains mostly bugfixes.

New features
------------

 * ssh(1), sshd(8): add a "global" ChannelTimeout type that watches
   all open channels and will close all open channels if there is no
   traffic on any of them for the specified interval. This is in
   addition to the existing per-channel timeouts added recently.

   This supports situations like having both session and x11
   forwarding channels open where one may be idle for an extended
   period but the other is actively used. The global timeout could
   close both channels when both have been idle for too long.

 * All: make DSA key support compile-time optional, defaulting to on.

Bugfixes
--------

 * sshd(8): don't append an unnecessary space to the end of subsystem
   arguments (bz3667)

 * ssh(1): fix the multiplexing "channel proxy" mode, broken when
   keystroke timing obfuscation was added. (GHPR#463)

 * ssh(1), sshd(8): fix spurious configuration parsing errors when
   options that accept array arguments are overridden (bz3657).

 * ssh-agent(1): fix potential spin in signal handler (bz3670)

 * Many fixes to manual pages and other documentation, including
   GHPR#462, GHPR#454, GHPR#442 and GHPR#441.

 * Greatly improve interop testing against PuTTY.

Portability
-----------

 * Improve the error message when the autoconf OpenSSL header check
   fails (bz#3668)

 * Improve detection of broken toolchain -fzero-call-used-regs support
   (bz3645).

 * Fix regress/misc/fuzz-harness fuzzers and make them compile without
   warnings when using clang16

Checksums:
==========

 - SHA1 (openssh-9.7.tar.gz) = 163272058edc20a8fde81661734a6684c9b4db11
 - SHA256 (openssh-9.7.tar.gz) = gXDWrF4wN2UWyPjyjvVhpjjKd7D2qI6LyZiIYhbJQVg=

 - SHA1 (openssh-9.7p1.tar.gz) = ce8985ea0ea2f16a5917fd982ade0972848373cc
 - SHA256 (openssh-9.7p1.tar.gz) = SQQm92bYKidj/KzY2D6j1weYdQx70q/y5X3FZg93P/0=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 9.6 / 9.6p1 (2023-12-18) 

OpenSSH 9.6 was released on 2023-12-18. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Changes since OpenSSH 9.5
=========================

This release contains a number of security fixes, some small features
and bugfixes.

Security
========

This release contains fixes for a newly-discovered weakness in the
SSH transport protocol, a logic error relating to constrained PKCS#11
keys in ssh-agent(1) and countermeasures for programs that invoke
ssh(1) with user or hostnames containing invalid characters.

 * ssh(1), sshd(8): implement protocol extensions to thwart the
   so-called "Terrapin attack" discovered by Fabian Bäumer, Marcus
   Brinkmann and Jörg Schwenk. This attack allows a MITM to effect a
   limited break of the integrity of the early encrypted SSH transport
   protocol by sending extra messages prior to the commencement of
   encryption, and deleting an equal number of consecutive messages
   immediately after encryption starts. A peer SSH client/server
   would not be able to detect that messages were deleted.

   While cryptographically novel, the security impact of this attack
   is fortunately very limited as it only allows deletion of
   consecutive messages, and deleting most messages at this stage of
   the protocol prevents user authentication from proceeding and
   results in a stuck connection.

   The most serious identified impact is that it lets a MITM to
   delete the SSH2_MSG_EXT_INFO message sent before authentication
   starts, allowing the attacker to disable a subset of the keystroke
   timing obfuscation features introduced in OpenSSH 9.5. There is no
   other discernible impact to session secrecy or session integrity.

   OpenSSH 9.6 addresses this protocol weakness through a new "strict
   KEX" protocol extension that will be automatically enabled when
   both the client and server support it. This extension makes
   two changes to the SSH transport protocol to improve the integrity
   of the initial key exchange.

   Firstly, it requires endpoints to terminate the connection if any
   unnecessary or unexpected message is received during key exchange
   (including messages that were previously legal but not strictly
   required like SSH2_MSG_DEBUG). This removes most malleability from
   the early protocol.

   Secondly, it resets the Message Authentication Code counter at the
   conclusion of each key exchange, preventing previously inserted
   messages from being able to make persistent changes to the
   sequence number across completion of a key exchange. Either of
   these changes should be sufficient to thwart the Terrapin Attack.

   More details of these changes are in the PROTOCOL file in the
   OpenSSH source distribition.

 * ssh-agent(1): when adding PKCS#11-hosted private keys while
   specifying destination constraints, if the PKCS#11 token returned
   multiple keys then only the first key had the constraints applied.
   Use of regular private keys, FIDO tokens and unconstrained keys
   are unaffected.

 * ssh(1): if an invalid user or hostname that contained shell
   metacharacters was passed to ssh(1), and a ProxyCommand,
   LocalCommand directive or "match exec" predicate referenced the
   user or hostname via %u, %h or similar expansion token, then
   an attacker who could supply arbitrary user/hostnames to ssh(1)
   could potentially perform command injection depending on what
   quoting was present in the user-supplied ssh_config(5) directive.

   This situation could arise in the case of git submodules, where
   a repository could contain a submodule with shell characters in
   its user/hostname. Git does not ban shell metacharacters in user
   or host names when checking out repositories from untrusted
   sources.

   Although we believe it is the user's responsibility to ensure
   validity of arguments passed to ssh(1), especially across a
   security boundary such as the git example above, OpenSSH 9.6 now
   bans most shell metacharacters from user and hostnames supplied
   via the command-line. This countermeasure is not guaranteed to be
   effective in all situations, as it is infeasible for ssh(1) to
   universally filter shell metacharacters potentially relevant to
   user-supplied commands.

   User/hostnames provided via ssh_config(5) are not subject to these
   restrictions, allowing configurations that use strange names to
   continue to be used, under the assumption that the user knows what
   they are doing in their own configuration files.

Potentially incompatible changes
--------------------------------

 * ssh(1), sshd(8): the RFC4254 connection/channels protocol provides
   a TCP-like window mechanism that limits the amount of data that
   can be sent without acceptance from the peer. In cases where this
   limit was exceeded by a non-conforming peer SSH implementation,
   ssh(1)/sshd(8) previously discarded the extra data. From OpenSSH
   9.6, ssh(1)/sshd(8) will now terminate the connection if a peer
   exceeds the window limit by more than a small grace factor. This
   change should have no effect of SSH implementations that follow
   the specification.

New features
------------

 * ssh(1): add a %j token that expands to the configured ProxyJump
   hostname (or the empty string if this option is not being used)
   that can be used in a number of ssh_config(5) keywords. bz3610

 * ssh(1): add ChannelTimeout support to the client, mirroring the
   same option in the server and allowing ssh(1) to terminate
   quiescent channels.

 * ssh(1), sshd(8), ssh-add(1), ssh-keygen(1): add support for
   reading ED25519 private keys in PEM PKCS8 format. Previously
   only the OpenSSH private key format was supported.

 * ssh(1), sshd(8): introduce a protocol extension to allow
   renegotiation of acceptable signature algorithms for public key
   authentication after the server has learned the username being
   used for authentication. This allows varying sshd_config(5)
   PubkeyAcceptedAlgorithms in a "Match user" block.

 * ssh-add(1), ssh-agent(1): add an agent protocol extension to allow
   specifying certificates when loading PKCS#11 keys. This allows the
   use of certificates backed by PKCS#11 private keys in all OpenSSH
   tools that support ssh-agent(1). Previously only ssh(1) supported
   this use-case.

Bugfixes
--------

 * ssh(1): when deciding whether to enable the keystroke timing
   obfuscation, enable it only if a channel with a TTY is active.

 * ssh(1): switch mainloop from poll(3) to ppoll(3) and mask signals
   before checking flags set in signal handler. Avoids potential
   race condition between signaling ssh to exit and polling. bz3531
 
 * ssh(1): when connecting to a destination with both the
   AddressFamily and CanonicalizeHostname directives in use,
   the AddressFamily directive could be ignored. bz5326

 * sftp(1): correct handling of the limits@openssh.com option when
   the server returned an unexpected message.

 * A number of fixes to the PuTTY and Dropbear regress/integration
   tests.

 * ssh(1): release GSS OIDs only at end of authentication, avoiding
   unnecessary init/cleanup cycles. bz2982

 * ssh_config(5): mention "none" is a valid argument to IdentityFile
   in the manual. bz3080

 * scp(1): improved debugging for paths from the server rejected for
   not matching the client's glob(3) pattern in old SCP/RCP protocol
   mode.

 * ssh-agent(1): refuse signing operations on destination-constrained
   keys if a previous session-bind operation has failed. This may
   prevent a fail-open situation in future if a user uses a mismatched
   ssh(1) client and ssh-agent(1) where the client supports a key type
   that the agent does not support.

Portability
-----------

 * Better identify unsupported and unstable compiler flags, such as
   -fzero-call-used-regs which has been unstable across a several
   clang releases.

 * A number of fixes to regression test reliability and log
   collection.

 * Update the OpenSSL dependency in the RPM specification.

 * sshd(8): for OpenSolaris systems that support privilege limitation
   via the getpflags() interface, prefer using the newer PRIV_XPOLICY
   to PRIV_LIMIT. bz2833

Checksums:
==========

 - SHA1 (openssh-9.6.tar.gz) = a6d4cb69811e879e2f158c2e597fd9f444b26506
 - SHA256 (openssh-9.6.tar.gz) = nejPUhSnG1R1sOmIBi/t+HMNvsRqfN/DJgjwIU2tvqg=

 - SHA1 (openssh-9.6p1.tar.gz) = de300d09ec79fdbf37de4e6672cce4161439f2c3
 - SHA256 (openssh-9.6p1.tar.gz) = kQIRwHJVqMWtZUORtA7lmABxDdgRndU2LeCThap6d3w=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 9.5 / 9.5p1 (2023-10-04) 

OpenSSH 9.5 was released on 2023-10-04. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Changes since OpenSSH 9.4
=========================

This release fixes a number of bugs and adds some small features.

Potentially incompatible changes
--------------------------------

 * ssh-keygen(1): generate Ed25519 keys by default. Ed25519 public keys
   are very convenient due to their small size. Ed25519 keys are
   specified in RFC 8709 and OpenSSH has supported them since version 6.5
   (January 2014).
    
 * sshd(8): the Subsystem directive now accurately preserves quoting of
   subsystem commands and arguments. This may change behaviour for exotic
   configurations, but the most common subsystem configuration
   (sftp-server) is unlikely to be affected.

New features
------------

 * ssh(1): add keystroke timing obfuscation to the client. This attempts
   to hide inter-keystroke timings by sending interactive traffic at
   fixed intervals (default: every 20ms) when there is only a small
   amount of data being sent. It also sends fake "chaff" keystrokes for
   a random interval after the last real keystroke. These are
   controlled by a new ssh_config ObscureKeystrokeTiming keyword.
    
 * ssh(1), sshd(8): Introduce a transport-level ping facility. This adds
   a pair of SSH transport protocol messages SSH2_MSG_PING/PONG to
   implement a ping capability. These messages use numbers in the "local
   extensions" number space and are advertised using a "ping@openssh.com"
   ext-info message with a string version number of "0".
    
 * sshd(8): allow override of Subsystem directives in sshd Match blocks.

Bugfixes
--------

 * scp(1): fix scp in SFTP mode recursive upload and download of
   directories that contain symlinks to other directories. In scp mode,
   the links would be followed, but in SFTP mode they were not. bz3611
    
 * ssh-keygen(1): handle cr+lf (instead of just cr) line endings in
   sshsig signature files.
    
 * ssh(1): interactive mode for ControlPersist sessions if they
   originally requested a tty.
    
 * sshd(8): make PerSourceMaxStartups first-match-wins
    
 * sshd(8): limit artificial login delay to a reasonable maximum (5s)
   and don't delay at all for the "none" authentication mechanism.cw
    bz3602
    
 * sshd(8): Log errors in kex_exchange_identification() with level
   verbose instead of error to reduce preauth log spam. All of those
   get logged with a more generic error message by sshpkt_fatal().
    
 * sshd(8): correct math for ClientAliveInterval that caused the probes
    to be sent less frequently than configured.
    
 * ssh(1): fix regression in OpenSSH 9.4 (mux.c r1.99) that caused
   multiplexed sessions to ignore SIGINT under some circumstances.

Portability
-----------

 * Avoid clang zero-call-used-regs=all bug on Apple compilers, which
   for some reason have version numbers that do not match the upstream
   clang version numbers. bz#3584

 * Fix configure test for zlib 1.3 and later/development versions. bz3604

Checksums:
==========

 - SHA1 (openssh-9.5.tar.gz) = 8a0bd3a91fac338d97d91817af58df731f6509a3
 - SHA256 (openssh-9.5.tar.gz) = sVMxeM3d6g65qBMktJIofxmK4Ipg9dblKif0VnhPeO0=

 - SHA1 (openssh-9.5p1.tar.gz) = 35c16dcc6e7d0a9465faa241476ef24f76b196cc
 - SHA256 (openssh-9.5p1.tar.gz) = 8Cbnt5un+1QPdRgq+W3IqPHbOV+SK7yfbKYDZyaGCGs=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 9.4 / 9.4p1 (2023-08-10) 

OpenSSH 9.4 was released on 2023-08-10. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Changes since OpenSSH 9.3p2
===========================

This release fixes a number of bugs and adds some small features.

Potentially incompatible changes
--------------------------------

 * This release removes support for older versions of libcrypto.
   OpenSSH now requires LibreSSL >= 3.1.0 or OpenSSL >= 1.1.1.
   Note that these versions are already deprecated by their upstream
   vendors.

 * ssh-agent(1): PKCS#11 modules must now be specified by their full
   paths. Previously dlopen(3) could search for them in system
   library directories.

New features
------------

 * ssh(1): allow forwarding Unix Domain sockets via ssh -W.

 * ssh(1): add support for configuration tags to ssh(1).
   This adds a ssh_config(5) "Tag" directive and corresponding
   "Match tag" predicate that may be used to select blocks of
   configuration similar to the pf.conf(5) keywords of the same
   name.

 * ssh(1): add a "match localnetwork" predicate. This allows matching
   on the addresses of available network interfaces and may be used to
   vary the effective client configuration based on network location.

 * ssh(1), sshd(8), ssh-keygen(1): infrastructure support for KRL
   extensions.  This defines wire formats for optional KRL extensions
   and implements parsing of the new submessages. No actual extensions
   are supported at this point.

 * sshd(8): AuthorizedPrincipalsCommand and AuthorizedKeysCommand now
   accept two additional %-expansion sequences: %D which expands to
   the routing domain of the connected session and %C which expands
   to the addresses and port numbers for the source and destination
   of the connection.

 * ssh-keygen(1): increase the default work factor (rounds) for the
   bcrypt KDF used to derive symmetric encryption keys for passphrase
   protected key files by 50%.

Bugfixes
--------

 * ssh-agent(1): improve isolation between loaded PKCS#11 modules
   by running separate ssh-pkcs11-helpers for each loaded provider.

 * ssh(1): make -f (fork after authentication) work correctly with
   multiplexed connections, including ControlPersist. bz3589 bz3589

 * ssh(1): make ConnectTimeout apply to multiplexing sockets and not
   just to network connections.

 * ssh-agent(1), ssh(1): improve defences against invalid PKCS#11
   modules being loaded by checking that the requested module
   contains the required symbol before loading it.

 * sshd(8): fix AuthorizedPrincipalsCommand when AuthorizedKeysCommand
   appears before it in sshd_config. Since OpenSSH 8.7 the
   AuthorizedPrincipalsCommand directive was incorrectly ignored in
   this situation. bz3574

 * sshd(8), ssh(1), ssh-keygen(1): remove vestigal support for KRL
   signatures When the KRL format was originally defined, it included
   support for signing of KRL objects. However, the code to sign KRLs
   and verify KRL signatures was never completed in OpenSSH. This
   release removes the partially-implemented code to verify KRLs.
   All OpenSSH tools now ignore KRL_SECTION_SIGNATURE sections in
   KRL files.

 * All: fix a number of memory leaks and unreachable/harmless integer
   overflows.

 * ssh-agent(1), ssh(1): don't truncate strings logged from PKCS#11
   modules; GHPR406

 * sshd(8), ssh(1): better validate CASignatureAlgorithms in
   ssh_config and sshd_config. Previously this directive would accept
   certificate algorithm names, but these were unusable in practice as
   OpenSSH does not support CA chains. bz3577

 * ssh(1): make `ssh -Q CASignatureAlgorithms` only list signature
   algorithms that are valid for CA signing. Previous behaviour was
   to list all signing algorithms, including certificate algorithms.

 * ssh-keyscan(1): gracefully handle systems where rlimits or the
   maximum number of open files is larger than INT_MAX; bz3581

 * ssh-keygen(1): fix "no comment" not showing on when running
   `ssh-keygen -l` on multiple keys where one has a comment and other
   following keys do not. bz3580

 * scp(1), sftp(1): adjust ftruncate() logic to handle servers that
   reorder requests. Previously, if the server reordered requests then
   the resultant file would be erroneously truncated.

 * ssh(1): don't incorrectly disable hostname canonicalization when
   CanonicalizeHostname=yes and ProxyJump was explicitly set to
   "none". bz3567

 * scp(1): when copying local->remote, check that the source file
   exists before opening an SFTP connection to the server. Based on
   GHPR#370

Portability
-----------

 * All: a number of build fixes for various platforms and
   configuration combinations.

 * sshd(8): provide a replacement for the SELinux matchpathcon()
   function, which is deprecated.

 * All: relax libcrypto version checks for OpenSSL >=3. Beyond
   OpenSSL 3.0, the ABI compatibility guarantees are wider (only
   the library major must match instead of major and minor in
   earlier versions).  bz#3548.

 * Tests: fix build problems for the sk-dummy.so FIDO provider module
   used in some tests.

Checksums:
==========

 - SHA1 (openssh-9.4.tar.gz) = d88126d8d7b8e5bf4656587ac4a16055560641cc
 - SHA256 (openssh-9.4.tar.gz) = 7eqFjx2hAunw+1Jy7f1JQXq//3AMr9B3dKtASDtq8go=

 - SHA1 (openssh-9.4p1.tar.gz) = 5dea1f3c88f9cfe53a711a3c893ee8b7d3ffecff
 - SHA256 (openssh-9.4p1.tar.gz) = Ngj9kIjbIWPOs+YAyFq3nQ3j0iHlkZLqGSPiMmOGaoU=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 9.3p2 (2023-07-19) 

OpenSSH 9.3p2 was released on 2023-07-19. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Changes since OpenSSH 9.3
=========================

This release fixes a security bug.

Security
========

Fix CVE-2023-38408 - a condition where specific libraries loaded via
ssh-agent(1)'s PKCS#11 support could be abused to achieve remote
code execution via a forwarded agent socket if the following
conditions are met:

* Exploitation requires the presence of specific libraries on
  the victim system.
* Remote exploitation requires that the agent was forwarded
  to an attacker-controlled system.

Exploitation can also be prevented by starting ssh-agent(1) with an
empty PKCS#11/FIDO allowlist (ssh-agent -P '') or by configuring
an allowlist that contains only specific provider libraries.

This vulnerability was discovered and demonstrated to be exploitable
by the Qualys Security Advisory team. 
 
In addition to removing the main precondition for exploitation,
this release removes the ability for remote ssh-agent(1) clients
to load PKCS#11 modules by default (see below).

Potentially-incompatible changes
--------------------------------

 * ssh-agent(8): the agent will now refuse requests to load PKCS#11
   modules issued by remote clients by default. A flag has been added
   to restore the previous behaviour "-Oallow-remote-pkcs11".

   Note that ssh-agent(8) depends on the SSH client to identify
   requests that are remote. The OpenSSH >=8.9 ssh(1) client does
   this, but forwarding access to an agent socket using other tools
   may circumvent this restriction.

Checksums:
==========

- SHA1 (openssh-9.3p2.tar.gz) = 219cf700c317f400bb20b001c0406056f7188ea4
- SHA256 (openssh-9.3p2.tar.gz) = IA6+FH9ss/EB/QzfngJEKvfdyimN/9n0VoeOfMrGdug=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 9.3 / 9.3p1 (2023-03-15) 

OpenSSH 9.3 was released on 2023-03-15. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Changes since OpenSSH 9.2
=========================

This release fixes a number of security bugs.

Security
========

This release contains fixes for a security problem and a memory
safety problem. The memory safety problem is not believed to be
exploitable, but we report most network-reachable memory faults as
security bugs.

 * ssh-add(1): when adding smartcard keys to ssh-agent(1) with the
   per-hop destination constraints (ssh-add -h ...) added in OpenSSH
   8.9, a logic error prevented the constraints from being
   communicated to the agent. This resulted in the keys being added
   without constraints. The common cases of non-smartcard keys and
   keys without destination constraints are unaffected. This problem
   was reported by Luci Stanescu.

 * ssh(1): Portable OpenSSH provides an implementation of the
   getrrsetbyname(3) function if the standard library does not
   provide it, for use by the VerifyHostKeyDNS feature. A
   specifically crafted DNS response could cause this function to
   perform an out-of-bounds read of adjacent stack data, but this
   condition does not appear to be exploitable beyond denial-of-
   service to the ssh(1) client.

   The getrrsetbyname(3) replacement is only included if the system's
   standard library lacks this function and portable OpenSSH was not
   compiled with the ldns library (--with-ldns). getrrsetbyname(3) is
   only invoked if using VerifyHostKeyDNS to fetch SSHFP records. This
   problem was found by the Coverity static analyzer.

New features
------------

 * ssh-keygen(1), ssh-keyscan(1): accept -Ohashalg=sha1|sha256 when
   outputting SSHFP fingerprints to allow algorithm selection. bz3493
    
 * sshd(8): add a `sshd -G` option that parses and prints the
   effective configuration without attempting to load private keys
   and perform other checks. This allows usage of the option before
   keys have been generated and for configuration evaluation and
   verification by unprivileged users.

Bugfixes
--------

 * scp(1), sftp(1): fix progressmeter corruption on wide displays;
   bz3534

 * ssh-add(1), ssh-keygen(1): use RSA/SHA256 when testing usability
   of private keys as some systems are starting to disable RSA/SHA1
   in libcrypto.

 * sftp-server(8): fix a memory leak. GHPR363

 * ssh(1), sshd(8), ssh-keyscan(1): remove vestigal protocol
   compatibility code and simplify what's left.

 * Fix a number of low-impact Coverity static analysis findings.
   These include several reported via bz2687

 * ssh_config(5), sshd_config(5): mention that some options are not
   first-match-wins.

 * Rework logging for the regression tests. Regression tests will now
   capture separate logs for each ssh and sshd invocation in a test.

 * ssh(1): make `ssh -Q CASignatureAlgorithms` work as the manpage
   says it should; bz3532.

 * ssh(1): ensure that there is a terminating newline when adding a
   new entry to known_hosts; bz3529

Portability
-----------

 * sshd(8): harden Linux seccomp sandbox. Move to an allowlist of
   mmap(2), madvise(2) and futex(2) flags, removing some concerning
   kernel attack surface.

 * sshd(8): improve Linux seccomp-bpf sandbox for older systems;
   bz3537

Checksums:
==========

- SHA1 (openssh-9.3.tar.gz) = 5f9d2f73ddfe94f3f0a78bdf46704b6ad7b66ec7
- SHA256 (openssh-9.3.tar.gz) = eRcXkFZByz70DUBUcyIdvU0pVxP2X280FrmV8pyUdrk=

- SHA1 (openssh-9.3p1.tar.gz) = 610959871bf8d6baafc3525811948f85b5dd84ab
- SHA256 (openssh-9.3p1.tar.gz) = 6bq6dwGnalHz2Fpiw4OjydzZf6kAuFm8fbEUwYaK+Kg=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 9.2 / 9.2p1 (2023-02-02) 

OpenSSH 9.2 was released on 2023-02-02. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Changes since OpenSSH 9.1
=========================

This release fixes a number of security bugs.

Security
========

This release contains fixes for two security problems and a memory
safety problem. The memory safety problem is not believed to be
exploitable, but we report most network-reachable memory faults as
security bugs.

 * sshd(8): fix a pre-authentication double-free memory fault
   introduced in OpenSSH 9.1. This is not believed to be exploitable,
   and it occurs in the unprivileged pre-auth process that is
   subject to chroot(2) and is further sandboxed on most major
   platforms.

 * ssh(8): in OpenSSH releases after 8.7, the PermitRemoteOpen option
   would ignore its first argument unless it was one of the special
   keywords "any" or "none", causing the permission list to fail open
   if only one permission was specified. bz3515

 * ssh(1): if the CanonicalizeHostname and CanonicalizePermittedCNAMEs
   options were enabled, and the system/libc resolver did not check
   that names in DNS responses were valid, then use of these options
   could allow an attacker with control of DNS to include invalid
   characters (possibly including wildcards) in names added to
   known_hosts files when they were updated. These names would still
   have to match the CanonicalizePermittedCNAMEs allow-list, so
   practical exploitation appears unlikely.

Potentially-incompatible changes
--------------------------------

 * ssh(1): add a new EnableEscapeCommandline ssh_config(5) option that
   controls whether the client-side ~C escape sequence that provides a
   command-line is available. Among other things, the ~C command-line
   could be used to add additional port-forwards at runtime.

   This option defaults to "no", disabling the ~C command-line that
   was previously enabled by default. Turning off the command-line
   allows platforms that support sandboxing of the ssh(1) client
   (currently only OpenBSD) to use a stricter default sandbox policy.

New features
------------

 * sshd(8): add support for channel inactivity timeouts via a new
   sshd_config(5) ChannelTimeout directive. This allows channels that
   have not seen traffic in a configurable interval to be
   automatically closed. Different timeouts may be applied to session,
   X11, agent and TCP forwarding channels.

 * sshd(8): add a sshd_config UnusedConnectionTimeout option to
   terminate client connections that have no open channels for a
   length of time. This complements the ChannelTimeout option above.
    
 * sshd(8): add a -V (version) option to sshd like the ssh client has.

 * ssh(1): add a "Host" line to the output of ssh -G showing the
   original hostname argument. bz3343
    
 * scp(1), sftp(1): add a -X option to both scp(1) and sftp(1) to
   allow control over some SFTP protocol parameters: the copy buffer
   length and the number of in-flight requests, both of which are used
   during upload/download. Previously these could be controlled in
   sftp(1) only. This makes them available in both SFTP protocol
   clients using the same option character sequence.
    
 * ssh-keyscan(1): allow scanning of complete CIDR address ranges,
   e.g.  "ssh-keyscan 192.168.0.0/24". If a CIDR range is passed, then
   it will be expanded to all possible addresses in the range
   including the all-0s and all-1s addresses. bz#976

 * ssh(1): support dynamic remote port forwarding in escape
   command-line's -R processing. bz#3499

Bugfixes
--------

 * ssh(1): when restoring non-blocking mode to stdio fds, restore
   exactly the flags that ssh started with and don't just clobber them
   with zero, as this could also remove the append flag from the set.
   bz3523
    
 * ssh(1): avoid printf("%s", NULL) if using UserKnownHostsFile=none
   and a hostkey in one of the system known hosts file changes.
    
 * scp(1): switch scp from using pipes to a socket-pair for
   communication with its ssh sub-processes, matching how sftp(1)
   operates.

 * sshd(8): clear signal mask early in main(); sshd may have been
   started with one or more signals masked (sigprocmask(2) is not
   cleared on fork/exec) and this could interfere with various things,
   e.g. the login grace timer. Execution environments that fail to
   clear the signal mask before running sshd are clearly broken, but
   apparently they do exist.
    
 * ssh(1): warn if no host keys for hostbased auth can be loaded.
    
 * sshd(8): Add server debugging for hostbased auth that is queued and
   sent to the client after successful authentication, but also logged
   to assist in diagnosis of HostbasedAuthentication problems. bz3507

 * ssh(1): document use of the IdentityFile option as being usable to
   list public keys as well as private keys. GHPR352
    
 * sshd(8): check for and disallow MaxStartups values less than or
   equal to zero during config parsing, rather than failing later at
   runtime.  bz3489
    
 * ssh-keygen(1): fix parsing of hex cert expiry times specified on
   the command-line when acting as a CA.
 
 * scp(1): when scp(1) is using the SFTP protocol for transport (the
   default), better match scp/rcp's handling of globs that don't match
   the globbed characters but do match literally (e.g. trying to
   transfer a file named "foo.[1]"). Previously scp(1) in SFTP mode
   would not match these pathnames but legacy scp/rcp mode would.
   bz3488
    
 * ssh-agent(1): document the "-O no-restrict-websafe" command-line
   option.

 * ssh(1): honour user's umask(2) if it is more restrictive then the
   ssh default (022).

Portability
-----------

 * sshd(8): allow writev(2) in the Linux seccomp sandbox. This seems
   to be used by recent glibcs at least in some configurations during
   error conditions. bz3512.

 * sshd(8): simply handling of SSH_CONNECTION PAM env var, removing
   global variable and checking the return value from pam_putenv.
   bz3508

 * sshd(8): disable SANDBOX_SECCOMP_FILTER_DEBUG that was mistakenly
   enabled during the OpenSSH 9.1 release cycle.

 * misc: update autotools and regenerate the config files using the
   latest autotools

 * all: use -fzero-call-used-regs=used on clang 15 instead of
   -fzero-call-used-reg=all, as some versions of clang 15 have
   miscompile code when it was enabled. bz3475

 * sshd(8): defer PRNG seeding until after the initial closefrom(2)
   call. PRNG seeding will initialize OpenSSL, and some engine
   providers (e.g. Intel's QAT) will open descriptors for their own
   use that closefrom(2) could clobber. bz3483

 * misc: in the poll(2)/ppoll(2) compatibility code, avoid assuming
   the layout of fd_set.

 * sftp-server(8), ssh-agent(1): fix ptrace(2) disabling on older
   FreeBSD kernels. Some versions do not support using id 0 to refer
   to the current PID for procctl, so try again with getpid()
   explicitly before failing.

 * configure.ac: fix -Wstrict-prototypes in configure test code.
   Clang 16 now warns on this and legacy prototypes will be removed
   in C23. GHPR355

 * configure.ac: fix setres*id checks to work with clang-16. glibc
   has the prototypes for setresuid behind _GNU_SOURCE, and clang 16
   will error out on implicit function definitions. bz3497

Checksums:
==========

- SHA1 (openssh-9.2.tar.gz) = e4b806b7c81b87d6c90afe97b3d016ba6cf3ba1c
- SHA256 (openssh-9.2.tar.gz) = yYe9uaaWSeetXGXOxuaaEiIsLnvITmGW+l5dgMZb9QU=

- SHA1 (openssh-9.2p1.tar.gz) = 3b172b8e971773a7018bbf3231f6589ae539ca4b
- SHA256 (openssh-9.2p1.tar.gz) = P2bb8WVftF9Q4cVtpiqwEhjCKIB7ITONY068351xz0Y=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 9.1 / 9.1p1 (2022-10-04) 

OpenSSH 9.1 was released on 2022-10-04. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Changes since OpenSSH 9.0
=========================

This release is focused on bug fixing.

Security
========

This release contains fixes for three minor memory safety problems.
None are believed to be exploitable, but we report most memory safety
problems as potential security vulnerabilities out of caution.

 * ssh-keyscan(1): fix a one-byte overflow in SSH- banner processing.
   Reported by Qualys

 * ssh-keygen(1): double free() in error path of file hashing step in
   signing/verify code; GHPR333

 * ssh-keysign(8): double-free in error path introduced in openssh-8.9

Potentially-incompatible changes
--------------------------------

 * The portable OpenSSH project now signs commits and release tags
   using git's recent SSH signature support. The list of developer
   signing keys is included in the repository as .git_allowed_signers
   and is cross-signed using the PGP key that is still used to sign
   release artifacts:
   https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

 * ssh(1), sshd(8): SetEnv directives in ssh_config and sshd_config
   are now first-match-wins to match other directives. Previously
   if an environment variable was multiply specified the last set
   value would have been used. bz3438

 * ssh-keygen(8): ssh-keygen -A (generate all default host key types)
   will no longer generate DSA keys, as these are insecure and have
   not been used by default for some years.


New features
------------

 * ssh(1), sshd(8): add a RequiredRSASize directive to set a minimum
   RSA key length. Keys below this length will be ignored for user
   authentication and for host authentication in sshd(8). 

   ssh(1) will terminate a connection if the server offers an RSA key
   that falls below this limit, as the SSH protocol does not include
   the ability to retry a failed key exchange.

 * sftp-server(8): add a "users-groups-by-id@openssh.com" extension
   request that allows the client to obtain user/group names that
   correspond to a set of uids/gids.

 * sftp(1): use "users-groups-by-id@openssh.com" sftp-server
   extension (when available) to fill in user/group names for
   directory listings.

 * sftp-server(8): support the "home-directory" extension request
   defined in draft-ietf-secsh-filexfer-extensions-00. This overlaps
   a bit with the existing "expand-path@openssh.com", but some other
   clients support it.

 * ssh-keygen(1), sshd(8): allow certificate validity intervals,
   sshsig verification times and authorized_keys expiry-time options
   to accept dates in the UTC time zone in addition to the default
   of interpreting them in the system time zone. YYYYMMDD and
   YYMMDDHHMM[SS] dates/times will be interpreted as UTC if suffixed
   with a 'Z' character.

   Also allow certificate validity intervals to be specified in raw
   seconds-since-epoch as hex value, e.g. -V 0x1234:0x4567890. This
   is intended for use by regress tests and other tools that call
   ssh-keygen as part of a CA workflow. bz3468

 * sftp(1): allow arguments to the sftp -D option, e.g. sftp -D
   "/usr/libexec/sftp-server -el debug3"

 * ssh-keygen(1): allow the existing -U (use agent) flag to work
   with "-Y sign" operations, where it will be interpreted to require
   that the private keys is hosted in an agent; bz3429

Bugfixes
--------

 * ssh-keygen(1): implement the "verify-required" certificate option.
   This was already documented when support for user-verified FIDO
   keys was added, but the ssh-keygen(1) code was missing.

 * ssh-agent(1): hook up the restrict_websafe command-line flag;
   previously the flag was accepted but never actually used.

 * sftp(1): improve filename tab completions: never try to complete
   names to non-existent commands, and better match the completion
   type (local or remote filename) against the argument position
   being completed.

 * ssh-keygen(1), ssh(1), ssh-agent(1): several fixes to FIDO key
   handling, especially relating to keys that request
   user-verification. These should reduce the number of unnecessary
   PIN prompts for keys that support intrinsic user verification.
   GHPR302, GHPR329

 * ssh-keygen(1): when enrolling a FIDO resident key, check if a
   credential with matching application and user ID strings already
   exists and, if so, prompt the user for confirmation before
   overwriting the credential. GHPR329

 * sshd(8): improve logging of errors when opening authorized_keys
   files. bz2042

 * ssh(1): avoid multiplexing operations that could cause SIGPIPE from
   causing the client to exit early. bz3454

 * ssh_config(5), sshd_config(5): clarify that the RekeyLimit
   directive applies to both transmitted and received data. GHPR328

 * ssh-keygen(1): avoid double fclose() in error path.

 * sshd(8): log an error if pipe() fails while accepting a
   connection. bz3447

 * ssh(1), ssh-keygen(1): fix possible NULL deref when built without
   FIDO support. bz3443

 * ssh-keyscan(1): add missing *-sk types to ssh-keyscan manpage.
   GHPR294.

 * sshd(8): ensure that authentication passwords are cleared from
   memory in error paths. GHPR286

 * ssh(1), ssh-agent(1): avoid possibility of notifier code executing
   kill(-1). GHPR286

 * ssh_config(5): note that the ProxyJump directive also accepts the
   same tokens as ProxyCommand. GHPR305.

 * scp(1): do not ftruncate(3) files early when in sftp mode. The
   previous behaviour of unconditionally truncating the destination
   file would cause "scp ~/foo localhost:foo" and the reverse
   "scp localhost:foo ~/foo" to delete all the contents of their
   destination. bz3431

 * ssh-keygen(1): improve error message when 'ssh-keygen -Y sign' is
   unable to load a private key; bz3429

 * sftp(1), scp(1): when performing operations that glob(3) a remote
   path, ensure that the implicit working directory used to construct
   that path escapes glob(3) characters. This prevents glob characters
   from being processed in places they shouldn't, e.g. "cd /tmp/a*/",
   "get *.txt" should have the get operation treat the path "/tmp/a*"
   literally and not attempt to expand it.

 * ssh(1), sshd(8): be stricter in which characters will be accepted
   in specifying a mask length; allow only 0-9. GHPR278

 * ssh-keygen(1): avoid printing hash algorithm twice when dumping a
   KRL

 * ssh(1), sshd(8): continue running local I/O for open channels
   during SSH transport rekeying. This should make ~-escapes work in
   the client (e.g. to exit) if the connection happened to have
   stalled during a rekey event.

 * ssh(1), sshd(8): avoid potential poll() spin during rekeying

 * Further hardening for sshbuf internals: disallow "reparenting" a
   hierarchical sshbuf and zero the entire buffer if reallocation
   fails. GHPR287

Portability
-----------

 * ssh(1), ssh-keygen(1), sshd(8): automatically enable the built-in
   FIDO security key support if libfido2 is found and usable, unless
   --without-security-key-builtin was requested.

 * ssh(1), ssh-keygen(1), sshd(8): many fixes to make the WinHello
   FIDO device usable on Cygwin. The windows://hello FIDO device will
   be automatically used by default on this platform unless requested
   otherwise, or when probing resident FIDO credentials (an operation
   not currently supported by WinHello).

 * Portable OpenSSH: remove workarounds for obsolete and unsupported
   versions of OpenSSL libcrypto. In particular, this release removes
   fallback support for OpenSSL that lacks AES-CTR or AES-GCM.

   Those AES cipher modes were added to OpenSSL prior to the minimum
   version currently supported by OpenSSH, so this is not expected to
   impact any currently supported configurations.

 * sshd(8): fix SANDBOX_SECCOMP_FILTER_DEBUG on current Linux/glibc

 * All: resync and clean up internal CSPRNG code.

 * scp(1), sftp(1), sftp-server(8): avoid linking these programs with
   unnecessary libraries. They are no longer linked against libz and
   libcrypto. This may be of benefit to space constrained systems
   using any of those components in isolation.

 * sshd(8): add AUDIT_ARCH_PPC to supported seccomp sandbox
   architectures.

 * configure: remove special casing of crypt(). configure will no
   longer search for crypt() in libcrypto, as it was removed from
   there years ago. configure will now only search libc and libcrypt.

 * configure: refuse to use OpenSSL 3.0.4 due to potential RCE in its
   RSA implementation (CVE-2022-2274) on x86_64.

 * All: request 1.1x API compatibility for OpenSSL >=3.x; GHPR#322

 * ssh(1), ssh-keygen(1), sshd(8): fix a number of missing includes
   required by the XMSS code on some platforms.

 * sshd(8): cache timezone data in capsicum sandbox.

Checksums:
==========

- SHA1 (openssh-9.1.tar.gz) = 3ae2d6a3a695d92778c4c4567dcd6ad481092f6c
- SHA256 (openssh-9.1.tar.gz) = QKfVArlcItV+e8V1Th85TL5//5d/AvOUhYOeHMDEGuE=

- SHA1 (openssh-9.1p1.tar.gz) = 15545440268967511d3194ebf20bcd0c7ff3fcc9
- SHA256 (openssh-9.1p1.tar.gz) = GfhQCcfj4jeH8CNvuxV4OSq01L+fjsX+a8HNfov90og=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 9.0 / 9.0p1 (2022-04-08) 

OpenSSH 9.0 was released on 2022-04-08. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Changes since OpenSSH 8.9
=========================

This release is focused on bug fixing.

Potentially-incompatible changes
--------------------------------

This release switches scp(1) from using the legacy scp/rcp protocol
to using the SFTP protocol by default.

Legacy scp/rcp performs wildcard expansion of remote filenames (e.g.
"scp host:* .") through the remote shell. This has the side effect of
requiring double quoting of shell meta-characters in file names
included on scp(1) command-lines, otherwise they could be interpreted
as shell commands on the remote side.

This creates one area of potential incompatibility: scp(1) when using
the SFTP protocol no longer requires this finicky and brittle quoting,
and attempts to use it may cause transfers to fail. We consider the
removal of the need for double-quoting shell characters in file names
to be a benefit and do not intend to introduce bug-compatibility for
legacy scp/rcp in scp(1) when using the SFTP protocol.

Another area of potential incompatibility relates to the use of remote
paths relative to other user's home directories, for example -
"scp host:~user/file /tmp". The SFTP protocol has no native way to
expand a ~user path. However, sftp-server(8) in OpenSSH 8.7 and later
support a protocol extension "expand-path@openssh.com" to support
this.

In case of incompatibility, the scp(1) client may be instructed to use
the legacy scp/rcp using the -O flag.

New features
------------

 * ssh(1), sshd(8): use the hybrid Streamlined NTRU Prime + x25519 key
   exchange method by default ("sntrup761x25519-sha512@openssh.com").
   The NTRU algorithm is believed to resist attacks enabled by future
   quantum computers and is paired with the X25519 ECDH key exchange
   (the previous default) as a backstop against any weaknesses in
   NTRU Prime that may be discovered in the future. The combination
   ensures that the hybrid exchange offers at least as good security
   as the status quo.

   We are making this change now (i.e. ahead of cryptographically-
   relevant quantum computers) to prevent "capture now, decrypt
   later" attacks where an adversary who can record and store SSH
   session ciphertext would be able to decrypt it once a sufficiently
   advanced quantum computer is available.

 * sftp-server(8): support the "copy-data" extension to allow server-
   side copying of files/data, following the design in
   draft-ietf-secsh-filexfer-extensions-00. bz2948

 * sftp(1): add a "cp" command to allow the sftp client to perform
   server-side file copies.

Bugfixes
--------

 * ssh(1), sshd(8): upstream: fix poll(2) spin when a channel's output
   fd closes without data in the channel buffer. bz3405 and bz3411

 * sshd(8): pack pollfd array in server listen/accept loop. Could
   cause the server to hang/spin when MaxStartups > RLIMIT_NOFILE

 * ssh-keygen(1): avoid NULL deref via the find-principals and
   check-novalidate operations. bz3409 and GHPR#307 respectively.

 * scp(1): fix a memory leak in argument processing. bz3404

 * sshd(8): don't try to resolve ListenAddress directives in the sshd
   re-exec path. They are unused after re-exec and parsing errors
   (possible for example if the host's network configuration changed)
   could prevent connections from being accepted.
    
 * sshd(8): when refusing a public key authentication request from a
   client for using an unapproved or unsupported signature algorithm
   include the algorithm name in the log message to make debugging
   easier.
    
Portability
-----------

 * sshd(8): refactor platform-specific locked account check, fixing
   an incorrect free() on platforms with both libiaf and shadow
   passwords (probably only Unixware) GHPR#284,

 * ssh(1), sshd(8): Fix possible integer underflow in scan_scaled(3)
   parsing of K/M/G/etc quantities. bz#3401.

 * sshd(8): provide killpg implementation (mostly for Tandem NonStop)
   GHPR#301.

 * Check for missing ftruncate prototype. GHPR#301

 * sshd(8): default to not using sandbox when cross compiling. On most
   systems poll(2) does not work when the number of FDs is reduced
   with setrlimit, so assume it doesn't when cross compiling and we
   can't run the test.  bz#3398.

 * sshd(8): allow ppoll_time64 in seccomp sandbox. Should fix sandbox
   violations on some (at least i386 and armhf) 32bit Linux platforms.
   bz#3396.

 * Improve detection of -fzero-call-used-regs=all support in
   configure script.


Checksums:
==========


 - SHA1 (openssh-9.0.tar.gz) = 05302aa4781e1a69db4261474ed940bd685afc24
 - SHA256 (openssh-9.0.tar.gz) = 9I/FrLf5Gij/4NIPts9A8yWVi0ienyyMqjqn8s0hyLk=

 - SHA1 (openssh-9.0p1.tar.gz) = 06dd658874dcd22d66311cf5999bd56c614de509
 - SHA256 (openssh-9.0p1.tar.gz) = A5dDAhYenszjIVPPoQAS8eZcjzdQ9XOnOrG+/Vlyooo=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Please note that the OpenPGP key used to sign releases has been
rotated for this release. The new key has been signed by the previous
key to provide continuity.

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 8.9 / 8.9p1 (2022-02-23) 

OpenSSH 8.9 was released on 2022-02-23. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Future deprecation notice
=========================

A near-future release of OpenSSH will switch scp(1) from using the
legacy scp/rcp protocol to using SFTP by default.

Legacy scp/rcp performs wildcard expansion of remote filenames (e.g.
"scp host:* .") through the remote shell. This has the side effect of
requiring double quoting of shell meta-characters in file names
included on scp(1) command-lines, otherwise they could be interpreted
as shell commands on the remote side.

This creates one area of potential incompatibility: scp(1) when using
the SFTP protocol no longer requires this finicky and brittle quoting,
and attempts to use it may cause transfers to fail. We consider the
removal of the need for double-quoting shell characters in file names
to be a benefit and do not intend to introduce bug-compatibility for
legacy scp/rcp in scp(1) when using the SFTP protocol.

Another area of potential incompatibility relates to the use of remote
paths relative to other user's home directories, for example -
"scp host:~user/file /tmp". The SFTP protocol has no native way to
expand a ~user path. However, sftp-server(8) in OpenSSH 8.7 and later
support a protocol extension "expand-path@openssh.com" to support
this.

Security Near Miss
==================

 * sshd(8): fix an integer overflow in the user authentication path
   that, in conjunction with other logic errors, could have yielded
   unauthenticated access under difficult to exploit conditions.

   This situation is not exploitable because of independent checks in
   the privilege separation monitor. Privilege separation has been
   enabled by default in since openssh-3.2.2 (released in 2002) and
   has been mandatory since openssh-7.5 (released in 2017). Moreover,
   portable OpenSSH has used toolchain features available in most
   modern compilers to abort on signed integer overflow since
   openssh-6.5 (released in 2014).

   Thanks to Malcolm Stagg for finding and reporting this bug.

Potentially-incompatible changes
================================

 * sshd(8), portable OpenSSH only: this release removes in-built
   support for MD5-hashed passwords. If you require these on your
   system then we recommend linking against libxcrypt or similar.

 * This release modifies the FIDO security key middleware interface
   and increments SSH_SK_VERSION_MAJOR.

Changes since OpenSSH 8.8
=========================

This release includes a number of new features.

New features
------------

 * ssh(1), sshd(8), ssh-add(1), ssh-agent(1): add a system for
   restricting forwarding and use of keys added to ssh-agent(1)
   A detailed description of the feature is available at
   https://www.openssh.com/agent-restrict.html and the protocol
   extensions are documented in the PROTOCOL and PROTOCOL.agent
   files in the source release.

 * ssh(1), sshd(8): add the sntrup761x25519-sha512@openssh.com hybrid
   ECDH/x25519 + Streamlined NTRU Prime post-quantum KEX to the
   default KEXAlgorithms list (after the ECDH methods but before the
   prime-group DH ones). The next release of OpenSSH is likely to
   make this key exchange the default method.
    
 * ssh-keygen(1): when downloading resident keys from a FIDO token,
   pass back the user ID that was used when the key was created and
   append it to the filename the key is written to (if it is not the
   default). Avoids keys being clobbered if the user created multiple
   resident keys with the same application string but different user
   IDs.

 * ssh-keygen(1), ssh(1), ssh-agent(1): better handling for FIDO keys
   on tokens that provide user verification (UV) on the device itself,
   including biometric keys, avoiding unnecessary PIN prompts.

 * ssh-keygen(1): add "ssh-keygen -Y match-principals" operation to
   perform matching of principals names against an allowed signers
   file. To be used towards a TOFU model for SSH signatures in git.

 * ssh-add(1), ssh-agent(1): allow pin-required FIDO keys to be added
   to ssh-agent(1). $SSH_ASKPASS will be used to request the PIN at
   authentication time.
    
 * ssh-keygen(1): allow selection of hash at sshsig signing time
   (either sha512 (default) or sha256).

 * ssh(1), sshd(8): read network data directly to the packet input
   buffer instead of indirectly via a small stack buffer. Provides a
   modest performance improvement.

 * ssh(1), sshd(8): read data directly to the channel input buffer,
   providing a similar modest performance improvement.

 * ssh(1): extend the PubkeyAuthentication configuration directive to
   accept yes|no|unbound|host-bound to allow control over one of the
   protocol extensions used to implement agent-restricted keys.

Bugfixes
--------

 * sshd(8): document that CASignatureAlgorithms, ExposeAuthInfo and
   PubkeyAuthOptions can be used in a Match block. PR#277.

 * sshd(8): fix possible string truncation when constructing paths to
   .rhosts/.shosts files with very long user home directory names.

 * ssh-keysign(1): unbreak for KEX algorithms that use SHA384/512
   exchange hashes
    
 * ssh(1): don't put the TTY into raw mode when SessionType=none,
   avoids ^C being unable to kill such a session. bz3360

 * scp(1): fix some corner-case bugs in SFTP-mode handling of
   ~-prefixed paths.

 * ssh(1): unbreak hostbased auth using RSA keys. Allow ssh(1) to
   select RSA keys when only RSA/SHA2 signature algorithms are
   configured (this is the default case). Previously RSA keys were
   not being considered in the default case.

 * ssh-keysign(1): make ssh-keysign use the requested signature
   algorithm and not the default for the key type. Part of unbreaking
   hostbased auth for RSA/SHA2 keys.

 * ssh(1): stricter UpdateHostkey signature verification logic on
   the client- side. Require RSA/SHA2 signatures for RSA hostkeys
   except when RSA/SHA1 was explicitly negotiated during initial
   KEX; bz3375

 * ssh(1), sshd(8): fix signature algorithm selection logic for
   UpdateHostkeys on the server side. The previous code tried to
   prefer RSA/SHA2 for hostkey proofs of RSA keys, but missed some
   cases. This will use RSA/SHA2 signatures for RSA keys if the
   client proposed these algorithms in initial KEX. bz3375

 * All: convert all uses of select(2)/pselect(2) to poll(2)/ppoll(2).
   This includes the mainloops in ssh(1), ssh-agent(1), ssh-agent(1)
   and sftp-server(8), as well as the sshd(8) listen loop and all
   other FD read/writability checks. On platforms with missing or
   broken poll(2)/ppoll(2) syscalls a select(2)-based compat shim is
   available.
    
 * ssh-keygen(1): the "-Y find-principals" command was verifying key
   validity when using ca certs but not with simple key lifetimes
   within the allowed signers file.
    
 * ssh-keygen(1): make sshsig verify-time argument parsing optional

 * sshd(8): fix truncation in rhosts/shosts path construction.

 * ssh(1), ssh-agent(1): avoid xmalloc(0) for PKCS#11 keyid for ECDSA
   keys (we already did this for RSA keys). Avoids fatal errors for
   PKCS#11 libraries that return empty keyid, e.g. Microchip ATECC608B
   "cryptoauthlib"; bz#3364

 * ssh(1), ssh-agent(1): improve the testing of credentials against
   inserted FIDO: ask the token whether a particular key belongs to
   it in cases where the token supports on-token user-verification
   (e.g. biometrics) rather than just assuming that it will accept it.

   Will reduce spurious "Confirm user presence" notifications for key
   handles that relate to FIDO keys that are not currently inserted in at
   least some cases. bz3366
    
 * ssh(1), sshd(8): correct value for IPTOS_DSCP_LE. It needs to
   allow for the preceding two ECN bits. bz#3373

 * ssh-keygen(1): add missing -O option to usage() for the "-Y sign"
   option.

 * ssh-keygen(1): fix a NULL deref when using the find-principals
   function, when matching an allowed_signers line that contains a
   namespace restriction, but no restriction specified on the
   command-line

 * ssh-agent(1): fix memleak in process_extension(); oss-fuzz
   issue #42719

 * ssh(1): suppress "Connection to xxx closed" messages when LogLevel
   is set to "error" or above. bz3378

 * ssh(1), sshd(8): use correct zlib flags when inflate(3)-ing
   compressed packet data. bz3372
 
 * scp(1): when recursively transferring files in SFTP mode, create the
   destination directory if it doesn't already exist to match scp(1) in
   legacy RCP mode behaviour.
    
 * scp(1): many improvements in error message consistency between scp(1)
   in SFTP mode vs legacy RCP mode.

 * sshd(8): fix potential race in SIGTERM handling PR#289

 * ssh(1), ssh(8): since DSA keys are deprecated, move them to the
   end of the default list of public keys so that they will be tried
   last. PR#295

 * ssh-keygen(1): allow 'ssh-keygen -Y find-principals' to match
   wildcard principals in allowed_signers files
    
Portability
-----------

 * ssh(1), sshd(8): don't trust closefrom(2) on Linux. glibc's
   implementation does not work in a chroot when the kernel does not
   have close_range(2). It tries to read from /proc/self/fd and when
   that fails dies with an assertion of sorts. Instead, call
   close_range(2) directly from our compat code and fall back if
   that fails.  bz#3349,

 * OS X poll(2) is broken; use compat replacement. For character-
   special devices like /dev/null, Darwin's poll(2) returns POLLNVAL
   when polled with POLLIN. Apparently this is Apple bug 3710161 -
   not public but a websearch will find other OSS projects
   rediscovering it periodically since it was first identified in
   2005.

 * Correct handling of exceptfds/POLLPRI in our select(2)-based
   poll(2)/ppoll(2) compat implementation.

 * Cygwin: correct checking of mbstowcs() return value.

 * Add a basic SECURITY.md that refers people to the openssh.com
   website.

 * Enable additional compiler warnings and toolchain hardening flags,
   including -Wbitwise-instead-of-logical, -Wmisleading-indentation,
   -fzero-call-used-regs and -ftrivial-auto-var-init.

 * HP/UX. Use compat getline(3) on HP-UX 10.x, where the libc version
   is not reliable.

Checksums:
==========

 - SHA1 (openssh-8.9.tar.gz) = 653310ba1a63959fe2df503fe7ad556445180127
 - SHA256 (openssh-8.9.tar.gz) = mJigktP+Bk0sB7uRPuWgjcCOYZ+mIMdvRlZe66irtQA=

 - SHA1 (openssh-8.9p1.tar.gz) = 205cdf0040a238047e2c49f43460e03d76e5d650
 - SHA256 (openssh-8.9p1.tar.gz) = /Ul2VLerFobaxnL7g9+0ukCW6LX/zazNJiOArli+xec=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Please note that the OpenPGP key used to sign releases has been
rotated for this release. The new key has been signed by the previous
key to provide continuity.

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 8.8 / 8.8p1 (2021-09-26) 

OpenSSH 8.8 was released on 2021-09-26. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Future deprecation notice
=========================

A near-future release of OpenSSH will switch scp(1) from using the
legacy scp/rcp protocol to using SFTP by default.

Legacy scp/rcp performs wildcard expansion of remote filenames (e.g.
"scp host:* .") through the remote shell. This has the side effect of
requiring double quoting of shell meta-characters in file names
included on scp(1) command-lines, otherwise they could be interpreted
as shell commands on the remote side.

This creates one area of potential incompatibility: scp(1) when using
the SFTP protocol no longer requires this finicky and brittle quoting,
and attempts to use it may cause transfers to fail. We consider the
removal of the need for double-quoting shell characters in file names
to be a benefit and do not intend to introduce bug- compatibility for
legacy scp/rcp in scp(1) when using the SFTP protocol.

Another area of potential incompatibility relates to the use of remote
paths relative to other user's home directories, for example -
"scp host:~user/file /tmp". The SFTP protocol has no native way to
expand a ~user path. However, sftp-server(8) in OpenSSH 8.7 and later
support a protocol extension "expand-path@openssh.com" to support
this.

Security
========

sshd(8) from OpenSSH 6.2 through 8.7 failed to correctly initialise
supplemental groups when executing an AuthorizedKeysCommand or
AuthorizedPrincipalsCommand, where a AuthorizedKeysCommandUser or
AuthorizedPrincipalsCommandUser directive has been set to run the
command as a different user. Instead these commands would inherit
the groups that sshd(8) was started with.

Depending on system configuration, inherited groups may allow
AuthorizedKeysCommand/AuthorizedPrincipalsCommand helper programs to
gain unintended privilege.

Neither AuthorizedKeysCommand nor AuthorizedPrincipalsCommand are
enabled by default in sshd_config(5).

Potentially-incompatible changes
================================

This release disables RSA signatures using the SHA-1 hash algorithm
by default. This change has been made as the SHA-1 hash algorithm is
cryptographically broken, and it is possible to create chosen-prefix
hash collisions for <USD$50K [1]

For most users, this change should be invisible and there is
no need to replace ssh-rsa keys. OpenSSH has supported RFC8332
RSA/SHA-256/512 signatures since release 7.2 and existing ssh-rsa keys
will automatically use the stronger algorithm where possible.

Incompatibility is more likely when connecting to older SSH
implementations that have not been upgraded or have not closely tracked
improvements in the SSH protocol. For these cases, it may be necessary
to selectively re-enable RSA/SHA1 to allow connection and/or user
authentication via the HostkeyAlgorithms and PubkeyAcceptedAlgorithms
options. For example, the following stanza in ~/.ssh/config will enable
RSA/SHA1 for host and user authentication for a single destination host:

    Host old-host
        HostkeyAlgorithms +ssh-rsa
	PubkeyAcceptedAlgorithms +ssh-rsa

We recommend enabling RSA/SHA1 only as a stopgap measure until legacy
implementations can be upgraded or reconfigured with another key type
(such as ECDSA or Ed25519).

[1] "SHA-1 is a Shambles: First Chosen-Prefix Collision on SHA-1 and
    Application to the PGP Web of Trust" Leurent, G and Peyrin, T
    (2020) https://eprint.iacr.org/2020/014.pdf

Changes since OpenSSH 8.7
=========================

This release is motivated primarily by the above deprecation and
security fix.

New features
------------
 * ssh(1): allow the ssh_config(5) CanonicalizePermittedCNAMEs
   directive to accept a "none" argument to specify the default
   behaviour.

Bugfixes
--------

 * scp(1): when using the SFTP protocol, continue transferring files
   after a transfer error occurs, better matching original scp/rcp
   behaviour.
    
 * ssh(1): fixed a number of memory leaks in multiplexing,

 * ssh-keygen(1): avoid crash when using the -Y find-principals
   command.

 * A number of documentation and manual improvements, including
   bz#3340, PR#139, PR#215, PR#241, PR#257

Portability
-----------

 * ssh-agent(1): on FreeBSD, use procctl to disable ptrace(2)

 * ssh(1)/sshd(8): some fixes to the pselect(2) replacement
   compatibility code. bz#3345

Checksums:
==========

 - SHA1 (openssh-8.8.tar.gz) = 732947082a8998047e839cc0b4c066bf0a7e1a5b
 - SHA256 (openssh-8.8.tar.gz) = AngyrPSQH255hnzU1l7y+LlVAUNcGWtuYQIFEl22nRo=

 - SHA1 (openssh-8.8p1.tar.gz) = 1eb964897a4372f6fb96c7effeb509ec71c379c9
 - SHA256 (openssh-8.8p1.tar.gz) = RZCJDqm7ms5Pca4zF4WjpYIyMkNRYZYO1fyGWI8zH+k=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Please note that the OpenPGP key used to sign releases has been
rotated for this release. The new key has been signed by the previous
key to provide continuity.

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 8.7 / 8.7p1 (2021-08-20) 

OpenSSH 8.7 was released on 2021-08-20. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Imminent deprecation notice
===========================

OpenSSH will disable the ssh-rsa signature scheme by default in the
next release.

In the SSH protocol, the "ssh-rsa" signature scheme uses the SHA-1
hash algorithm in conjunction with the RSA public key algorithm.
It is now possible[1] to perform chosen-prefix attacks against the
SHA-1 algorithm for less than USD$50K.

Note that the deactivation of "ssh-rsa" signatures does not necessarily
require cessation of use for RSA keys. In the SSH protocol, keys may be
capable of signing using multiple algorithms. In particular, "ssh-rsa"
keys are capable of signing using "rsa-sha2-256" (RSA/SHA256),
"rsa-sha2-512" (RSA/SHA512) and "ssh-rsa" (RSA/SHA1). Only the last of
these is being turned off by default.

This algorithm is unfortunately still used widely despite the
existence of better alternatives, being the only remaining public key
signature algorithm specified by the original SSH RFCs that is still
enabled by default.

The better alternatives include:

 * The RFC8332 RSA SHA-2 signature algorithms rsa-sha2-256/512. These
   algorithms have the advantage of using the same key type as
   "ssh-rsa" but use the safe SHA-2 hash algorithms. These have been
   supported since OpenSSH 7.2 and are already used by default if the
   client and server support them.

 * The RFC8709 ssh-ed25519 signature algorithm. It has been supported
   in OpenSSH since release 6.5.

 * The RFC5656 ECDSA algorithms: ecdsa-sha2-nistp256/384/521. These
   have been supported by OpenSSH since release 5.7.

To check whether a server is using the weak ssh-rsa public key
algorithm, for host authentication, try to connect to it after
removing the ssh-rsa algorithm from ssh(1)'s allowed list:

    ssh -oHostKeyAlgorithms=-ssh-rsa user@host

If the host key verification fails and no other supported host key
types are available, the server software on that host should be
upgraded.

OpenSSH recently enabled the UpdateHostKeys option by default to
assist the client by automatically migrating to better algorithms.

[1] "SHA-1 is a Shambles: First Chosen-Prefix Collision on SHA-1 and
    Application to the PGP Web of Trust" Leurent, G and Peyrin, T
    (2020) https://eprint.iacr.org/2020/014.pdf

Potentially-incompatible changes
================================

This release includes a number of changes that may affect existing
configurations:

 * scp(1): this release changes the behaviour of remote to remote
   copies (e.g. "scp host-a:/path host-b:") to transfer through the
   local host by default. This was previously available via the -3
   flag. This mode avoids the need to expose credentials on the
   origin hop, avoids triplicate interpretation of filenames by the
   shell (by the local system, the copy origin and the destination)
   and, in conjunction with the SFTP support for scp(1) mentioned
   below, allows use of all authentication methods to the remote
   hosts (previously, only non-interactive methods could be used).
   A -R flag has been added to select the old behaviour.

 * ssh(1)/sshd(8): both the client and server are now using a
   stricter configuration file parser. The new parser uses more
   shell-like rules for quotes, space and escape characters. It is
   also more strict in rejecting configurations that include options
   lacking arguments. Previously some options (e.g. DenyUsers) could
   appear on a line with no subsequent arguments. This release will
   reject such configurations. The new parser will also reject
   configurations with unterminated quotes and multiple '='
   characters after the option name.

 * ssh(1): when using SSHFP DNS records for host key verification,
   ssh(1) will verify all matching records instead of just those
   with the specific signature type requested. This may cause host
   key verification problems if stale SSHFP records of a different
   or legacy signature type exist alongside other records for a
   particular host. bz#3322

 * ssh-keygen(1): when generating a FIDO key and specifying an
   explicit attestation challenge (using -Ochallenge), the challenge
   will now be hashed by the builtin security key middleware. This
   removes the (undocumented) requirement that challenges be exactly
   32 bytes in length and matches the expectations of libfido2.

 * sshd(8): environment="..." directives in authorized_keys files are
   now first-match-wins and limited to 1024 discrete environment
   variable names.

Changes since OpenSSH 8.6
=========================

This release contains a mix of new features and bug-fixes.

New features
------------

 - scp(1): experimental support for transfers using the SFTP protocol
   as a replacement for the venerable SCP/RCP protocol that it has
   traditionally used. SFTP offers more predictable filename handling
   and does not require expansion of glob(3) patterns via the shell
   on the remote side.

   SFTP support may be enabled via a temporary scp -s flag. It is
   intended for SFTP to become the default transfer mode in the
   near future, at which time the -s flag will be removed. The -O
   flag exists to force use of the original SCP/RCP protocol for
   cases where SFTP may be unavailable or incompatible.

 - sftp-server(8): add a protocol extension to support expansion of
   ~/ and ~user/ prefixed paths. This was added to support these
   paths when used by scp(1) while in SFTP mode.

 - ssh(1): add a ForkAfterAuthentication ssh_config(5) counterpart to
   the ssh(1) -f flag. GHPR#231

 - ssh(1): add a StdinNull directive to ssh_config(5) that allows the
   config file to do the same thing as -n does on the ssh(1) command-
   line. GHPR#231

 - ssh(1): add a SessionType directive to ssh_config, allowing the
    configuration file to offer equivalent control to the -N (no
    session) and -s (subsystem) command-line flags. GHPR#231

 - ssh-keygen(1): allowed signers files used by ssh-keygen(1)
   signatures now support listing key validity intervals alongside
   they key, and ssh-keygen(1) can optionally check during signature
   verification whether a specified time falls inside this interval.
   This feature is intended for use by git to support signing and
   verifying objects using ssh keys.

 - ssh-keygen(8): support printing of the full public key in a sshsig
   signature via a -Oprint-pubkey flag.

Bugfixes
--------

 * ssh(1)/sshd(8): start time-based re-keying exactly on schedule in
   the client and server mainloops. Previously the re-key timeout
   could expire but re-keying would not start until a packet was sent
   or received, causing a spin in select() if the connection was
   quiescent.

 * ssh-keygen(1): avoid Y2038 problem in printing certificate
   validity lifetimes. Dates past 2^31-1 seconds since epoch were
   displayed incorrectly on some platforms. bz#3329

 * scp(1): allow spaces to appear in usernames for local to remote
   and scp -3 remote to remote copies. bz#1164

 * ssh(1)/sshd(8): remove references to ChallengeResponseAuthentication
   in favour of KbdInteractiveAuthentication. The former is what was in
   SSHv1, the latter is what is in SSHv2 (RFC4256) and they were
   treated as somewhat but not entirely equivalent. We retain the old
   name as a deprecated alias so configuration files continue to work
   as well as a reference in the man page for people looking for it.
   bz#3303

 * ssh(1)/ssh-add(1)/ssh-keygen(1): fix decoding of X.509 subject name
   when extracting a key from a PKCS#11 certificate. bz#3327

 * ssh(1): restore blocking status on stdio fds before close. ssh(1)
   needs file descriptors in non-blocking mode to operate but it was
   not restoring the original state on exit. This could cause
   problems with fds shared with other programs via the shell,
   bz#3280 and GHPR#246

 * ssh(1)/sshd(8): switch both client and server mainloops from
   select(3) to pselect(3). Avoids race conditions where a signal
   may arrive immediately before select(3) and not be processed until
   an event fires. bz#2158

 * ssh(1): sessions started with ControlPersist were incorrectly
   executing a shell when the -N (no shell) option was specified.
   bz#3290

 * ssh(1): check if IPQoS or TunnelDevice are already set before
   overriding. Prevents values in config files from overriding values
   supplied on the command line. bz#3319

 * ssh(1): fix debug message when finding a private key to match a
   certificate being attempted for user authentication. Previously it
   would print the certificate's path, whereas it was supposed to be
   showing the private key's path. GHPR#247

 * sshd(8): match host certificates against host public keys, not
   private keys. Allows use of certificates with private keys held in
   a ssh-agent.  bz#3524

 * ssh(1): add a workaround for a bug in OpenSSH 7.4 sshd(8), which
   allows RSA/SHA2 signatures for public key authentication but fails
   to advertise this correctly via SSH2_MSG_EXT_INFO. This causes
   clients of these server to incorrectly match
   PubkeyAcceptedAlgorithmse and potentially refuse to offer valid
   keys. bz#3213

 * sftp(1)/scp(1): degrade gracefully if a sftp-server offers the
   limits@openssh.com extension but fails when the client tries to
   invoke it. bz#3318

 * ssh(1): allow ssh_config SetEnv to override $TERM, which is
   otherwise handled specially by the protocol. Useful in ~/.ssh/config
   to set TERM to something generic (e.g. "xterm" instead of
   "xterm-256color") for destinations that lack terminfo entries.

 * sftp-server(8): the limits@openssh.com extension was incorrectly
   marked as an operation that writes to the filesystem, which made it
   unavailable in sftp-server read-only mode. bz#3318

 * ssh(1): fix SEGV in UpdateHostkeys debug() message, triggered when
   the update removed more host keys than remain present.

 * many manual page fixes.

Portability
-----------

 * ssh(1): move closefrom() to before first malloc. When built against
   tcmalloc, the closefrom() would stomp on file descriptors created
   for tcmalloc's internal use. bz#3321

 * sshd(8): handle GIDs > 2^31 in getgrouplist. When compiled in 32bit
   mode, the getgrouplist implementation may fail for GIDs greater than
   LONG_MAX.

 * ssh(1): xstrdup environment variable used by ForwardAgent. bz#3328

 * sshd(8): don't sigdie() in signal handler in privsep child process;
   this can end up causing sandbox violations per bz3286

Checksums:
==========

 - SHA1 (openssh-8.7.tar.gz) = 61bfa5a55e2b7e851b1d463aa432e4ff508f61cc
 - SHA256 (openssh-8.7.tar.gz) = Q5jPfCaYhTKhkDPYH32jtjFN7qhNhkNHYC9awg2THLs=

 - SHA1 (openssh-8.7p1.tar.gz) = 8719032c1e47732c8fdb14adfb24b5e9e71de802
 - SHA256 (openssh-8.7p1.tar.gz) = fKNLi7JK6eUPM3krcJGzhB1+G0QP9XvJ+r3fAeLtHiQ=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Please note that the OpenPGP key used to sign releases has been
rotated for this release. The new key has been signed by the previous
key to provide continuity.

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 8.6 / 8.6p1 (2021-04-19) 

OpenSSH 8.6 was released on 2021-04-19. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Future deprecation notice
=========================

It is now possible[1] to perform chosen-prefix attacks against the
SHA-1 algorithm for less than USD$50K.

In the SSH protocol, the "ssh-rsa" signature scheme uses the SHA-1
hash algorithm in conjunction with the RSA public key algorithm.
OpenSSH will disable this signature scheme by default in the near
future.

Note that the deactivation of "ssh-rsa" signatures does not necessarily
require cessation of use for RSA keys. In the SSH protocol, keys may be
capable of signing using multiple algorithms. In particular, "ssh-rsa"
keys are capable of signing using "rsa-sha2-256" (RSA/SHA256),
"rsa-sha2-512" (RSA/SHA512) and "ssh-rsa" (RSA/SHA1). Only the last of
these is being turned off by default.

This algorithm is unfortunately still used widely despite the
existence of better alternatives, being the only remaining public key
signature algorithm specified by the original SSH RFCs that is still
enabled by default.

The better alternatives include:

 * The RFC8332 RSA SHA-2 signature algorithms rsa-sha2-256/512. These
   algorithms have the advantage of using the same key type as
   "ssh-rsa" but use the safe SHA-2 hash algorithms. These have been
   supported since OpenSSH 7.2 and are already used by default if the
   client and server support them.

 * The RFC8709 ssh-ed25519 signature algorithm. It has been supported
   in OpenSSH since release 6.5.

 * The RFC5656 ECDSA algorithms: ecdsa-sha2-nistp256/384/521. These
   have been supported by OpenSSH since release 5.7.

To check whether a server is using the weak ssh-rsa public key
algorithm, for host authentication, try to connect to it after
removing the ssh-rsa algorithm from ssh(1)'s allowed list:

    ssh -oHostKeyAlgorithms=-ssh-rsa user@host

If the host key verification fails and no other supported host key
types are available, the server software on that host should be
upgraded.

OpenSSH recently enabled the UpdateHostKeys option by default to assist
the client by automatically migrating to better algorithms.

[1] "SHA-1 is a Shambles: First Chosen-Prefix Collision on SHA-1 and
    Application to the PGP Web of Trust" Leurent, G and Peyrin, T
    (2020) https://eprint.iacr.org/2020/014.pdf

Security
========

 * sshd(8): OpenSSH 8.5 introduced the LogVerbose keyword. When this
   option was enabled with a set of patterns that activated logging
   in code that runs in the low-privilege sandboxed sshd process, the
   log messages were constructed in such a way that printf(3) format
   strings could effectively be specified the low-privilege code.

   An attacker who had successfully exploited the low-privilege
   process could use this to escape OpenSSH's sandboxing and attack
   the high-privilege process. Exploitation of this weakness is
   highly unlikely in practice as the LogVerbose option is not
   enabled by default and is typically only used for debugging. No
   vulnerabilities in the low-privilege process are currently known
   to exist.

   Thanks to Ilja Van Sprundel for reporting this bug.

Changes since OpenSSH 8.5
=========================

This release contains mostly bug fixes.

New features
------------

 * sftp-server(8): add a new limits@openssh.com protocol extension
   that allows a client to discover various server limits, including
   maximum packet size and maximum read/write length.

 * sftp(1): use the new limits@openssh.com extension (when available)
   to select better transfer lengths in the client.

 * sshd(8): Add ModuliFile keyword to sshd_config to specify the
   location of the "moduli" file containing the groups for DH-GEX.

 * unit tests: Add a TEST_SSH_ELAPSED_TIMES environment variable to
   enable printing of the elapsed time in seconds of each test.

Bugfixes
--------

 * ssh_config(5), sshd_config(5): sync CASignatureAlgorithms lists in
   manual pages with the current default. GHPR#174

 * ssh(1): ensure that pkcs11_del_provider() is called before exit.
   GHPR#234

 * ssh(1), sshd(8): fix problems in string->argv conversion. Multiple
   backslashes were not being dequoted correctly and quoted space in
   the middle of a string was being incorrectly split. GHPR#223

 * ssh(1): return non-zero exit status when killed by signal; bz#3281

 * sftp-server(8): increase maximum SSH2_FXP_READ to match the maximum
   packet size. Also handle zero-length reads that are not explicitly
   banned by the spec.

Portability
-----------

 * sshd(8): don't mistakenly exit on transient read errors on the
   network socket (e.g. EINTR, EAGAIN); bz3297

 * Create a dedicated contrib/gnome-ssk-askpass3.c source instead of
   building it from the same file as used for GNOME2. Use the GNOME3
   gdk_seat_grab() to manage keyboard/mouse/server grabs for better
   compatibility with Wayland.

 * Fix portability build errors bz3293 bz3292 bz3291 bz3278

 * sshd(8): soft-disallow the fstatat64 syscall in the Linux
   seccomp-bpf sandbox. bz3276

 * unit tests: enable autoopt and misc unit tests that were
   previously skipped

Checksums:
==========

 - SHA1 (openssh-8.6.tar.gz) = a3e93347eed6296faaaceb221e8786391530fccb
 - SHA256 (openssh-8.6.tar.gz) = ihmgdEgKfCBRpC0qzdQRwYownrpBf+rsihvk4Rmim8M=

 - SHA1 (openssh-8.6p1.tar.gz) = 8f9f0c94317baeb97747d6258f3997b4542762c0
 - SHA256 (openssh-8.6p1.tar.gz) = w+bk2hYhdiyFDQO0fu0eSN/0zJYI3etUcgKiNN+O164=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Please note that the OpenPGP key used to sign releases has been
rotated for this release. The new key has been signed by the previous
key to provide continuity.

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 8.5 / 8.5p1 (2021-03-03) 

OpenSSH 8.5 was released on 2021-03-03. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Future deprecation notice
=========================

It is now possible[1] to perform chosen-prefix attacks against the
SHA-1 algorithm for less than USD$50K.

In the SSH protocol, the "ssh-rsa" signature scheme uses the SHA-1
hash algorithm in conjunction with the RSA public key algorithm.
OpenSSH will disable this signature scheme by default in the near
future.

Note that the deactivation of "ssh-rsa" signatures does not necessarily
require cessation of use for RSA keys. In the SSH protocol, keys may be
capable of signing using multiple algorithms. In particular, "ssh-rsa"
keys are capable of signing using "rsa-sha2-256" (RSA/SHA256),
"rsa-sha2-512" (RSA/SHA512) and "ssh-rsa" (RSA/SHA1). Only the last of
these is being turned off by default.

This algorithm is unfortunately still used widely despite the
existence of better alternatives, being the only remaining public key
signature algorithm specified by the original SSH RFCs that is still
enabled by default.

The better alternatives include:

 * The RFC8332 RSA SHA-2 signature algorithms rsa-sha2-256/512. These
   algorithms have the advantage of using the same key type as
   "ssh-rsa" but use the safe SHA-2 hash algorithms. These have been
   supported since OpenSSH 7.2 and are already used by default if the
   client and server support them.

 * The RFC8709 ssh-ed25519 signature algorithm. It has been supported
   in OpenSSH since release 6.5.

 * The RFC5656 ECDSA algorithms: ecdsa-sha2-nistp256/384/521. These
   have been supported by OpenSSH since release 5.7.

To check whether a server is using the weak ssh-rsa public key
algorithm, for host authentication, try to connect to it after
removing the ssh-rsa algorithm from ssh(1)'s allowed list:

    ssh -oHostKeyAlgorithms=-ssh-rsa user@host

If the host key verification fails and no other supported host key
types are available, the server software on that host should be
upgraded.

This release enables the UpdateHostKeys option by default to assist
the client by automatically migrating to better algorithms.

[1] "SHA-1 is a Shambles: First Chosen-Prefix Collision on SHA-1 and
    Application to the PGP Web of Trust" Leurent, G and Peyrin, T
    (2020) https://eprint.iacr.org/2020/014.pdf

Security
========

 * ssh-agent(1): fixed a double-free memory corruption that was
   introduced in OpenSSH 8.2 . We treat all such memory faults as
   potentially exploitable. This bug could be reached by an attacker
   with access to the agent socket.

   On modern operating systems where the OS can provide information
   about the user identity connected to a socket, OpenSSH ssh-agent
   and sshd limit agent socket access only to the originating user
   and root. Additional mitigation may be afforded by the system's
   malloc(3)/free(3) implementation, if it detects double-free
   conditions.

   The most likely scenario for exploitation is a user forwarding an
   agent either to an account shared with a malicious user or to a
   host with an attacker holding root access.

 * Portable sshd(8): Prevent excessively long username going to PAM.
   This is a mitigation for a buffer overflow in Solaris' PAM username
   handling (CVE-2020-14871), and is only enabled for Sun-derived PAM
   implementations.  This is not a problem in sshd itself, it only
   prevents sshd from being used as a vector to attack Solaris' PAM.
   It does not prevent the bug in PAM from being exploited via some
   other PAM application. GHPR#212


Potentially-incompatible changes
================================

This release includes a number of changes that may affect existing
configurations:

 * ssh(1), sshd(8): this release changes the first-preference signature
   algorithm from ECDSA to ED25519.

 * ssh(1), sshd(8): set the TOS/DSCP specified in the configuration
   for interactive use prior to TCP connect. The connection phase of
   the SSH session is time-sensitive and often explicitly interactive.
   The ultimate interactive/bulk TOS/DSCP will be set after
   authentication completes.

 * ssh(1), sshd(8): remove the pre-standardization cipher
   rijndael-cbc@lysator.liu.se. It is an alias for aes256-cbc before
   it was standardized in RFC4253 (2006), has been deprecated and
   disabled by default since OpenSSH 7.2 (2016) and was only briefly
   documented in ssh.1 in 2001.

 * ssh(1), sshd(8): update/replace the experimental post-quantum
   hybrid key exchange method based on Streamlined NTRU Prime coupled
   with X25519.

   The previous sntrup4591761x25519-sha512@tinyssh.org method is
   replaced with sntrup761x25519-sha512@openssh.com. Per its
   designers, the sntrup4591761 algorithm was superseded almost two
   years ago by sntrup761.

   (note this both the updated method and the one that it replaced are
   disabled by default)

 * ssh(1): disable CheckHostIP by default. It provides insignificant
   benefits while making key rotation significantly more difficult,
   especially for hosts behind IP-based load-balancers.

Changes since OpenSSH 8.4
=========================

New features
------------

 * ssh(1): this release enables UpdateHostkeys by default subject to
   some conservative preconditions:
    - The key was matched in the UserKnownHostsFile (and not in the
      GlobalKnownHostsFile).
    - The same key does not exist under another name.
    - A certificate host key is not in use.
    - known_hosts contains no matching wildcard hostname pattern.
    - VerifyHostKeyDNS is not enabled.
    - The default UserKnownHostsFile is in use.

   We expect some of these conditions will be modified or relaxed in
   future.

 * ssh(1), sshd(8): add a new LogVerbose configuration directive for
   that allows forcing maximum debug logging by file/function/line
   pattern-lists.

 * ssh(1): when prompting the user to accept a new hostkey, display
   any other host names/addresses already associated with the key.

 * ssh(1): allow UserKnownHostsFile=none to indicate that no
   known_hosts file should be used to identify host keys.

 * ssh(1): add a ssh_config KnownHostsCommand option that allows the
   client to obtain known_hosts data from a command in addition to
   the usual files.

 * ssh(1): add a ssh_config PermitRemoteOpen option that allows the
   client to restrict the destination when RemoteForward is used
   with SOCKS.

 * ssh(1): for FIDO keys, if a signature operation fails with a
   "incorrect PIN" reason and no PIN was initially requested from the
   user, then request a PIN and retry the operation. This supports
   some biometric devices that fall back to requiring PIN when reading
   of the biometric failed, and devices that require PINs for all
   hosted credentials.

 * sshd(8): implement client address-based rate-limiting via new
   sshd_config(5) PerSourceMaxStartups and PerSourceNetBlockSize
   directives that provide more fine-grained control on a per-origin
   address basis than the global MaxStartups limit.

Bugfixes
--------

 * ssh(1): Prefix keyboard interactive prompts with "(user@host)" to
   make it easier to determine which connection they are associated
   with in cases like scp -3, ProxyJump, etc. bz#3224

 * sshd(8): fix sshd_config SetEnv directives located inside Match
   blocks. GHPR#201

 * ssh(1): when requesting a FIDO token touch on stderr, inform the
   user once the touch has been recorded.

 * ssh(1): prevent integer overflow when ridiculously large
   ConnectTimeout values are specified, capping the effective value
   (for most platforms) at 24 days. bz#3229

 * ssh(1): consider the ECDSA key subtype when ordering host key
   algorithms in the client.

 * ssh(1), sshd(8): rename the PubkeyAcceptedKeyTypes keyword to
   PubkeyAcceptedAlgorithms. The previous name incorrectly suggested
   that it control allowed key algorithms, when this option actually
   specifies the signature algorithms that are accepted. The previous
   name remains available as an alias. bz#3253

 * ssh(1), sshd(8): similarly, rename HostbasedKeyTypes (ssh) and
   HostbasedAcceptedKeyTypes (sshd) to HostbasedAcceptedAlgorithms.

 * sftp-server(8): add missing lsetstat@openssh.com documentation
   and advertisement in the server's SSH2_FXP_VERSION hello packet.

 * ssh(1), sshd(8): more strictly enforce KEX state-machine by
   banning packet types once they are received. Fixes memleak caused
   by duplicate SSH2_MSG_KEX_DH_GEX_REQUEST (oss-fuzz #30078).

 * sftp(1): allow the full range of UIDs/GIDs for chown/chgrp on 32bit
   platforms instead of being limited by LONG_MAX. bz#3206

 * Minor man page fixes (capitalization, commas, etc.) bz#3223

 * sftp(1): when doing an sftp recursive upload or download of a
   read-only directory, ensure that the directory is created with
   write and execute permissions in the interim so that the transfer
   can actually complete, then set the directory permission as the
   final step. bz#3222

 * ssh-keygen(1): document the -Z, check the validity of its argument
   earlier and provide a better error message if it's not correct.
   bz#2879

 * ssh(1): ignore comments at the end of config lines in ssh_config,
   similar to what we already do for sshd_config. bz#2320

 * sshd_config(5): mention that DisableForwarding is valid in a
   sshd_config Match block. bz3239

 * sftp(1): fix incorrect sorting of "ls -ltr" under some
   circumstances. bz3248.

 * ssh(1), sshd(8): fix potential integer truncation of (unlikely)
   timeout values. bz#3250

 * ssh(1): make hostbased authentication send the signature algorithm
   in its SSH2_MSG_USERAUTH_REQUEST packets instead of the key type.
   This make HostbasedAcceptedAlgorithms do what it is supposed to -
   filter on signature algorithm and not key type.

Portability
-----------

 * sshd(8): add a number of platform-specific syscalls to the Linux
   seccomp-bpf sandbox. bz#3232 bz#3260

 * sshd(8): remove debug message from sigchld handler that could cause
   deadlock on some platforms. bz#3259

 * Sync contrib/ssh-copy-id with upstream.

 * unittests: add a hostname function for systems that don't have it.
   Some systems don't have a hostname command (it's not required by
   POSIX). The do have uname -n (which is), but not all of those have
   it report the FQDN.

Checksums:
==========

 - SHA1 (openssh-8.5.tar.gz) = 04cae43c389fb411227c01219e4eb46e3113f34e
 - SHA256 (openssh-8.5.tar.gz) = 5qB2CgzNG4io4DmChTjHgCWqRWvEOvCKJskLdJCz+SU=

 - SHA1 (openssh-8.5p1.tar.gz) = 72eadcbe313b07b1dd3b693e41d3cd56d354e24e
 - SHA256 (openssh-8.5p1.tar.gz) = 9S8/QdQpqpkY44zyAK8iXM3Y5m8FLaVyhwyJc3ZG7CU=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available from the mirror sites:
https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc

Please note that the OpenPGP key used to sign releases has been
rotated for this release. The new key has been signed by the previous
key to provide continuity.

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 8.4 / 8.4p1 (2020-09-27) 

OpenSSH 8.4 was released on 2020-09-27. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Future deprecation notice
=========================

It is now possible[1] to perform chosen-prefix attacks against the
SHA-1 algorithm for less than USD$50K. For this reason, we will be
disabling the "ssh-rsa" public key signature algorithm by default in a
near-future release.

This algorithm is unfortunately still used widely despite the
existence of better alternatives, being the only remaining public key
signature algorithm specified by the original SSH RFCs.

The better alternatives include:

 * The RFC8332 RSA SHA-2 signature algorithms rsa-sha2-256/512. These
   algorithms have the advantage of using the same key type as
   "ssh-rsa" but use the safe SHA-2 hash algorithms. These have been
   supported since OpenSSH 7.2 and are already used by default if the
   client and server support them.

 * The ssh-ed25519 signature algorithm. It has been supported in
   OpenSSH since release 6.5.

 * The RFC5656 ECDSA algorithms: ecdsa-sha2-nistp256/384/521. These
   have been supported by OpenSSH since release 5.7.

To check whether a server is using the weak ssh-rsa public key
algorithm, for host authentication, try to connect to it after
removing the ssh-rsa algorithm from ssh(1)'s allowed list:

    ssh -oHostKeyAlgorithms=-ssh-rsa user@host

If the host key verification fails and no other supported host key
types are available, the server software on that host should be
upgraded.

We intend to enable UpdateHostKeys by default in the next OpenSSH
release. This will assist the client by automatically migrating to
better algorithms. Users may consider enabling this option manually.

[1] "SHA-1 is a Shambles: First Chosen-Prefix Collision on SHA-1 and
    Application to the PGP Web of Trust" Leurent, G and Peyrin, T
    (2020) https://eprint.iacr.org/2020/014.pdf

Security
========

 * ssh-agent(1): restrict ssh-agent from signing web challenges for
   FIDO/U2F keys.

   When signing messages in ssh-agent using a FIDO key that has an
   application string that does not start with "ssh:", ensure that the
   message being signed is one of the forms expected for the SSH protocol
   (currently public key authentication and sshsig signatures).

   This prevents ssh-agent forwarding on a host that has FIDO keys
   attached granting the ability for the remote side to sign challenges
   for web authentication using those keys too.

   Note that the converse case of web browsers signing SSH challenges is
   already precluded because no web RP can have the "ssh:" prefix in the
   application string that we require.

 * ssh-keygen(1): Enable FIDO 2.1 credProtect extension when generating
   a FIDO resident key.

   The recent FIDO 2.1 Client to Authenticator Protocol introduced a
   "credProtect" feature to better protect resident keys. We use this
   option to require a PIN prior to all operations that may retrieve
   a resident key from a FIDO token.

Potentially-incompatible changes
================================

This release includes a number of changes that may affect existing
configurations:

 * For FIDO/U2F support, OpenSSH recommends the use of libfido2 1.5.0
   or greater. Older libraries have limited support at the expense of
   disabling particular features. These include resident keys, PIN-
   required keys and multiple attached tokens.

 * ssh-keygen(1): the format of the attestation information optionally
   recorded when a FIDO key is generated has changed. It now includes
   the authenticator data needed to validate attestation signatures. 

 * The API between OpenSSH and the FIDO token middleware has changed
   and the SSH_SK_VERSION_MAJOR version has been incremented as a
   result. Third-party middleware libraries must support the current
   API version (7) to work with OpenSSH 8.4.

 * The portable OpenSSH distribution now requires automake to rebuild
   the configure script and supporting files. This is not required when
   simply building portable OpenSSH from a release tar file.

Changes since OpenSSH 8.3
=========================

New features
------------

 * ssh(1), ssh-keygen(1): support for FIDO keys that require a PIN for
   each use. These keys may be generated using ssh-keygen using a new
   "verify-required" option. When a PIN-required key is used, the user
   will be prompted for a PIN to complete the signature operation.

 * sshd(8): authorized_keys now supports a new "verify-required"
   option to require FIDO signatures assert that the token verified
   that the user was present before making the signature. The FIDO
   protocol supports multiple methods for user-verification, but
   currently OpenSSH only supports PIN verification.

 * sshd(8), ssh-keygen(1): add support for verifying FIDO webauthn
   signatures. Webauthn is a standard for using FIDO keys in web
   browsers. These signatures are a slightly different format to plain
   FIDO signatures and thus require explicit support.

 * ssh(1): allow some keywords to expand shell-style ${ENV}
   environment variables. The supported keywords are CertificateFile,
   ControlPath, IdentityAgent and IdentityFile, plus LocalForward and
   RemoteForward when used for Unix domain socket paths. bz#3140

 * ssh(1), ssh-agent(1): allow some additional control over the use of
   ssh-askpass via a new $SSH_ASKPASS_REQUIRE environment variable,
   including forcibly enabling and disabling its use. bz#69

 * ssh(1): allow ssh_config(5)'s AddKeysToAgent keyword accept a time
   limit for keys in addition to its current flag options. Time-
   limited keys will automatically be removed from ssh-agent after
   their expiry time has passed.

 * scp(1), sftp(1): allow the -A flag to explicitly enable agent
   forwarding in scp and sftp. The default remains to not forward an
   agent, even when ssh_config enables it.

 * ssh(1): add a '%k' TOKEN that expands to the effective HostKey of
   the destination. This allows, e.g., keeping host keys in individual
   files using "UserKnownHostsFile ~/.ssh/known_hosts.d/%k". bz#1654

 * ssh(1): add %-TOKEN, environment variable and tilde expansion to
   the UserKnownHostsFile directive, allowing the path to be
   completed by the configuration (e.g. bz#1654)

 * ssh-keygen(1): allow "ssh-add -d -" to read keys to be deleted
   from stdin. bz#3180

 * sshd(8): improve logging for MaxStartups connection throttling.
   sshd will now log when it starts and stops throttling and periodically
   while in this state. bz#3055

Bugfixes
--------

 * ssh(1), ssh-keygen(1): better support for multiple attached FIDO
   tokens. In cases where OpenSSH cannot unambiguously determine which
   token to direct a request to, the user is now required to select a
   token by touching it. In cases of operations that require a PIN to
   be verified, this avoids sending the wrong PIN to the wrong token
   and incrementing the token's PIN failure counter (tokens
   effectively erase their keys after too many PIN failures).

 * sshd(8): fix Include before Match in sshd_config; bz#3122

 * ssh(1): close stdin/out/error when forking after authentication
   completes ("ssh -f ...") bz#3137

 * ssh(1), sshd(8): limit the amount of channel input data buffered,
   avoiding peers that advertise large windows but are slow to read
   from causing high memory consumption.

 * ssh-agent(1): handle multiple requests sent in a single write() to
   the agent.

 * sshd(8): allow sshd_config longer than 256k

 * sshd(8): avoid spurious "Unable to load host key" message when sshd
   load a private key but no public counterpart

 * ssh(1): prefer the default hostkey algorithm list whenever we have
   a hostkey that matches its best-preference algorithm.

 * sshd(1): when ordering the hostkey algorithms to request from a
   server, prefer certificate types if the known_hosts files contain a key
   marked as a @cert-authority; bz#3157

 * ssh(1): perform host key fingerprint comparisons for the "Are you
   sure you want to continue connecting (yes/no/[fingerprint])?"
   prompt with case sensitivity.

 * sshd(8): ensure that address/masklen mismatches in sshd_config
   yield fatal errors at daemon start time rather than later when
   they are evaluated.

 * ssh-keygen(1): ensure that certificate extensions are lexically
   sorted. Previously if the user specified a custom extension then
   the everything would be in order except the custom ones. bz#3198

 * ssh(1): also compare username when checking for JumpHost loops.
   bz#3057

 * ssh-keygen(1): preserve group/world read permission on known_hosts
   files across runs of "ssh-keygen -Rf /path". The old behaviour was
   to remove all rights for group/other. bz#3146

 * ssh-keygen(1): Mention the [-a rounds] flag in the ssh-keygen
   manual page and usage().

 * sshd(8): explicitly construct path to ~/.ssh/rc rather than
   relying on it being relative to the current directory, so that it
   can still be found if the shell startup changes its directory.
   bz#3185

 * sshd(8): when redirecting sshd's log output to a file, undo this
   redirection after the session child process is forked(). Fixes
   missing log messages when using this feature under some
   circumstances.

 * sshd(8): start ClientAliveInterval bookkeeping before first pass
   through select() loop; fixed theoretical case where busy sshd may
   ignore timeouts from client.

 * ssh(1): only reset the ServerAliveInterval check when we receive
   traffic from the server and ignore traffic from a port forwarding
   client, preventing a client from keeping a connection alive when
   it should be terminated. bz#2265

 * ssh-keygen(1): avoid spurious error message when ssh-keygen
   creates files outside ~/.ssh

 * sftp-client(1): fix off-by-one error that caused sftp downloads to
   make one more concurrent request that desired. This prevented using
   sftp(1) in unpipelined request/response mode, which is useful when
   debugging. bz#3054

 * ssh(1), sshd(8): handle EINTR in waitfd() and timeout_connect()
   helpers. bz#3071

 * ssh(1), ssh-keygen(1): defer creation of ~/.ssh until we attempt to
   write to it so we don't leave an empty .ssh directory when it's not
   needed. bz#3156

 * ssh(1), sshd(8): fix multiplier when parsing time specifications
   when handling seconds after other units. bz#3171

Portability
-----------

 * sshd(8): always send any PAM account messages. If the PAM account
   stack returns any messages, always send them to the user and not
   just if the check succeeds. bz#2049

 * Implement some backwards compatibility for libfido2 libraries
   older than 1.5.0. Note that use of an older library will result
   in the loss of certain features including resident key support,
   PIN support and support for multiple attached tokens.

 * configure fixes for XCode 12

 * gnome-ssh-askpass3: ensure the "close" button is not focused by
   default for SSH_ASKPASS_PROMPT=none prompts. Avoids space/enter
   accidentally dismissing FIDO touch notifications.

 * gnome-ssh-askpass3: allow some control over textarea colour via
   $GNOME_SSH_ASKPASS_FG_COLOR and $GNOME_SSH_ASKPASS_BG_COLOR
   environment variables.

 * sshd(8): document another PAM spec problem in a frustrated comment

 * sshd(8): support NetBSD's utmpx.ut_ss address field. bz#960

 * Add the ssh-sk-helper binary and its manpage to the RPM spec file

 * Detect the Frankenstein monster of Linux/X32 and allow the sandbox
   to function there. bz#3085

Checksums:
==========

- SHA1 (openssh-8.4.tar.gz) = 71675139df6807f396e6bd92ff8cb9b0356385d8
- SHA256 (openssh-8.4.tar.gz) = JhBgLYkyRge/zQK8ylBSRcOYvrV/tHwQcvVXfExGB70=

- SHA1 (openssh-8.4p1.tar.gz) = 69305059e10a60693ebe6f17731f962c9577535c
- SHA256 (openssh-8.4p1.tar.gz) = WgHSLkB+scBbqKj3xlTTiKE+nyJuTtM704dI2vodKyQ=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available as RELEASE_KEY.asc from
the mirror sites.

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 8.3 / 8.3p1 (2020-05-27) 

OpenSSH 8.3 was released on 2020-05-27. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Future deprecation notice
=========================

It is now possible[1] to perform chosen-prefix attacks against the
SHA-1 algorithm for less than USD$50K. For this reason, we will be
disabling the "ssh-rsa" public key signature algorithm by default in a
near-future release.

This algorithm is unfortunately still used widely despite the
existence of better alternatives, being the only remaining public key
signature algorithm specified by the original SSH RFCs.

The better alternatives include:

 * The RFC8332 RSA SHA-2 signature algorithms rsa-sha2-256/512. These
   algorithms have the advantage of using the same key type as
   "ssh-rsa" but use the safe SHA-2 hash algorithms. These have been
   supported since OpenSSH 7.2 and are already used by default if the
   client and server support them.

 * The ssh-ed25519 signature algorithm. It has been supported in
   OpenSSH since release 6.5.

 * The RFC5656 ECDSA algorithms: ecdsa-sha2-nistp256/384/521. These
   have been supported by OpenSSH since release 5.7.

To check whether a server is using the weak ssh-rsa public key
algorithm, for host authentication, try to connect to it after
removing the ssh-rsa algorithm from ssh(1)'s allowed list:

    ssh -oHostKeyAlgorithms=-ssh-rsa user@host

If the host key verification fails and no other supported host key
types are available, the server software on that host should be
upgraded.

A future release of OpenSSH will enable UpdateHostKeys by default
to allow the client to automatically migrate to better algorithms.
Users may consider enabling this option manually. Vendors of devices
that implement the SSH protocol should ensure that they support the
new signature algorithms for RSA keys.

[1] "SHA-1 is a Shambles: First Chosen-Prefix Collision on SHA-1 and
    Application to the PGP Web of Trust" Leurent, G and Peyrin, T
    (2020) https://eprint.iacr.org/2020/014.pdf

Security
========

 * scp(1): when receiving files, scp(1) could be become desynchronised
   if a utimes(2) system call failed. This could allow file contents
   to be interpreted as file metadata and thereby permit an adversary
   to craft a file system that, when copied with scp(1) in a
   configuration that caused utimes(2) to fail (e.g. under a SELinux
   policy or syscall sandbox), transferred different file names and
   contents to the actual file system layout.

   Exploitation of this is not likely as utimes(2) does not fail under
   normal circumstances. Successful exploitation is not silent - the
   output of scp(1) would show transfer errors followed by the actual
   file(s) that were received.

   Finally, filenames returned from the peer are (since openssh-8.0)
   matched against the user's requested destination, thereby
   disallowing a successful exploit from writing files outside the
   user's selected target glob (or directory, in the case of a
   recursive transfer). This ensures that this attack can achieve no
   more than a hostile peer is already able to achieve within the scp
   protocol.

Potentially-incompatible changes
================================

This release includes a number of changes that may affect existing
configurations:

 * sftp(1): reject an argument of "-1" in the same way as ssh(1) and
   scp(1) do instead of accepting and silently ignoring it.

Changes since OpenSSH 8.2
=========================

The focus of this release is bug fixing.

New Features
------------

 * sshd(8): make IgnoreRhosts a tri-state option: "yes" to ignore
   rhosts/shosts, "no" allow rhosts/shosts or (new) "shosts-only"
   to allow .shosts files but not .rhosts.

 * sshd(8): allow the IgnoreRhosts directive to appear anywhere in a
   sshd_config, not just before any Match blocks; bz3148

 * ssh(1): add %TOKEN percent expansion for the LocalFoward and
   RemoteForward keywords when used for Unix domain socket forwarding.
   bz#3014

 * all: allow loading public keys from the unencrypted envelope of a
   private key file if no corresponding public key file is present.
    
 * ssh(1), sshd(8): prefer to use chacha20 from libcrypto where
   possible instead of the (slower) portable C implementation included
   in OpenSSH.

 * ssh-keygen(1): add ability to dump the contents of a binary key
   revocation list via "ssh-keygen -lQf /path" bz#3132

Bugfixes
--------

 * ssh(1): fix IdentitiesOnly=yes to also apply to keys loaded from
   a PKCS11Provider; bz#3141

 * ssh-keygen(1): avoid NULL dereference when trying to convert an
   invalid RFC4716 private key.

 * scp(1): when performing remote-to-remote copies using "scp -3",
   start the second ssh(1) channel with BatchMode=yes enabled to
   avoid confusing and non-deterministic ordering of prompts.

 * ssh(1), ssh-keygen(1): when signing a challenge using a FIDO token,
   perform hashing of the message to be signed in the middleware layer
   rather than in OpenSSH code. This permits the use of security key
   middlewares that perform the hashing implicitly, such as Windows
   Hello.

 * ssh(1): fix incorrect error message for "too many known hosts
   files." bz#3149

 * ssh(1): make failures when establishing "Tunnel" forwarding
   terminate the connection when ExitOnForwardFailure is enabled;
   bz#3116

 * ssh-keygen(1): fix printing of fingerprints on private keys and add
   a regression test for same.

 * sshd(8): document order of checking AuthorizedKeysFile (first) and
   AuthorizedKeysCommand (subsequently, if the file doesn't match);
   bz#3134

 * sshd(8): document that /etc/hosts.equiv and /etc/shosts.equiv are
   not considered for HostbasedAuthentication when the target user is
   root; bz#3148
 
 * ssh(1), ssh-keygen(1): fix NULL dereference in private certificate
   key parsing (oss-fuzz #20074).

 * ssh(1), sshd(8): more consistency between sets of %TOKENS are
   accepted in various configuration options.

 * ssh(1), ssh-keygen(1): improve error messages for some common
   PKCS#11 C_Login failure cases; bz#3130

 * ssh(1), sshd(8): make error messages for problems during SSH banner
   exchange consistent with other SSH transport-layer error messages
   and ensure they include the relevant IP addresses bz#3129

 * various: fix a number of spelling errors in comments and debug/error
   messages

 * ssh-keygen(1), ssh-add(1): when downloading FIDO2 resident keys
   from a token, don't prompt for a PIN until the token has told us
   that it needs one. Avoids double-prompting on devices that
   implement on-device authentication.

 * sshd(8), ssh-keygen(1): no-touch-required FIDO certificate option
   should be an extension, not a critical option.
    
 * ssh(1), ssh-keygen(1), ssh-add(1): offer a better error message
   when trying to use a FIDO key function and SecurityKeyProvider is
   empty.

 * ssh-add(1), ssh-agent(8): ensure that a key lifetime fits within
   the values allowed by the wire format (u32). Prevents integer
   wraparound of the timeout values. bz#3119

 * ssh(1): detect and prevent trivial configuration loops when using
    ProxyJump. bz#3057.
    
Portability
-----------

 * Detect systems where signals flagged with SA_RESTART will interrupt
   select(2). POSIX permits implementations to choose whether
   select(2) will return when interrupted with a SA_RESTART-flagged
   signal, but OpenSSH requires interrupting behaviour.

 * Several compilation fixes for HP/UX and AIX.

 * On platforms that do not support setting process-wide routing
   domains (all excepting OpenBSD at present), fail to accept a
   configuration attempts to set one at process start time rather than
   fatally erroring at run time. bz#3126

 * Improve detection of egrep (used in regression tests) on platforms
   that offer a poor default one (e.g. Solaris).

 * A number of shell portability fixes for the regression tests.

 * Fix theoretical infinite loop in the glob(3) replacement
   implementation.

 * Fix seccomp sandbox compilation problems for some Linux
   configurations bz#3085

 * Improved detection of libfido2 and some compilation fixes for some
   configurations when --with-security-key-builtin is selected.

Checksums:
==========

 - SHA1 (openssh-8.3.tar.gz) = 46c63b7ddbe46a0666222f7988c993866c31fcca
 - SHA256 (openssh-8.3.tar.gz) = M6CnZ+duGs4bzDio8hQNLwyLQChV+3wkUEO8HWLV35c=

 - SHA1 (/openssh-8.3p1.tar.gz) = 04c7adb9986f16746588db8988b910530c589819
 - SHA256 (openssh-8.3p1.tar.gz) = 8r774Ecv5+t10jNA6xdTHLazqsJAdeIGa0H4FOEjh7I=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available as RELEASE_KEY.asc from
the mirror sites.

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 8.2 / 8.2p1 (2020-02-14) 

OpenSSH 8.2 was released on 2020-02-14. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
https://www.openssh.com/donations.html

Future deprecation notice
=========================

It is now possible[1] to perform chosen-prefix attacks against the
SHA-1 hash algorithm for less than USD$50K. For this reason, we will
be disabling the "ssh-rsa" public key signature algorithm that depends
on SHA-1 by default in a near-future release.

This algorithm is unfortunately still used widely despite the
existence of better alternatives, being the only remaining public key
signature algorithm specified by the original SSH RFCs.

The better alternatives include:

 * The RFC8332 RSA SHA-2 signature algorithms rsa-sha2-256/512. These
   algorithms have the advantage of using the same key type as
   "ssh-rsa" but use the safe SHA-2 hash algorithms. These have been
   supported since OpenSSH 7.2 and are already used by default if the
   client and server support them.

 * The ssh-ed25519 signature algorithm. It has been supported in
   OpenSSH since release 6.5.

 * The RFC5656 ECDSA algorithms: ecdsa-sha2-nistp256/384/521. These
   have been supported by OpenSSH since release 5.7.

To check whether a server is using the weak ssh-rsa public key
algorithm for host authentication, try to connect to it after
removing the ssh-rsa algorithm from ssh(1)'s allowed list:

    ssh -oHostKeyAlgorithms=-ssh-rsa user@host

If the host key verification fails and no other supported host key
types are available, the server software on that host should be
upgraded.

A future release of OpenSSH will enable UpdateHostKeys by default
to allow the client to automatically migrate to better algorithms.
Users may consider enabling this option manually.

[1] "SHA-1 is a Shambles: First Chosen-Prefix Collision on SHA-1 and
    Application to the PGP Web of Trust" Leurent, G and Peyrin, T
    (2020) https://eprint.iacr.org/2020/014.pdf

Security
========

 * ssh(1), sshd(8), ssh-keygen(1): this release removes the "ssh-rsa"
   (RSA/SHA1) algorithm from those accepted for certificate signatures
   (i.e. the client and server CASignatureAlgorithms option) and will
   use the rsa-sha2-512 signature algorithm by default when the
   ssh-keygen(1) CA signs new certificates.

   Certificates are at special risk to the aforementioned SHA1
   collision vulnerability as an attacker has effectively unlimited
   time in which to craft a collision that yields them a valid
   certificate, far more than the relatively brief LoginGraceTime
   window that they have to forge a host key signature.

   The OpenSSH certificate format includes a CA-specified (typically
   random) nonce value near the start of the certificate that should
   make exploitation of chosen-prefix collisions in this context
   challenging, as the attacker does not have full control over the
   prefix that actually gets signed. Nonetheless, SHA1 is now a
   demonstrably broken algorithm and further improvements in attacks
   are highly likely.

   OpenSSH releases prior to 7.2 do not support the newer RSA/SHA2
   algorithms and will refuse to accept certificates signed by an
   OpenSSH 8.2+ CA using RSA keys unless the unsafe algorithm is
   explicitly selected during signing ("ssh-keygen -t ssh-rsa").
   Older clients/servers may use another CA key type such as
   ssh-ed25519 (supported since OpenSSH 6.5) or one of the
   ecdsa-sha2-nistp256/384/521 types (supported since OpenSSH 5.7)
   instead if they cannot be upgraded.

Potentially-incompatible changes
================================

This release includes a number of changes that may affect existing
configurations:

 * ssh(1), sshd(8): the above removal of "ssh-rsa" from the accepted
   CASignatureAlgorithms list.

 * ssh(1), sshd(8): this release removes diffie-hellman-group14-sha1
   from the default key exchange proposal for both the client and
   server.

 * ssh-keygen(1): the command-line options related to the generation
   and screening of safe prime numbers used by the
   diffie-hellman-group-exchange-* key exchange algorithms have
   changed. Most options have been folded under the -O flag.

 * sshd(8): the sshd listener process title visible to ps(1) has
   changed to include information about the number of connections that
   are currently attempting authentication and the limits configured
   by MaxStartups.

 * ssh-sk-helper(8): this is a new binary. It is used by the FIDO/U2F
   support to provide address-space isolation for token middleware
   libraries (including the internal one). It needs to be installed
   in the expected path, typically under /usr/libexec or similar.

Changes since OpenSSH 8.1
=========================

This release contains some significant new features.

FIDO/U2F Support
----------------

This release adds support for FIDO/U2F hardware authenticators to
OpenSSH. U2F/FIDO are open standards for inexpensive two-factor
authentication hardware that are widely used for website
authentication.  In OpenSSH FIDO devices are supported by new public
key types "ecdsa-sk" and "ed25519-sk", along with corresponding
certificate types.

ssh-keygen(1) may be used to generate a FIDO token-backed key, after
which they may be used much like any other key type supported by
OpenSSH, so long as the hardware token is attached when the keys are
used. FIDO tokens also generally require the user explicitly authorise
operations by touching or tapping them.

Generating a FIDO key requires the token be attached, and will usually
require the user tap the token to confirm the operation:

  $ ssh-keygen -t ecdsa-sk -f ~/.ssh/id_ecdsa_sk
  Generating public/private ecdsa-sk key pair.
  You may need to touch your security key to authorize key generation.
  Enter file in which to save the key (/home/djm/.ssh/id_ecdsa_sk): 
  Enter passphrase (empty for no passphrase): 
  Enter same passphrase again: 
  Your identification has been saved in /home/djm/.ssh/id_ecdsa_sk
  Your public key has been saved in /home/djm/.ssh/id_ecdsa_sk.pub

This will yield a public and private key-pair. The private key file
should be useless to an attacker who does not have access to the
physical token. After generation, this key may be used like any other
supported key in OpenSSH and may be listed in authorized_keys, added
to ssh-agent(1), etc. The only additional stipulation is that the FIDO
token that the key belongs to must be attached when the key is used.

FIDO tokens are most commonly connected via USB but may be attached
via other means such as Bluetooth or NFC. In OpenSSH, communication
with the token is managed via a middleware library, specified by the
SecurityKeyProvider directive in ssh/sshd_config(5) or the
$SSH_SK_PROVIDER environment variable for ssh-keygen(1) and
ssh-add(1). The API for this middleware is documented in the sk-api.h
and PROTOCOL.u2f files in the source distribution.

OpenSSH includes a middleware ("SecurityKeyProvider=internal") with
support for USB tokens. It is automatically enabled in OpenBSD and may
be enabled in portable OpenSSH via the configure flag
--with-security-key-builtin. If the internal middleware is enabled
then it is automatically used by default. This internal middleware
requires that libfido2 (https://github.com/Yubico/libfido2)and its
dependencies be installed. We recommend that packagers of portable
OpenSSH enable the built-in middleware, as it provides the
lowest-friction experience for users.

Note: FIDO/U2F tokens are required to implement the ECDSA-P256
"ecdsa-sk" key type, but hardware support for Ed25519 "ed25519-sk" is
less common. Similarly, not all hardware tokens support some of the
optional features such as resident keys.

The protocol-level changes to support FIDO/U2F keys in SSH are
documented in the PROTOCOL.u2f file in the OpenSSH source
distribution.

There are a number of supporting changes to this feature:

 * ssh-keygen(1): add a "no-touch-required" option when generating
   FIDO-hosted keys, that disables their default behaviour of
   requiring a physical touch/tap on the token during authentication.
   Note: not all tokens support disabling the touch requirement.

 * sshd(8): add a sshd_config PubkeyAuthOptions directive that
   collects miscellaneous public key authentication-related options
   for sshd(8). At present it supports only a single option
   "no-touch-required". This causes sshd to skip its default check for
   FIDO/U2F keys that the signature was authorised by a touch or press
   event on the token hardware.

 * ssh(1), sshd(8), ssh-keygen(1): add a "no-touch-required" option
   for authorized_keys and a similar extension for certificates. This
   option disables the default requirement that FIDO key signatures
   attest that the user touched their key to authorize them, mirroring
   the similar PubkeyAuthOptions sshd_config option.

 * ssh-keygen(1): add support for the writing the FIDO attestation
   information that is returned when new keys are generated via the
   "-O write-attestation=/path" option. FIDO attestation certificates
   may be used to verify that a FIDO key is hosted in trusted
   hardware. OpenSSH does not currently make use of this information,
   beyond optionally writing it to disk.

FIDO2 resident keys
-------------------

FIDO/U2F OpenSSH keys consist of two parts: a "key handle" part stored
in the private key file on disk, and a per-device private key that is
unique to each FIDO/U2F token and that cannot be exported from the
token hardware. These are combined by the hardware at authentication
time to derive the real key that is used to sign authentication
challenges.

For tokens that are required to move between computers, it can be
cumbersome to have to move the private key file first. To avoid this
requirement, tokens implementing the newer FIDO2 standard support
"resident keys", where it is possible to effectively retrieve the key
handle part of the key from the hardware.

OpenSSH supports this feature, allowing resident keys to be generated
using the ssh-keygen(1) "-O resident" flag. This will produce a
public/private key pair as usual, but it will be possible to retrieve
the private key part from the token later. This may be done using
"ssh-keygen -K", which will download all available resident keys from
the tokens attached to the host and write public/private key files
for them. It is also possible to download and add resident keys
directly to ssh-agent(1) without writing files to the file-system
using "ssh-add -K".

Resident keys are indexed on the token by the application string and
user ID. By default, OpenSSH uses an application string of "ssh:" and
an empty user ID. If multiple resident keys on a single token are
desired then it may be necessary to override one or both of these
defaults using the ssh-keygen(1) "-O application=" or "-O user="
options. Note: OpenSSH will only download and use resident keys whose
application string begins with "ssh:"

Storing both parts of a key on a FIDO token increases the likelihood
of an attacker being able to use a stolen token device. For this
reason, tokens should enforce PIN authentication before allowing
download of keys, and users should set a PIN on their tokens before
creating any resident keys.

Other New Features
------------------

 * sshd(8): add an Include sshd_config keyword that allows including
   additional configuration files via glob(3) patterns. bz2468

 * ssh(1)/sshd(8): make the LE (low effort) DSCP code point available
   via the IPQoS directive; bz2986,

 * ssh(1): when AddKeysToAgent=yes is set and the key contains no
   comment, add the key to the agent with the key's path as the
   comment. bz2564
    
 * ssh-keygen(1), ssh-agent(1): expose PKCS#11 key labels and X.509
   subjects as key comments, rather than simply listing the PKCS#11
   provider library path. PR138

 * ssh-keygen(1): allow PEM export of DSA and ECDSA keys; bz3091

 * ssh(1), sshd(8): make zlib compile-time optional, available via the
   Makefile.inc ZLIB flag on OpenBSD or via the --with-zlib configure
   option for OpenSSH portable.

 * sshd(8): when clients get denied by MaxStartups, send a
   notification prior to the SSH2 protocol banner according to
   RFC4253 section 4.2.

 * ssh(1), ssh-agent(1): when invoking the $SSH_ASKPASS prompt
   program, pass a hint to the program to describe the type of
   desired prompt.  The possible values are "confirm" (indicating
   that a yes/no confirmation dialog with no text entry should be
   shown), "none" (to indicate an informational message only), or
   blank for the original ssh-askpass behaviour of requesting a
   password/phrase.

 * ssh(1): allow forwarding a different agent socket to the path
   specified by $SSH_AUTH_SOCK, by extending the existing ForwardAgent
   option to accepting an explicit path or the name of an environment
   variable in addition to yes/no.
   
 * ssh-keygen(1): add a new signature operations "find-principals" to
   look up the principal associated with a signature from an allowed-
   signers file.
    
 * sshd(8): expose the number of currently-authenticating connections
   along with the MaxStartups limit in the process title visible to
   "ps".

Bugfixes
--------

 * sshd(8): make ClientAliveCountMax=0 have sensible semantics: it
   will now disable connection killing entirely rather than the
   current behaviour of instantly killing the connection after the
   first liveness test regardless of success. bz2627
    
 * sshd(8): clarify order of AllowUsers / DenyUsers vs AllowGroups /
   DenyGroups in the sshd(8) manual page. bz1690

 * sshd(8): better describe HashKnownHosts in the manual page. bz2560

 * sshd(8): clarify that permitopen=/PermitOpen do no name or
   address translation in the manual page. bz3099

 * sshd(8): allow the UpdateHostKeys feature to function when
   multiple known_hosts files are in use. When updating host keys,
   ssh will now search subsequent known_hosts files, but will add
   updated host keys to the first specified file only. bz2738
    
 * All: replace all calls to signal(2) with a wrapper around
   sigaction(2). This wrapper blocks all other signals during the
   handler preventing races between handlers, and sets SA_RESTART
   which should reduce the potential for short read/write operations.
    
 * sftp(1): fix a race condition in the SIGCHILD handler that could
   turn in to a kill(-1); bz3084

 * sshd(8): fix a case where valid (but extremely large) SSH channel
   IDs were being incorrectly rejected. bz3098

 * ssh(1): when checking host key fingerprints as answers to new
   hostkey prompts, ignore whitespace surrounding the fingerprint
   itself.

 * All: wait for file descriptors to be readable or writeable during
   non-blocking connect, not just readable. Prevents a timeout when
   the server doesn't immediately send a banner (e.g. multiplexers
   like sslh)
 
 * sshd_config(5): document the sntrup4591761x25519-sha512@tinyssh.org
   key exchange algorithm. PR#151

Portability
-----------

 * sshd(8): multiple adjustments to the Linux seccomp sandbox:
   - Non-fatally deny IPC syscalls in sandbox
   - Allow clock_gettime64() in sandbox (MIPS / glibc >= 2.31)
   - Allow clock_nanosleep_time64 in sandbox (ARM) bz3100
   - Allow clock_nanosleep() in sandbox (recent glibc) bz3093

 * Explicit check for memmem declaration and fix up declaration if the
   system headers lack it. bz3102
 
Checksums:
==========

 - SHA1 (openssh-8.2.tar.gz) = 0daae2a8c47c489a8784f2c38c4b39e6159ba678
 - SHA256 (openssh-8.2.tar.gz) = +UmInEIoHJqYqWneMb/kgRbLcq8WDCo7+ooYcjzW4jg=

 - SHA1 (openssh-8.2p1.tar.gz) = d1ab35a93507321c5db885e02d41ce1414f0507c
 - SHA256 (openssh-8.2p1.tar.gz) = Q5JRUebPbO4UUBkMDpr03Da0HBJzdhnt/4vOvf9k5nE=

Note: the openssh-8.2 tarball for OpenBSD that was initially released
advertised an incorrect version for "ssh -V" and the sshd server
banner. The above tarball replace the incorrect release, which has
been renamed to openssh-8.2.tar.gz.incorrect. These are the checksums
for the original, incorrect tarball:

 - SHA1 (openssh-8.2.tar.gz) = 77584c22fbb89269398acdf53c1e554400584ba8
 - SHA256 (openssh-8.2.tar.gz) = UttLaaSYXVK1O65cYvyQzyQ5sCfuJ4Lwrs8zNsPrluQ=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available as RELEASE_KEY.asc from
the mirror sites.

Reporting Bugs:
===============

- Please read https://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 8.1 / 8.1p1 (2019-10-09) 

OpenSSH 8.1 was released on 2019-10-09. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Security
========

 * ssh(1), sshd(8), ssh-add(1), ssh-keygen(1): an exploitable integer
   overflow bug was found in the private key parsing code for the XMSS
   key type. This key type is still experimental and support for it is
   not compiled by default. No user-facing autoconf option exists in
   portable OpenSSH to enable it. This bug was found by Adam Zabrocki
   and reported via SecuriTeam's SSD program.

 * ssh(1), sshd(8), ssh-agent(1): add protection for private keys at
   rest in RAM against speculation and memory side-channel attacks like
   Spectre, Meltdown and Rambleed. This release encrypts private keys
   when they are not in use with a symmetric key that is derived from a
   relatively large "prekey" consisting of random data (currently 16KB).

Potentially-incompatible changes
================================

This release includes a number of changes that may affect existing
configurations:

 * ssh-keygen(1): when acting as a CA and signing certificates with
   an RSA key, default to using the rsa-sha2-512 signature algorithm.
   Certificates signed by RSA keys will therefore be incompatible
   with OpenSSH versions prior to 7.2 unless the default is
   overridden (using "ssh-keygen -t ssh-rsa -s ...").

Changes since OpenSSH 8.0
=========================

This release is focused on bug-fixing.

New Features
------------

 * ssh(1): Allow %n to be expanded in ProxyCommand strings

 * ssh(1), sshd(8): Allow prepending a list of algorithms to the
   default set by starting the list with the '^' character, E.g.
   "HostKeyAlgorithms ^ssh-ed25519"

 * ssh-keygen(1): add an experimental lightweight signature and
   verification ability. Signatures may be made using regular ssh keys
   held on disk or stored in a ssh-agent and verified against an
   authorized_keys-like list of allowed keys. Signatures embed a
   namespace that prevents confusion and attacks between different
   usage domains (e.g. files vs email).

 * ssh-keygen(1): print key comment when extracting public key from a
   private key.  bz#3052

 * ssh-keygen(1): accept the verbose flag when searching for host keys
   in known hosts (i.e. "ssh-keygen -vF host") to print the matching
   host's random-art signature too. bz#3003

 * All: support PKCS8 as an optional format for storage of private
   keys to disk.  The OpenSSH native key format remains the default,
   but PKCS8 is a superior format to PEM if interoperability with
   non-OpenSSH software is required, as it may use a less insecure
   key derivation function than PEM's.

Bugfixes
--------

 * ssh(1): if a PKCS#11 token returns no keys then try to login and
   refetch them. Based on patch from Jakub Jelen; bz#2430

 * ssh(1): produce a useful error message if the user's shell is set
   incorrectly during "match exec" processing. bz#2791

 * sftp(1): allow the maximum uint32 value for the argument passed
   to -b which allows better error messages from later validation.
   bz#3050

 * ssh(1): avoid pledge sandbox violations in some combinations of
   remote forwarding, connection multiplexing and ControlMaster.

 * ssh-keyscan(1): include SHA2-variant RSA key algorithms in KEX
   proposal; allows ssh-keyscan to harvest keys from servers that
   disable old SHA1 ssh-rsa. bz#3029

 * sftp(1): print explicit "not modified" message if a file was
   requested for resumed download but was considered already complete.
   bz#2978

 * sftp(1): fix a typo and make <esc><right> move right to the
   closest end of a word just like <esc><left> moves left to the
   closest beginning of a word.

 * sshd(8): cap the number of permitopen/permitlisten directives
   allowed to appear on a single authorized_keys line.

 * All: fix a number of memory leaks (one-off or on exit paths).

 * Regression tests: a number of fixes and improvements, including
   fixes to the interop tests, adding the ability to run most tests
   on builds that disable OpenSSL support, better support for running
   tests under Valgrind and a number of bug-fixes.

 * ssh(1), sshd(8): check for convtime() refusing to accept times that
   resolve to LONG_MAX Reported by Kirk Wolf bz2977

 * ssh(1): slightly more instructive error message when the user
   specifies multiple -J options on the command-line. bz3015

 * ssh-agent(1): process agent requests for RSA certificate private
   keys using correct signature algorithm when requested. bz3016

 * sftp(1): check for user@host when parsing sftp target. This
   allows user@[1.2.3.4] to work without a path.  bz#2999

 * sshd(8): enlarge format buffer size for certificate serial
   number so the log message can record any 64-bit integer without
   truncation. bz#3012

 * sshd(8): for PermitOpen violations add the remote host and port to
   be able to more easily ascertain the source of the request. Add the
   same logging for PermitListen violations which where not previously
   logged at all.

 * scp(1), sftp(1): use the correct POSIX format style for left
   justification for the transfer progress meter. bz#3002

 * sshd(8) when examining a configuration using sshd -T, assume any
   attribute not provided by -C does not match, which allows it to work
   when sshd_config contains a Match directive with or without -C.
   bz#2858

 * ssh(1), ssh-keygen(1): downgrade PKCS#11 "provider returned no
   slots" warning from log level error to debug. This is common when
   attempting to enumerate keys on smartcard readers with no cards
   plugged in. bz#3058

 * ssh(1), ssh-keygen(1): do not unconditionally log in to PKCS#11
   tokens. Avoids spurious PIN prompts for keys not selected for
   authentication in ssh(1) and when listing public keys available in
   a token using ssh-keygen(1). bz#3006

Portability
-----------

 * ssh(1): fix SIGWINCH delivery of Solaris for multiplexed sessions
   bz#3030

 * ssh(1), sshd(8): fix typo that prevented detection of Linux VRF

 * sshd(8): add no-op implementation of pam_putenv to avoid build
   breakage on platforms where the PAM implementation lacks this
   function (e.g. HP-UX). bz#3008

 * sftp-server(8): fix Solaris privilege sandbox from preventing
   the legacy sftp rename operation from working (was refusing to
   allow hard links to files owned by other users). bz#3036

 * All: add a proc_pidinfo()-based closefrom() for OS X to avoid
   the need to brute-force close all high-numbered file descriptors.
   bz#3049

 * sshd(8): in the Linux seccomp-bpf sandbox, allow mprotect(2) with
   PROT_(READ|WRITE|NONE) only. This syscall is used by some hardened
   heap allocators. Github PR#142

 * sshd(8): in the Linux seccomp-bpf sandbox, allow the s390-specific
   ioctl for ECC hardware support.

 * All: use "doc" man page format if the mandoc(1) tool is present on
   the system. Previously configure would not select the "doc" man
   page format if mandoc was present but nroff was not.

 * sshd(8): don't install duplicate STREAMS modules on Solaris; check
   if STREAMS modules are already installed on a pty before installing
   since when compiling with XPG>=4 they will likely be installed
   already. Prevents hangs and duplicate lines on the terminal.
   bz#2945 and bz#2998,

Checksums:
==========

 - SHA1 (openssh-8.1.tar.gz) = bf7b0c65a7c0afa5ba9c787f345b8a24fa459add
 - SHA256 (openssh-8.1.tar.gz) = vamkKxZTFfgxQXSxGeJ1vbuot0H3Vx9bNBgrvChSrFg=

 - SHA1 (openssh-8.1p1.tar.gz) = c44b96094869f177735ae053d92bd5fcab1319de
 - SHA256 (openssh-8.1p1.tar.gz) = AvXb7zg10HU1VvlzzVe0wZtrH2zSTANEXiOsd8obk/8=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available as RELEASE_KEY.asc from
the mirror sites.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 8.0 / 8.0p1 (2019-04-17) 

OpenSSH 8.0 was released on 2019-04-17. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Security
========

This release contains mitigation for a weakness in the scp(1) tool
and protocol (CVE-2019-6111): when copying files from a remote system
to a local directory, scp(1) did not verify that the filenames that
the server sent matched those requested by the client. This could
allow a hostile server to create or clobber unexpected local files
with attacker-controlled content.

This release adds client-side checking that the filenames sent from
the server match the command-line request,

The scp protocol is outdated, inflexible and not readily fixed. We
recommend the use of more modern protocols like sftp and rsync for
file transfer instead.

Potentially-incompatible changes
================================

This release includes a number of changes that may affect existing
configurations:

 * scp(1): Relating to the above changes to scp(1); the scp protocol
   relies on the remote shell for wildcard expansion, so there is no
   infallible way for the client's wildcard matching to perfectly
   reflect the server's. If there is a difference between client and
   server wildcard expansion, the client may refuse files from the
   server. For this reason, we have provided a new "-T" flag to scp
   that disables these client-side checks at the risk of
   reintroducing the attack described above.

 * sshd(8): Remove support for obsolete "host/port" syntax. Slash-
   separated host/port was added in 2001 as an alternative to
   host:port syntax for the benefit of IPv6 users. These days there
   are established standards for this like [::1]:22 and the slash
   syntax is easily mistaken for CIDR notation, which OpenSSH
   supports for some things. Remove the slash notation from
   ListenAddress and PermitOpen; bz#2335

Changes since OpenSSH 7.9
=========================

This release is focused on new features and internal refactoring.

New Features
------------

 * ssh(1), ssh-agent(1), ssh-add(1): Add support for ECDSA keys in
   PKCS#11 tokens.

 * ssh(1), sshd(8): Add experimental quantum-computing resistant
   key exchange method, based on a combination of Streamlined NTRU
   Prime 4591^761 and X25519.

 * ssh-keygen(1): Increase the default RSA key size to 3072 bits,
   following NIST Special Publication 800-57's guidance for a
   128-bit equivalent symmetric security level.

 * ssh(1): Allow "PKCS11Provider=none" to override later instances of
   the PKCS11Provider directive in ssh_config; bz#2974

 * sshd(8): Add a log message for situations where a connection is
   dropped for attempting to run a command but a sshd_config
   ForceCommand=internal-sftp restriction is in effect; bz#2960

 * ssh(1): When prompting whether to record a new host key, accept
   the key fingerprint as a synonym for "yes". This allows the user
   to paste a fingerprint obtained out of band at the prompt and
   have the client do the comparison for you.

 * ssh-keygen(1): When signing multiple certificates on a single
   command-line invocation, allow automatically incrementing the
   certificate serial number.

 * scp(1), sftp(1): Accept -J option as an alias to ProxyJump on
   the scp and sftp command-lines.

 * ssh-agent(1), ssh-pkcs11-helper(8), ssh-add(1): Accept "-v"
   command-line flags to increase the verbosity of output; pass
   verbose flags though to subprocesses, such as ssh-pkcs11-helper
   started from ssh-agent.

 * ssh-add(1): Add a "-T" option to allowing testing whether keys in
   an agent are usable by performing a signature and a verification.

 * sftp-server(8): Add a "lsetstat@openssh.com" protocol extension
   that replicates the functionality of the existing SSH2_FXP_SETSTAT
   operation but does not follow symlinks. bz#2067

 * sftp(1): Add "-h" flag to chown/chgrp/chmod commands to request
   they do not follow symlinks.

 * sshd(8): Expose $SSH_CONNECTION in the PAM environment. This makes
   the connection 4-tuple available to PAM modules that wish to use
   it in decision-making. bz#2741

 * sshd(8): Add a ssh_config "Match final" predicate Matches in same
   pass as "Match canonical" but doesn't require hostname
   canonicalisation be enabled. bz#2906

 * sftp(1): Support a prefix of '@' to suppress echo of sftp batch
   commands; bz#2926

 * ssh-keygen(1): When printing certificate contents using
   "ssh-keygen -Lf /path/certificate", include the algorithm that
   the CA used to sign the cert.

Bugfixes
--------

 * sshd(8): Fix authentication failures when sshd_config contains
   "AuthenticationMethods any" inside a Match block that overrides
   a more restrictive default.

 * sshd(8): Avoid sending duplicate keepalives when ClientAliveCount
   is enabled.

 * sshd(8): Fix two race conditions related to SIGHUP daemon restart.
   Remnant file descriptors in recently-forked child processes could
   block the parent sshd's attempt to listen(2) to the configured
   addresses. Also, the restarting parent sshd could exit before any
   child processes that were awaiting their re-execution state had
   completed reading it, leaving them in a fallback path.

 * ssh(1): Fix stdout potentially being redirected to /dev/null when
   ProxyCommand=- was in use.

 * sshd(8): Avoid sending SIGPIPE to child processes if they attempt
   to write to stderr after their parent processes have exited;
   bz#2071

 * ssh(1): Fix bad interaction between the ssh_config ConnectTimeout
   and ConnectionAttempts directives - connection attempts after the
   first were ignoring the requested timeout; bz#2918

 * ssh-keyscan(1): Return a non-zero exit status if no keys were
   found; bz#2903

 * scp(1): Sanitize scp filenames to allow UTF-8 characters without
   terminal control sequences;  bz#2434

 * sshd(8): Fix confusion between ClientAliveInterval and time-based
   RekeyLimit that could cause connections to be incorrectly closed.
   bz#2757

 * ssh(1), ssh-add(1): Correct some bugs in PKCS#11 token PIN
   handling at initial token login. The attempt to read the PIN
   could be skipped in some cases, particularly on devices with
   integrated PIN readers. This would lead to an inability to
   retrieve keys from these tokens. bz#2652

 * ssh(1), ssh-add(1): Support keys on PKCS#11 tokens that set the
   CKA_ALWAYS_AUTHENTICATE flag by requiring a fresh login after the
   C_SignInit operation. bz#2638

 * ssh(1): Improve documentation for ProxyJump/-J, clarifying that
   local configuration does not apply to jump hosts.

 * ssh-keygen(1): Clarify manual - ssh-keygen -e only writes
   public keys, not private.

 * ssh(1), sshd(8): be more strict in processing protocol banners,
   allowing \r characters only immediately before \n.

 * Various: fix a number of memory leaks, including bz#2942 and
   bz#2938

 * scp(1), sftp(1): fix calculation of initial bandwidth limits.
   Account for bytes written before the timer starts and adjust the
   schedule on which recalculations are performed. Avoids an initial
   burst of traffic and yields more accurate bandwidth limits;
   bz#2927

 * sshd(8): Only consider the ext-info-c extension during the initial
   key eschange. It shouldn't be sent in subsequent ones, but if it
   is present we should ignore it. This prevents sshd from sending a
   SSH_MSG_EXT_INFO for REKEX for buggy these clients. bz#2929

 * ssh-keygen(1): Clarify manual that ssh-keygen -F (find host in 
   authorized_keys) and -R (remove host from authorized_keys) options
   may accept either a bare hostname or a [hostname]:port combo.
   bz#2935

 * ssh(1): Don't attempt to connect to empty SSH_AUTH_SOCK; bz#2936

 * sshd(8): Silence error messages when sshd fails to load some of
   the default host keys. Failure to load an explicitly-configured
   hostkey is still an error, and failure to load any host key is
   still fatal. pr/103

 * ssh(1): Redirect stderr of ProxyCommands to /dev/null when ssh is
   started with ControlPersist; prevents random ProxyCommand output
   from interfering with session output.

 * ssh(1): The ssh client was keeping a redundant ssh-agent socket
   (leftover from authentication) around for the life of the
   connection; bz#2912

 * sshd(8): Fix bug in HostbasedAcceptedKeyTypes and
   PubkeyAcceptedKeyTypes options. If only RSA-SHA2 signature types
   were specified, then authentication would always fail for RSA keys
   as the monitor checks only the base key (not the signature
   algorithm) type against *AcceptedKeyTypes. bz#2746

 * ssh(1): Request correct signature types from ssh-agent when
   certificate keys and RSA-SHA2 signatures are in use.

Portability
-----------

 * sshd(8): On Cygwin, run as SYSTEM where possible, using S4U for
   token creation if it supports MsV1_0 S4U Logon.

 * sshd(8): On Cygwin, use custom user/group matching code that
   respects the OS' behaviour of case-insensitive matching.

 * sshd(8): Don't set $MAIL if UsePAM=yes as PAM typically specifies
   the user environment if it's enabled; bz#2937

 * sshd(8) Cygwin: Change service name to cygsshd to avoid collision
   with Microsoft's OpenSSH port.

 * Allow building against OpenSSL -dev (3.x)

 * Fix a number of build problems against version configurations and
   versions of OpenSSL. Including bz#2931 and bz#2921

 * Improve warnings in cygwin service setup. bz#2922

 * Remove hardcoded service name in cygwin setup. bz#2922

Checksums:
==========

 - SHA1 (openssh-8.0.tar.gz) = 8aaa99091fc7e5a92a4a320e1e5521046b3f95f0
 - SHA256 (openssh-8.0.tar.gz) = 1xvSJk1KYSnOLPYEUzyCVwTEQ7MHOaCO65DzeNuuLdo=

 - SHA1 (openssh-8.0p1.tar.gz) = 756dbb99193f9541c9206a667eaa27b0fa184a4f
 - SHA256 (openssh-8.0p1.tar.gz) = vZQ4eeaUmOgDHra39E0IzcN9WaeraJqgtDcyDDSB/Wg=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available as RELEASE_KEY.asc from
the mirror sites.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 7.9 / 7.9p1 (2018-10-19) 

OpenSSH 7.9 was released on 2018-10-19. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Potentially-incompatible changes
================================

This release includes a number of changes that may affect existing
configurations:

 * ssh(1), sshd(8): the setting of the new CASignatureAlgorithms
   option (see below) bans the use of DSA keys as certificate
   authorities.

 * sshd(8): the authentication success/failure log message has
   changed format slightly. It now includes the certificate
   fingerprint (previously it included only key ID and CA key
   fingerprint).

Changes since OpenSSH 7.8
=========================

This is primarily a bugfix release.

New Features
------------

 * ssh(1), sshd(8): allow most port numbers to be specified using
   service names from getservbyname(3) (typically /etc/services).

 * ssh(1): allow the IdentityAgent configuration directive to accept
   environment variable names. This supports the use of multiple
   agent sockets without needing to use fixed paths.

 * sshd(8): support signalling sessions via the SSH protocol.
   A limited subset of signals is supported and only for login or
   command sessions (i.e. not subsystems) that were not subject to
   a forced command via authorized_keys or sshd_config. bz#1424

 * ssh(1): support "ssh -Q sig" to list supported signature options.
   Also "ssh -Q help" to show the full set of supported queries.

 * ssh(1), sshd(8): add a CASignatureAlgorithms option for the
   client and server configs to allow control over which signature
   formats are allowed for CAs to sign certificates. For example,
   this allows banning CAs that sign certificates using the RSA-SHA1
   signature algorithm.

 * sshd(8), ssh-keygen(1): allow key revocation lists (KRLs) to
   revoke keys specified by SHA256 hash.

 * ssh-keygen(1): allow creation of key revocation lists directly
   from base64-encoded SHA256 fingerprints. This supports revoking
   keys using only the information contained in sshd(8)
   authentication log messages.

Bugfixes
--------

 * ssh(1), ssh-keygen(1): avoid spurious "invalid format" errors when
   attempting to load PEM private keys while using an incorrect
   passphrase. bz#2901

 * sshd(8): when a channel closed message is received from a client,
   close the stderr file descriptor at the same time stdout is
   closed. This avoids stuck processes if they were waiting for
   stderr to close and were insensitive to stdin/out closing. bz#2863

 * ssh(1): allow ForwardX11Timeout=0 to disable the untrusted X11
   forwarding timeout and support X11 forwarding indefinitely.
   Previously the behaviour of ForwardX11Timeout=0 was undefined.

 * sshd(8): when compiled with GSSAPI support, cache supported method
   OIDs regardless of whether GSSAPI authentication is enabled in the
   main section of sshd_config. This avoids sandbox violations if
   GSSAPI authentication was later enabled in a Match block. bz#2107

 * sshd(8): do not fail closed when configured with a text key
   revocation list that contains a too-short key. bz#2897

 * ssh(1): treat connections with ProxyJump specified the same as
   ones with a ProxyCommand set with regards to hostname
   canonicalisation (i.e. don't try to canonicalise the hostname
   unless CanonicalizeHostname is set to 'always'). bz#2896

 * ssh(1): fix regression in OpenSSH 7.8 that could prevent public-
   key authentication using certificates hosted in a ssh-agent(1)
   or against sshd(8) from OpenSSH <7.8.

Portability
-----------

 * All: support building against the openssl-1.1 API (releases 1.1.0g
   and later). The openssl-1.0 API will remain supported at least
   until OpenSSL terminates security patch support for that API version.

 * sshd(8): allow the futex(2) syscall in the Linux seccomp sandbox;
   apparently required by some glibc/OpenSSL combinations.

 * sshd(8): handle getgrouplist(3) returning more than
   _SC_NGROUPS_MAX groups. Some platforms consider this limit more
   as a guideline.

Checksums:
==========

 - SHA1 (openssh-7.9.tar.gz) = 7c50a86b8f591decd172ed7f5527abc533098dec
 - SHA256 (openssh-7.9.tar.gz) = nSVigtHGn3+xKXRqpSnp4YOyEPPAb+pCHdWS9Eh/IPY=

 - SHA1 (openssh-7.9p1.tar.gz) = 993aceedea8ecabb1d0dd7293508a361891c4eaa
 - SHA256 (openssh-7.9p1.tar.gz) = a0s7oiU9hO03ccgFByjVl8kc/OiYcTvre2SjBbbxGq0=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available as RELEASE_KEY.asc from
the mirror sites.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 7.8 / 7.8p1 (2018-08-24) 

OpenSSH 7.8 was released on 2018-08-24. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Potentially-incompatible changes
================================

This release includes a number of changes that may affect existing
configurations:

 * ssh-keygen(1): write OpenSSH format private keys by default
   instead of using OpenSSL's PEM format. The OpenSSH format,
   supported in OpenSSH releases since 2014 and described in the
   PROTOCOL.key file in the source distribution, offers substantially
   better protection against offline password guessing and supports
   key comments in private keys. If necessary, it is possible to write
   old PEM-style keys by adding "-m PEM" to ssh-keygen's arguments
   when generating or updating a key.

 * sshd(8): remove internal support for S/Key multiple factor
   authentication. S/Key may still be used via PAM or BSD auth.

 * ssh(1): remove vestigal support for running ssh(1) as setuid. This
   used to be required for hostbased authentication and the (long
   gone) rhosts-style authentication, but has not been necessary for
   a long time. Attempting to execute ssh as a setuid binary, or with
   uid != effective uid will now yield a fatal error at runtime.

 * sshd(8): the semantics of PubkeyAcceptedKeyTypes and the similar
   HostbasedAcceptedKeyTypes options have changed. These now specify
   signature algorithms that are accepted for their respective
   authentication mechanism, where previously they specified accepted
   key types. This distinction matters when using the RSA/SHA2
   signature algorithms "rsa-sha2-256", "rsa-sha2-512" and their
   certificate counterparts. Configurations that override these
   options but omit these algorithm names may cause unexpected
   authentication failures (no action is required for configurations
   that accept the default for these options).

 * sshd(8): the precedence of session environment variables has
   changed. ~/.ssh/environment and environment="..." options in
   authorized_keys files can no longer override SSH_* variables set
   implicitly by sshd.

 * ssh(1)/sshd(8): the default IPQoS used by ssh/sshd has changed.
   They will now use DSCP AF21 for interactive traffic and CS1 for
   bulk.  For a detailed rationale, please see the commit message:
   https://cvsweb.openbsd.org/log/src/usr.bin/ssh/readconf.c,v#rev1.284

Changes since OpenSSH 7.7
=========================

This is primarily a bugfix release.

New Features
------------

 * ssh(1)/sshd(8): add new signature algorithms "rsa-sha2-256-cert-
   v01@openssh.com" and "rsa-sha2-512-cert-v01@openssh.com" to
   explicitly force use of RSA/SHA2 signatures in authentication.

 * sshd(8): extend the PermitUserEnvironment option to accept a
   whitelist of environment variable names in addition to global
   "yes" or "no" settings.

 * sshd(8): add a PermitListen directive to sshd_config(5) and a 
   corresponding permitlisten= authorized_keys option that control
   which listen addresses and port numbers may be used by remote
   forwarding (ssh -R ...).

 * sshd(8): add some countermeasures against timing attacks used for
   account validation/enumeration. sshd will enforce a minimum time
   or each failed authentication attempt consisting of a global 5ms
   minimum plus an additional per-user 0-4ms delay derived from a
   host secret.

 * sshd(8): add a SetEnv directive to allow an administrator to
   explicitly specify environment variables in sshd_config.
   Variables set by SetEnv override the default and client-specified
   environment.

 * ssh(1): add a SetEnv directive to request that the server sets
   an environment variable in the session. Similar to the existing
   SendEnv option, these variables are set subject to server
   configuration.

 * ssh(1): allow "SendEnv -PATTERN" to clear environment variables
   previously marked for sending to the server. bz#1285

 * ssh(1)/sshd(8): make UID available as a %-expansion everywhere
   that the username is available currently. bz#2870

 * ssh(1): allow setting ProxyJump=none to disable ProxyJump
   functionality. bz#2869

Bugfixes
--------

 * sshd(8): avoid observable differences in request parsing that could
   be used to determine whether a target user is valid.

 * all: substantial internal refactoring

 * ssh(1)/sshd(8): fix some memory leaks; bz#2366

 * ssh(1): fix a pwent clobber (introduced in openssh-7.7) that could
   occur during key loading, manifesting as crash on some platforms.

 * sshd_config(5): clarify documentation for AuthenticationMethods
   option; bz#2663

 * ssh(1): ensure that the public key algorithm sent in a
   public key SSH_MSG_USERAUTH_REQUEST matches the content of the
   signature blob. Previously, these could be inconsistent when a
   legacy or non-OpenSSH ssh-agent returned a RSA/SHA1 signature
   when asked to make a RSA/SHA2 signature.

 * sshd(8): fix failures to read authorized_keys caused by faulty
   supplemental group caching. bz#2873

 * scp(1): apply umask to directories, fixing potential mkdir/chmod
   race when copying directory trees bz#2839

 * ssh-keygen(1): return correct exit code when searching for and
   hashing known_hosts entries in a single operation; bz#2772

 * ssh(1): prefer the ssh binary pointed to via argv[0] to $PATH when
   re-executing ssh for ProxyJump. bz#2831

 * sshd(8): do not ban PTY allocation when a sshd session is
   restricted because the user password is expired as it breaks
   password change dialog. (regression in openssh-7.7).

 * ssh(1)/sshd(8): fix error reporting from select() failures. 

 * ssh(1): improve documentation for -w (tunnel) flag, emphasising
   that -w implicitly sets Tunnel=point-to-point. bz#2365

 * ssh-agent(1): implement EMFILE mitigation for ssh-agent. ssh-agent
   will no longer spin when its file descriptor limit is exceeded.
   bz#2576

 * ssh(1)/sshd(8): disable SSH2_MSG_DEBUG messages for Twisted Conch
   clients. Twisted Conch versions that lack a version number in
   their identification strings will mishandle these messages when
   running on Python 2.x (https://twistedmatrix.com/trac/ticket/9422)
 * sftp(1): notify user immediately when underlying ssh process dies
   expectedly. bz#2719

 * ssh(1)/sshd(8): fix tunnel forwarding; regression in 7.7 release.
   bz#2855

 * ssh-agent(1): don't kill ssh-agent's listening socket entirely if
   it fails to accept(2) a connection. bz#2837

 * sshd(8): relax checking of authorized_keys environment="..."
   options to allow underscores in variable names (regression
   introduced in 7.7). bz#2851

 * ssh(1): add some missing options in the configuration dump output
   (ssh -G). bz#2835

Portability
-----------

 * sshd(8): Expose details of completed authentication to PAM auth
   modules via SSH_AUTH_INFO_0 in the PAM environment. bz#2408

 * Fix compilation problems caused by fights between zlib and OpenSSL
   colliding uses of "free_func"

 * Improve detection of unsupported compiler options. Recently these
   may have manifested as "unsupported -Wl,-z,retpoline" warnings
   during linking.

 * sshd(8): some sandbox support for Linux/s390 bz#2752.

 * regress tests: unbreak key-options.sh test on platforms without
   openpty(3). bz#2856

 * use getrandom(2) for PRNG seeding when built without OpenSSL.

Checksums:
==========

- SHA1 (openssh-7.8.tar.gz) = ed5511cd42b543cd15166a9cbc56705f23b847e7
- SHA256 (openssh-7.8.tar.gz) = TDqIsMEmghsBUNCrSCPyCxChfitntyOLXNC694py1XE

- SHA1 (openssh-7.8p1.tar.gz) = 27e267e370315561de96577fccae563bc2c37a60
- SHA256 (openssh-7.8p1.tar.gz) = GkhLsVFSwYO7JRThEqow3TQTjDz7Ay7uVJCmbFBxRMo

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available as RELEASE_KEY.asc from
the mirror sites.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 7.7 / 7.7p1 (2018-04-02) 

OpenSSH 7.7 was released on 2018-04-02. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Potentially-incompatible changes
================================

This release includes a number of changes that may affect existing
configurations:

 * ssh(1)/sshd(8): Drop compatibility support for some very old SSH
   implementations, including ssh.com <=2.* and OpenSSH <= 3.*. These
   versions were all released in or before 2001 and predate the final
   SSH RFCs. The support in question isn't necessary for RFC-compliant
   SSH implementations.

Changes since OpenSSH 7.6
=========================

This is primarily a bugfix release.

New Features
------------

 * All: Add experimental support for PQC XMSS keys (Extended Hash-
   Based Signatures) based on the algorithm described in
   https://tools.ietf.org/html/draft-irtf-cfrg-xmss-hash-based-signatures-12
   The XMSS signature code is experimental and not compiled in by
   default.

 * sshd(8): Add a "rdomain" criteria for the sshd_config Match keyword
   to allow conditional configuration that depends on which routing
   domain a connection was received on (currently supported on OpenBSD
   and Linux).

 * sshd_config(5): Add an optional rdomain qualifier to the
   ListenAddress directive to allow listening on different routing
   domains. This is supported only on OpenBSD and Linux at present.

 * sshd_config(5): Add RDomain directive to allow the authenticated
   session to be placed in an explicit routing domain. This is only
   supported on OpenBSD at present.

 * sshd(8): Add "expiry-time" option for authorized_keys files to
   allow for expiring keys.

 * ssh(1): Add a BindInterface option to allow binding the outgoing
   connection to an interface's address (basically a more usable
   BindAddress)

 * ssh(1): Expose device allocated for tun/tap forwarding via a new
   %T expansion for LocalCommand. This allows LocalCommand to be used
   to prepare the interface.

 * sshd(8): Expose the device allocated for tun/tap forwarding via a
   new SSH_TUNNEL environment variable. This allows automatic setup of
   the interface and surrounding network configuration automatically on
   the server.

 * ssh(1)/scp(1)/sftp(1): Add URI support to ssh, sftp and scp, e.g.
   ssh://user@host or sftp://user@host/path.  Additional connection
   parameters described in draft-ietf-secsh-scp-sftp-ssh-uri-04 are not
   implemented since the ssh fingerprint format in the draft uses the
   deprecated MD5 hash with no way to specify the any other algorithm.

 * ssh-keygen(1): Allow certificate validity intervals that specify
   only a start or stop time (instead of both or neither).

 * sftp(1): Allow "cd" and "lcd" commands with no explicit path
   argument. lcd will change to the local user's home directory as
   usual. cd will change to the starting directory for session (because
   the protocol offers no way to obtain the remote user's home
   directory). bz#2760

 * sshd(8): When doing a config test with sshd -T, only require the
   attributes that are actually used in Match criteria rather than (an
   incomplete list of) all criteria.

Bugfixes
--------

 * ssh(1)/sshd(8): More strictly check signature types during key
   exchange against what was negotiated. Prevents downgrade of RSA
   signatures made with SHA-256/512 to SHA-1.

 * sshd(8): Fix support for client that advertise a protocol version
   of "1.99" (indicating that they are prepared to accept both SSHv1 and
   SSHv2). This was broken in OpenSSH 7.6 during the removal of SSHv1
   support. bz#2810

 * ssh(1): Warn when the agent returns a ssh-rsa (SHA1) signature when
   a rsa-sha2-256/512 signature was requested. This condition is possible
   when an old or non-OpenSSH agent is in use. bz#2799

 * ssh-agent(1): Fix regression introduced in 7.6 that caused ssh-agent
   to fatally exit if presented an invalid signature request message.

 * sshd_config(5): Accept yes/no flag options case-insensitively, as
   has been the case in ssh_config(5) for a long time. bz#2664

 * ssh(1): Improve error reporting for failures during connection.
   Under some circumstances misleading errors were being shown. bz#2814

 * ssh-keyscan(1): Add -D option to allow printing of results directly
   in SSHFP format. bz#2821

 * regress tests: fix PuTTY interop test broken in last release's SSHv1
   removal. bz#2823

 * ssh(1): Compatibility fix for some servers that erroneously drop the
   connection when the IUTF8 (RFC8160) option is sent.

 * scp(1): Disable RemoteCommand and RequestTTY in the ssh session
   started by scp (sftp was already doing this.)

 * ssh-keygen(1): Refuse to create a certificate with an unusable
   number of principals.

 * ssh-keygen(1): Fatally exit if ssh-keygen is unable to write all the
   public key during key generation. Previously it would silently
   ignore errors writing the comment and terminating newline.

 * ssh(1): Do not modify hostname arguments that are addresses by
   automatically forcing them to lower-case. Instead canonicalise them
   to resolve ambiguities (e.g. ::0001 => ::1) before they are matched
   against known_hosts. bz#2763

 * ssh(1): Don't accept junk after "yes" or "no" responses to hostkey
   prompts. bz#2803

 * sftp(1): Have sftp print a warning about shell cleanliness when
   decoding the first packet fails, which is usually caused by shells
   polluting stdout of non-interactive startups. bz#2800

 * ssh(1)/sshd(8): Switch timers in packet code from using wall-clock
   time to monotonic time, allowing the packet layer to better function
   over a clock step and avoiding possible integer overflows during
   steps.

 * Numerous manual page fixes and improvements.

Portability
-----------

 * sshd(8): Correctly detect MIPS ABI in use at configure time. Fixes
   sandbox violations on some environments.

 * sshd(8): Remove UNICOS support. The hardware and software are literal
   museum pieces and support in sshd is too intrusive to justify
   maintaining.

 * All: Build and link with "retpoline" flags when available to mitigate
   the "branch target injection" style (variant 2) of the Spectre
   branch-prediction vulnerability.

 * All: Add auto-generated dependency information to Makefile.

 * Numerous fixed to the RPM spec files.

Checksums:
==========

- SHA1 (openssh-7.7.tar.gz) = 24812e05fa233014c847c7775748316e7f8a836c
- SHA256 (openssh-7.7.tar.gz) = T4ua1L/vgAYqwB0muRahvnm5ZUr3PLY9nPljaG8egvo=

- SHA1 (openssh-7.7p1.tar.gz) = 446fe9ed171f289f0d62197dffdbfdaaf21c49f2
- SHA256 (openssh-7.7p1.tar.gz) = 1zvn5oTpnvzQJL4Vowv/y+QbASsvezyQhK7WIXdea48=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available as RELEASE_KEY.asc from
the mirror sites.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH 7.6 / 7.6p1 (2017-10-03) 

OpenSSH 7.6 was released on 2017-10-03. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Potentially-incompatible changes
================================

This release includes a number of changes that may affect existing
configurations:

 * ssh(1): delete SSH protocol version 1 support, associated
   configuration options and documentation.

 * ssh(1)/sshd(8): remove support for the hmac-ripemd160 MAC.

 * ssh(1)/sshd(8): remove support for the arcfour, blowfish and CAST
   ciphers.

 * Refuse RSA keys <1024 bits in length and improve reporting for keys
   that do not meet this requirement.

 * ssh(1): do not offer CBC ciphers by default.

Changes since OpenSSH 7.5
=========================

This is primarily a bugfix release. It also contains substantial
internal refactoring.

Security
--------

 * sftp-server(8): in read-only mode, sftp-server was incorrectly
   permitting creation of zero-length files. Reported by Michal
   Zalewski.

New Features
------------

 * ssh(1): add RemoteCommand option to specify a command in the ssh
   config file instead of giving it on the client's command line. This
   allows the configuration file to specify the command that will be
   executed on the remote host.

 * sshd(8): add ExposeAuthInfo option that enables writing details of
   the authentication methods used (including public keys where
   applicable) to a file that is exposed via a $SSH_USER_AUTH
   environment variable in the subsequent session.

 * ssh(1): add support for reverse dynamic forwarding. In this mode,
   ssh will act as a SOCKS4/5 proxy and forward connections
   to destinations requested by the remote SOCKS client. This mode
   is requested using extended syntax for the -R and RemoteForward
   options and, because it is implemented solely at the client,
   does not require the server be updated to be supported.

 * sshd(8): allow LogLevel directive in sshd_config Match blocks;
   bz#2717

 * ssh-keygen(1): allow inclusion of arbitrary string or flag
   certificate extensions and critical options.

 * ssh-keygen(1): allow ssh-keygen to use a key held in ssh-agent as
   a CA when signing certificates. bz#2377

 * ssh(1)/sshd(8): allow IPQoS=none in ssh/sshd to not set an explicit
   ToS/DSCP value and just use the operating system default.

 * ssh-add(1): added -q option to make ssh-add quiet on success.

 * ssh(1): expand the StrictHostKeyChecking option with two new
   settings. The first "accept-new" will automatically accept
   hitherto-unseen keys but will refuse connections for changed or
   invalid hostkeys. This is a safer subset of the current behaviour
   of StrictHostKeyChecking=no. The second setting "off", is a synonym
   for the current behaviour of StrictHostKeyChecking=no: accept new
   host keys, and continue connection for hosts with incorrect
   hostkeys. A future release will change the meaning of
   StrictHostKeyChecking=no to the behaviour of "accept-new". bz#2400

 * ssh(1): add SyslogFacility option to ssh(1) matching the equivalent
   option in sshd(8). bz#2705

Bugfixes
--------

 * ssh(1): use HostKeyAlias if specified instead of hostname for
   matching host certificate principal names; bz#2728

 * sftp(1): implement sorting for globbed ls; bz#2649

 * ssh(1): add a user@host prefix to client's "Permission denied"
   messages, useful in particular when using "stacked" connections
   (e.g. ssh -J) where it's not clear which host is denying. bz#2720

 * ssh(1): accept unknown EXT_INFO extension values that contain \0
   characters. These are legal, but would previously cause fatal
   connection errors if received.

 * ssh(1)/sshd(8): repair compression statistics printed at
   connection exit

 * sftp(1): print '?' instead of incorrect link count (that the
   protocol doesn't provide) for remote listings. bz#2710

 * ssh(1): return failure rather than fatal() for more cases during
   session multiplexing negotiations. Causes the session to fall back
   to a non-mux connection if they occur. bz#2707

 * ssh(1): mention that the server may send debug messages to explain
   public key authentication problems under some circumstances; bz#2709

 * Translate OpenSSL error codes to better report incorrect passphrase
   errors when loading private keys; bz#2699

 * sshd(8): adjust compatibility patterns for WinSCP to correctly
   identify versions that implement only the legacy DH group exchange
   scheme. bz#2748

 * ssh(1): print the "Killed by signal 1" message only at LogLevel
   verbose so that it is not shown at the default level; prevents it
   from appearing during ssh -J and equivalent ProxyCommand configs.
   bz#1906, bz#2744

 * ssh-keygen(1): when generating all hostkeys (ssh-keygen -A), clobber
   existing keys if they exist but are zero length. zero-length keys
   could previously be made if ssh-keygen failed or was interrupted part
   way through generating them. bz#2561

 * ssh(1): fix pledge(2) violation in the escape sequence "~&" used to
   place the current session in the background.

 * ssh-keyscan(1): avoid double-close() on file descriptors; bz#2734

 * sshd(8): avoid reliance on shared use of pointers shared between
   monitor and child sshd processes. bz#2704

 * sshd_config(8): document available AuthenticationMethods; bz#2453

 * ssh(1): avoid truncation in some login prompts; bz#2768

 * sshd(8): Fix various compilations failures, inc bz#2767

 * ssh(1): make "--" before the hostname terminate argument processing
   after the hostname too.

 * ssh-keygen(1): switch from aes256-cbc to aes256-ctr for encrypting
   new-style private keys. Fixes problems related to private key
   handling for no-OpenSSL builds. bz#2754

 * ssh(1): warn and do not attempt to use keys when the public and
   private halves do not match. bz#2737

 * sftp(1): don't print verbose error message when ssh disconnects
   from under sftp. bz#2750

 * sshd(8): fix keepalive scheduling problem: activity on a forwarded
   port from preventing the keepalive from being sent; bz#2756

 * sshd(8): when started without root privileges, don't require the
   privilege separation user or path to exist. Makes running the
   regression tests easier without touching the filesystem.

 * Make integrity.sh regression tests more robust against timeouts.
   bz#2658

 * ssh(1)/sshd(8): correctness fix for channels implementation: accept
   channel IDs greater than 0x7FFFFFFF.

Portability
-----------

 * sshd(9): drop two more privileges in the Solaris sandbox:
   PRIV_DAX_ACCESS and PRIV_SYS_IB_INFO; bz#2723

 * sshd(8): expose list of completed authentication methods to PAM
   via the SSH_AUTH_INFO_0 PAM environment variable. bz#2408

 * ssh(1)/sshd(8): fix several problems in the tun/tap forwarding code,
   mostly to do with host/network byte order confusion. bz#2735

 * Add --with-cflags-after and --with-ldflags-after configure flags to
   allow setting CFLAGS/LDFLAGS after configure has completed. These
   are useful for setting sanitiser/fuzzing options that may interfere
   with configure's operation.

 * sshd(8): avoid Linux seccomp violations on ppc64le over the
   socketcall syscall.

 * Fix use of ldns when using ldns-config; bz#2697

 * configure: set cache variables when cross-compiling. The cross-
   compiling fallback message was saying it assumed the test passed,
   but it wasn't actually set the cache variables and this would
   cause later tests to fail.

 * Add clang libFuzzer harnesses for public key parsing and signature
   verification.

Checksums:
==========

 - SHA1 (openssh-7.6.tar.gz) = 157fe3989a245c58fcdb34d9fe722a3c4e14c008
 - SHA1 (openssh-7.6p1.tar.gz) = a6984bc2c72192bed015c8b879b35dd9f5350b3b

 - SHA256 (openssh-7.6.tar.gz) = Xu3bdpCcu65vM2FnW7b6IKLgd4Kvf2P3WBTMw+I7Bao=
 - SHA256 (openssh-7.6p1.tar.gz) = oyPK7t3+FFuqoNsW6Y14Sx+8fdQ2pr8fR539XNHSFyM=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available as RELEASE_KEY.asc from
the mirror sites.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de
Raadt, Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre,
Tim Rice and Ben Lindstrom.

OpenSSH 7.5 / 7.5p1 (2017-03-20) 

OpenSSH 7.5 was released on 2017-03-20. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support. OpenSSH also includes
transitional support for the legacy SSH 1.3 and 1.5 protocols
that may be enabled at compile-time.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Future deprecation notice
=========================

We plan on retiring more legacy cryptography in future releases,
specifically:

 * In the next major release (expected June-August), removing remaining
   support for the SSH v.1 protocol (currently client-only and compile-
   time disabled).

 * In the same release, removing support for Blowfish and RC4 ciphers
   and the RIPE-MD160 HMAC. (These are currently run-time disabled).

 * In the same release, removing the remaining CBC ciphers from being
   offered by default in the client (These have not been offered in
   sshd by default for several years).

 * Refusing all RSA keys smaller than 1024 bits (the current minimum
   is 768 bits)

This list reflects our current intentions, but please check the final
release notes for future releases.

Potentially-incompatible changes
================================

This release includes a number of changes that may affect existing
configurations:

 * This release deprecates the sshd_config UsePrivilegeSeparation
   option, thereby making privilege separation mandatory. Privilege
   separation has been on by default for almost 15 years and
   sandboxing has been on by default for almost the last five.

 * The format of several log messages emitted by the packet code has
   changed to include additional information about the user and
   their authentication state. Software that monitors ssh/sshd logs
   may need to account for these changes. For example:

   Connection closed by user x 1.1.1.1 port 1234 [preauth]
   Connection closed by authenticating user x 10.1.1.1 port 1234 [preauth]
   Connection closed by invalid user x 1.1.1.1 port 1234 [preauth]

   Affected messages include connection closure, timeout, remote
   disconnection, negotiation failure and some other fatal messages
   generated by the packet code.

 * [Portable OpenSSH only] This version removes support for building
   against OpenSSL versions prior to 1.0.1. OpenSSL stopped supporting
   versions prior to 1.0.1 over 12 months ago (i.e. they no longer
   receive fixes for security bugs).

Changes since OpenSSH 7.4
=========================

This is a bugfix release.

Security
--------

 * ssh(1), sshd(8): Fix weakness in CBC padding oracle countermeasures
   that allowed a variant of the attack fixed in OpenSSH 7.3 to proceed.
   Note that the OpenSSH client disables CBC ciphers by default, sshd
   offers them as lowest-preference options and will remove them by
   default entriely in the next release. Reported by Jean Paul
   Degabriele, Kenny Paterson, Martin Albrecht and Torben Hansen of
   Royal Holloway, University of London.

 * sftp-client(1): [portable OpenSSH only] On Cygwin, a client making
   a recursive file transfer could be manipulated by a hostile server to
   perform a path-traversal attack. creating or modifying files outside
   of the intended target directory. Reported by Jann Horn of Google
   Project Zero.

New Features
------------

 * ssh(1), sshd(8): Support "=-" syntax to easily remove methods from
   algorithm lists, e.g. Ciphers=-*cbc. bz#2671

Bugfixes
--------

 * sshd(1): Fix NULL dereference crash when key exchange start
   messages are sent out of sequence.

 * ssh(1), sshd(8): Allow form-feed characters to appear in
   configuration files.
 
 * sshd(8): Fix regression in OpenSSH 7.4 support for the
   server-sig-algs extension, where SHA2 RSA signature methods were
   not being correctly advertised. bz#2680

 * ssh(1), ssh-keygen(1): Fix a number of case-sensitivity bugs in
   known_hosts processing. bz#2591 bz#2685

 * ssh(1): Allow ssh to use certificates accompanied by a private key
   file but no corresponding plain *.pub public key. bz#2617

 * ssh(1): When updating hostkeys using the UpdateHostKeys option,
   accept RSA keys if HostkeyAlgorithms contains any RSA keytype.
   Previously, ssh could ignore RSA keys when only the ssh-rsa-sha2-*
   methods were enabled in HostkeyAlgorithms and not the old ssh-rsa
   method. bz#2650
    
 * ssh(1): Detect and report excessively long configuration file
   lines. bz#2651

 * Merge a number of fixes found by Coverity and reported via Redhat
   and FreeBSD. Includes fixes for some memory and file descriptor
   leaks in error paths. bz#2687
    
 * ssh-keyscan(1): Correctly hash hosts with a port number. bz#2692

 * ssh(1), sshd(8): When logging long messages to stderr, don't truncate
   "\r\n" if the length of the message exceeds the buffer. bz#2688

 * ssh(1): Fully quote [host]:port in generated ProxyJump/-J command-
   line; avoid confusion over IPv6 addresses and shells that treat
   square bracket characters specially.
    
 * ssh-keygen(1): Fix corruption of known_hosts when running
   "ssh-keygen -H" on a known_hosts containing already-hashed entries.

 * Fix various fallout and sharp edges caused by removing SSH protocol
   1 support from the server, including the server banner string being
   incorrectly terminated with only \n (instead of \r\n), confusing
   error messages from ssh-keyscan bz#2583 and a segfault in sshd
   if protocol v.1 was enabled for the client and sshd_config
   contained references to legacy keys bz#2686.

 * ssh(1), sshd(8): Free fd_set on connection timeout. bz#2683

 * sshd(8): Fix Unix domain socket forwarding for root (regression in
   OpenSSH 7.4).
    
 * sftp(1): Fix division by zero crash in "df" output when server
   returns zero total filesystem blocks/inodes.
 
 * ssh(1), ssh-add(1), ssh-keygen(1), sshd(8): Translate OpenSSL errors
   encountered during key loading to more meaningful error codes.
   bz#2522 bz#2523

 * ssh-keygen(1): Sanitise escape sequences in key comments sent to
   printf but preserve valid UTF-8 when the locale supports it;
   bz#2520

 * ssh(1), sshd(8): Return reason for port forwarding failures where
   feasible rather than always "administratively prohibited". bz#2674

 * sshd(8): Fix deadlock when AuthorizedKeysCommand or
   AuthorizedPrincipalsCommand produces a lot of output and a key is
   matched early. bz#2655

 * Regression tests: several reliability fixes. bz#2654 bz#2658 bz#2659
    
 * ssh(1): Fix typo in ~C error message for bad port forward
   cancellation. bz#2672

 * ssh(1): Show a useful error message when included config files
   can't be opened; bz#2653

 * sshd(8): Make sshd set GSSAPIStrictAcceptorCheck=yes as the manual page
   (previously incorrectly) advertised. bz#2637

 * sshd_config(5): Repair accidentally-deleted mention of %k token
   in AuthorizedKeysCommand; bz#2656

 * sshd(8): Remove vestiges of previously removed LOGIN_PROGRAM; bz#2665

 * ssh-agent(1): Relax PKCS#11 whitelist to include libexec and
   common 32-bit compatibility library directories.

 * sftp-client(1): Fix non-exploitable integer overflow in SSH2_FXP_NAME
   response handling.

 * ssh-agent(1): Fix regression in 7.4 of deleting PKCS#11-hosted
   keys. It was not possible to delete them except by specifying
   their full physical path. bz#2682

Portability
-----------

 * sshd(8): Avoid sandbox errors for Linux S390 systems using an ICA
   crypto coprocessor.

 * sshd(8): Fix non-exploitable weakness in seccomp-bpf sandbox arg
   inspection.

 * ssh(1): Fix X11 forwarding on OSX where X11 was being started by
   launchd. bz#2341

 * ssh-keygen(1), ssh(1), sftp(1): Fix output truncation for various that
   contain non-printable characters where the codeset in use is ASCII.

 * build: Fix builds that attempt to link a kerberised libldns. bz#2603

 * build: Fix compilation problems caused by unconditionally defining
   _XOPEN_SOURCE in wide character detection.

 * sshd(8): Fix sandbox violations for clock_gettime VSDO syscall
   fallback on some Linux/X32 kernels. bz#2142

Checksums:
==========

 - SHA1 (openssh-7.5.tar.gz) = 81384df377e38551f7659a4c250383d0bbd25341
 - SHA1 (openssh-7.5p1.tar.gz) = 5e8f185d00afb4f4f89801e9b0f8b9cee9d87ebd

 - SHA256 (openssh-7.5.tar.gz) = Gmk8jOdGdKa7NixUN5J+bTMfeum5Vx8Nv+leAdQNq3U=
 - SHA256 (openssh-7.5p1.tar.gz) = mEbjxfq58FR0ALTSwBeZL5FCIrP9H47ubH3GvF5Z+fA=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available as RELEASE_KEY.asc from
the mirror sites.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de
Raadt, Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre,
Tim Rice and Ben Lindstrom.

OpenSSH 7.4 / 7.4p1 (2016-12-19) 

OpenSSH 7.4 was released on 2016-12-19. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support. OpenSSH also includes
transitional support for the legacy SSH 1.3 and 1.5 protocols
that may be enabled at compile-time.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Future deprecation notice
=========================

We plan on retiring more legacy cryptography in future releases,
specifically:

 * In approximately August 2017, removing remaining support for the
   SSH v.1 protocol (client-only and currently compile-time disabled).

 * In the same release, removing support for Blowfish and RC4 ciphers
   and the RIPE-MD160 HMAC. (These are currently run-time disabled).

 * Refusing all RSA keys smaller than 1024 bits (the current minimum
   is 768 bits)

 * The next release of OpenSSH will remove support for running sshd(8)
   with privilege separation disabled.

 * The next release of portable OpenSSH will remove support for
   OpenSSL version prior to 1.0.1.

This list reflects our current intentions, but please check the final
release notes for future releases.

Potentially-incompatible changes
================================

This release includes a number of changes that may affect existing
configurations:

 * This release removes server support for the SSH v.1 protocol.

 * ssh(1): Remove 3des-cbc from the client's default proposal. 64-bit
   block ciphers are not safe in 2016 and we don't want to wait until
   attacks like SWEET32 are extended to SSH. As 3des-cbc was the
   only mandatory cipher in the SSH RFCs, this may cause problems
   connecting to older devices using the default configuration,
   but it's highly likely that such devices already need explicit
   configuration for key exchange and hostkey algorithms already
   anyway.
    
 * sshd(8): Remove support for pre-authentication compression.
   Doing compression early in the protocol probably seemed reasonable
   in the 1990s, but today it's clearly a bad idea in terms of both
   cryptography (cf. multiple compression oracle attacks in TLS) and
   attack surface. Pre-auth compression support has been disabled by
   default for >10 years. Support remains in the client.
    
 * ssh-agent will refuse to load PKCS#11 modules outside a whitelist
   of trusted paths by default. The path whitelist may be specified
   at run-time.

 * sshd(8): When a forced-command appears in both a certificate and
   an authorized keys/principals command= restriction, sshd will now
   refuse to accept the certificate unless they are identical.
   The previous (documented) behaviour of having the certificate
   forced-command override the other could be a bit confusing and
   error-prone.
    
 * sshd(8): Remove the UseLogin configuration directive and support
   for having /bin/login manage login sessions.
    
Changes since OpenSSH 7.3
=========================

This is primarily a bugfix release.

Security
--------

 * ssh-agent(1): Will now refuse to load PKCS#11 modules from paths
   outside a trusted whitelist (run-time configurable). Requests to
   load modules could be passed via agent forwarding and an attacker
   could attempt to load a hostile PKCS#11 module across the forwarded
   agent channel: PKCS#11 modules are shared libraries, so this would
   result in code execution on the system running the ssh-agent if the
   attacker has control of the forwarded agent-socket (on the host
   running the sshd server) and the ability to write to the filesystem
   of the host running ssh-agent (usually the host running the ssh
   client). Reported by Jann Horn of Project Zero.

 * sshd(8): When privilege separation is disabled, forwarded Unix-
   domain sockets would be created by sshd(8) with the privileges of
   'root' instead of the authenticated user. This release refuses
   Unix-domain socket forwarding when privilege separation is disabled
   (Privilege separation has been enabled by default for 14 years).
   Reported by Jann Horn of Project Zero.

 * sshd(8): Avoid theoretical leak of host private key material to
   privilege-separated child processes via realloc() when reading
   keys. No such leak was observed in practice for normal-sized keys,
   nor does a leak to the child processes directly expose key material
   to unprivileged users. Reported by Jann Horn of Project Zero.
 
 * sshd(8): The shared memory manager used by pre-authentication
   compression support had a bounds checks that could be elided by
   some optimising compilers. Additionally, this memory manager was
   incorrectly accessible when pre-authentication compression was
   disabled. This could potentially allow attacks against the
   privileged monitor process from the sandboxed privilege-separation
   process (a compromise of the latter would be required first).
   This release removes support for pre-authentication compression
   from sshd(8). Reported by Guido Vranken using the Stack unstable
   optimisation identification tool (http://css.csail.mit.edu/stack/)
 * sshd(8): Fix denial-of-service condition where an attacker who
   sends multiple KEXINIT messages may consume up to 128MB per
   connection. Reported by Shi Lei of Gear Team, Qihoo 360.

 * sshd(8): Validate address ranges for AllowUser and DenyUsers
   directives at configuration load time and refuse to accept invalid
   ones. It was previously possible to specify invalid CIDR address
   ranges (e.g. user@127.1.2.3/55) and these would always match,
   possibly resulting in granting access where it was not intended.
   Reported by Laurence Parry.

New Features
------------

 * ssh(1): Add a proxy multiplexing mode to ssh(1) inspired by the
   version in PuTTY by Simon Tatham. This allows a multiplexing
   client to communicate with the master process using a subset of
   the SSH packet and channels protocol over a Unix-domain socket,
   with the main process acting as a proxy that translates channel
   IDs, etc.  This allows multiplexing mode to run on systems that
   lack file- descriptor passing (used by current multiplexing
   code) and potentially, in conjunction with Unix-domain socket
   forwarding, with the client and multiplexing master process on
   different machines. Multiplexing proxy mode may be invoked using
   "ssh -O proxy ..."

 * sshd(8): Add a sshd_config DisableForwarding option that disables
   X11, agent, TCP, tunnel and Unix domain socket forwarding, as well
   as anything else we might implement in the future. Like the
   'restrict' authorized_keys flag, this is intended to be a simple
   and future-proof way of restricting an account.

 * sshd(8), ssh(1): Support the "curve25519-sha256" key exchange
   method. This is identical to the currently-supported method named
   "curve25519-sha256@libssh.org".

 * sshd(8): Improve handling of SIGHUP by checking to see if sshd is
   already daemonised at startup and skipping the call to daemon(3)
   if it is. This ensures that a SIGHUP restart of sshd(8) will
   retain the same process-ID as the initial execution. sshd(8) will
   also now unlink the PidFile prior to SIGHUP restart and re-create
   it after a successful restart, rather than leaving a stale file in
   the case of a configuration error. bz#2641

 * sshd(8): Allow ClientAliveInterval and ClientAliveCountMax
   directives to appear in sshd_config Match blocks.

 * sshd(8): Add %-escapes to AuthorizedPrincipalsCommand to match
   those supported by AuthorizedKeysCommand (key, key type,
   fingerprint, etc.) and a few more to provide access to the
   contents of the certificate being offered.

 * Added regression tests for string matching, address matching and
   string sanitisation functions.

 * Improved the key exchange fuzzer harness.
 
Bugfixes
--------

 * ssh(1): Allow IdentityFile to successfully load and use
   certificates that have no corresponding bare public key. bz#2617
   certificate id_rsa-cert.pub (and no id_rsa.pub).

 * ssh(1): Fix public key authentication when multiple
   authentication is in use and publickey is not just the first
   method attempted. bz#2642

 * regress: Allow the PuTTY interop tests to run unattended. bz#2639
 
 * ssh-agent(1), ssh(1): improve reporting when attempting to load
   keys from PKCS#11 tokens with fewer useless log messages and more
   detail in debug messages. bz#2610

 * ssh(1): When tearing down ControlMaster connections, don't
   pollute stderr when LogLevel=quiet.

 * sftp(1): On ^Z wait for underlying ssh(1) to suspend before
   suspending sftp(1) to ensure that ssh(1) restores the terminal mode
   correctly if suspended during a password prompt.

 * ssh(1): Avoid busy-wait when ssh(1) is suspended during a password
   prompt.

 * ssh(1), sshd(8): Correctly report errors during sending of ext-
   info messages.

 * sshd(8): fix NULL-deref crash if sshd(8) received an out-of-
   sequence NEWKEYS message.

 * sshd(8): Correct list of supported signature algorithms sent in
   the server-sig-algs extension. bz#2547

 * sshd(8): Fix sending ext_info message if privsep is disabled.

 * sshd(8): more strictly enforce the expected ordering of privilege
   separation monitor calls used for authentication and allow them
   only when their respective authentication methods are enabled
   in the configuration

 * sshd(8): Fix uninitialised optlen in getsockopt() call; harmless
   on Unix/BSD but potentially crashy on Cygwin.

 * Fix false positive reports caused by explicit_bzero(3) not being
   recognised as a memory initialiser when compiled with
   -fsanitize-memory.
    
 * sshd_config(5): Use 2001:db8::/32, the official IPv6 subnet for
   configuration examples.

Portability
-----------

 * On environments configured with Turkish locales, fall back to the
   C/POSIX locale to avoid errors in configuration parsing caused by
   that locale's unique handling of the letters 'i' and 'I'. bz#2643

 * sftp-server(8), ssh-agent(1): Deny ptrace on OS X using
   ptrace(PT_DENY_ATTACH, ..)

 * ssh(1), sshd(8): Unbreak AES-CTR ciphers on old (~0.9.8) OpenSSL.

 * Fix compilation for libcrypto compiled without RIPEMD160 support.

 * contrib: Add a gnome-ssh-askpass3 with GTK+3 support. bz#2640
    
 * sshd(8): Improve PRNG reseeding across privilege separation and
   force libcrypto to obtain a high-quality seed before chroot or
   sandboxing.

 * All: Explicitly test for broken strnvis. NetBSD added an strnvis
   and unfortunately made it incompatible with the existing one in
   OpenBSD and Linux's libbsd (the former having existed for over ten
   years). Try to detect this mess, and assume the only safe option
   if we're cross compiling.

Checksums:
==========

 - SHA1 (openssh-7.4.tar.gz) = 1e2073f95d5ead8f2814b4b6c0700bcd533c410f
 - SHA1 (openssh-7.4p1.tar.gz) = 2330bbf82ed08cf3ac70e0acf00186ef3eeb97e0

 - SHA256 (openssh-7.4.tar.gz) = +GEXh7Xr2J87cq1uA97hF9e+3lfOQ2LKxXGdmFXREf0
 - SHA256 (openssh-7.4p1.tar.gz) = Gx/EoU4gJCkxgZJO0khy5vLgYpPz6JJqN2uK7EgfGdE=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available as RELEASE_KEY.asc from
the mirror sites.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de
Raadt, Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre,
Tim Rice and Ben Lindstrom.

OpenSSH 7.3 / 7.3p1 (2016-08-01) 

OpenSSH 7.3 was released on 2016-08-01. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support. OpenSSH also includes
transitional support for the legacy SSH 1.3 and 1.5 protocols
that may be enabled at compile-time.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Future deprecation notice
=========================

We plan on retiring more legacy cryptography in a near-future
release, specifically:

 * Refusing all RSA keys smaller than 1024 bits (the current minimum
   is 768 bits)
 * Removing server-side support for the SSH v.1 protocol (currently
   compile-time disabled).
 * In approximately 1 year, removing all support for the SSH v.1
   protocol (currently compile-time disabled).

This list reflects our current intentions, but please check the final
release notes for future releases.

Changes since OpenSSH 7.2
=========================

This is primarily a bugfix release.

Security
--------

 * sshd(8): Mitigate a potential denial-of-service attack against
   the system's crypt(3) function via sshd(8). An attacker could
   send very long passwords that would cause excessive CPU use in
   crypt(3). sshd(8) now refuses to accept password authentication
   requests of length greater than 1024 characters. Independently
   reported by Tomas Kuthan (Oracle), Andres Rojas and Javier Nieto.

 * sshd(8): Mitigate timing differences in password authentication
   that could be used to discern valid from invalid account names
   when long passwords were sent and particular password hashing
   algorithms are in use on the server. CVE-2016-6210, reported by
   EddieEzra.Harari at verint.com

 * ssh(1), sshd(8): Fix observable timing weakness in the CBC padding
   oracle countermeasures. Reported by Jean Paul Degabriele, Kenny
   Paterson, Torben Hansen and Martin Albrecht. Note that CBC ciphers
   are disabled by default and only included for legacy compatibility.

 * ssh(1), sshd(8): Improve operation ordering of MAC verification for
   Encrypt-then-MAC (EtM) mode transport MAC algorithms to verify the
   MAC before decrypting any ciphertext. This removes the possibility
   of timing differences leaking facts about the plaintext, though no
   such leakage has been observed.  Reported by Jean Paul Degabriele,
   Kenny Paterson, Torben Hansen and Martin Albrecht.
    
 * sshd(8): (portable only) Ignore PAM environment vars when
   UseLogin=yes. If PAM is configured to read user-specified
   environment variables and UseLogin=yes in sshd_config, then a
   hostile local user may attack /bin/login via LD_PRELOAD or
   similar environment variables set via PAM. CVE-2015-8325,
   found by Shayan Sadigh.

New Features
------------

 * ssh(1): Add a ProxyJump option and corresponding -J command-line
   flag to allow simplified indirection through a one or more SSH
   bastions or "jump hosts".

 * ssh(1): Add an IdentityAgent option to allow specifying specific
   agent sockets instead of accepting one from the environment.
    
 * ssh(1): Allow ExitOnForwardFailure and ClearAllForwardings to be
   optionally overridden when using ssh -W. bz#2577

 * ssh(1), sshd(8): Implement support for the IUTF8 terminal mode as
   per draft-sgtatham-secsh-iutf8-00.
    
 * ssh(1), sshd(8): Add support for additional fixed Diffie-Hellman
   2K, 4K and 8K groups from draft-ietf-curdle-ssh-kex-sha2-03.

 * ssh-keygen(1), ssh(1), sshd(8): support SHA256 and SHA512 RSA
   signatures in certificates;
    
 * ssh(1): Add an Include directive for ssh_config(5) files.

 * ssh(1): Permit UTF-8 characters in pre-authentication banners sent
   from the server. bz#2058

Bugfixes
--------

 * ssh(1), sshd(8): Reduce the syslog level of some relatively common
   protocol events from LOG_CRIT. bz#2585

 * sshd(8): Refuse AuthenticationMethods="" in configurations and
   accept AuthenticationMethods=any for the default behaviour of not
   requiring multiple authentication. bz#2398

 * sshd(8): Remove obsolete and misleading "POSSIBLE BREAK-IN
   ATTEMPT!" message when forward and reverse DNS don't match. bz#2585

 * ssh(1): Close ControlPersist background process stderr except
   in debug mode or when logging to syslog. bz#1988

 * misc: Make PROTOCOL description for direct-streamlocal@openssh.com
   channel open messages match deployed code. bz#2529

 * ssh(1): Deduplicate LocalForward and RemoteForward entries to fix
   failures when both ExitOnForwardFailure and hostname
   canonicalisation are enabled. bz#2562

 * sshd(8): Remove fallback from moduli to obsolete "primes" file
   that was deprecated in 2001. bz#2559.

 * sshd_config(5): Correct description of UseDNS: it affects ssh
   hostname processing for authorized_keys, not known_hosts; bz#2554
    
 * ssh(1): Fix authentication using lone certificate keys in an agent
   without corresponding private keys on the filesystem. bz#2550

 * sshd(8): Send ClientAliveInterval pings when a time-based
   RekeyLimit is set; previously keepalive packets were not being
   sent. bz#2252
    
Portability
-----------

 * ssh(1), sshd(8): Fix compilation by automatically disabling ciphers
   not supported by OpenSSL. bz#2466

 * misc: Fix compilation failures on some versions of AIX's compiler
   related to the definition of the VA_COPY macro. bz#2589

 * sshd(8): Whitelist more architectures to enable the seccomp-bpf
   sandbox. bz#2590

 * ssh-agent(1), sftp-server(8): Disable process tracing on Solaris
   using setpflags(__PROC_PROTECT, ...). bz#2584

 * sshd(8): On Solaris, don't call Solaris setproject() with
   UsePAM=yes it's PAM's responsibility. bz#2425

Checksums:
==========

 - SHA1 (openssh-7.3.tar.gz) = b1641e5265d9ec68a9a19decc3a7edd1203cbd33
 - SHA256 (openssh-7.3.tar.gz) = vS0X35qrX9OOPBkyDMYhOje/DBwHBVEV7nv5rkzw4vM=

 - SHA1 (openssh-7.3p1.tar.gz) = bfade84283fcba885e2084343ab19a08c7d123a5
 - SHA256 (openssh-7.3p1.tar.gz) = P/uYmm3KppWUw7VQ1IVaWi4XGMzd5/XjY4e0JCIPvsw=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available as RELEASE_KEY.asc from
the mirror sites.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de
Raadt, Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre,
Tim Rice and Ben Lindstrom.

OpenSSH 7.2p2 (2016-03-10) 

Portable OpenSSH 7.2p2 was released on 2016-03-10. It will be available
from the mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support. OpenSSH also includes
transitional support for the legacy SSH 1.3 and 1.5 protocols that
may be enabled at compile-time.

Once again, we would like to thank the OpenSSH community for
their continued support of the project, especially those who
contributed code or patches, reported bugs, tested snapshots or
donated to the project. More information on donations may be found
at: http://www.openssh.com/donations.html

Changes since OpenSSH 7.2p1
===========================

This release fixes a security bug:

 * sshd(8): sanitise X11 authentication credentials to avoid xauth
   command injection when X11Forwarding is enabled.

   Full details of the vulnerability are available at:
   http://www.openssh.com/txt/x11fwd.adv

Checksums:
==========

 - SHA1 (openssh-7.2p2.tar.gz) = 70e35d7d6386fe08abbd823b3a12a3ca44ac6d38
 - SHA256 (openssh-7.2p2.tar.gz) = pyeB0aBDh2oiT/GwAy2qQJTYdWWmhSh1nBwsq1SCVIw=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available as RELEASE_KEY.asc from
the mirror sites.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de
Raadt, Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre,
Tim Rice and Ben Lindstrom.

OpenSSH 7.2 / 7.2p1 (2016-02-29) 

OpenSSH 7.2 was released on 2016-02-29. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support. OpenSSH also includes
transitional support for the legacy SSH 1.3 and 1.5 protocols
that may be enabled at compile-time.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Future deprecation notice
=========================

We plan on retiring more legacy cryptography in a near-future
release, specifically:

 * Refusing all RSA keys smaller than 1024 bits (the current minimum
   is 768 bits)

This list reflects our current intentions, but please check the final
release notes for future releases.

Potentially-incompatible changes
================================

This release disables a number of legacy cryptographic algorithms
by default in ssh:

 * Several ciphers blowfish-cbc, cast128-cbc, all arcfour variants
   and the rijndael-cbc aliases for AES.

 * MD5-based and truncated HMAC algorithms.

These algorithms are already disabled by default in sshd.

Changes since OpenSSH 7.1p2
===========================

This is primarily a bugfix release.

Security
--------

 * ssh(1), sshd(8): remove unfinished and unused roaming code (was
   already forcibly disabled in OpenSSH 7.1p2).
 
 * ssh(1): eliminate fallback from untrusted X11 forwarding to
   trusted forwarding when the X server disables the SECURITY
   extension.

 * ssh(1), sshd(8): increase the minimum modulus size supported for
   diffie-hellman-group-exchange to 2048 bits.

 * sshd(8): pre-auth sandboxing is now enabled by default (previous
   releases enabled it for new installations via sshd_config).

New Features
------------

 * all: add support for RSA signatures using SHA-256/512 hash
   algorithms based on draft-rsa-dsa-sha2-256-03.txt and
   draft-ssh-ext-info-04.txt.

 * ssh(1): Add an AddKeysToAgent client option which can be set to
   'yes', 'no', 'ask', or 'confirm', and defaults to 'no'.  When
   enabled, a private key that is used during authentication will be
   added to ssh-agent if it is running (with confirmation enabled if
   set to 'confirm').
 
 * sshd(8): add a new authorized_keys option "restrict" that includes
   all current and future key restrictions (no-*-forwarding, etc.).
   Also add permissive versions of the existing restrictions, e.g.
   "no-pty" -> "pty". This simplifies the task of setting up
   restricted keys and ensures they are maximally-restricted,
   regardless of any permissions we might implement in the future.
    
 * ssh(1): add ssh_config CertificateFile option to explicitly list
   certificates. bz#2436
 
 * ssh-keygen(1): allow ssh-keygen to change the key comment for all
   supported formats.

 * ssh-keygen(1): allow fingerprinting from standard input, e.g.
   "ssh-keygen -lf -"

 * ssh-keygen(1): allow fingerprinting multiple public keys in a
   file, e.g. "ssh-keygen -lf ~/.ssh/authorized_keys" bz#1319

 * sshd(8): support "none" as an argument for sshd_config
   Foreground and ChrootDirectory. Useful inside Match blocks to
   override a global default. bz#2486

 * ssh-keygen(1): support multiple certificates (one per line) and
   reading from standard input (using "-f -") for "ssh-keygen -L"
    
 * ssh-keyscan(1): add "ssh-keyscan -c ..." flag to allow fetching
   certificates instead of plain keys.
 
 * ssh(1): better handle anchored FQDNs (e.g. 'cvs.openbsd.org.') in
   hostname canonicalisation - treat them as already canonical and
   remove the trailing '.' before matching ssh_config.

Bugfixes
--------

 * sftp(1): existing destination directories should not terminate
   recursive uploads (regression in openssh 6.8) bz#2528

 * ssh(1), sshd(8): correctly send back SSH2_MSG_UNIMPLEMENTED
   replies to unexpected messages during key exchange. bz#2949

 * ssh(1): refuse attempts to set ConnectionAttempts=0, which does
   not make sense and would cause ssh to print an uninitialised stack
   variable. bz#2500

 * ssh(1): fix errors when attempting to connect to scoped IPv6
   addresses with hostname canonicalisation enabled.

 * sshd_config(5): list a couple more options usable in Match blocks.
   bz#2489

 * sshd(8): fix "PubkeyAcceptedKeyTypes +..." inside a Match block.
    
 * ssh(1): expand tilde characters in filenames passed to -i options
   before checking whether or not the identity file exists. Avoids
   confusion for cases where shell doesn't expand (e.g. "-i ~/file"
   vs. "-i~/file"). bz#2481

 * ssh(1): do not prepend "exec" to the shell command run by "Match
   exec" in a config file, which could cause some commands to fail
   in certain environments. bz#2471

 * ssh-keyscan(1): fix output for multiple hosts/addrs on one line
   when host hashing or a non standard port is in use bz#2479
 
 * sshd(8): skip "Could not chdir to home directory" message when
   ChrootDirectory is active. bz#2485

 * ssh(1): include PubkeyAcceptedKeyTypes in ssh -G config dump.
    
 * sshd(8): avoid changing TunnelForwarding device flags if they are
   already what is needed; makes it possible to use tun/tap
   networking as non-root user if device permissions and interface
   flags are pre-established

 * ssh(1), sshd(8): RekeyLimits could be exceeded by one packet.
   bz#2521

 * ssh(1): fix multiplexing master failure to notice client exit.

 * ssh(1), ssh-agent(1): avoid fatal() for PKCS11 tokens that present
   empty key IDs. bz#1773

 * sshd(8): avoid printf of NULL argument. bz#2535  

 * ssh(1), sshd(8): allow RekeyLimits larger than 4GB. bz#2521
 
 * ssh-keygen(1): sshd(8): fix several bugs in (unused) KRL signature
   support.

 * ssh(1), sshd(8): fix connections with peers that use the key
   exchange guess feature of the protocol. bz#2515

 * sshd(8): include remote port number in log messages. bz#2503

 * ssh(1): don't try to load SSHv1 private key when compiled without
   SSHv1 support. bz#2505

 * ssh-agent(1), ssh(1): fix incorrect error messages during key
   loading and signing errors. bz#2507

 * ssh-keygen(1): don't leave empty temporary files when performing
   known_hosts file edits when known_hosts doesn't exist.

 * sshd(8): correct packet format for tcpip-forward replies for
   requests that don't allocate a port bz#2509

 * ssh(1), sshd(8): fix possible hang on closed output. bz#2469
    
 * ssh(1): expand %i in ControlPath to UID. bz#2449

 * ssh(1), sshd(8): fix return type of openssh_RSA_verify. bz#2460
 
 * ssh(1), sshd(8): fix some option parsing memory leaks. bz#2182

 * ssh(1): add a some debug output before DNS resolution; it's a
   place where ssh could previously silently stall in cases of
   unresponsive DNS servers. bz#2433
    
 * ssh(1): remove spurious newline in visual hostkey. bz#2686
 
 * ssh(1): fix printing (ssh -G ...) of HostKeyAlgorithms=+...
 
 * ssh(1): fix expansion of HostkeyAlgorithms=+...

Documentation
-------------

 * ssh_config(5), sshd_config(5): update default algorithm lists to
   match current reality. bz#2527

 * ssh(1): mention -Q key-plain and -Q key-cert query options.
   bz#2455

 * sshd_config(8): more clearly describe what AuthorizedKeysFile=none
   does.

 * ssh_config(5): better document ExitOnForwardFailure. bz#2444  

 * sshd(5): mention internal DH-GEX fallback groups in manual.
   bz#2302

 * sshd_config(5): better description for MaxSessions option.
   bz#2531

Portability
-----------

 * ssh(1), sftp-server(8), ssh-agent(1), sshd(8): Support Illumos/
   Solaris fine-grained privileges. Including a pre-auth privsep
   sandbox and several pledge() emulations. bz#2511

 * Renovate redhat/openssh.spec, removing deprecated options and
   syntax.

 * configure: allow --without-ssl-engine with --without-openssl
 
 * sshd(8): fix multiple authentication using S/Key. bz#2502

 * sshd(8): read back from libcrypto RAND_* before dropping
   privileges.  Avoids sandboxing violations with BoringSSL.

 * Fix name collision with system-provided glob(3) functions.
   bz#2463

 * Adapt Makefile to use ssh-keygen -A when generating host keys.
   bz#2459
 
 * configure: correct default value for --with-ssh1 bz#2457

 * configure: better detection of _res symbol bz#2259

 * support getrandom() syscall on Linux

Checksums:
==========

 - SHA1 (openssh-7.2.tar.gz) = 9567d00fffe655010c087aeb80c830cecbbecca6
 - SHA256 (openssh-7.2.tar.gz) = 99GsHA8NwSGuEJhMc7hAOQ510y1xfGx27uJqyw73sCI=

 - SHA1 (openssh-7.2p1.tar.gz) = d30a6fd472199ab5838a7668c0c5fd885fb8d371
 - SHA256 (openssh-7.2p1.tar.gz) = lzzDey81l+TPWZsJ5gTnnA/l2bb1laJOke0GYoYLSsM=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available as RELEASE_KEY.asc from
the mirror sites.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de
Raadt, Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre,
Tim Rice and Ben Lindstrom.

OpenSSH 7.1p2 (2016-01-14) 

OpenSSH 7.1p2 was released on 2016-01-14. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support. OpenSSH also includes
transitional support for the legacy SSH 1.3 and 1.5 protocols
that may be enabled at compile-time.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Changes since OpenSSH 7.1p1
===========================

 * SECURITY: ssh(1): The OpenSSH client code between 5.4 and 7.1
   contains experimental support for resuming SSH-connections (roaming).

   The matching server code has never been shipped, but the client
   code was enabled by default and could be tricked by a malicious
   server into leaking client memory to the server, including private
   client user keys.

   The authentication of the server host key prevents exploitation
   by a man-in-the-middle, so this information leak is restricted
   to connections to malicious or compromised servers.

   MITIGATION: For OpenSSH >= 5.4 the vulnerable code in the client
   can be completely disabled by adding 'UseRoaming no' to the global
   ssh_config(5) file, or to user configuration in ~/.ssh/config,
   or by passing -oUseRoaming=no on the command line.

   PATCH: See below for a patch to disable this feature (Disabling
   Roaming in the Source Code).

   This problem was reported by the Qualys Security Advisory team.

 * SECURITY: Fix an out of-bound read access in the packet handling
   code. Reported by Ben Hawkes.

 * PROTOCOL: Correctly interpret the 'first_kex_follows' option during
   the initial key exchange. Reported by Matt Johnston.

 * Further use of explicit_bzero has been added in various buffer
   handling code paths to guard against compilers aggressively
   doing dead-store removal.


Checksums:
==========

 - SHA1 (openssh-7.1p2.tar.gz) = 9202f5a2a50c8a55ecfb830609df1e1fde97f758
 - SHA256 (openssh-7.1p2.tar.gz) = dd75f024dcf21e06a0d6421d582690bf987a1f6323e32ad6619392f3bfde6bbd

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available as RELEASE_KEY.asc from
the mirror sites.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

Disabling Roaming in the Source Code:
=====================================

--- readconf.c	30 Jul 2015 00:01:34 -0000	1.239
+++ readconf.c	13 Jan 2016 23:17:23 -0000
@@ -1648,7 +1648,7 @@ initialize_options(Options * options)
 	options->tun_remote = -1;
 	options->local_command = NULL;
 	options->permit_local_command = -1;
-	options->use_roaming = -1;
+	options->use_roaming = 0;
 	options->visual_host_key = -1;
 	options->ip_qos_interactive = -1;
 	options->ip_qos_bulk = -1;
@@ -1819,8 +1819,7 @@ fill_default_options(Options * options)
 		options->tun_remote = SSH_TUNID_ANY;
 	if (options->permit_local_command == -1)
 		options->permit_local_command = 0;
-	if (options->use_roaming == -1)
-		options->use_roaming = 1;
+	options->use_roaming = 0;
 	if (options->visual_host_key == -1)
 		options->visual_host_key = 0;
 	if (options->ip_qos_interactive == -1)
--- ssh.c	30 Jul 2015 00:01:34 -0000	1.420
+++ ssh.c	13 Jan 2016 23:17:23 -0000
@@ -1882,9 +1882,6 @@ ssh_session2(void)
 			fork_postauth();
 	}
 
-	if (options.use_roaming)
-		request_roaming();
-
 	return client_loop(tty_flag, tty_flag ?
 	    options.escape_char : SSH_ESCAPECHAR_NONE, id);
 }

OpenSSH 7.1 / 7.1p1 (2015-08-21) 

OpenSSH 7.1 was released on 2015-08-21. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support. OpenSSH also includes
transitional support for the legacy SSH 1.3 and 1.5 protocols
that may be enabled at compile-time.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Future deprecation notice
=========================

We plan on retiring more legacy cryptography in the next release
including:

 * Refusing all RSA keys smaller than 1024 bits (the current minimum
   is 768 bits)

 * Several ciphers will be disabled by default: blowfish-cbc,
   cast128-cbc, all arcfour variants and the rijndael-cbc aliases
   for AES.

 * MD5-based HMAC algorithms will be disabled by default.

This list reflects our current intentions, but please check the final
release notes for OpenSSH 7.2 when it is released.

Changes since OpenSSH 7.0
=========================

This is a bugfix release.

Security
--------

 * sshd(8): OpenSSH 7.0 contained a logic error in PermitRootLogin=
   prohibit-password/without-password that could, depending on
   compile-time configuration, permit password authentication to
   root while preventing other forms of authentication. This problem
   was reported by Mantas Mikulenas.

Bugfixes
--------

 * ssh(1), sshd(8): add compatibility workarounds for FuTTY

 * ssh(1), sshd(8): refine compatibility workarounds for WinSCP

 * Fix a number of memory faults (double-free, free of uninitialised
   memory, etc) in ssh(1) and ssh-keygen(1). Reported by Mateusz
   Kocielski.

Checksums:
==========

 - SHA1 (openssh-7.1.tar.gz) = 06c1db39f33831fe004726e013b2cf84f1889042
 - SHA256 (openssh-7.1.tar.gz) = H7U1se9EoBmhkKi2i7lqpMX9QHdDTsgpu7kd5VZUGSY=

 - SHA1 (openssh-7.1p1.tar.gz) = ed22af19f962262c493fcc6ed8c8826b2761d9b6
 - SHA256 (openssh-7.1p1.tar.gz) = /AptLR0GPVxm3/2VJJPQzaJWytIE9oHeD4TvhbKthCg=

Please note that the SHA256 signatures are base64 encoded and not
hexadecimal (which is the default for most checksum tools). The PGP
key used to sign the releases is available as RELEASE_KEY.asc from
the mirror sites.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 7.0 / 7.0p1 (2015-08-11) 

OpenSSH 7.0 was released on 2015-08-11. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol 2.0 implementation and
includes sftp client and server support. OpenSSH also includes
transitional support for the legacy SSH 1.3 and 1.5 protocols
that may be enabled at compile-time.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Future deprecation notice
=========================

We plan on retiring more legacy cryptography in the next release
including:

 * Refusing all RSA keys smaller than 1024 bits (the current minimum
   is 768 bits)

 * Several ciphers will be disabled by default: blowfish-cbc,
   cast128-cbc, all arcfour variants and the rijndael-cbc aliases
   for AES.

 * MD5-based HMAC algorithms will be disabled by default.

This list reflects our current intentions, but please check the final
release notes for OpenSSH 7.1 when it is released.

Changes since OpenSSH 6.9
=========================

This focus of this release is primarily to deprecate weak, legacy
and/or unsafe cryptography.

Security
--------

 * sshd(8): OpenSSH 6.8 and 6.9 incorrectly set TTYs to be world-
   writable. Local attackers may be able to write arbitrary messages
   to logged-in users, including terminal escape sequences.
   Reported by Nikolay Edigaryev.

 * sshd(8): Portable OpenSSH only: Fixed a privilege separation
   weakness related to PAM support. Attackers who could successfully
   compromise the pre-authentication process for remote code
   execution and who had valid credentials on the host could
   impersonate other users.  Reported by Moritz Jodeit.

 * sshd(8): Portable OpenSSH only: Fixed a use-after-free bug
   related to PAM support that was reachable by attackers who could
   compromise the pre-authentication process for remote code
   execution. Also reported by Moritz Jodeit.

 * sshd(8): fix circumvention of MaxAuthTries using keyboard-
   interactive authentication. By specifying a long, repeating
   keyboard-interactive "devices" string, an attacker could request
   the same authentication method be tried thousands of times in
   a single pass. The LoginGraceTime timeout in sshd(8) and any
   authentication failure delays implemented by the authentication
   mechanism itself were still applied. Found by Kingcope.

Potentially-incompatible Changes
--------------------------------

 * Support for the legacy SSH version 1 protocol is disabled by
   default at compile time.

 * Support for the 1024-bit diffie-hellman-group1-sha1 key exchange
   is disabled by default at run-time. It may be re-enabled using
   the instructions at http://www.openssh.com/legacy.html

 * Support for ssh-dss, ssh-dss-cert-* host and user keys is disabled
   by default at run-time. These may be re-enabled using the
   instructions at http://www.openssh.com/legacy.html

 * Support for the legacy v00 cert format has been removed.

 * The default for the sshd_config(5) PermitRootLogin option has
   changed from "yes" to "prohibit-password".

 * PermitRootLogin=without-password/prohibit-password now bans all
   interactive authentication methods, allowing only public-key,
   hostbased and GSSAPI authentication (previously it permitted
   keyboard-interactive and password-less authentication if those
   were enabled).

New Features
------------

 * ssh_config(5): add PubkeyAcceptedKeyTypes option to control which
   public key types are available for user authentication.

 * sshd_config(5): add HostKeyAlgorithms option to control which
   public key types are offered for host authentications.

 * ssh(1), sshd(8): extend Ciphers, MACs, KexAlgorithms,
   HostKeyAlgorithms, PubkeyAcceptedKeyTypes and HostbasedKeyTypes
   options to allow appending to the default set of algorithms
   instead of replacing it. Options may now be prefixed with a '+'
   to append to the default, e.g. "HostKeyAlgorithms=+ssh-dss".

 * sshd_config(5): PermitRootLogin now accepts an argument of
   'prohibit-password' as a less-ambiguous synonym of 'without-
   password'.

Bugfixes
--------

 * ssh(1), sshd(8): add compatibility workarounds for Cisco and more
   PuTTY versions. bz#2424

 * Fix some omissions and errors in the PROTOCOL and PROTOCOL.mux
   documentation relating to Unix domain socket forwarding;
   bz#2421 bz#2422

 * ssh(1): Improve the ssh(1) manual page to include a better
   description of Unix domain socket forwarding; bz#2423

 * ssh(1), ssh-agent(1): skip uninitialised PKCS#11 slots, fixing
   failures to load keys when they are present. bz#2427

 * ssh(1), ssh-agent(1): do not ignore PKCS#11 hosted keys that with
   empty CKA_ID; bz#2429

 * sshd(8): clarify documentation for UseDNS option; bz#2045

Portable OpenSSH
----------------

 * Check realpath(3) behaviour matches what sftp-server requires and
   use a replacement if necessary.

Checksums:
==========

 - SHA1 (openssh-7.0.tar.gz) = a19ff0bad2a67348b1d01a38a9580236120b7099
 - SHA256 (openssh-7.0.tar.gz) = 4F6HV/ZqT465f3sMB2vIkXO+wrYtL5hnqzAymfbZ1Jk=

 - SHA1 (openssh-7.0p1.tar.gz) = d8337c9eab91d360d104f6dd805f8b32089c063c
 - SHA256 (openssh-7.0p1.tar.gz) = /VkySToZ9MgRU9gS7k4EK0m707dZqz2TRKvswrwUheU=

Please note that the PGP key used to sign releases was recently rotated.
The new key has been signed by the old key to provide continuity. It is
available from the mirror sites as RELEASE_KEY.asc.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 6.9 / 6.9p1 (2015-07-01) 

OpenSSH 6.9 was released on 2015-07-01. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Future Deprecation Notice
=========================

The 7.0 release of OpenSSH, due for release in late July, will
deprecate several features, some of which may affect compatibility
or existing configurations. The intended changes are as follows:

 * The default for the sshd_config(5) PermitRootLogin option will
   change from "yes" to "no".

 * Support for the legacy version 1.x of the SSH protocol will be
   disabled at compile time by default.

 * Support for the 1024-bit diffie-hellman-group1-sha1 key exchange
   will be run-time disabled by default.

 * Support for ssh-dss, ssh-dss-cert-* host and user keys will be
   run-time disabled by default.

 * Support for the legacy v00 cert format will be removed

 * Several ciphers will be disabled by default: blowfish-cbc,
   cast128-cbc, all arcfour variants and the rijndael-cbc aliases
   for AES

 * Refusing all RSA keys smaller than 1024 bits (the current minimum
   is 768 bits)

This list reflects our current intentions, but please check the final
release notes for OpenSSH 7.0 when it is released.

Changes since OpenSSH 6.8
=========================

This is primarily a bugfix release.

Security
--------

 * ssh(1): when forwarding X11 connections with ForwardX11Trusted=no,
   connections made after ForwardX11Timeout expired could be permitted
   and no longer subject to XSECURITY restrictions because of an
   ineffective timeout check in ssh(1) coupled with "fail open"
   behaviour in the X11 server when clients attempted connections with
   expired credentials. This problem was reported by Jann Horn.

 * ssh-agent(1): fix weakness of agent locking (ssh-add -x) to
   password guessing by implementing an increasing failure delay,
   storing a salted hash of the password rather than the password
   itself and using a timing-safe comparison function for verifying
   unlock attempts. This problem was reported by Ryan Castellucci.

New Features
------------

 * ssh(1), sshd(8): promote chacha20-poly1305@openssh.com to be the
   default cipher

 * sshd(8): support admin-specified arguments to AuthorizedKeysCommand;
   bz#2081

 * sshd(8): add AuthorizedPrincipalsCommand that allows retrieving
   authorized principals information from a subprocess rather than
   a file.

 * ssh(1), ssh-add(1): support PKCS#11 devices with external PIN
   entry devices bz#2240

 * sshd(8): allow GSSAPI host credential check to be relaxed for
   multihomed hosts via GSSAPIStrictAcceptorCheck option; bz#928

 * ssh-keygen(1): support "ssh-keygen -lF hostname" to search
   known_hosts and print key hashes rather than full keys.

 * ssh-agent(1): add -D flag to leave ssh-agent in foreground without
   enabling debug mode; bz#2381

Bugfixes
--------

 * ssh(1), sshd(8): deprecate legacy SSH2_MSG_KEX_DH_GEX_REQUEST_OLD
   message and do not try to use it against some 3rd-party SSH
   implementations that use it (older PuTTY, WinSCP).

 * Many fixes for problems caused by compile-time deactivation of
   SSH1 support (including bz#2369)

 * ssh(1), sshd(8): cap DH-GEX group size at 4Kbits for Cisco
   implementations as some would fail when attempting to use group
   sizes >4K; bz#2209

 * ssh(1): fix out-of-bound read in EscapeChar configuration option
   parsing; bz#2396

 * sshd(8): fix application of PermitTunnel, LoginGraceTime,
   AuthenticationMethods and StreamLocalBindMask options in Match
   blocks

 * ssh(1), sshd(8): improve disconnection message on TCP reset;
   bz#2257

 * ssh(1): remove failed remote forwards established by muliplexing
   from the list of active forwards; bz#2363

 * sshd(8): make parsing of authorized_keys "environment=" options
   independent of PermitUserEnv being enabled; bz#2329

 * sshd(8): fix post-auth crash with permitopen=none; bz#2355

 * ssh(1), ssh-add(1), ssh-keygen(1): allow new-format private keys
   to be encrypted with AEAD ciphers; bz#2366

 * ssh(1): allow ListenAddress, Port and AddressFamily configuration
   options to appear in any order; bz#86

 * sshd(8): check for and reject missing arguments for VersionAddendum
   and ForceCommand; bz#2281

 * ssh(1), sshd(8): don't treat unknown certificate extensions as
   fatal; bz#2387

 * ssh-keygen(1): make stdout and stderr output consistent; bz#2325

 * ssh(1): mention missing DISPLAY environment in debug log when X11
   forwarding requested; bz#1682

 * sshd(8): correctly record login when UseLogin is set; bz#378

 * sshd(8): Add some missing options to sshd -T output and fix output
   of VersionAddendum and HostCertificate. bz#2346   

 * Document and improve consistency of options that accept a "none"
   argument" TrustedUserCAKeys, RevokedKeys (bz#2382),
   AuthorizedPrincipalsFile (bz#2288)

 * ssh(1): include remote username in debug output; bz#2368

 * sshd(8): avoid compatibility problem with some versions of Tera
   Term, which would crash when they received the hostkeys notification
   message (hostkeys-00@openssh.com)

 * sshd(8): mention ssh-keygen -E as useful when comparing legacy MD5
   host key fingerprints; bz#2332

 * ssh(1): clarify pseudo-terminal request behaviour and use make
   manual language consistent; bz#1716

 * ssh(1): document that the TERM environment variable is not subject
   to SendEnv and AcceptEnv; bz#2386

Portable OpenSSH
----------------

 * sshd(8): Format UsePAM setting when using sshd -T, part of bz#2346

 * Look for '${host}-ar' before 'ar', making cross-compilation easier;
   bz#2352.

 * Several portable compilation fixes: bz#2402, bz#2337, bz#2370

 * moduli(5): update DH-GEX moduli

Checksums:
==========

 - SHA1 (openssh-6.9.tar.gz) = cd5fcb93411025bbc4b4b57753b622769dfb1e0d
 - SHA256 (openssh-6.9.tar.gz) = itCMw0aE/xvrGKWhzRD2UM/9kzIOyFaH2dIWMfX8agQ=

 - SHA1 (openssh-6.9p1.tar.gz) = 86ab57f00d0fd9bf302760f2f6deac1b6e9df265
 - SHA256 (openssh-6.9p1.tar.gz) = bgdN9TjzV9RAvmz5PcWBoh8i054jbyF/zY6su2yJbP4=

Please note that the PGP key used to sign releases was recently rotated.
The new key has been signed by the old key to provide continuity. It is
available from the mirror sites as RELEASE_KEY.asc.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 6.8 / 6.8p1 (2015-03-18) 

OpenSSH 6.8 was released on 2015-03-18. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Changes since OpenSSH 6.7
=========================

This is a major release, containing a number of new features as
well as a large internal re-factoring.

Potentially-incompatible changes
--------------------------------

 * sshd(8): UseDNS now defaults to 'no'. Configurations that match
   against the client host name (via sshd_config or authorized_keys)
   may need to re-enable it or convert to matching against addresses.

New Features
------------

 * Much of OpenSSH's internal code has been re-factored to be more
   library-like. These changes are mostly not user-visible, but
   have greatly improved OpenSSH's testability and internal layout.

 * Add FingerprintHash option to ssh(1) and sshd(8), and equivalent
   command-line flags to the other tools to control algorithm used
   for key fingerprints. The default changes from MD5 to SHA256 and
   format from hex to base64.

   Fingerprints now have the hash algorithm prepended. An example of
   the new format: SHA256:mVPwvezndPv/ARoIadVY98vAC0g+P/5633yTC4d/wXE
   Please note that visual host keys will also be different.

 * ssh(1), sshd(8): Experimental host key rotation support. Add a
   protocol extension for a server to inform a client of all its
   available host keys after authentication has completed. The client
   may record the keys in known_hosts, allowing it to upgrade to better
   host key algorithms and a server to gracefully rotate its keys.

   The client side of this is controlled by a UpdateHostkeys config
   option (default off).

 * ssh(1): Add a ssh_config HostbasedKeyType option to control which
   host public key types are tried during host-based authentication.

 * ssh(1), sshd(8): fix connection-killing host key mismatch errors
   when sshd offers multiple ECDSA keys of different lengths.

 * ssh(1): when host name canonicalisation is enabled, try to
   parse host names as addresses before looking them up for
   canonicalisation. fixes bz#2074 and avoiding needless DNS
   lookups in some cases.

 * ssh-keygen(1), sshd(8): Key Revocation Lists (KRLs) no longer
   require OpenSSH to be compiled with OpenSSL support.

 * ssh(1), ssh-keysign(8): Make ed25519 keys work for host based
   authentication.

 * sshd(8): SSH protocol v.1 workaround for the Meyer, et al,
   Bleichenbacher Side Channel Attack. Fake up a bignum key before
   RSA decryption.

 * sshd(8): Remember which public keys have been used for
   authentication and refuse to accept previously-used keys.
   This allows AuthenticationMethods=publickey,publickey to require
   that users authenticate using two _different_ public keys.

 * sshd(8): add sshd_config HostbasedAcceptedKeyTypes and
   PubkeyAcceptedKeyTypes options to allow sshd to control what
   public key types will be accepted. Currently defaults to all.

 * sshd(8): Don't count partial authentication success as a failure
   against MaxAuthTries.

 * ssh(1): Add RevokedHostKeys option for the client to allow
   text-file or KRL-based revocation of host keys.

 * ssh-keygen(1), sshd(8): Permit KRLs that revoke certificates by
   serial number or key ID without scoping to a particular CA.

 * ssh(1): Add a "Match canonical" criteria that allows ssh_config
   Match blocks to trigger only in the second config pass.

 * ssh(1): Add a -G option to ssh that causes it to parse its
   configuration and dump the result to stdout, similar to "sshd -T".

 * ssh(1): Allow Match criteria to be negated. E.g. "Match !host".

 * The regression test suite has been extended to cover more OpenSSH
   features. The unit tests have been expanded and now cover key
   exchange.

Bugfixes

 * ssh-keyscan(1): ssh-keyscan has been made much more robust again
   servers that hang or violate the SSH protocol.

 * ssh(1), ssh-keygen(1): Fix regression bz#2306: Key path names were
   being lost as comment fields.

 * ssh(1): Allow ssh_config Port options set in the second config
   parse phase to be applied (they were being ignored). bz#2286

 * ssh(1): Tweak config re-parsing with host canonicalisation - make
   the second pass through the config files always run when host name
   canonicalisation is enabled (and not whenever the host name
   changes) bz#2267

 * ssh(1): Fix passing of wildcard forward bind addresses when
   connection multiplexing is in use; bz#2324;

 * ssh-keygen(1): Fix broken private key conversion from non-OpenSSH
   formats; bz#2345.

 * ssh-keygen(1): Fix KRL generation bug when multiple CAs are in
   use.

 * Various fixes to manual pages: bz#2288, bz#2316, bz#2273

Portable OpenSSH

 * Support --without-openssl at configure time

   Disables and removes dependency on OpenSSL. Many features,
   including SSH protocol 1 are not supported and the set of crypto
   options is greatly restricted. This will only work on systems
   with native arc4random or /dev/urandom.

   Considered highly experimental for now.

 * Support --without-ssh1 option at configure time

   Allows disabling support for SSH protocol 1.

 * sshd(8): Fix compilation on systems with IPv6 support in utmpx; bz#2296

 * Allow custom service name for sshd on Cygwin. Permits the use of
   multiple sshd running with different service names.

Checksums:
==========

 - SHA1 (openssh-6.8.tar.gz) = 99903c6ca76e0a2c044711017f81127e12459d37
 - SHA256 (openssh-6.8.tar.gz) = N1uzVarFbrm2CzAwuDu3sRoszmqpK+5phAChP/QNyuw=

 - SHA1 (openssh-6.8p1.tar.gz) = cdbc51e46a902b30d263b05fdc71340920e91c92
 - SHA256 (openssh-6.8p1.tar.gz) = P/ZM5z7hJEgLW/dnuYMNfTwDu8tqvnFrePAZLDfOFg4=

Please note that the PGP key used to sign releases was recently rotated.
The new key has been signed by the old key to provide continuity. It is
available from the mirror sites as RELEASE_KEY.asc.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 6.7 / 6.7p1 (2014-10-06) 

OpenSSH 6.7 was released on 2014-10-06. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Changes since OpenSSH 6.6
=========================

Potentially-incompatible changes

 * sshd(8): The default set of ciphers and MACs has been altered to
   remove unsafe algorithms. In particular, CBC ciphers and arcfour*
   are disabled by default.

   The full set of algorithms remains available if configured
   explicitly via the Ciphers and MACs sshd_config options.

 * sshd(8): Support for tcpwrappers/libwrap has been removed.

 * OpenSSH 6.5 and 6.6 have a bug that causes ~0.2% of connections
   using the curve25519-sha256@libssh.org KEX exchange method to fail
   when connecting with something that implements the specification
   correctly. OpenSSH 6.7 disables this KEX method when speaking to
   one of the affected versions.

New Features

 * Major internal refactoring to begin to make part of OpenSSH usable
   as a library. So far the wire parsing, key handling and KRL code
   has been refactored. Please note that we do not consider the API
   stable yet, nor do we offer the library in separable form.

 * ssh(1), sshd(8): Add support for Unix domain socket forwarding.
   A remote TCP port may be forwarded to a local Unix domain socket
   and vice versa or both ends may be a Unix domain socket.

 * ssh(1), ssh-keygen(1): Add support for SSHFP DNS records for
   ED25519 key types.

 * sftp(1): Allow resumption of interrupted uploads.

 * ssh(1): When rekeying, skip file/DNS lookups of the hostkey if it
   is the same as the one sent during initial key exchange; bz#2154

 * sshd(8): Allow explicit ::1 and 127.0.0.1 forwarding bind
   addresses when GatewayPorts=no; allows client to choose address
   family; bz#2222

 * sshd(8): Add a sshd_config PermitUserRC option to control whether
   ~/.ssh/rc is executed, mirroring the no-user-rc authorized_keys
   option; bz#2160

 * ssh(1): Add a %C escape sequence for LocalCommand and ControlPath
   that expands to a unique identifier based on a hash of the tuple of
   (local host, remote user, hostname, port). Helps avoid exceeding
   miserly pathname limits for Unix domain sockets in multiplexing
   control paths; bz#2220

 * sshd(8): Make the "Too many authentication failures" message
   include the user, source address, port and protocol in a format
   similar to the authentication success / failure messages; bz#2199

 * Added unit and fuzz tests for refactored code. These are run
   automatically in portable OpenSSH via the "make tests" target.

Bugfixes

 * sshd(8): Fix remote forwarding with the same listen port but
   different listen address.

 * ssh(1): Fix inverted test that caused PKCS#11 keys that were
   explicitly listed in ssh_config or on the commandline not to be
   preferred.

 * ssh-keygen(1): Fix bug in KRL generation: multiple consecutive
   revoked certificate serial number ranges could be serialised to an
   invalid format. Readers of a broken KRL caused by this bug will
   fail closed, so no should-have-been-revoked key will be accepted.

 * ssh(1): Reflect stdio-forward ("ssh -W host:port ...") failures in
   exit status. Previously we were always returning 0; bz#2255

 * ssh(1), ssh-keygen(1): Make Ed25519 keys' title fit properly in the
   randomart border; bz#2247

 * ssh-agent(1): Only cleanup agent socket in the main agent process
   and not in any subprocesses it may have started (e.g. forked
   askpass). Fixes agent sockets being zapped when askpass processes
   fatal(); bz#2236

 * ssh-add(1): Make stdout line-buffered; saves partial output getting
   lost when ssh-add fatal()s part-way through (e.g. when listing keys
   from an agent that supports key types that ssh-add doesn't);
   bz#2234

 * ssh-keygen(1): When hashing or removing hosts, don't choke on
   @revoked markers and don't remove @cert-authority markers; bz#2241

 * ssh(1): Don't fatal when hostname canonicalisation fails and a
   ProxyCommand is in use; continue and allow the ProxyCommand to
   connect anyway (e.g. to a host with a name outside the DNS behind
   a bastion)

 * scp(1): When copying local->remote fails during read, don't send
   uninitialised heap to the remote end.

 * sftp(1): Fix fatal "el_insertstr failed" errors when tab-completing
   filenames with  a single quote char somewhere in the string;
   bz#2238

 * ssh-keyscan(1): Scan for Ed25519 keys by default.

 * ssh(1): When using VerifyHostKeyDNS with a DNSSEC resolver, down-
   convert any certificate keys to plain keys and attempt SSHFP
   resolution.  Prevents a server from skipping SSHFP lookup and
   forcing a new-hostkey dialog by offering only certificate keys.
     
 * sshd(8): Avoid crash at exit via NULL pointer reference; bz#2225

 * Fix some strict-alignment errors.

Portable OpenSSH

 * Portable OpenSSH now supports building against libressl-portable.

 * Portable OpenSSH now requires openssl 0.9.8f or greater. Older
   versions are no longer supported.

 * In the OpenSSL version check, allow fix version upgrades (but not
   downgrades. Debian bug #748150.

 * sshd(8): On Cygwin, determine privilege separation user at runtime,
   since it may need to be a domain account.

 * sshd(8): Don't attempt to use vhangup on Linux. It doesn't work for
   non-root users, and for them it just messes up the tty settings.

 * Use CLOCK_BOOTTIME in preference to CLOCK_MONOTONIC when it is
   available. It considers time spent suspended, thereby ensuring
   timeouts (e.g. for expiring agent keys) fire correctly.  bz#2228

 * Add support for ed25519 to opensshd.init init script.

 * sftp-server(8): On platforms that support it, use prctl() to
   prevent sftp-server from accessing /proc/self/{mem,maps}

Checksums:
==========

 - SHA1 (openssh-6.7.tar.gz) = 315497b27a0186e4aef67987cfc9f3d9ba561cd8
 - SHA256 (openssh-6.7.tar.gz) = /me/hPxDw9Tfd3siNKQubSQph84qiKwftiMsgj6nh5E=

 - SHA1 (openssh-6.7p1.tar.gz) = 14e5fbed710ade334d65925e080d1aaeb9c85bf6
 - SHA256 (openssh-6.7p1.tar.gz) = svg5Tq6Fjau9732sELma7ADJVGJ1PoA0LlMLu29yVQc=

Please note that the PGP key used to sign releases was recently rotated.
The new key has been signed by the old key to provide continuity. It is
available from the mirror sites as RELEASE_KEY.asc.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 6.6 / 6.6p1 (2014-03-15) 

OpenSSH 6.6 was released on 2014-03-15. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Changes since OpenSSH 6.6
=========================

This is primarily a bugfix release.

Security:

 * sshd(8): when using environment passing with a sshd_config(5)
   AcceptEnv pattern with a wildcard. OpenSSH prior to 6.6 could be
   tricked into accepting any environment variable that contains the
   characters before the wildcard character.

New / changed features:

 * ssh(1), sshd(8): this release removes the J-PAKE authentication code.
   This code was experimental, never enabled and had been unmaintained
   for some time.

 * ssh(1): when processing Match blocks, skip 'exec' clauses other clauses
   predicates failed to match.

 * ssh(1): if hostname canonicalisation is enabled and results in the
   destination hostname being changed, then re-parse ssh_config(5) files
   using the new destination hostname. This gives 'Host' and 'Match'
   directives that use the expanded hostname a chance to be applied.

Bugfixes:

 * ssh(1): avoid spurious "getsockname failed: Bad file descriptor" in
   ssh -W. bz#2200, debian#738692

 * sshd(8): allow the shutdown(2) syscall in seccomp-bpf and systrace
   sandbox modes, as it is reachable if the connection is terminated
   during the pre-auth phase.

 * ssh(1), sshd(8): fix unsigned overflow that in SSH protocol 1 bignum
   parsing. Minimum key length checks render this bug unexploitable to
   compromise SSH 1 sessions.

 * sshd_config(5): clarify behaviour of a keyword that appears in
   multiple matching Match blocks. bz#2184

 * ssh(1): avoid unnecessary hostname lookups when canonicalisation is
   disabled. bz#2205

 * sshd(8): avoid sandbox violation crashes in GSSAPI code by caching
   the supported list of GSSAPI mechanism OIDs before entering the
   sandbox. bz#2107

 * ssh(1): fix possible crashes in SOCKS4 parsing caused by assumption
   that the SOCKS username is nul-terminated.

 * ssh(1): fix regression for UsePrivilegedPort=yes when BindAddress is
   not specified.

 * ssh(1), sshd(8): fix memory leak in ECDSA signature verification.

 * ssh(1): fix matching of 'Host' directives in ssh_config(5) files
   to be case-insensitive again (regression in 6.5).

Portable OpenSSH:

 * sshd(8): don't fatal if the FreeBSD Capsicum is offered by the
   system headers and libc but is not supported by the kernel.
 * Fix build using the HP-UX compiler.

Checksums:
==========

 - SHA1 (openssh-6.6.tar.gz) = bf932d798324ff2502409d3714d0ad8d65c7e1e7
 - SHA256 (openssh-6.6.tar.gz) = jaSJE5aiQRm+91dV6EvVGr/ozo33tbxyjjFSiu+Cy80=

 - SHA1 (openssh-6.6p1.tar.gz) = b850fd1af704942d9b3c2eff7ef6b3a59b6a6b6e
 - SHA256 (openssh-6.6p1.tar.gz) = SMHwZktFNIdQOABMxPNVW4MpwqgcHfSNtcUXgA3iA7s=

Please note that the PGP key used to sign releases was recently rotated.
The new key has been signed by the old key to provide continuity. It is
available from the mirror sites as RELEASE_KEY.asc.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 6.5 / 6.5p1 (2014-01-30) 

Changes since OpenSSH 6.4
=========================

This is a feature-focused release.

New features:

 * ssh(1), sshd(8): Add support for key exchange using elliptic-curve
   Diffie Hellman in Daniel Bernstein's Curve25519. This key exchange
   method is the default when both the client and server support it.

 * ssh(1), sshd(8): Add support for Ed25519 as a public key type.
   Ed25519 is a elliptic curve signature scheme that offers
   better security than ECDSA and DSA and good performance. It may be
   used for both user and host keys.

 * Add a new private key format that uses a bcrypt KDF to better
   protect keys at rest. This format is used unconditionally for
   Ed25519 keys, but may be requested when generating or saving
   existing keys of other types via the -o ssh-keygen(1) option.
   We intend to make the new format the default in the near future.
   Details of the new format are in the PROTOCOL.key file.

 * ssh(1), sshd(8): Add a new transport cipher
   "chacha20-poly1305@openssh.com" that combines Daniel Bernstein's
   ChaCha20 stream cipher and Poly1305 MAC to build an authenticated
   encryption mode. Details are in the PROTOCOL.chacha20poly1305 file.

 * ssh(1), sshd(8): Refuse RSA keys from old proprietary clients and
   servers that use the obsolete RSA+MD5 signature scheme. It will
   still be possible to connect with these clients/servers but only
   DSA keys will be accepted, and OpenSSH will refuse connection
   entirely in a future release.

 * ssh(1), sshd(8): Refuse old proprietary clients and servers that
   use a weaker key exchange hash calculation.

 * ssh(1): Increase the size of the Diffie-Hellman groups requested
   for each symmetric key size. New values from NIST Special
   Publication 800-57 with the upper limit specified by RFC4419.

 * ssh(1), ssh-agent(1): Support PKCS#11 tokens that only provide
   X.509 certs instead of raw public keys (requested as bz#1908).

 * ssh(1): Add a ssh_config(5) "Match" keyword that allows
   conditional configuration to be applied by matching on hostname,
   user and result of arbitrary commands.

 * ssh(1): Add support for client-side hostname canonicalisation
   using a set of DNS suffixes and rules in ssh_config(5). This
   allows unqualified names to be canonicalised to fully-qualified
   domain names to eliminate ambiguity when looking up keys in
   known_hosts or checking host certificate names.

 * sftp-server(8): Add the ability to whitelist and/or blacklist sftp
   protocol requests by name.

 * sftp-server(8): Add a sftp "fsync@openssh.com" to support calling
   fsync(2) on an open file handle.

 * sshd(8): Add a ssh_config(5) PermitTTY to disallow TTY allocation,
   mirroring the longstanding no-pty authorized_keys option.

 * ssh(1): Add a ssh_config ProxyUseFDPass option that supports the
   use of ProxyCommands that establish a connection and then pass a
   connected file descriptor back to ssh(1). This allows the
   ProxyCommand to exit rather than staying around to transfer data.

Bugfixes:

 * ssh(1), sshd(8): Fix potential stack exhaustion caused by nested
   certificates.

 * ssh(1): bz#1211: make BindAddress work with UsePrivilegedPort.

 * sftp(1): bz#2137: fix the progress meter for resumed transfer.

 * ssh-add(1): bz#2187: do not request smartcard PIN when removing
   keys from ssh-agent.

 * sshd(8): bz#2139: fix re-exec fallback when original sshd binary
   cannot be executed.

 * ssh-keygen(1): Make relative-specified certificate expiry times
   relative to current time and not the validity start time.

 * sshd(8): bz#2161: fix AuthorizedKeysCommand inside a Match block.

 * sftp(1): bz#2129: symlinking a file would incorrectly canonicalise
   the target path.

 * ssh-agent(1): bz#2175: fix a use-after-free in the PKCS#11 agent
   helper executable.

 * sshd(8): Improve logging of sessions to include the user name,
   remote host and port, the session type (shell, command, etc.) and
   allocated TTY (if any).

 * sshd(8): bz#1297: tell the client (via a debug message) when
   their preferred listen address has been overridden by the
   server's GatewayPorts setting.

 * sshd(8): bz#2162: include report port in bad protocol banner
   message.

 * sftp(1): bz#2163: fix memory leak in error path in do_readdir().

 * sftp(1): bz#2171: don't leak file descriptor on error.

 * sshd(8): Include the local address and port in "Connection from
   ..." message (only shown at loglevel>=verbose).

Portable OpenSSH:

 * Please note that this is the last version of Portable OpenSSH that
   will support versions of OpenSSL prior to 0.9.6. Support (i.e.
   SSH_OLD_EVP) will be removed following the 6.5p1 release.

 * Portable OpenSSH will attempt compile and link as a Position
   Independent Executable on Linux, OS X and OpenBSD on recent gcc-
   like compilers. Other platforms and older/other compilers may
   request this using the --with-pie configure flag.

 * A number of other toolchain-related hardening options are used
   automatically if available, including -ftrapv to abort on signed
   integer overflow and options to write-protect dynamic linking
   information.  The use of these options may be disabled using the
   --without-hardening configure flag.

 * If the toolchain supports it, one of the -fstack-protector-strong,
   -fstack-protector-all or -fstack-protector compilation flag are
   used to add guards to mitigate attacks based on stack overflows.
   The use of these options may be disabled using the
   --without-stackprotect configure option.

 * sshd(8): Add support for pre-authentication sandboxing using the
   Capsicum API introduced in FreeBSD 10.

 * Switch to a ChaCha20-based arc4random() PRNG for platforms that do
   not provide their own.

 * sshd(8): bz#2156: restore Linux oom_adj setting when handling
   SIGHUP to maintain behaviour over restart.

 * sshd(8): bz#2032: use local username in krb5_kuserok check rather
   than full client name which may be of form user@REALM.

 * ssh(1), sshd(8): Test for both the presence of ECC NID numbers in
   OpenSSL and that they actually work. Fedora (at least) has
   NID_secp521r1 that doesn't work.

 * bz#2173: use pkg-config --libs to include correct -L location for
   libedit.

Checksums:
==========

 - SHA1 (openssh-6.5.tar.gz) = 0a375e20d895670489a9241f8faa57670214fbed
 - SHA256 (openssh-6.5.tar.gz) = sK5q2rB0o5JCbEmbeE/6N9DtJkT81dwmeuhogT4i900=

 - SHA1 (openssh-6.5p1.tar.gz) = 3363a72b4fee91b29cf2024ff633c17f6cd2f86d
 - SHA256 (openssh-6.5p1.tar.gz) = oRle1V25RSUtWhcw1KKipcHJpqoB7y5a91CpYmI9kCc=

Please note that the PGP key used to sign releases has been rotated.
The new key has been signed by the old key to provide continuity. It
is available from the mirror sites as RELEASE_KEY.asc.

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 6.4 / 6.4p1 (2013-11-08) 

Changes since OpenSSH 6.3
=========================

This release fixes a security bug:

 * sshd(8): fix a memory corruption problem triggered during rekeying
   when an AES-GCM cipher is selected. Full details of the vulnerability
   are available at: http://www.openssh.com/txt/gcmrekey.adv

Checksums:
==========

 - SHA1 (openssh-6.4.tar.gz) = 4caf1a50eb3a3da821c16298c4aaa576fe24210c
 - SHA1 (openssh-6.4p1.tar.gz) = cf5fe0eb118d7e4f9296fbc5d6884965885fc55d

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 6.3 / 6.3p1 (2013-09-13) 

Changes since OpenSSH 6.2
=========================

This release is predominantly a bugfix release:

Features:

 * sshd(8): add ssh-agent(1) support to sshd(8); allows encrypted hostkeys,
   or hostkeys on smartcards.

 * ssh(1)/sshd(8): allow optional time-based rekeying via a second argument
   to the existing RekeyLimit option. RekeyLimit is now supported in
   sshd_config as well as on the client.

 * sshd(8): standardise logging of information during user authentication.

   The presented key/cert and the remote username (if available) is now
   logged in the authentication success/failure message on the same log
   line as the local username, remote host/port and protocol in use.
   Certificates contents and the key fingerprint of the signing CA are
   logged too.

   Including all relevant information on a single line simplifies log
   analysis as it is no longer necessary to relate information scattered
   across multiple log entries.

 * ssh(1): add the ability to query which ciphers, MAC algorithms, key
   types and key exchange methods are supported in the binary.

 * ssh(1): support ProxyCommand=- to allow support cases where stdin and
   stdout already point to the proxy.

 * ssh(1): allow IdentityFile=none

 * ssh(1)/sshd(8): add -E option to ssh and sshd to append debugging logs
   to a specified file instead of stderr or syslog.

 * sftp(1): add support for resuming partial downloads using the "reget"
   command and on the sftp commandline or on the "get" commandline using
   the "-a" (append) option.

 * ssh(1): add an "IgnoreUnknown" configuration option to selectively
   suppress errors arising from unknown configuration directives.

 * sshd(8): add support for submethods to be appended to required
   authentication methods listed via AuthenticationMethods.

Bugfixes:

 * sshd(8): fix refusal to accept certificate if a key of a different type
   to the CA key appeared in authorized_keys before the CA key.

 * ssh(1)/ssh-agent(1)/sshd(8): Use a monotonic time source for timers so
   that things like keepalives and rekeying will work properly over clock
   steps.

 * sftp(1): update progressmeter when data is acknowledged, not when it's
   sent. bz#2108

 * ssh(1)/ssh-keygen(1): improve error messages when the current user does
   not exist in /etc/passwd; bz#2125

 * ssh(1): reset the order in which public keys are tried after partial
   authentication success.

 * ssh-agent(1): clean up socket files after SIGINT when in debug mode;
   bz#2120

 * ssh(1) and others: avoid confusing error messages in the case of broken
   system resolver configurations; bz#2122

 * ssh(1): set TCP nodelay for connections started with -N; bz#2124

 * ssh(1): correct manual for permission requirements on ~/.ssh/config;
   bz#2078

 * ssh(1): fix ControlPersist timeout not triggering in cases where TCP
   connections have hung. bz#1917

 * ssh(1): properly detach a ControlPersist master from its controlling
   terminal.

 * sftp(1): avoid crashes in libedit when it has been compiled with multi-
   byte character support. bz#1990

 * sshd(8): when running sshd -D, close stderr unless we have explicitly
   requested logging to stderr. bz#1976,

 * ssh(1): fix incomplete bzero; bz#2100

 * sshd(8): log and error and exit if ChrootDirectory is specified and
   running without root privileges.

 * Many improvements to the regression test suite. In particular log files
   are now saved from ssh and sshd after failures.

 * Fix a number of memory leaks. bz#1967 bz#2096 and others

 * sshd(8): fix public key authentication when a :style is appended to
   the requested username.

 * ssh(1): do not fatally exit when attempting to cleanup multiplexing-
   created channels that are incompletely opened. bz#2079

Portable OpenSSH:

 * Major overhaul of contrib/cygwin/README

 * Fix unaligned accesses in umac.c for strict-alignment architectures.
   bz#2101

 * Enable -Wsizeof-pointer-memaccess if the compiler supports it. bz#2100

 * Fix broken incorrect commandline reporting errors. bz#1448

 * Only include SHA256 and ECC-based key exchange methods if libcrypto has
   the required support.

 * Fix crash in SOCKS5 dynamic forwarding code on strict-alignment
   architectures.

 * A number of portability fixes for Android:
   * Don't try to use lastlog on Android; bz#2111
   * Fall back to using openssl's DES_crypt function on platforms that don't
     have a native crypt() function; bz#2112
   * Test for fd_mask, howmany and NFDBITS rather than trying to enumerate
     the platforms that don't have them. bz#2085
   * Replace S_IWRITE, which isn't standardized, with S_IWUSR, which is.
     bz#2085
   * Add a null implementation of endgrent for platforms that don't have
     it (eg Android) bz#2087
   * Support platforms, such as Android, that lack struct passwd.pw_gecos.
     bz#2086

Checksums:
==========

 - SHA1 (openssh-6.3.tar.gz) = 8a6ef99ffc80c19e9afe9fe1e857370f6adcf450
 - SHA1 (openssh-6.3p1.tar.gz) = 70845ca79474258cab29dbefae13d93e41a83ccb

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 6.2p2 (2013-05-16) 

Changes since OpenSSH 6.2p1
===========================

This is a bugfix release:

Bugfixes:

 * ssh(1): Only warn for missing identity files that were explicitly
   specified.

 * Fix bug in contributed contrib/ssh-copy-id script that could result in
   "rm *" being called on mktemp failure. bz#2105

 * sshd(8): Quiet disconnect notifications on the server from error() back
   to logit() from error() for normal, client-initiated disconnections.
   bz#2057

 * Avoid conflicting definitions of __int64 on Cygwin

Checksums:
==========

 - SHA1 (openssh-6.2p2.tar.gz) = c2b4909eba6f5ec6f9f75866c202db47f3b501ba

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 6.2 / 6.2p1 (2013-03-22) 

Changes since OpenSSH 6.1
=========================

This release introduces a number of new features:

Features:

 * ssh(1)/sshd(8): Added support for AES-GCM authenticated encryption in
   SSH protocol 2. The new cipher is available as aes128-gcm@openssh.com
   and aes256-gcm@openssh.com. It uses an identical packet format to the
   AES-GCM mode specified in RFC 5647, but uses simpler and different
   selection rules during key exchange.

 * ssh(1)/sshd(8): Added support for encrypt-then-mac (EtM) MAC modes
   for SSH protocol 2. These modes alter the packet format and compute
   the MAC over the packet length and encrypted packet rather than over
   the plaintext data. These modes are considered more secure and are
   used by default when available.

 * ssh(1)/sshd(8): Added support for the UMAC-128 MAC as
   "umac-128@openssh.com" and "umac-128-etm@openssh.com". The latter
   being an encrypt-then-mac mode.

 * sshd(8): Added support for multiple required authentication in SSH
   protocol 2 via an AuthenticationMethods option. This option lists
   one or more comma-separated lists of authentication method names.
   Successful completion of all the methods in any list is required for
   authentication to complete. This allows, for example, requiring a
   user having to authenticate via public key or GSSAPI before they
   are offered password authentication.

 * sshd(8)/ssh-keygen(1): Added support for Key Revocation Lists
   (KRLs), a compact binary format to represent lists of revoked keys
   and certificates that take as little as one bit per certificate when
   revoking by serial number. KRLs may be generated using ssh-keygen(1)
   and are loaded into sshd(8) via the existing RevokedKeys sshd_config
   option.

 * ssh(1): IdentitiesOnly now applies to keys obtained from a
   PKCS11Provider. This allows control of which keys are offered from
   tokens using IdentityFile.

 * sshd(8): sshd_config(5)'s AllowTcpForwarding now accepts "local"
   and "remote" in addition to its previous "yes"/"no" keywords to allow
   the server to specify whether just local or remote TCP forwarding is
   enabled.

 * sshd(8): Added a sshd_config(5) option AuthorizedKeysCommand to
   support fetching authorized_keys from a command in addition to (or
   instead of) from the filesystem. The command is run under an account
   specified by an AuthorizedKeysCommandUser sshd_config(5) option.

 * sftp-server(8): Now supports a -d option to allow the starting
   directory to be something other than the user's home directory.

 * ssh-keygen(1): Now allows fingerprinting of keys hosted in PKCS#11
   tokens using "ssh-keygen -lD pkcs11_provider".

 * ssh(1): When SSH protocol 2 only is selected (the default), ssh(1)
   now immediately sends its SSH protocol banner to the server without
   waiting to receive the server's banner, saving time when connecting.

 * ssh(1): Added ~v and ~V escape sequences to raise and lower the
   logging level respectively.

 * ssh(1): Made the escape command help (~?) context sensitive so that
   only commands that will work in the current session are shown.

 * ssh-keygen(1): When deleting host lines from known_hosts using
   "ssh-keygen -R host", ssh-keygen(1) now prints details of which lines
   were removed.
    
Bugfixes:

 * ssh(1): Force a clean shutdown of ControlMaster client sessions when
   the ~. escape sequence is used. This means that ~. should now work in
   mux clients even if the server is no longer responding.

 * ssh(1): Correctly detect errors during local TCP forward setup in
   multiplexed clients. bz#2055

 * ssh-add(1): Made deleting explicit keys "ssh-add -d" symmetric with
   adding keys with respect to certificates. It now tries to delete the
   corresponding certificate and respects the -k option to allow deleting
   of the key only.

 * sftp(1): Fix a number of parsing and command-editing bugs, including
   bz#1956

 * ssh(1): When muxmaster is run with -N, ensured that it shuts down
   gracefully when a client sends it "-O stop" rather than hanging around.
   bz#1985

 * ssh-keygen(1): When screening moduli candidates, append to the file
   rather than overwriting to allow resumption. bz#1957

 * ssh(1): Record "Received disconnect" messages at ERROR rather than
   INFO priority. bz#2057.

 * ssh(1): Loudly warn if explicitly-provided private key is unreadable.
   bz#1981

Portable OpenSSH:

 * sshd(8): The Linux seccomp-filter sandbox is now supported on ARM
   platforms where the kernel supports it.

 * sshd(8): The seccomp-filter sandbox will not be enabled if the system
   headers support it at compile time, regardless of whether it can be
   enabled then. If the run-time system does not support seccomp-filter,
   sshd will fall back to the rlimit pseudo-sandbox.

 * ssh(1): Don't link in the Kerberos libraries. They aren't necessary
   on the client, just on sshd(8). bz#2072

 * Fix GSSAPI linking on Solaris, which uses a differently-named GSSAPI
   library. bz#2073

 * Fix compilation on systems with openssl-1.0.0-fips.

 * Fix a number of errors in the RPM spec files.

Checksums:
==========

 - SHA1 (openssh-6.2.tar.gz) = b3f6cd774d345f22f6d0038cc9464cce131a0676
 - SHA1 (openssh-6.2p1.tar.gz) = 8824708c617cc781b2bb29fa20bd905fd3d2a43d

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 6.1 / 6.1p1 (2012-08-29) 

OpenSSH 6.1 was released on 2012-08-29. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Changes since OpenSSH 6.0
=========================

This is primarily a bugfix release.

Features:

 * sshd(8): This release turns on pre-auth sandboxing sshd by default for
   new installs, by setting UsePrivilegeSeparation=sandbox in sshd_config.
 * ssh-keygen(1): Add options to specify starting line number and number of
   lines to process when screening moduli candidates, allowing processing
   of different parts of a candidate moduli file in parallel
 * sshd(8): The Match directive now supports matching on the local (listen)
   address and port upon which the incoming connection was received via
   LocalAddress and LocalPort clauses.
 * sshd(8): Extend sshd_config Match directive to allow setting AcceptEnv
   and {Allow,Deny}{Users,Groups}
 * Add support for RFC6594 SSHFP DNS records for ECDSA key types. bz#1978
 * ssh-keygen(1): Allow conversion of RSA1 keys to public PEM and PKCS8
 * sshd(8): Allow the sshd_config PermitOpen directive to accept "none" as
   an argument to refuse all port-forwarding requests.
 * sshd(8): Support "none" as an argument for AuthorizedPrincipalsFile
 * ssh-keyscan(1): Look for ECDSA keys by default. bz#1971
 * sshd(8): Add "VersionAddendum" to sshd_config to allow server operators
   to append some arbitrary text to the server SSH protocol banner.

Bugfixes:

 * ssh(1)/sshd(8): Don't spin in accept() in situations of file
   descriptor exhaustion. Instead back off for a while.
 * ssh(1)/sshd(8): Remove hmac-sha2-256-96 and hmac-sha2-512-96 MACs as
   they were removed from the specification. bz#2023,
 * sshd(8): Handle long comments in config files better. bz#2025
 * ssh(1): Delay setting tty_flag so RequestTTY options are correctly
   picked up. bz#1995
 * sshd(8): Fix handling of /etc/nologin incorrectly being applied to root
   on platforms that use login_cap.

Portable OpenSSH:

 * sshd(8): Allow sshd pre-auth sandboxing to fall-back to the rlimit
   sandbox from the Linux SECCOMP filter sandbox when the latter is
   not available in the kernel.
 * ssh(1): Fix NULL dereference when built with LDNS and using DNSSEC to
   retrieve a CNAME SSHFP record.
 * Fix cross-compilation problems related to pkg-config. bz#1996

Checksums:
==========

 - SHA1 (openssh-6.1.tar.gz) = 7ed5b491cfebcaee2273d1f872314107273c2167
 - SHA1 (openssh-6.1p1.tar.gz) = 751c92c912310c3aa9cadc113e14458f843fc7b3

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 6.0 / 6.0p1 (2012-04-22) 

OpenSSH 6.0 was released on 2012-04-22. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Changes since OpenSSH 5.9
=========================

This is primarily a bugfix release.

Features:

 * ssh-keygen(1): Add optional checkpoints for moduli screening
 * ssh-add(1): new -k option to load plain keys (skipping certificates)
 * sshd(8): Add wildcard support to PermitOpen, allowing things like
   "PermitOpen localhost:*".  bz #1857
 * ssh(1): support for cancelling local and remote port forwards via the
   multiplex socket. Use ssh -O cancel -L xx:xx:xx -R yy:yy:yy user@host"
   to request the cancellation of the specified forwardings
 * support cancellation of local/dynamic forwardings from ~C commandline

Bugfixes:

 * ssh(1): ensure that $DISPLAY contains only valid characters before
   using it to extract xauth data so that it can't be used to play local
   shell metacharacter games.
 * ssh(1): unbreak remote portforwarding with dynamic allocated listen ports
 * scp(1): uppress adding '--' to remote commandlines when the first
   argument does not start with '-'. saves breakage on some
   difficult-to-upgrade embedded/router platforms
 * ssh(1)/sshd(8): fix typo in IPQoS parsing: there is no "AF14" class,
   but there is an "AF21" class
 * ssh(1)/sshd(8): do not permit SSH2_MSG_SERVICE_REQUEST/ACCEPT during
   rekeying
 * ssh(1): skip attempting to create ~/.ssh when -F is passed
 * sshd(8): unbreak stdio forwarding when ControlPersist is in use; bz#1943
 * sshd(1): send tty break to pty master instead of (probably already
   closed) slave side; bz#1859
 * sftp(1): silence error spam for "ls */foo" in directory with files;
   bz#1683
 * Fixed a number of memory and file descriptor leaks

Portable OpenSSH:

 * Add a new privilege separation sandbox implementation for Linux's
   new seccomp sandbox, automatically enabled on platforms that support
   it. (Note: privilege separation sandboxing is still experimental)
 * Fix compilation problems on FreeBSD, where libutil contained openpty()
   but not login().
 * ssh-keygen(1): don't fail in -A on platforms that don't support ECC
 * Add optional support for LDNS, a BSD licensed DNS resolver library
   which supports DNSSEC
 * Relax OpenSSL version check to allow running OpenSSH binaries on
   systems with OpenSSL libraries with a newer "fix" or "patch" level
   than the binaries were originally compiled on (previous check only
   allowed movement within "patch" releases). bz#1991
 * Fix builds using contributed Redhat spec file. bz#1992

Checksums:
==========

 - SHA1 (openssh-6.0.tar.gz) = 5d30aba0423c44e89924bb44c5d2153635506a9f
 - SHA1 (openssh-6.0p1.tar.gz) = f691e53ef83417031a2854b8b1b661c9c08e4422

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 5.9 / 5.9p1 (2011-09-06) 

OpenSSH 5.9 was released on 2011-09-06. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Changes since OpenSSH 5.8
=========================

Features:

 * Introduce sandboxing of the pre-auth privsep child using an optional
   sshd_config(5) "UsePrivilegeSeparation=sandbox" mode that enables
   mandatory restrictions on the syscalls the privsep child can perform.
   This intention is to prevent a compromised privsep child from being
   used to attack other hosts (by opening sockets and proxying) or
   probing local kernel attack surface.
    
   Three concrete sandbox implementation are provided (selected at
   configure time): systrace, seatbelt and rlimit.

   The systrace sandbox uses systrace(4) in unsupervised "fast-path"
   mode, where a list of permitted syscalls is supplied. Any syscall not
   on the list results in SIGKILL being sent to the privsep child. Note
   that this requires a kernel with the new SYSTR_POLICY_KILL option
   (only OpenBSD has this mode at present).
 
   The seatbelt sandbox uses OS X/Darwin sandbox(7) facilities with a
   strict (kSBXProfilePureComputation) policy that disables access to
   filesystem and network resources.

   The rlimit sandbox is a fallback choice for platforms that don't
   support a better one; it uses setrlimit() to reset the hard-limit
   of file descriptors and processes to zero, which should prevent
   the privsep child from forking or opening new network connections.

   Sandboxing of the privilege separated child process is currently
   experimental but should become the default in a future release.
   Native sandboxes for other platforms are welcome (e.g. Capsicum,
   Linux pid/net namespaces, etc.)

 * Add new SHA256-based HMAC transport integrity modes from
   http://www.ietf.org/id/draft-dbider-sha2-mac-for-ssh-02.txt
   These modes are hmac-sha2-256, hmac-sha2-256-96, hmac-sha2-512,
   and hmac-sha2-512-96, and are available by default in ssh(1) and
   sshd(8)

 * The pre-authentication sshd(8) privilege separation slave process
   now logs via a socket shared with the master process, avoiding the
   need to maintain /dev/log inside the chroot.

 * ssh(1) now warns when a server refuses X11 forwarding

 * sshd_config(5)'s AuthorizedKeysFile now accepts multiple paths,
   separated by whitespace. The undocumented AuthorizedKeysFile2
   option is deprecated (though the default for AuthorizedKeysFile
   includes .ssh/authorized_keys2)

 * sshd_config(5): similarly deprecate UserKnownHostsFile2 and
   GlobalKnownHostsFile2 by making UserKnownHostsFile and
   GlobalKnownHostsFile accept multiple options and default to
   include known_hosts2

 * Retain key comments when loading v.2 keys. These will be visible
   in "ssh-add -l" and other places. bz#439

 * ssh(1) and sshd(8): set IPv6 traffic class from IPQoS (as well as
   IPv4 ToS/DSCP). bz#1855

 * ssh_config(5)'s ControlPath option now expands %L to the host
   portion of the destination host name.

 * ssh_config(5) "Host" options now support negated Host matching, e.g.
     
     Host *.example.org !c.example.org
        User mekmitasdigoat
     
   Will match "a.example.org", "b.example.org", but not "c.example.org"

 * ssh_config(5): a new RequestTTY option provides control over when a
   TTY is requested for a connection, similar to the existing -t/-tt/-T
   ssh(1) commandline options.

 * sshd(8): allow GSSAPI authentication to detect when a server-side
   failure causes authentication failure and don't count such failures
   against MaxAuthTries; bz#1244

 * ssh-keygen(1): Add -A option. For each of the key types (rsa1, rsa,
   dsa and ecdsa) for which host keys do not exist, generate the host
   keys with the default key file path, an empty passphrase, default
   bits for the key type, and default comment. This is useful for
   system initialisation scripts.

 * ssh(1): Allow graceful shutdown of multiplexing: request that a mux
   server removes its listener socket and refuse future multiplexing
   requests but don't kill existing connections. This may be requested
   using "ssh -O stop ..."

 * ssh-add(1) now accepts keys piped from standard input. E.g.
   "ssh-add - < /path/to/key"
 
 * ssh-keysign(8) now signs hostbased authentication
   challenges correctly using ECDSA keys; bz#1858

 * sftp(1): document that sftp accepts square brackets to delimit
   addresses (useful for IPv6); bz#1847a

 * ssh(1): when using session multiplexing, the master process will
   change its process title to reflect the control path in use and
   when a ControlPersist-ed master is waiting to close; bz#1883 and
   bz#1911

 * Other minor bugs fixed: 1849 1861 1862 1869 1875 1878 1879 1892
   1900 1905 1913

Portable OpenSSH Bugfixes:

 * Fix a compilation error in the SELinux support code. bz#1851

 * This release removes support for ssh-rand-helper. OpenSSH now
   obtains its random numbers directly from OpenSSL or from
   a PRNGd/EGD instance specified at configure time.

 * sshd(8) now resets the SELinux process execution context before
   executing passwd for password changes; bz#1891

 * Since gcc >= 4.x ignores all -Wno-options options, test only the
   corresponding -W-option when trying to determine whether it is
   accepted; bz#1901

 * Add ECDSA key generation to the Cygwin ssh-{host,user}-config
   scripts.

 * Updated .spec and init files for Linux; bz#1920

 * Improved SELinux error messages in context change failures and
   suppress error messages when attempting to change from the
   "unconfined_t" type; bz#1924 bz#1919

 * Fix build errors on platforms without dlopen(); bz#1929

Checksums:
==========

 - SHA1 (openssh-5.9.tar.gz) = bc0cb728bbc394769f9a2ce5b8cd99dc41e12632
 - SHA1 (openssh-5.9p1.tar.gz) = ac4e0055421e9543f0af5da607a72cf5922dcc56

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 5.8p2 (2011-05-03) 

Portable OpenSSH 5.8p2 was released on 2011-05-03. It will be available
from the mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Changes since OpenSSH 5.8p1
===========================

Security:

 * Fix local private host key compromise on platforms without host-
   level randomness support (e.g. /dev/random) reported by Tomas Mraz

   On hosts that did not have a randomness source configured in
   OpenSSL and were not configured to use EGD/PRNGd (using the
   --with-prngd-socket configure option), the ssh-rand-helper command
   was being implicitly executed by ssh-keysign with open file
   descriptors to the host private keys. An attacker could use
   ptrace(2) to attach to ssh-rand-helper and exfiltrate the keys.

   Most modern operating systems are not vulnerable. In particular,
   *BSD, Linux, OS X and Cygwin do not use ssh-rand-helper.

   A full advisory for this issue is available at:
   http://www.openssh.com/txt/portable-keysign-rand-helper.adv

Portable OpenSSH Bugfixes:

 * Fix compilation failure when enabling SELinux support.

 * Revised Cygwin ssh-{host,user}-config that include ECDSA key
   support.

 * Revised Cygwin ssh-host-config to be more thorough in error checking
   and reporting.

Checksums:
==========

 - SHA1 (openssh-5.8p2.tar.gz) = 64798328d310e4f06c9f01228107520adbc8b3e5

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 5.8 / 5.8p1 (2011-02-04) 

OpenSSH 5.8 was released on 2011-02-04. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Changes since OpenSSH 5.7
=========================

Security:

 * Fix vulnerability in legacy certificate signing introduced in
   OpenSSH-5.6 and found by Mateusz Kocielski.

   Legacy certificates signed by OpenSSH 5.6 or 5.7 included data from
   the stack in place of a random nonce field. The contents of the stack
   do not appear to contain private data at this point, but this cannot
   be stated with certainty for all platform, library and compiler
   combinations. In particular, there exists a risk that some bytes from
   the privileged CA key may be accidentally included.

   A full advisory for this issue is available at:
   http://www.openssh.com/txt/legacy-cert.adv

Portable OpenSSH Bugfixes:

 * Fix compilation failure when enabling SELinux support.

 * Do not attempt to call SELinux functions when SELinux is disabled.
   bz#1851

Checksums:
==========

 - SHA1 (openssh-5.8.tar.gz) = 205dece2c8b41c69b082eb65320d359987aae25b
 - SHA1 (openssh-5.8p1.tar.gz) = adebb2faa9aba2a3a3c8b401b2b19677ab53f0de

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 5.7 / 5.7p1 (2011-01-24) 

OpenSSH 5.7 was released on 2011-01-24. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Changes since OpenSSH 5.6
=========================

Features:

 * Implement Elliptic Curve Cryptography modes for key exchange (ECDH)
   and host/user keys (ECDSA) as specified by RFC5656. ECDH and ECDSA
   offer better performance than plain DH and DSA at the same equivalent
   symmetric key length, as well as much shorter keys.
     
   Only the mandatory sections of RFC5656 are implemented, specifically
   the three REQUIRED curves nistp256, nistp384 and nistp521 and only
   ECDH and ECDSA. Point compression (optional in RFC5656) is NOT
   implemented.
     
   Certificate host and user keys using the new ECDSA key types are
   supported - an ECDSA key may be certified, and an ECDSA key may act
   as a CA to sign certificates.

   ECDH in a 256 bit curve field is the preferred key agreement
   algorithm when both the client and server support it. ECDSA host
   keys are preferred when learning a host's keys for the first time,
   or can be learned using ssh-keyscan(1).
     
 * sftp(1)/sftp-server(8): add a protocol extension to support a hard
   link operation. It is available through the "ln" command in the
   client. The old "ln" behaviour of creating a symlink is available
   using its "-s" option or through the preexisting "symlink" command

 * scp(1): Add a new -3 option to scp: Copies between two remote hosts
   are transferred through the local host.  Without this option the
   data is copied directly between the two remote hosts. 

 * ssh(1): automatically order the hostkeys requested by the client
   based on which hostkeys are already recorded in known_hosts. This
   avoids hostkey warnings when connecting to servers with new ECDSA
   keys, since these are now preferred when learning hostkeys for the
   first time.

 * ssh(1)/sshd(8): add a new IPQoS option to specify arbitrary
   TOS/DSCP/QoS values instead of hardcoding lowdelay/throughput.
   bz#1733

 * sftp(1): the sftp client is now significantly faster at performing
   directory listings, using OpenBSD glob(3) extensions to preserve
   the results of stat(3) operations performed in the course of its
   execution rather than performing expensive round trips to fetch
   them again afterwards.

 * ssh(1): "atomically" create the listening mux socket by binding it on
   a temporary name and then linking it into position after listen() has
   succeeded. This allows the mux clients to determine that the server
   socket is either ready or stale without races. stale server sockets
   are now automatically removed. (also fixes bz#1711)

 * ssh(1)/sshd(8): add a KexAlgorithms knob to the client and server
   configuration to allow selection of which key exchange methods are
   used by ssh(1) and sshd(8) and their order of preference.

 * sftp(1)/scp(1): factor out bandwidth limiting code from scp(1) into
   a generic bandwidth limiter that can be attached using the atomicio
   callback mechanism and use it to add a bandwidth limit option to
   sftp(1). bz#1147
 
BugFixes:

 * ssh(1)/ssh-agent(1): honour $TMPDIR for client xauth and ssh-agent
   temporary directories. bz#1809

 * ssh(1): avoid NULL deref on receiving a channel request on an unknown
   or invalid channel; bz#1842

 * sshd(8): remove a debug() that pollutes stderr on client connecting
   to a server in debug mode; bz#1719

 * scp(1): pass through ssh command-line flags and options when doing
   remote-remote transfers, e.g. to enable agent forwarding which is
   particularly useful in this case; bz#1837

 * sftp-server(8): umask should be parsed as octal

 * sftp(1): escape '[' in filename tab-completion

 * ssh(1): Typo in confirmation message.  bz#1827

 * sshd(8): prevent free() of string in .rodata when overriding
   AuthorizedKeys in a Match block

 * sshd(8): Use default shell /bin/sh if $SHELL is ""

 * ssh(1): kill proxy command on fatal() (we already killed it on
   clean exit);

 * ssh(1): install a SIGCHLD handler to reap expired child process;
   bz#1812

 * Support building against openssl-1.0.0a

Portable OpenSSH Bugfixes:

 * Use mandoc as preferred manpage formatter if it is present, followed
   by nroff and groff respectively.

 * sshd(8): Relax permission requirement on btmp logs to allow group
   read/write

 * bz#1840: fix warning when configuring --with-ssl-engine

 * sshd(8): Use correct uid_t/pid_t types instead of int. bz#1817

 * sshd(8): bz#1824: Add Solaris Project support.

 * sshd(8): Check is_selinux_enabled for exact return code since it can
   apparently return -1 under some conditions.

Checksums:
==========

 - SHA1 (openssh-5.7.tar.gz) = 67cb91772a33fb3a004b39bcdb9148218365494c
 - SHA1 (openssh-5.7p1.tar.gz) = 423e27475f06e1055847dfff7f61e1ac632b5372

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 5.6 / 5.6p1 (2010-08-23) 

OpenSSH 5.6 was released on 2010-08-23. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Changes since OpenSSH 5.5
=========================

Features:

 * Added a ControlPersist option to ssh_config(5) that automatically
   starts a background ssh(1) multiplex master when connecting. This
   connection can stay alive indefinitely, or can be set to
   automatically close after a user-specified duration of inactivity.

 * Hostbased authentication may now use certificate host keys. CA keys
   must be specified in a known_hosts file using the @cert-authority
   marker as described in sshd(8).

 * ssh-keygen(1) now supports signing certificate using a CA key that
   has been stored in a PKCS#11 token.

 * ssh(1) will now log the hostname and address that we connected to at
   LogLevel=verbose after authentication is successful to mitigate
   "phishing" attacks by servers with trusted keys that accept
   authentication silently and automatically before presenting fake
   password/passphrase prompts.

   Note that, for such an attack to be successful, the user must have
   disabled StrictHostKeyChecking (enabled by default) or an attacker
   must have access to a trusted host key for the destination server.

 * Expand %h to the hostname in ssh_config Hostname options. While this
   sounds useless, it is actually handy for working with unqualified
   hostnames:
     
     Host *.*
        Hostname %h
     Host *
        Hostname %h.example.org
     
 * Allow ssh-keygen(1) to import (-i) and export (-e) of PEM and PKCS#8
   keys in addition to RFC4716 (SSH.COM) encodings via a new -m option 
   (bz#1749)

 * sshd(8) will now queue debug messages for bad ownership or
   permissions on the user's keyfiles encountered during authentication
   and will send them after authentication has successfully completed.
   These messages may be viewed in ssh(1) at LogLevel=debug or higher.

 * ssh(1) connection multiplexing now supports remote forwarding with
   dynamic port allocation and can report the allocated port back to
   the user:

     LPORT=`ssh -S muxsocket -R0:localhost:25 -O forward somehost`

 * sshd(8) now supports indirection in matching of principal names
   listed in certificates. By default, if a certificate has an
   embedded principals list then the username on the server must match
   one of the names in the list for it to be accepted for
   authentication.

   sshd(8) now has a new AuthorizedPrincipalsFile option to specify a
   file containing a list of names that may be accepted in place of the
   username when authorizing a certificate trusted via the
   sshd_config(5) TrustedCAKeys option. Similarly, authentication
   using a CA trusted in ~/.ssh/authorized_keys now accepts a
   principals="name1[,name2,...]" to specify a list of permitted names.
     
   If either option is absent, the current behaviour of requiring the
   username to appear in principals continues to apply. These options
   are useful for role accounts, disjoint account namespaces and
   "user@realm"-style naming policies in certificates.
 
 * Additional sshd_config(5) options are now valid inside Match blocks:

     AuthorizedKeysFile
     AuthorizedPrincipalsFile
     HostbasedUsesNameFromPacketOnly
     PermitTunnel

 * Revised the format of certificate keys. The new format, identified as
   ssh-{dss,rsa}-cert-v01@openssh.com includes the following changes:
     
     - Adding a serial number field. This may be specified by the CA at
       the time of certificate signing.

     - Moving the nonce field to the beginning of the certificate where
       it can better protect against chosen-prefix attacks on the
       signature hash (currently infeasible against the SHA1 hash used)
     
     - Renaming the "constraints" field to "critical options"
     
     - Addng a new non-critical "extensions" field. The "permit-*"
       options are now extensions, rather than critical options to
       permit non-OpenSSH implementation of this key format to degrade
       gracefully when encountering keys with options they do not
       recognize.
     
   The older format is still supported for authentication and may still
   be used when signing certificates (use "ssh-keygen -t v00 ...").
   The v00 format, introduced in OpenSSH 5.4, will be supported for at
   least one year from this release, after which it will be deprecated
   and removed.
     
BugFixes:

 * The PKCS#11 code now retries a lookup for a private key if there is
   no matching key with CKA_SIGN attribute enabled; this fixes
   MuscleCard support (bz#1736)
    
 * Unbreak strdelim() skipping past quoted strings (bz#1757). For
   example, the following directive was not parsed correctly:

       AllowUsers "blah blah" blah

 * sftp(1): fix swapped args in upload_dir_internal(), breaking
   recursive upload depth checks and causing verbose printing of
   transfers to always be turned on (bz#1797)

 * Fix a longstanding problem where if you suspend scp(1) at the
   password/passphrase prompt the terminal mode is not restored.

 * Fix a PKCS#11 crash on some smartcards by validating the length
   returned for C_GetAttributValue (bz#1773)

 * sftp(1): fix ls in working directories that contain globbing
   characters in their pathnames (bz#1655)

 * Print warning for missing home directory when ChrootDirectory=none
   (bz#1564)

 * sftp(1): fix a memory leak in do_realpath() error path (bz#1771)

 * ssk-keygen(1): Standardise error messages when attempting to open
   private key files to include "progname: filename: error reason"
   (bz#1783)

 * Replace verbose and overflow-prone Linebuf code with
   read_keyfile_line() (bz#1565)

 * Include the user name on "subsystem request for ..." log messages

 * ssh(1) and sshd(8): remove hardcoded limit of 100 permitopen clauses
   and port forwards per direction (bz#1327)

 * sshd(8): ignore stderr output from subsystems to avoid hangs if a
   subsystem or shell initialisation writes to stderr (bz#1750)

 * Skip the initial check for access with an empty password when
   PermitEmptyPasswords=no (bz#1638)

 * sshd(8): fix logspam when key options (from="..." especially) deny
   non-matching keys (bz#1765)

 * ssh-keygen(1): display a more helpful error message when $HOME is
   inaccessible while trying to create .ssh directory (bz#1740)

 * ssh(1): fix hang when terminating a mux slave using ~. (bz#1758)

 * ssh-keygen(1): refuse to generate keys longer than
   OPENSSL_[RD]SA_MAX_MODULUS_BITS, since we would refuse to use
   them anyway (bz#1516)

 * Suppress spurious tty warning when using -O and stdin is not a tty
   (bz#1746)

 * Kill channel when pty allocation requests fail. Fixed stuck client
   if the server refuses pty allocation (bz#1698)

Portable OpenSSH Bugfixes:

 * sshd(8): increase the maximum username length for login recording
   to 512 characters (bz#1579)

 * Initialize the values to be returned from PAM to sane values in
   case the PAM method doesn't write to them. (bz#1795) 

 * Let configure find OpenSSL libraries in a lib64 subdirectory.
   (bz#1756)

Checksums:
==========

 - SHA1 (openssh-5.6.tar.gz) = fa5ac394b874d6709031306b6ac5c48399697f7f
 - SHA1 (openssh-5.6p1.tar.gz) = 347dd39c91c3529f41dae63714d452fb95efea1e

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 5.5 / 5.5p1 (2010-04-16) 

OpenSSH 5.5 was released on 2010-04-16. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed code
or patches, reported bugs, tested snapshots or donated to the project.
More information on donations may be found at:
http://www.openssh.com/donations.html

This is a bugfix release.

Changes since OpenSSH 5.4
=========================

 * Unbreak sshd_config's AuthorizedKeysFile option for $HOME-relative paths

 * Fix compilation failures on platforms that lack dlopen()

 * Include a language tag when sending a protocol 2 disconnection message.

 * Make logging of certificates used for user authentication more clear and
   consistent between CAs specified using TrustedUserCAKeys and
   authorized_keys

Portable OpenSSH:

 * Allow contrib/ssh-copy-id to fail gracefully when there are no keys in
   the ssh-agent. bz#1723

 * Explicitly link libX11 into contrib/gnome-ssh-askpass2. bz#1725

 * Allow ChrootDirectory to work in SELinux platforms. bz#1726

 * Add configure.ac stanza for Haiku OS. bz#1741

 * Enable utmpx support on FreeBSD where possible. bz#1732

 * Use pkg-config to determine libedit linker flags where possible. bz#1744

Checksums:
==========

 - SHA1 (openssh-5.5.tar.gz) = 59864a048b09ad1b6e65a74d5d385d8189ab8c74
 - SHA1 (openssh-5.5p1.tar.gz) = 361c6335e74809b26ea096b34062ba8ff6c97cd6

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 5.4 / 5.4p1 (2010-03-08) 

OpenSSH 5.4 was released on 2010-03-08. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed code
or patches, reported bugs, tested snapshots or donated to the project.
More information on donations may be found at:
http://www.openssh.com/donations.html

This is a major feature and bugfix release.

Changes since OpenSSH 5.3
=========================

Features:

 * After a transition period of about 10 years, this release disables
   SSH protocol 1 by default. Clients and servers that need to use the
   legacy protocol must explicitly enable it in ssh_config / sshd_config
   or on the command-line.

 * Remove the libsectok/OpenSC-based smartcard code and add support for
   PKCS#11 tokens. This support is automatically enabled on all
   platforms that support dlopen(3) and was inspired by patches written
   by Alon Bar-Lev. Details in the ssh(1) and ssh-add(1) manpages.

 * Add support for certificate authentication of users and hosts using a
   new, minimal OpenSSH certificate format (not X.509). Certificates
   contain a public key, identity information and some validity
   constraints and are signed with a standard SSH public key using
   ssh-keygen(1). CA keys may be marked as trusted in authorized_keys
   or via a TrustedUserCAKeys option in sshd_config(5) (for user
   authentication), or in known_hosts (for host authentication).

   Documentation for certificate support may be found in ssh-keygen(1),
   sshd(8) and ssh(1) and a description of the protocol extensions in
   PROTOCOL.certkeys.

 * Added a 'netcat mode' to ssh(1): "ssh -W host:port ..." This connects
   stdio on the client to a single port forward on the server. This
   allows, for example, using ssh as a ProxyCommand to route connections
   via intermediate servers. bz#1618

 * Add the ability to revoke keys in sshd(8) and ssh(1). User keys may
   be revoked using a new sshd_config(5) option "RevokedKeys". Host keys
   are revoked through known_hosts (details in the sshd(8) man page).
   Revoked keys cannot be used for user or host authentication and will
   trigger a warning if used.

 * Rewrite the ssh(1) multiplexing support to support non-blocking
   operation of the mux master, improve the resilience of the master to
   malformed messages sent to it by the slave and add support for
   requesting port- forwardings via the multiplex protocol. The new
   stdio-to-local forward mode ("ssh -W host:port ...") is also
   supported. The revised multiplexing protocol is documented in the
   file PROTOCOL.mux in the source distribution.

 * Add a 'read-only' mode to sftp-server(8) that disables open in write
   mode and all other fs-modifying protocol methods. bz#430

 * Allow setting an explicit umask on the sftp-server(8) commandline to
   override whatever default the user has. bz#1229

 * Many improvements to the sftp(1) client, many of which were
   implemented by Carlos Silva through the Google Summer of Code
   program:
   - Support the "-h" (human-readable units) flag for ls
   - Implement tab-completion of commands, local and remote filenames
   - Support most of scp(1)'s commandline arguments in sftp(1), as a
     first step towards making sftp(1) a drop-in replacement for scp(1).
     Note that the rarely-used "-P sftp_server_path" option has been
     moved to "-D sftp_server_path" to make way for "-P port" to match
     scp(1).
   - Add recursive transfer support for get/put and on the commandline

 * New RSA keys will be generated with a public exponent of RSA_F4 ==
   (2**16)+1 == 65537 instead of the previous value 35.

 * Passphrase-protected SSH protocol 2 private keys are now protected
   with AES-128 instead of 3DES. This applied to newly-generated keys
   as well as keys that are reencrypted (e.g. by changing their
   passphrase).

Bugfixes:

 * Hold authentication debug messages until after successful
   authentication. Fixes a minor information leak of environment
   variables specified in authorized_keys if an attacker happens to
   know the public key in use.
 * When using ChrootDirectory, make sure we test for the existence of
   the user's shell inside the chroot and not outside (bz#1679)
 * Cache user and group name lookups in sftp-server using
   user_from_[ug]id(3) to improve performance on hosts where these
   operations are slow (e.g. NIS or LDAP). bz#1495
 * Fix problem that prevented passphrase reading from being interrupted
   in some circumstances; bz#1590
 * Ignore and log any Protocol 1 keys where the claimed size is not
   equal to the actual size.
 * Make HostBased authentication work with a ProxyCommand. bz#1569
 * Avoid run-time failures when specifying hostkeys via a relative
   path by prepending the current working directory in these cases.
   bz#1290
 * Do not prompt for a passphrase if we fail to open a keyfile, and log
   the reason why the open failed to debug. bz#1693
 * Document that the PubkeyAuthentication directive is allowed in a
   sshd_config(5) Match block. bz#1577
 * When converting keys, truncate key comments at 72 chars as per
   RFC4716. bz#1630
 * Do not allow logins if /etc/nologin exists but is not readable by the
   user logging in.
 * Output a debug log if sshd(8) can't open an existing authorized_keys.
   bz#1694
 * Quell tc[gs]etattr warnings when forcing a tty (ssh -tt), since we
   usually don't actually have a tty to read/set; bz#1686
 * Prevent sftp from crashing when given a "-" without a command.
   Also, allow whitespace to follow a "-". bz#1691
 * After sshd receives a SIGHUP, ignore subsequent HUPs while sshd
   re-execs itself. Prevents two HUPs in quick succession from resulting
   in sshd dying. bz#1692
 * Clarify in sshd_config(5) that StrictModes does not apply to
   ChrootDirectory. Permissions and ownership are always checked when
   chrooting. bz#1532
 * Set close-on-exec on various descriptors so they don't get leaked to
   child processes. bz#1643
 * Fix very rare race condition in x11/agent channel allocation: don't
   read after the end of the select read/write fdset and make sure a
   reused FD is not touched before the pre-handlers are called.
 * Fix incorrect exit status when multiplexing and channel ID 0 is
   recycled. bz#1570
 * Fail with an error when an attempt is made to connect to a server
   with ForceCommand=internal-sftp with a shell session (i.e. not a
   subsystem session). Avoids stuck client when attempting to ssh to
   such a service. bz#1606:
 * Warn but do not fail if stat()ing the subsystem binary fails. This
   helps with chrootdirectory+forcecommand=sftp-server and restricted
   shells. bz #1599
 * Change "Connecting to host..." message to "Connected to host."
   and delay it until after the sftp protocol connection has been
   established. Avoids confusing sequence of messages when the
   underlying ssh connection experiences problems. bz#1588
 * Use the HostKeyAlias rather than the hostname specified on the
   commandline when prompting for passwords. bz#1039
 * Correct off-by-one in percent_expand(): we would fatal() when trying
   to expand EXPAND_MAX_KEYS, allowing only EXPAND_MAX_KEYS-1 to
   actually work. Note that nothing in OpenSSH actually uses close to
   this limit at present. bz#1607
 * Fix passing of empty options from scp(1) and sftp(1) to the
   underlying ssh(1). Also add support for the stop option "--".
 * Fix an incorrect magic number and typo in PROTOCOL; bz#1688
 * Don't escape backslashes when displaying the SSH2 banner. bz#1533
 * Don't unnecessarily dup() the in and out fds for sftp-server. bz#1566
 * Force use of the correct hash function for random-art signature
   display as it was inheriting the wrong one when bubblebabble
   signatures were activated. bz#1611
 * Do not fall back to adding keys without constraints (ssh-add -c /
   -t ...) when the agent refuses the constrained add request. bz#1612
 * Fix a race condition in ssh-agent that could result in a wedged or
   spinning agent. bz#1633
 * Flush stdio before exec() to ensure that everying (motd
   in particular) has made it out before the streams go away. bz#1596
 * Set FD_CLOEXEC on in/out sockets in sshd(8). bz#1706

Portable OpenSSH Bugfixes:

 * Use system's kerberos principal name on AIX if it's available.
   bz#1583
 * Disable OOM-killing of the listening sshd on Linux. bz#1470
 * Use pkg-config for opensc config if it's available. bz#1160
 * Unbreak Redhat spec to allow building without askpass. bz#1677
 * If PidFile is set in sshd_config, use it in SMF init file. bz#1628
 * Print error and usage() when ssh-rand-helper is passed command-
   line arguments as none are supported. bz#1568
 * Add missing setsockopt() to set IPV6_V6ONLY for local forwarding
   with GatwayPorts=yes. bz#1648
 * Make GNOME 2 askpass dialog desktop-modal. bz#1645
 * If SELinux is enabled set the security context to "sftpd_t" before
   running the internal sftp server. bz#1637
 * Correctly check libselinux for necessary SELinux functions; bz#1713
 * Unbreak builds on Redhat using the supplied openssh.spec; bz#1731
 * Fix incorrect privilege dropping order on AIX that prevented
   chroot operation; bz#1567
 * Call aix_setauthdb/aix_restoredb at the correct times on AIX to
   prevent authentication failure; bz#1710

Checksums:
==========

 - SHA1 (openssh-5.4.tar.gz) = 1776832d902f7b4c7863afd41a5ec7a14efe95d6
 - SHA1 (openssh-5.4p1.tar.gz) = 2a3042372f08afb1415ceaec8178213276a36302

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 5.3 / 5.3p1 (2009-10-01) 

OpenSSH 5.3 was released on 2009-10-01. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

This release marks the 10th anniversary of the OpenSSH project.
We would like to thank the OpenSSH community for their support,
especially those who will continue to contribute code or patches,
report bugs, test snapshots or donate to the project during the
next 10 years.  More information on donations may be found at:
http://www.openssh.com/donations.html

This is a bugfix release, no new features have been added.

Changes since OpenSSH 5.2
=========================

General Bugfixes:

 * Do not limit home directory paths to 256 characters. bz#1615

 * Several minor documentation and correctness fixes.

Portable OpenSSH Bugfixes:

 * This release removes for support for very old versions of Cygwin and
   for Windows 95/98/ME

 * Move the deletion of PAM credentials on logout to after the session
   close. bz#1534

 * Make PrintLastLog work on AIX. bz#1595

 * Avoid compile errors on FreeBSD from conflicts in glob.h. bz#1634

 * Delay dropping of root privileges on AIX so chroot and pam_open_session
   work correctly. bz#1249 and bz#1567

 * Increase client IO buffer on Cygwin to 64K, realising a significant
   performance improvement.
 
 * Roll back bz#1241 (better handling for expired passwords on Tru64).
   The change broke password logins on some configurations.

 * Accept ENOSYS as a fallback error when attempting atomic
   rename(). bz#1535

 * Fix passing of variables to recursive make(1) invocations on Solaris.
   bz#1505

 * Skip the tcgetattr call on the pty master on Solaris, since it never
   succeeds and can hang if large amounts of data is sent to the slave
   (eg a copy-paste). bz#1528 

 * Fix detection of krb5-config. bz#1639

 * Fix test for server-assigned remote forwarding port for non-root users.
   bz#1578

 * Fix detection of libresolv on OSX 10.6.

Checksums:
==========

 - SHA1 (openssh-5.3.tar.gz) = f1b9a280565e916c1f84fd4d944313ec926242a2
 - SHA1 (openssh-5.3p1.tar.gz) = d411fde2584ef6022187f565360b2c63a05602b5

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 5.2 / 5.2p1 (2009-02-23) 

OpenSSH 5.2 was released on 2009-02-23. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We have also recently completed another Internet SSH usage scan, the 
results of which may be found at http://www.openssh.com/usage.html

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

The focus of this release has been on bugfixes as the previous
openssh-5.1 release introduced many new features and made some
invasive changes.

Changes since OpenSSH 5.1
=========================

Security:

 * This release changes the default cipher order to prefer the AES CTR
   modes and the revised "arcfour256" mode to CBC mode ciphers that are
   susceptible to CPNI-957037 "Plaintext Recovery Attack Against SSH".

 * This release also adds countermeasures to mitigate CPNI-957037-style
   attacks against the SSH protocol's use of CBC-mode ciphers. Upon
   detection of an invalid packet length or Message Authentication
   Code, ssh/sshd will continue reading up to the maximum supported
   packet length rather than immediately terminating the connection.
   This eliminates most of the known differences in behaviour that
   leaked information about the plaintext of injected data which formed
   the basis of this attack. We believe that these attacks are rendered
   infeasible by these changes.

New features:

 * Added a -y option to ssh(1) to force logging to syslog rather than
   stderr, which is useful when running daemonised (ssh -f)

 * The sshd_config(5) ForceCommand directive now accepts commandline
   arguments for the internal-sftp server.

 * The ssh(1) ~C escape commandline now support runtime creation of
   dynamic (-D) port forwards.

 * Support the SOCKS4A protocol in ssh(1) dynamic (-D) forwards.
   (bz#1482)

 * Support remote port forwarding with a listen port of '0'. This
   informs the server that it should dynamically allocate a listen
   port and report it back to the client. (bz#1003)

 * sshd(8) now supports setting PermitEmptyPasswords and
   AllowAgentForwarding in Match blocks

Bug and documentation fixes

 * Repair a ssh(1) crash introduced in openssh-5.1 when the client is
   sent a zero-length banner (bz#1496)

 * Due to interoperability problems with certain
   broken SSH implementations, the eow@openssh.com and
   no-more-sessions@openssh.com protocol extensions are now only sent
   to peers that identify themselves as OpenSSH.

 * Make ssh(1) send the correct channel number for
   SSH2_MSG_CHANNEL_SUCCESS and SSH2_MSG_CHANNEL_FAILURE messages to
   avoid triggering 'Non-public channel' error messages on sshd(8) in
   openssh-5.1.

 * Avoid printing 'Non-public channel' warnings in sshd(8), since the
   ssh(1) has sent incorrect channel numbers since ~2004 (this reverts
   a behaviour introduced in openssh-5.1).

 * Avoid double-free in ssh(1) ~C escape -L handler (bz#1539)

 * Correct fail-on-error behaviour in sftp(1) batchmode for remote
   stat operations. (bz#1541)

 * Disable nonfunctional ssh(1) ~C escape handler in multiplex slave
   connections. (bz#1543)

 * Avoid hang in ssh(1) when attempting to connect to a server that
   has MaxSessions=0 set.

 * Multiple fixes to sshd(8) configuration test (-T) mode

 * Several core and portable OpenSSH bugs fixed: 1380, 1412, 1418,
   1419, 1421, 1490, 1491, 1492, 1514, 1515, 1518, 1520, 1538, 1540

 * Many manual page improvements.

Checksums:
==========

 - SHA1 (openssh-5.2.tar.gz) = 260074ed466e95f054ac05a4406f613d08575217
 - SHA1 (openssh-5.2p1.tar.gz) = 8273a0237db98179fbdc412207ff8eb14ff3d6de

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 5.1 / 5.1p1 (2008-07-22) 

OpenSSH 5.1 was released on 2008-07-22. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We have also recently completed another Internet SSH usage scan, the 
results of which may be found at http://www.openssh.com/usage.html

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots or donated to the
project. More information on donations may be found at:
http://www.openssh.com/donations.html

Changes since OpenSSH 5.0
=========================

Security:

 * sshd(8): Avoid X11 man-in-the-middle attack on HP/UX (and possibly
   other platforms) when X11UseLocalhost=no

   When attempting to bind(2) to a port that has previously been bound
   with SO_REUSEADDR set, most operating systems check that either the
   effective user-id matches the previous bind (common on BSD-derived
   systems) or that the bind addresses do not overlap (Linux and
   Solaris).

   Some operating systems, such as HP/UX, do not perform these checks
   and are vulnerable to an X11 man-in-the-middle attack when the
   sshd_config(5) option X11UseLocalhost has been set to "no" - an
   attacker may establish a more-specific bind, which will be used in
   preference to sshd's wildcard listener.

   Modern BSD operating systems, Linux, OS X and Solaris implement the
   above checks and are not vulnerable to this attack, nor are systems
   where the X11UseLocalhost has been left at the default value of
   "yes".

   Portable OpenSSH 5.1 avoids this problem for all operating systems
   by not setting SO_REUSEADDR when X11UseLocalhost is set to no.

   This vulnerability was reported by sway2004009 AT hotmail.com.

New features:

 * Introduce experimental SSH Fingerprint ASCII Visualisation to ssh(1)
   and ssh-keygen(1). Visual fingerprinnt display is controlled by a new
   ssh_config(5) option "VisualHostKey". The intent is to render
   SSH host keys in a visual form that is amenable to easy recall and
   rejection of changed host keys. This technique inspired by the
   graphical hash visualisation schemes known as "random art[*]", and
   by Dan Kaminsky's musings at 23C3 in Berlin.

   Fingerprint visualisation in is currently disabled by default, as the
   algorithm used to generate the random art is still subject to change.

   [*] "Hash Visualization: a New Technique to improve Real-World
       Security", Perrig A. and Song D., 1999, International Workshop on
       Cryptographic Techniques and E-Commerce (CrypTEC '99)
   http://sparrow.ece.cmu.edu/~adrian/projects/validation/validation.pdf

 * sshd_config(5) now supports CIDR address/masklen matching in "Match
   address" blocks, with a fallback to classic wildcard matching. For 
   example:
     Match address 192.0.2.0/24,3ffe:ffff::/32,!10.*
         PasswordAuthentication yes

 * sshd(8) now supports CIDR matching in ~/.ssh/authorized_keys
   from="..." restrictions, also with a fallback to classic wildcard
   matching.

 * Added an extended test mode (-T) to sshd(8) to request that it write
   its effective configuration to stdout and exit. Extended test mode
   also supports the specification of connection parameters (username,
   source address and hostname) to test the application of
   sshd_config(5) Match rules. 

 * ssh(1) now prints the number of bytes transferred and the overall
   connection throughput for SSH protocol 2 sessions when in verbose
   mode (previously these statistics were displayed for protocol 1
   connections only).

 * sftp-server(8) now supports extension methods statvfs@openssh.com and
   fstatvfs@openssh.com that implement statvfs(2)-like operations.
   (bz#1399)

 * sftp(1) now has a "df" command to the sftp client that uses the
   statvfs@openssh.com to produce a df(1)-like display of filesystem
   space and inode utilisation (requires statvfs@openssh.com support on
   the server)

 * Added a MaxSessions option to sshd_config(5) to allow control of the
   number of multiplexed sessions supported over a single TCP connection.
   This allows increasing the number of allowed sessions above the
   previous default of 10, disabling connection multiplexing 
   (MaxSessions=1) or disallowing login/shell/subsystem sessions
   entirely (MaxSessions=0).

 * Added a no-more-sessions@openssh.com global request extension that is
   sent from ssh(1) to sshd(8) when the client knows that it will never
   request another session (i.e. when session multiplexing is disabled). 
   This allows a server to disallow further session requests and
   terminate the session in cases where the client has been hijacked.

 * ssh-keygen(1) now supports the use of the -l option in combination
   with -F to search for a host in ~/.ssh/known_hosts and display its 
   fingerprint.

 * ssh-keyscan(1) now defaults to "rsa" (protocol 2) keys, instead of
   "rsa1".

 * Added an AllowAgentForwarding option to sshd_config(8) to control
   whether authentication agent forwarding is permitted. Note that this
   is a loose control, as a client may install their own unofficial
   forwarder.

 * ssh(1) and sshd(8): avoid unnecessary malloc/copy/free when receiving
   network data, resulting in a ~10% speedup

 * ssh(1) and sshd(8) will now try additional addresses when connecting
   to a port forward destination whose DNS name resolves to more than
   one address. The previous behaviour was to try the only first address
   and give up if that failed. (bz#383)

 * ssh(1) and sshd(8) now support signalling that channels are
   half-closed for writing, through a channel protocol extension
   notification "eow@openssh.com". This allows propagation of closed
   file descriptors, so that commands such as:
       "ssh -2 localhost od /bin/ls | true"
   do not send unnecessary data over the wire. (bz#85)

 * sshd(8): increased the default size of ssh protocol 1 ephemeral keys 
   from 768 to 1024 bits.

 * When ssh(1) has been requested to fork after authentication
   ("ssh -f") with ExitOnForwardFailure enabled, delay the fork until
   after replies for any -R forwards have been seen. Allows for robust
   detection of -R forward failure when using -f. (bz#92)

 * "Match group" blocks in sshd_config(5) now support negation of
   groups. E.g. "Match group staff,!guests" (bz#1315)

 * sftp(1) and sftp-server(8) now allow chmod-like operations to set 
   set[ug]id/sticky bits. (bz#1310)

 * The MaxAuthTries option is now permitted in sshd_config(5) match
   blocks.

 * Multiplexed ssh(1) sessions now support a subset of the ~ escapes
   that are available to a primary connection. (bz#1331)

 * ssh(1) connection multiplexing will now fall back to creating a new
   connection in most error cases. (bz#1439 bz#1329)

 * Added some basic interoperability tests against Twisted Conch.

 * Documented OpenSSH's extensions to and deviations from the published
   SSH protocols (the PROTOCOL file in the distribution)

 * Documented OpenSSH's ssh-agent protocol (PROTOCOL.agent).

Bug and documentation fixes

 * Make ssh(1) deal more gracefully with channel requests that fail.
   Previously it would optimistically assume that requests would always 
   succeed, which could cause hangs if they did not (e.g. when the
   server runs out of file descriptors). (bz#1384)

 * ssh(1) now reports multiplexing errors via the multiplex slave's
   stderr where possible (subject to LogLevel in the mux master).

 * ssh(1) and sshd(8) now send terminate protocol banners with CR+LF for
   protocol 2 to comply with RFC 4253. Previously they were terminated
   with CR alone. Protocol 1 banners remain CR terminated. (bz#1443)

 * Merged duplicate authentication file checks in sshd(8) and refuse to
   read authorised_keys and .shosts from non-regular files. (bz#1438)

 * Ensure that sshd(8)'s umask disallows at least group and world write,
   even if a more permissive one has been inherited. (bz#1433)

 * Suppress the warning message from sshd(8) when changing to a
   non-existent user home directory after chrooting. (bz#1461)

 * Mention that scp(1) follows symlinks when performing recursive
   copies. (bz#1466)

 * Prevent sshd(8) from erroneously applying public key restrictions
   leaned from ~/.ssh/authorized_keys to other authentication methods
   when public key authentication subsequently fails. (bz#1472)

 * Fix protocol keepalive timeouts - in some cases, keepalive packets
   were being sent, but the connection was not being closed when the
   limit for missing replies was exceeded. (bz#1465)

 * Fix ssh(1) sending invalid TTY modes when a TTY was forced (ssh -tt)
   but stdin was not a TTY. (bz#1199)

 * ssh(1) will now exit with a non-zero exit status if
   ExitOnForwardFailure was set and forwardings were disabled due to a
   failed host key check.

 * Fix MaxAuthTries tests to disallow a free authentication try to
   clients that skipped the protocol 2 "none" authentication method.
   (part of bz#1432)

 * Make keepalive timeouts apply while synchronously waiting
   for a packet, particularly during key renegotiation. (bz#1363)

 * sshd(8) has been audited to eliminate fd leaks and calls to fatal()
   in conditions of file descriptor exhaustion.

Portable OpenSSH-specific bugfixes

 * Avoid a sshd(8) hang-on-exit on Solaris caused by depending on the 
   success of isatty() on a PTY master (undefined behaviour). Probably 
   affected other platforms too. (bz#1463)

 * Fixed test for locked accounts on HP/UX with shadowed
   passwords disabled. (bz#1083)

 * Disable poll() fallback in atomiciov for Tru64. readv
   doesn't seem to be a comparable object there, which lead to
   compilation errors. (bz#1386)

 * Fall back to racy rename if link returns EXDEV. (bz#1447)

 * Explicitly handle EWOULDBLOCK wherever we handle EAGAIN, on
   some platforms (HP nonstop) it is a distinct errno. (bz#1467)

 * Avoid NULL dereferences in ancient sigaction replacement
   code. (bz#1240)

 * Avoid linking against libgssapi, which despite its name
   doesn't seem to implement all of GSSAPI. (bz#1276)

 * Use explicit noreturn attribute instead of __dead, fixing
   compilation problems on Interix. (bz#1112)

 * Added support password expiry on Tru64 SIA systems. (bz#1241)

 * Fixed an UMAC alignment problem that manifested on Itanium 
   platforms. (bz#1462)

 * The sftp-server(8) manual now describes the requirements for
   transfer logging in chroot environments. (bz#1488)

 * Strip trailing dot from hostnames when the sshd_config(5)
   HostbasedUsesNameFromPacketOnly option is set. (bz#1200)
   
Checksums:
==========

 - SHA1 (openssh-5.1.tar.gz) = 1e5b43844ed015e4fbbbe25cfad6f5377c60e759
 - SHA1 (openssh-5.1p1.tar.gz) = 877ea5b283060fe0160e376ea645e8e168047ff5

Reporting Bugs:
===============

- Please read http://www.openssh.com/report.html
  Security bugs should be reported directly to openssh@openssh.com

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 5.0 / 5.0p1 (2008-04-03) 

OpenSSH 5.0 was released on 2008-04-03. It is available from the
mirrors listed at https://www.openssh.com/.
We apologise for any inconvenience resulting from this release
being made so shortly after 4.9. Unfortunately we only learned of
the below security issue from the public CVE report. The Debian
OpenSSH maintainers responsible for handling the initial report of
this bug failed to report it via either the private OpenSSH security
contact list (openssh@openssh.com) or the portable OpenSSH Bugzilla
(http://bugzilla.mindrot.org/).
We ask anyone wishing to report security bugs in OpenSSH to please use
the openssh@openssh.com contact and to practice responsible disclosure.

OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots and purchased
T-shirts or posters.

T-shirt, poster and CD sales directly support the project. Pictures
and more information can be found at:
        http://www.openbsd.org/tshirts.html and
	http://www.openbsd.org/orders.html

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu

Changes since OpenSSH 4.9:
============================

Security:

 * CVE-2008-1483: Avoid possible hijacking of X11-forwarded connections
   by refusing to listen on a port unless all address families bind
   successfully.

Checksums:
==========

 - SHA1 (openssh-5.0.tar.gz) = 729fb3168edf6a68408223b5ed82e59d13b57c47
 - SHA1 (openssh-5.0p1.tar.gz) = 121cea3a730c0b0353334b6f46f438de30ab4928

Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 4.9 / 4.9p1 (2008-03-31) 

OpenSSH 4.9 was released on 2008-03-31. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots and purchased
T-shirts or posters.

T-shirt, poster and CD sales directly support the project. Pictures
and more information can be found at:
        http://www.openbsd.org/tshirts.html and
	http://www.openbsd.org/orders.html

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu

Note that OpenSSH 4.8 was an OpenBSD-only release shipped with the
OpenBSD 4.3 CD.

Changes since OpenSSH 4.7:
============================

Security:

 * Disable execution of ~/.ssh/rc for sessions where a command has been
   forced by the sshd_config ForceCommand directive. Users who had
   write access to this file could use it to execute arbitrary commands.
   This behaviour was documented, but was an unsafe default and an extra
   hassle for administrators.

New features:

  * Added chroot(2) support for sshd(8), controlled by a new option
    "ChrootDirectory". Please refer to sshd_config(5) for details, and
    please use this feature carefully. (bz#177 bz#1352)
  * Linked sftp-server(8) into sshd(8). The internal sftp server is
    used when the command "internal-sftp" is specified in a Subsystem
    or ForceCommand declaration. When used with ChrootDirectory, the
    internal sftp server requires no special configuration of files
    inside the chroot environment. Please refer to sshd_config(5) for
    more information.
  * Added a "no-user-rc" option for authorized_keys to disable execution
    of ~/.ssh/rc
  * Added a protocol extension method "posix-rename@openssh.com" for
    sftp-server(8) to perform POSIX atomic rename() operations.
    (bz#1400)
  * Removed the fixed limit of 100 file handles in sftp-server(8). The
    server will now dynamically allocate handles up to the number of
    available file descriptors. (bz#1397)
  * ssh(8) will now skip generation of SSH protocol 1 ephemeral server
    keys when in inetd mode and protocol 2 connections are negotiated.
    This speeds up protocol 2 connections to inetd-mode servers that
    also allow Protocol 1 (bz#440)
  * Accept the PermitRootLogin directive in a sshd_config(5) Match
    block. Allows for, e.g. permitting root only from the local
    network.
  * Reworked sftp(1) argument splitting and escaping to be more
    internally consistent (i.e. between sftp commands) and more
    consistent with sh(1). Please note that this will change the
    interpretation of some quoted strings, especially those with
    embedded backslash escape sequences. (bz#778)
  * Support "Banner=none" in sshd_config(5) to disable sending of a
    pre-login banner (e.g. in a Match block).
  * ssh(1) ProxyCommands are now executed with $SHELL rather than
    /bin/sh.
  * ssh(1)'s ConnectTimeout option is now applied to both the TCP
    connection and the SSH banner exchange (previously it just covered
    the TCP connection). This allows callers of ssh(1) to better detect
    and deal with stuck servers that accept a TCP connection but don't
    progress the protocol, and also makes ConnectTimeout useful for
    connections via a ProxyCommand.
  * Many new regression tests, including interop tests against PuTTY's
    plink.
  * Support BSM auditing on Mac OS X

The following bugs have been fixed in this release:

   - scp(1) incorrectly reported "stalled" on slow copies. (bz#799)
   - scp(1) date underflow for timestamps before epoch. (bz#828)
   - scp(1) and sftp(1) progressmeter type fixes. (bz#842)
   - SSH2_MSG_UNIMPLEMENTED packets did not correctly reset the client
     keepalive logic, causing disconnections on servers that did not
     explicitly implement "keepalive@openssh.com". (bz#1307)
   - ssh(1) used the obsolete SIG DNS RRtype for host keys in DNS,
     instead of the current standard RRSIG. (bz#1317)
   - Extract magic buffer size constants in scp(1) to #defines.
     (bz#1333)
   - Correctly drain ACKs when a sftp(1) upload write fails midway,
     avoids a fatal() exit from what should be a recoverable condition.
     (bz#1354)
   - Avoid pointer arithmetic and strict aliasing warnings. (bz#1355)
   - Fixed packet size advertisements. Previously TCP and agent
     forwarding incorrectly advertised the channel window size as the
     packet size, causing fatal errors under some conditions. (bz#1360)
   - Document KbdInteractiveAuthentication in sshd_config(5). (bz#1367)
   - Fixed sftp(1) file handle leak on download when the local file
     could not be opened. (bz#1375)
   - Fixed ssh-keygen(1) selective host key hashing (i.e.
     "ssh-keygen -HF hostname") to not include any IP address in the
     data to be hashed. (bz#1376)
   - Fix clobbering of struct passwd from multiple getpwid calls,
     resulting in fatal errors inside tilde_expand_filename. (bz#1377)
   - Fix parsing of port-forwarding specifications to correctly
     detect errors in either specified port number. (bz#1378)
   - Fix memory leak in ssh(1) ~ escape commandline handling. (bz#1379)
   - Make ssh(1) skip listening on the IPv6 wildcard address when a
     binding address of 0.0.0.0 is used against an old SSH server that
     does not support the RFC4254 syntax for wildcard bind addresses.
     (bz#1381)
   - Remove extra backslashes in the RB_PROTOTYPE macro definition.
     (bz#1385)
   - Support ssh(1) RekeyLimits up to the maximum allowed by the
     protocol: 2**32-1. (bz#1390)
   - Enable IPV6_V6ONLY socket option on sshd(8) listen socket, as is
     already done for X11/TCP forwarding sockets. (bz#1392)
   - Fix FD leak that could hang a ssh(1) connection multiplexing
     master. (bz#1398)
   - Improve error messages when hostname resolution fails due to a
     system error. (bz#1417)
   - Make ssh(1) -q option documentation consistent with reality.
     (bz#1427 bz#1429)

Portable OpenSSH bugs fixed:

   - Fixed sshd(8) PAM support not calling pam_session_close(), or
     failing to call it with root privileges. (bz#926)
   - Made sshd(8) use configuration-specified SyslogFacility for
     hosts_access(3) messages for denied connections. (bz#1042)
   - Implement getgrouplist(3) for AIX, enabling NSS LDAP to work on
     this platform. (bz#1081)
   - Fix compilation errors on AIX due to misdefinition of LLONG_MAX.
     (bz#1347)
   - Force use of local glob(3) implementation on Mac OS X and FreeBSD,
     as the platform versions lack features or have unexpected
     behaviour. (bz#1407)
   - Reduce stdout/stderr noise from ssh-copy-id. (bz#1431)
   - Fix activation of OpenSSL engine support when requested in
     configure. (bz#1437)

Checksums:
==========

 - SHA1 (openssh-4.9.tar.gz) = fa7d1b3dcb093bd0dfc643b33b1a57a26f459373
 - SHA1 (openssh-4.9p1.tar.gz) = 91575878883065bd777f82b47e0d481ac69ee7fe

Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 4.8 / 4.8p1 (2008-03-31) 

OpenSSH 4.8 was an OpenBSD-only release, included on the OpenBSD 4.3
CD only.

Changes since OpenSSH 4.8:
============================

New features:

  * Added chroot(2) support for sshd(8), controlled by a new option
    "ChrootDirectory". Please refer to sshd_config(5) for details, and
    please use this feature carefully. (bz#177 bz#1352)
  * Linked sftp-server(8) into sshd(8). The internal sftp server is
    used when the command "internal-sftp" is specified in a Subsystem
    or ForceCommand declaration. When used with ChrootDirectory, the
    internal sftp server requires no special configuration of files
    inside the chroot environment. Please refer to sshd_config(5) for
    more information.
  * Added a protocol extension method "posix-rename@openssh.com" for
    sftp-server(8) to perform POSIX atomic rename() operations.
    (bz#1400)
  * Removed the fixed limit of 100 file handles in sftp-server(8). The
    server will now dynamically allocate handles up to the number of
    available file descriptors. (bz#1397)
  * ssh(8) will now skip generation of SSH protocol 1 ephemeral server
    keys when in inetd mode and protocol 2 connections are negotiated.
    This speeds up protocol 2 connections to inetd-mode servers that
    also allow Protocol 1 (bz#440)
  * Accept the PermitRootLogin directive in a sshd_config(5) Match
    block. Allows for, e.g. permitting root only from the local
    network.
  * Reworked sftp(1) argument splitting and escaping to be more
    internally consistent (i.e. between sftp commands) and more
    consistent with sh(1). Please note that this will change the
    interpretation of some quoted strings, especially those with
    embedded backslash escape sequences. (bz#778)
  * Support "Banner=none" in sshd_config(5) to disable sending of a
    pre-login banner (e.g. in a Match block).
  * ssh(1) ProxyCommands are now executed with $SHELL rather than
    /bin/sh.
  * ssh(1)'s ConnectTimeout option is now applied to both the TCP
    connection and the SSH banner exchange (previously it just covered
    the TCP connection). This allows callers of ssh(1) to better detect
    and deal with stuck servers that accept a TCP connection but don't
    progress the protocol, and also makes ConnectTimeout useful for
    connections via a ProxyCommand.
  * Many new regression tests, including interop tests against PuTTY's
    plink.
  * Support BSM auditing on Mac OS X

The following bugs have been fixed in this release:

   - scp(1) incorrectly reported "stalled" on slow copies. (bz#799)
   - scp(1) date underflow for timestamps before epoch. (bz#828)
   - scp(1) and sftp(1) progressmeter type fixes. (bz#842)
   - SSH2_MSG_UNIMPLEMENTED packets did not correctly reset the client
     keepalive logic, causing disconnections on servers that did not
     explicitly implement "keepalive@openssh.com". (bz#1307)
   - ssh(1) used the obsolete SIG DNS RRtype for host keys in DNS,
     instead of the current standard RRSIG. (bz#1317)
   - Extract magic buffer size constants in scp(1) to #defines.
     (bz#1333)
   - Correctly drain ACKs when a sftp(1) upload write fails midway,
     avoids a fatal() exit from what should be a recoverable condition.
     (bz#1354)
   - Avoid pointer arithmetic and strict aliasing warnings. (bz#1355)
   - Fixed packet size advertisements. Previously TCP and agent
     forwarding incorrectly advertised the channel window size as the
     packet size, causing fatal errors under some conditions. (bz#1360)
   - Document KbdInteractiveAuthentication in sshd_config(5). (bz#1367)
   - Fixed sftp(1) file handle leak on download when the local file
     could not be opened. (bz#1375)
   - Fixed ssh-keygen(1) selective host key hashing (i.e.
     "ssh-keygen -HF hostname") to not include any IP address in the
     data to be hashed. (bz#1376)
   - Fix clobbering of struct passwd from multiple getpwid calls,
     resulting in fatal errors inside tilde_expand_filename. (bz#1377)
   - Fix parsing of port-forwarding specifications to correctly
     detect errors in either specified port number. (bz#1378)
   - Fix memory leak in ssh(1) ~ escape commandline handling. (bz#1379)
   - Make ssh(1) skip listening on the IPv6 wildcard address when a
     binding address of 0.0.0.0 is used against an old SSH server that
     does not support the RFC4254 syntax for wildcard bind addresses.
     (bz#1381)
   - Remove extra backslashes in the RB_PROTOTYPE macro definition.
     (bz#1385)
   - Support ssh(1) RekeyLimits up to the maximum allowed by the
     protocol: 2**32-1. (bz#1390)
   - Enable IPV6_V6ONLY socket option on sshd(8) listen socket, as is
     already done for X11/TCP forwarding sockets. (bz#1392)
   - Fix FD leak that could hang a ssh(1) connection multiplexing
     master. (bz#1398)
   - Improve error messages when hostname resolution fails due to a
     system error. (bz#1417)
   - Make ssh(1) -q option documentation consistent with reality.
     (bz#1427 bz#1429)

Portable OpenSSH bugs fixed:

   - Fixed sshd(8) PAM support not calling pam_session_close(), or
     failing to call it with root privileges. (bz#926)
   - Made sshd(8) use configuration-specified SyslogFacility for
     hosts_access(3) messages for denied connections. (bz#1042)
   - Implement getgrouplist(3) for AIX, enabling NSS LDAP to work on
     this platform. (bz#1081)
   - Fix compilation errors on AIX due to misdefinition of LLONG_MAX.
     (bz#1347)
   - Force use of local glob(3) implementation on Mac OS X and FreeBSD,
     as the platform versions lack features or have unexpected
     behaviour. (bz#1407)
   - Reduce stdout/stderr noise from ssh-copy-id. (bz#1431)
   - Fix activation of OpenSSL engine support when requested in
     configure. (bz#1437)

Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 4.7 / 4.7p1 (2007-09-05) 

OpenSSH 4.7 was released on 2007-09-05. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots and purchased
T-shirts or posters.

T-shirt, poster and CD sales directly support the project. Pictures
and more information can be found at:
        http://www.openbsd.org/tshirts.html and
        http://www.openbsd.org/orders.html

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu

Changes since OpenSSH 4.6:
============================

Security bugs resolved in this release:

 * Prevent ssh(1) from using a trusted X11 cookie if creation of an
   untrusted cookie fails; found and fixed by Jan Pechanec.

Other changes, new functionality and fixes in this release:

 * sshd(8) in new installations defaults to SSH Protocol 2 only.
   Existing installations are unchanged.

 * The SSH channel window size has been increased, and both ssh(1)
   sshd(8) now send window updates more aggressively. These improves
   performance on high-BDP (Bandwidth Delay Product) networks.

 * ssh(1) and sshd(8) now preserve MAC contexts between packets, which
   saves 2 hash calls per packet and results in 12-16% speedup for
   arcfour256/hmac-md5.

 * A new MAC algorithm has been added, UMAC-64 (RFC4418) as
   "umac-64@openssh.com". UMAC-64 has been measured to be 
   approximately 20% faster than HMAC-MD5.

 * A -K flag was added to ssh(1) to set GSSAPIAuthentication=Yes

 * Failure to establish a ssh(1) TunnelForward is now treated as a
   fatal error when the ExitOnForwardFailure option is set.

 * ssh(1) returns a sensible exit status if the control master goes
   away without passing the full exit status. (bz #1261)

 * The following bugs have been fixed in this release:

   - When using a ProxyCommand in ssh(1), set the outgoing hostname with
     gethostname(2), allowing hostbased authentication to work (bz #616)
   - Make scp(1) skip FIFOs rather than hanging (bz #856)
   - Encode non-printing characters in scp(1) filenames.
     these could cause copies to be aborted with a "protocol error"
     (bz #891)
   - Handle SIGINT in sshd(8) privilege separation child process to
     ensure that wtmp and lastlog records are correctly updated
     (bz #1196)
   - Report GSSAPI mechanism in errors, for libraries that support
     multiple mechanisms (bz #1220)
   - Improve documentation for ssh-add(1)'s -d option (bz #1224)
   - Rearrange and tidy GSSAPI code, removing server-only code being
     linked into the client. (bz #1225)
   - Delay execution of ssh(1)'s LocalCommand until after all forwadings
     have been established. (bz #1232)
   - In scp(1), do not truncate non-regular files (bz #1236)
   - Improve exit message from ControlMaster clients. (bz #1262)
   - Prevent sftp-server(8) from reading until it runs out of buffer
     space, whereupon it would exit with a fatal error. (bz #1286)

 * Portable OpenSSH bugs fixed:

   - Fix multiple inclusion of paths.h on AIX 5.1 systems. (bz #1243)
   - Implement getpeereid for Solaris using getpeerucred. Solaris
     systems will now refuse ssh-agent(1) and ssh(1) ControlMaster
     clients from different, non-root users (bz #1287)
   - Fix compilation warnings by including string.h if found. (bz #1294)
   - Remove redefinition of _res in getrrsetbyname.c for platforms that
     already define it. (bz #1299)
   - Fix spurious "chan_read_failed for istate 3" errors from sshd(8),
     a side-effect of the "hang on exit" fix introduced in 4.6p1.
     (bz #1306)
   - pam_end() was not being called if authentication failed (bz #1322)
   - Fix SELinux support when SELinux is in permissive mode. Previously
     sshd(8) was treating SELinux errors as always fatal. (bz #1325)
   - Ensure that pam_setcred(..., PAM_ESTABLISH_CRED) is called before
     pam_setcred(..., PAM_REINITIALIZE_CRED), fixing pam_dhkeys.
     (bz #1339)
   - Fix privilege separation on QNX - pre-auth only, this platform does
     not support file descriptor passing needed for post-auth privilege
     separation. (bz #1343)

Thanks to everyone who has contributed patches, reported bugs and
tested releases.

Checksums:
==========

- SHA1 (openssh-4.7.tar.gz) = 9ebaab9b31e01bd0d04425dc23536bcc78f8d990
- SHA1 (openssh-4.7p1.tar.gz) = 58357db9e64ba6382bef3d73d1d386fcdc0508f4

Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 4.6 / 4.6p1 (2007-03-09) 

OpenSSH 4.6 was released on 2007-03-09. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots and purchased
T-shirts or posters.

T-shirt, poster and CD sales directly support the project. Pictures
and more information can be found at:
        http://www.openbsd.org/tshirts.html and
	http://www.openbsd.org/orders.html

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu

Changes since OpenSSH 4.5:
============================

 * sshd now allows the enabling and disabling of authentication
   methods on a per user, group, host and network basis via the
   Match directive in sshd_config.

 * The following bugs have been fixed in this release:

   - Clear SIGALRM when restarting due to SIGHUP. Prevents stray
     signal from taking down sshd if a connection was pending at
     the time SIGHUP was received
   - sftp returned a zero exit status when upload failed due to write
     errors (bugzilla #1252)
   - fixed an inconsistent check for a terminal when displaying scp
     progress meter (bugzilla #1265)
   - Parsing of time values in Match blocks was incorrectly applied
     to the global configuration (bugzilla #1275)
   - Allow multiple forwarding options to work when specified in a
     PermitOpen directive (bugzilla #1267)
   - Interoperate with ssh.com versions that do not support binding
     remote port forwarding sessions to a hostname (bugzilla #1019)

 * Portable OpenSSH bugs fixed:

   - "hang on exit" when background processes are running at the time
     of exit on a ttyful/login session (bugzilla #52)
   - Fix typos in the ssh-rand-helper(8) man page (bugzilla #1259)
   - Check that some SIG records have been returned in getrrsetbyname
     (bugzilla #1281)
   - Fix contrib/findssl for platforms that lack "which" (bugzilla 
     #1237)
   - Work around bug in OpenSSL 0.9.8e that broke aes256-ctr,
     aes192-ctr, arcfour256 (bugzilla #1291)

Checksums:
==========

- SHA1 (openssh-4.6.tar.gz) = c1700845be464a769428f34ef727c1f530728afc
- SHA1 (openssh-4.6p1.tar.gz) = b2aefeb1861b4688b1777436035239ec32a47da8


Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 4.5 / 4.5p1 (2006-11-07) 

OpenSSH 4.5 was released on 2006-11-07. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots and purchased
T-shirts or posters.

T-shirt, poster and CD sales directly support the project. Pictures
and more information can be found at:
        http://www.openbsd.org/tshirts.html and
	http://www.openbsd.org/orders.html

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu

Changes since OpenSSH 4.4:
============================

This is a bugfix only release. No new features have been added.

Security bugs resolved in this release:

 * Fix a bug in the sshd privilege separation monitor that weakened its
   verification of successful authentication. This bug is not known to
   be exploitable in the absence of additional vulnerabilities.
 
This release includes the following non-security fixes:

 * Several compilation fixes for portable OpenSSH

 * Fixes to Solaris SMF/process contract support (bugzilla #1255)

Thanks to everyone who has contributed patches, reported bugs and
tested releases.

Checksums:
==========

- SHA1 (openssh-4.5.tar.gz) = def3de1557181062d788695b9371d02635af39fb
- SHA1 (openssh-4.5p1.tar.gz) = 2eefcbbeb9e4fa16fa4500dec107d1a09d3d02d7

Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 4.4 / 4.4p1 (2006-09-27) 

OpenSSH 4.4 was released on 2006-09-27. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots and purchased
T-shirts or posters.

T-shirt, poster and CD sales directly support the project. Pictures
and more information can be found at:
        http://www.openbsd.org/tshirts.html and
	http://www.openbsd.org/orders.html

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu

Changes since OpenSSH 4.3:
============================

Security bugs resolved in this release:

 * Fix a pre-authentication denial of service found by Tavis Ormandy,
   that would cause sshd(8) to spin until the login grace time
   expired.

 * Fix an unsafe signal handler reported by Mark Dowd. The signal
   handler was vulnerable to a race condition that could be exploited
   to perform a pre-authentication denial of service. On portable
   OpenSSH, this vulnerability could theoretically lead to
   pre-authentication remote code execution if GSSAPI authentication
   is enabled, but the likelihood of successful exploitation appears
   remote.

 * On portable OpenSSH, fix a GSSAPI authentication abort that could
   be used to determine the validity of usernames on some platforms.

This release includes the following new functionality and fixes:

 * Implemented conditional configuration in sshd_config(5) using the
   "Match" directive. This allows some configuration options to be
   selectively overridden if specific criteria (based on user, group,
   hostname and/or address) are met. So far a useful subset of post-
   authentication options are supported and more are expected to be
   added in future releases.

 * Add support for Diffie-Hellman group exchange key agreement with a
   final hash of SHA256.

 * Added a "ForceCommand" directive to sshd_config(5). Similar to the
   command="..." option accepted in ~/.ssh/authorized_keys, this forces
   the execution of the specified command regardless of what the user
   requested. This is very useful in conjunction with the new "Match"
   option.

 * Add a "PermitOpen" directive to sshd_config(5). This mirrors the
   permitopen="..." authorized_keys option, allowing fine-grained
   control over the port-forwardings that a user is allowed to
   establish.

 * Add optional logging of transactions to sftp-server(8).

 * ssh(1) will now record port numbers for hosts stored in
   ~/.ssh/authorized_keys when a non-standard port has been requested.

 * Add an "ExitOnForwardFailure" option to cause ssh(1) to exit (with
   a non-zero exit code) when requested port forwardings could not be
   established.

 * Extend sshd_config(5) "SubSystem" declarations to allow the
   specification of command-line arguments.

 * Replacement of all integer overflow susceptible invocations of
   malloc(3) and realloc(3) with overflow-checking equivalents.

 * Many manpage fixes and improvements

 * New portable OpenSSH-specific features:

   - Add optional support for SELinux, controlled using the
     --with-selinux configure option (experimental)

   - Add optional support for Solaris process contracts, enabled
     using the --with-solaris-contracts configure option (experimental)
     This option will also include SMF metadata in Solaris packages
     built using the "make package" target

   - Add optional support for OpenSSL hardware accelerators (engines),
     enabled using the --with-ssl-engine configure option.

 * Bugs from http://bugzilla.mindrot.org fixed:
    #482  - readconf doesn't accept paths with spaces in them.
    #906  - syslog messages from sshd [net] lost.
    #975  - Kerberos authentication timing can leak information
            about account validity.
    #981  - Flow stop in SSH2.
    #1102 - C program 'write' with zero length hangs.
    #1129 - sshd hangs for command-only invocations due to
            fork/child signals.
    #1131 - error "buffer_append_space:alloc not supported"
    #1138 - Passphrase asked for (but ignored) if key file permissions
            too liberal..
    #1156 - Closes connection after C-c is pressed on QNX.
    #1157 - ssh-keygen doesn't handle DOS line breaks.
    #1159 - %u and %h not handled in IdentityFile.
    #1161 - scp -r fails.
    #1162 - Inappropriate sequence of syslog messages.
    #1166 - openssh-4.3p1 has some issues compiling.
    #1171 - configure can't always figure out LLONG_MAX..
    #1173 - scp reports lost connection for very large files.
    #1177 - Incorrect sshrc file location in Makefile.in.
    #1179 - sshd incorrectly rejects  connections due to IP options.
    #1181 - configure should detect when openssl-0.9.8x needs -ldl.
    #1186 - ssh tries multiple times to open unprotected keys.
    #1188 - keyboard-interactive should not allow retry after
            pam_acct_mgmt fails.
    #1193 - Open ssh will not allow changing of passwords on usernames
            greater than 8 characters..
    #1201 - Bind address information is not specified in command line
            help messages.
    #1203 - configure.ac is missing an open [.
    #1207 - sshd does not clear unsuccessful login count on
            non-interactive logins.
    #1218 - GSSAPI client code permits SPNEGO usage.
    #1221 - Banner only suppressed at log level = QUIET (used to be
            at log level < INFO).

 * Fixes to memory and file descriptor leaks reported by the Coverity
   static analysis tool

 * Fixes to inconsistent pointer checks reported by the Stanford
   SATURN tool

Thanks to everyone who has contributed patches, reported bugs and
tested releases.

Checksums:
==========

- SHA1 (openssh-4.4.tar.gz) = 2294b5e5a591420aa05ff607c1890ab622ace878
- SHA1 (openssh-4.4p1.tar.gz) = 6a52b1dee1c2c9862923c0008d201d98a7fd9d6c

Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 4.3p2 (2006-02-11) 

Portable OpenSSH 4.3p2 was released on 2006-02-11. It will be available
from the mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We have also recently completed another Internet SSH usage scan, the 
results of which may be found at http://www.openssh.com/usage.html

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots and purchased 
T-shirts or posters.

T-shirt, poster and CD sales directly support the project. Pictures 
and more information can be found at:
        http://www.openbsd.org/tshirts.html and
	http://www.openbsd.org/orders.html

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu

Changes since Portable OpenSSH 4.3p1:
====================================

This is a release of Portable OpenSSH only, to resolve some 
portability bugs. There are no new features, only fixes: 

 * Explicitly test for egrep in ./configure, fixing a problem in 4.3p1
   that caused some platforms to fail to detect the available fields
   in utmp/wtmp/lastlog records. This bug manifested as missing or
   empty login/logout records (as seen by last(1), etc.)

 * Fix for logout records not being updated on platforms without 
   support for post-authentication privilege separation (e.g. Cygwin)

 * Fixed compilation problems on Ultrix, NewsOS and QNX

Thanks to everyone who has contributed patches, reported bugs or test
releases.

Checksums:
==========

- SHA1 (openssh-4.3p2.tar.gz) = 2b5b0751fd578283ba7b106025c0ba391fd72f1f

Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 4.3 / 4.3p1 (2006-02-01) 

OpenSSH 4.3 was released on 2006-02-01. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We have also recently completed another Internet SSH usage scan, the 
results of which may be found at http://www.openssh.com/usage.html

Once again, we would like to thank the OpenSSH community for their
continued support of the project, especially those who contributed
code or patches, reported bugs, tested snapshots and purchased 
T-shirts or posters.

T-shirt, poster and CD sales directly support the project. Pictures 
and more information can be found at:
        http://www.openbsd.org/tshirts.html and
	http://www.openbsd.org/orders.html

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu

Changes since OpenSSH 4.2:
============================ 

Security bugs resolved in this release:

 * CVE-2006-0225: scp (as does rcp, on which it is based) invoked a
   subshell to perform local to local, and remote to remote copy
   operations. This subshell exposed filenames to shell expansion
   twice; allowing a local attacker to create filenames containing
   shell metacharacters that, if matched by a wildcard, could lead
   to execution of attacker-specified commands with the privilege of
   the user running scp (Bugzilla #1094)

This is primarily a bug-fix release, only one new feature has been
added: 

 * Add support for tunneling arbitrary network packets over a
   connection between an OpenSSH client and server via tun(4) virtual
   network interfaces. This allows the use of OpenSSH (4.3+) to create
   a true VPN between the client and server providing real network
   connectivity at layer 2 or 3. This feature is experimental and is 
   currently supported on OpenBSD, Linux, NetBSD (IPv4 only) and 
   FreeBSD. Other operating systems with tun/tap interface capability 
   may be added in future portable OpenSSH releases. Please refer to 
   the README.tun file in the source distribution for further details
   and usage examples.

Some of the other bugs resolved and internal improvements are:

 * Reduce default key length for new DSA keys generated by ssh-keygen 
   back to 1024 bits. DSA is not specified for longer lengths and does 
   not fully benefit from simply making keys longer. As per FIPS 186-2 
   Change Notice 1, ssh-keygen will refuse to generate a new DSA key 
   smaller or larger than 1024 bits

 * Fixed X forwarding failing to start when a the X11 client is executed
   in background at the time of session exit (Bugzilla #1086)

 * Change ssh-keygen to generate a protocol 2 RSA key when invoked
   without arguments (Bugzilla #1064)

 * Fix timing variance for valid vs. invalid accounts when attempting
   Kerberos authentication (Bugzilla #975)

 * Ensure that ssh always returns code 255 on internal error (Bugzilla 
   #1137)

 * Cleanup wtmp files on SIGTERM when not using privsep (Bugzilla #1029)

 * Set SO_REUSEADDR on X11 listeners to avoid problems caused by
   lingering sockets from previous session (X11 applications can
   sometimes not connect to 127.0.0.1:60xx) (Bugzilla #1076)

 * Ensure that fds 0, 1 and 2 are always attached in all programs, by
   duping /dev/null to them if necessary.

 * Xauth list invocation had bogus "." argument (Bugzilla #1082)

 * Remove internal assumptions on key exchange hash algorithm and output
   length, preparing OpenSSH for KEX methods with alternate hashes.

 * Ignore junk sent by a server before  it sends the "SSH-" banner
   (Bugzilla #1067)

 * The manpages has been significantly improves and rearranged, in 
   addition to other specific manpage fixes:
   #1037 - Man page entries for -L and -R should mention -g.
   #1077 - Descriptions for "ssh -D" and DynamicForward should mention
           they can specify "bind_address" optionally.
   #1088 - Incorrect descriptions in ssh_config man page for 
           ControlMaster=no.
   #1121 - Several corrections for ssh_agent manpages

 * Lots of cleanups, including fixes to memory leaks on error paths
   (Bugzilla #1109, #1110, #1111 and more) and possible crashes (#1092)

 * Portable OpenSSH-specific fixes:

   - Pass random seed during re-exec for each connection: speeds up
     processing of new connections on platforms using the OpenSSH's
     builtin entropy collector (ssh-rand-helper)

   - PAM fixes and improvements:
     #1045 - Missing option for ignoring the /etc/nologin file
     #1087 - Show PAM password expiry message from LDAP on login
     #1028 - Forward final non-query conversations to client
     #1126 - Prevent user from being forced to change an expired
             password repeatedly on AIX in some PAM configurations.
     #1045 - Do not check /etc/nologin when PAM is enabled, instead 
             allow PAM to handle it.  Note that on platforms using 
             PAM, the pam_nologin module should be used in sshd's 
             session stack in order to maintain past behaviour

   - Portability-related fixes:
     #989  - Fix multiplexing regress test on Solaris
     #1097 - Cross-compile fixes.
     #1096 - ssh-keygen broken on HPUX.
     #1098 - $MAIL being set incorrectly for HPUX server login.
     #1104 - Compile error on Tru64 Unix 4.0f
     #1106 - Updated .spec file and startup for SuSE.
     #1122 - Use _GNU_SOURCE define in favor of __USE_GNU, fixing
             compilation problems on glibc 2.4

Thanks to everyone who has contributed patches, reported bugs or test
releases.

Checksums:
==========

- SHA1 (openssh-4.3.tar.gz) = 0cb66e56805d66b51511455423bab88aa58a1455
- SHA1 (openssh-4.3p1.tar.gz) = b1f379127829e7e820955b2825130edd1601ba59

Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and
Ben Lindstrom.

OpenSSH 4.2 / 4.2p1 (2005-09-01) 

OpenSSH 4.2 was released on 2005-09-01. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support of the project, especially those who contributed source,
reported bugs, tested snapshots and purchased T-shirts or posters.

T-shirt, poster and CD sales directly support the project. Pictures 
and more information can be found at:
        http://www.openbsd.org/tshirts.html and
	http://www.openbsd.org/orders.html

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu

Changes since OpenSSH 4.1:
============================ 

  - SECURITY: Fix a bug introduced in OpenSSH 4.0 that caused 
    GatewayPorts to be incorrectly activated for dynamic ("-D") port 
    forwardings when no listen address was explicitly specified.

  - SECURITY: sshd in OpenSSH versions prior to 4.2 allow GSSAPI 
    credentials to be delegated to users who log in with methods 
    other than GSSAPI authentication (e.g. public key) when the 
    client requests it. This behaviour has been changed in OpenSSH 
    4.2 to only delegate credentials to users who authenticate
    using the GSSAPI method. This eliminates the risk of credentials 
    being inadvertently exposed to an untrusted user/host (though 
    users should not activate GSSAPIDelegateCredentials to begin
    with when the remote user or host is untrusted)

  - Added a new compression method that delays the start of zlib
    compression until the user has been authenticated successfully. 
    The new method ("Compression delayed") is on by default in the 
    server. This eliminates the risk of any zlib vulnerability 
    leading to a compromise of the server from unauthenticated users.

    NB. Older OpenSSH (<3.5) versions have a bug that will cause them
    to refuse to connect to any server that does not offer compression
    when the client has compression requested. Since the new "delayed"
    server mode isn't supported by these older clients, they will
    refuse to connect to a new server unless compression is disabled
    (on the client end) or the original compression method is enabled
    on the server ("Compression yes" in sshd_config)

  - Another round of proactive changes for signed vs unsigned integer
    bugs has been completed, including changing the atomicio() API to
    encourage safer programming. This work is ongoing.

  - Added support for the improved arcfour cipher modes from
    draft-harris-ssh-arcfour-fixes-02. The improves the cipher's
    resistance to a number of attacks by discarding early keystream
    output.

  - Increase the default size of new RSA/DSA keys generated by
    ssh-keygen from 1024 to 2048 bits.

  - Many bugfixes and improvements to connection multiplexing,
    including:

    - Added ControlMaster=auto/autoask options to support opportunistic
      multiplexing (see the ssh_config(5) manpage for details).

    - The client will now gracefully fallback to starting a new TCP
      connection if it cannot connect to a specified multiplexing
      control socket

    - Added %h (target hostname), %p (target port) and %r (remote
      username) expansion sequences to ControlPath. Also allow
      ControlPath=none to disable connection multiplexing.

    - Implemented support for X11 and agent forwarding over multiplexed
      connections. Because of protocol limitations, the slave
      connections inherit the master's DISPLAY and SSH_AUTH_SOCK rather
      than distinctly forwarding their own.

  - Portable OpenSSH: Added support for long passwords (> 8-char) on
    UnixWare 7.

  - The following bugs from http://bugzilla.mindrot.org/ were closed:

     #471  - Misleading error message if /dev/tty perms wrong
     #623  - Don't use $HOME in manpages
     #829  - Don't allocate a tty if -n option is set
     #1025 - Correctly handle disabled special character in ttymodes
     #1033 - Fix compile-time warnings
     #1046 - AIX 5.3 Garbage on Login
     #1054 - Don't terminate connection on getpeername() failure
     #1076 - GSSAPIDelegateCredentials issue mentioned above

  - Lots of other improvements and fixes. Please refer to the ChangeLog
    for details

Thanks to everyone who has contributed patches, problem or test reports.

Checksums:
==========

- SHA1 (openssh-4.2.tar.gz) = d2bd777986a30e446268ceeb24cddbf2edf51b21
- SHA1 (openssh-4.2p1.tar.gz) = 5e7231cfa8ec673ea856ce291b78fac8b380eb78

Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Ben Lindstrom, Darren Tucker and Tim Rice.

OpenSSH 4.1 / 4.1p1 (2005-05-26) 

OpenSSH 4.1 was released on 2005-05-26. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support to the project, especially those who contributed source and
bought T-shirts or posters.

We have a new design of T-shirt available, more info on
        http://www.openbsd.org/tshirts.html#18

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu


Changes since OpenSSH 4.0:
============================ 

* This is a bugfix release, no new features have been added. Some notable
  fixes are:

  - Fix segfault when using forwardings configured in ssh_config(5) and 
    ClearAllForwardings (bugzilla #996)

  - Limit input buffer size for channels. A peer could send more data
    than the buffer code was willing to accept. This would cause OpenSSH
    to abort the connection (bugzilla #896)

* Several improvements to the regression tests

* Portable OpenSSH:

  - OpenSSH will now always normalise IPv4 in IPv6 mapped addresses back to 
    IPv4 addresses. This means that IPv4 addresses in log messages on IPv6
    enabled machines will no longer be prefixed by "::ffff:" and AllowUsers,
    DenyUsers, AllowGroups, DenyGroups will match IPv4-style addresses only 
    for 4-in-6 mapped connections. This ensures a consistent representation
    of IPv4 addresses regardless of whether or not the machine is IPv6
    enabled.

* Other bugfixes, including bugzilla #950, #997, #998, #999, #1005, #1006, 
  #1024, and #1038

Changes since OpenSSH 3.9:
============================ 

* ssh(1) now allows the optional specification of an address to bind to 
  in port forwarding connections (local, remote and dynamic). Please 
  refer to the documentation for the -L and -R options in the ssh(1) 
  manual page and the LocalForward and RemoteForward options in the 
  ssh_config(5) manpage. (Bugzilla #413)

* To control remote bindings while retaining backwards compatibility,
  sshd(8)'s GatewayPorts option has been extended. To allow client
  specified bind addresses for remote (-R) port forwardings, the server
  must be configured with "GatewayPorts clientspecified".

* ssh(1) and ssh-keyscan(1) now support hashing of host names and 
  addresses added to known_hosts files, controlled by the ssh(1) 
  HashKnownHosts configuration directive. This option improves user 
  privacy by hiding which hosts have been visited. At present this 
  option is off by default, but may be turned on once it receives 
  sufficient testing.

* Added options for managing keys in known_hosts files to ssh-keygen(1),
  including the ability to search for hosts by name, delete hosts by
  name and convert an unhashed known_hosts file into one with hashed
  names. These are particularly useful for managing known_hosts files
  with hashed hostnames.

* Improve account and password expiry support in sshd(8). The server
  will now warn in advance for both account and password expiry.

* sshd(8) will now log the source of connections denied by AllowUsers,
  DenyUsers, AllowGroups and DenyGroups (Bugzilla #909)

* Added AddressFamily option to sshd(8) to allow global control over
  IPv4/IPv6 usage. (Bugzilla #989)

* Improved sftp(1) client, including bugfixes and optimisations for the 
  ``ls'' command and command history and editing support using libedit.

* Improved the handling of bad data in authorized_keys files,
  eliminating fatal errors on corrupt or very large keys. (Bugzilla
  #884)

* Improved connection multiplexing support in ssh(1). Several bugs 
  have been fixed and a new "command mode" has been added to allow the
  control of a running multiplexing master connection, including 
  checking that it is up, determining its PID and asking it to exit.

* Have scp(1) and sftp(1) wait for the spawned ssh to exit before they
  exit themselves.  This prevents ssh from being unable to restore 
  terminal modes (not normally a problem on OpenBSD but common with 
  -Portable on POSIX platforms). (Bugzilla #950)

* Portable OpenSSH:

  - Add *EXPERIMENTAL* BSM audit support for Solaris systems 
    (Bugzilla #125)

  - Enable IPv6 on AIX where possible (see README.platform for
    details), working around a misfeature of AIX's getnameinfo. 
    (Bugzilla #835)

  - Teach sshd(8) to write failed login records to btmp for
    unsuccessful auth attempts. Currently this is only for password,
    keyboard-interactive and challenge/response authentication methods
    and only on Linux and HP-UX.

  - sshd(8) now sends output from failing PAM session modules to the 
    user before exiting, similar to the way /etc/nologin is handled

  - Store credentials from gssapi-with-mic authentication early enough
    to be available to PAM session modules when privsep=yes.

Checksums:
==========

- SHA1 (openssh-4.1.tar.gz) = 62fc9596b20244bb559d5fee3ff3ecc0dfd557cb
- SHA1 (openssh-4.1p1.tar.gz) = e85d389da8ad8290f5031b8f9972e2623c674e46

Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Ben Lindstrom, Darren Tucker and Tim Rice.

OpenSSH 4.0 / 4.0p1 (2005-03-09) 

OpenSSH 4.0 was released on 2005-03-09. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support to the project, especially those who contributed source and
bought T-shirts or posters.

We have a new design of T-shirt available, more info on
        http://www.openbsd.org/tshirts.html#18

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu


Changes since OpenSSH 3.9:
============================ 

* ssh(1) now allows the optional specification of an address to bind to 
  in port forwarding connections (local, remote and dynamic). Please 
  refer to the documentation for the -L and -R options in the ssh(1) 
  manual page and the LocalForward and RemoteForward options in the 
  ssh_config(5) manpage. (Bugzilla #413)

* To control remote bindings while retaining backwards compatibility,
  sshd(8)'s GatewayPorts option has been extended. To allow client
  specified bind addresses for remote (-R) port forwardings, the server
  must be configured with "GatewayPorts clientspecified".

* ssh(1) and ssh-keyscan(1) now support hashing of host names and 
  addresses added to known_hosts files, controlled by the ssh(1) 
  HashKnownHosts configuration directive. This option improves user 
  privacy by hiding which hosts have been visited. At present this 
  option is off by default, but may be turned on once it receives 
  sufficient testing.

* Added options for managing keys in known_hosts files to ssh-keygen(1),
  including the ability to search for hosts by name, delete hosts by
  name and convert an unhashed known_hosts file into one with hashed
  names. These are particularly useful for managing known_hosts files
  with hashed hostnames.

* Improve account and password expiry support in sshd(8). The server
  will now warn in advance for both account and password expiry.

* sshd(8) will now log the source of connections denied by AllowUsers,
  DenyUsers, AllowGroups and DenyGroups (Bugzilla #909)

* Added AddressFamily option to sshd(8) to allow global control over
  IPv4/IPv6 usage. (Bugzilla #989)

* Improved sftp(1) client, including bugfixes and optimisations for the 
  ``ls'' command and command history and editing support using libedit.

* Improved the handling of bad data in authorized_keys files,
  eliminating fatal errors on corrupt or very large keys. (Bugzilla
  #884)

* Improved connection multiplexing support in ssh(1). Several bugs 
  have been fixed and a new "command mode" has been added to allow the
  control of a running multiplexing master connection, including 
  checking that it is up, determining its PID and asking it to exit.

* Have scp(1) and sftp(1) wait for the spawned ssh to exit before they
  exit themselves.  This prevents ssh from being unable to restore 
  terminal modes (not normally a problem on OpenBSD but common with 
  -Portable on POSIX platforms). (Bugzilla #950)

* Portable OpenSSH:

  - Add *EXPERIMENTAL* BSM audit support for Solaris systems 
    (Bugzilla #125)

  - Enable IPv6 on AIX where possible (see README.platform for
    details), working around a misfeature of AIX's getnameinfo. 
    (Bugzilla #835)

  - Teach sshd(8) to write failed login records to btmp for
    unsuccessful auth attempts. Currently this is only for password,
    keyboard-interactive and challenge/response authentication methods
    and only on Linux and HP-UX.

  - sshd(8) now sends output from failing PAM session modules to the 
    user before exiting, similar to the way /etc/nologin is handled

  - Store credentials from gssapi-with-mic authentication early enough
    to be available to PAM session modules when privsep=yes.

Checksums:
==========

- MD5 (openssh-4.0.tgz) = 7dbf15fe7c294672e8822127f50107d0
- MD5 (openssh-4.0p1.tar.gz) = 7b36f28fc16e1b7f4ba3c1dca191ac92

Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Ben Lindstrom, Darren Tucker and Tim Rice.

OpenSSH 3.9 / 3.9p1 (2004-08-18) 

OpenSSH 3.9 was released on 2004-08-18. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support to the project, especially those who contributed source and
bought T-shirts or posters.

We have a new design of T-shirt available, more info on
        http://www.openbsd.org/tshirts.html#18

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu


Changes since OpenSSH 3.8:
============================ 

* Added new "IdentitiesOnly" option to ssh(1), which specifies that it should
  use keys specified in ssh_config, rather than any keys in ssh-agent(1)

* Make sshd(8) re-execute itself on accepting a new connection. This security
  measure ensures that all execute-time randomisations are reapplied for each 
  connection rather than once, for the master process' lifetime. This includes
  mmap and malloc mappings, shared library addressing, shared library mapping 
  order, ProPolice and StackGhost cookies on systems that support such things

* Add strict permission and ownership checks to programs reading ~/.ssh/config
  NB ssh(1) will now exit instead of trying to process a config with poor 
  ownership or permissions

* Implemented the ability to pass selected environment variables between the
  client and the server. See "AcceptEnv" in sshd_config(5) and "SendEnv" in 
  ssh_config(5) for details
 
* Added a "MaxAuthTries" option to sshd(8), allowing control over the maximum
  number of authentication attempts permitted per connection

* Added support for cancellation of active remote port forwarding sessions. 
  This may be performed using the ~C escape character, see "Escape Characters"
  in ssh(1) for details
 
* Many sftp(1) interface improvements, including greatly enhanced "ls" support 
  and the ability to cancel active transfers using SIGINT (^C)

* Implement session multiplexing: a single ssh(1) connection can now carry 
  multiple login/command/file transfer sessions. Refer to the "ControlMaster" 
  and "ControlPath" options in ssh_config(5) for more information

* The sftp-server has improved support for non-POSIX filesystems (e.g. FAT)

* Portable OpenSSH: Re-introduce support for PAM password authentication, in 
  addition to the keyboard-interactive driver. PAM password authentication 
  is less flexible, and doesn't support pre-authentication password expiry but
  runs in-process so Kerberos tokens, etc are retained

* Improved and more extensive regression tests

* Many bugfixes and small improvements

Checksums:
==========

- MD5 (openssh-3.9.tgz) = 93f48bfcc1560895ae53de6bfc41689b
- MD5 (openssh-3.9p1.tar.gz) = 8e1774d0b52aff08f817f3987442a16e


Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Ben Lindstrom, Darren Tucker and Tim Rice.

OpenSSH 3.8.1p1 (2004-04-19) 

OpenSSH 3.8.1p1 was released on 2004-04-19. It is available from 
the mirrors listed at https://www.openssh.com/.

This release is a bug-fix release for the portable version. There are
no feature additions and no corresponding OpenBSD-only release.

OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support to the project, especially those who contributed source, help
with testing and have bought T-shirts or posters.

We have a new design of T-shirt available, more info on
        http://www.openbsd.org/tshirts.html#18

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu

Bugs fixed since OpenSSH 3.8p1:
=============================== 

Bug #673 - Fix compilation on NetBSD with S/Key enabled

Bug #748 - Detect and workaround broken name resolution on HP-UX 

Bug #802 - Fix linking on Tru64 when compiled with SIA support

Bug #808 - Fix PAM crash on expired password when not authenticated using 
           pam/kbdint mechanism

Bug #810 - Fix erroneous clearing of TZ environment variable

Bug #811 - Improve locked password detection across Linux variants

Bug #820 - Fix utmp corruption on Irix

Bug #825 - Fix disconnection problem when using IPv4-in-IPv6 mapped
           addresses on Solaris.

- Fix compilation on OS X systems with Kerberos/GSSAPI

- Many more minor fixes, please refer to the ChangeLog file for details

Checksums:
==========

- MD5 (openssh-3.8.1p1.tar.gz) = 1dbfd40ae683f822ae917eebf171ca42


Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Ben Lindstrom, Darren Tucker and Tim Rice.

OpenSSH 3.8.1 / 3.8.1p1 

3.8.1p1 is a Portable-only release, there is no corresponding 3.8.1 release.
See http://www.openssh.com/txt/release-3.8.1p1 for the Portable release notes.

OpenSSH 3.8 / 3.8p1 (2004-02-24) 

OpenSSH 3.8 was released on 2004-02-24. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support to the project, especially those who contributed source and
bought T-shirts or posters.

We have a new design of T-shirt available, more info on
        http://www.openbsd.org/tshirts.html#18

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu


Changes since OpenSSH 3.7.1:
============================ 

* sshd(8) now supports forced changes of expired passwords via
  /usr/bin/passwd or keyboard-interactive authentication.

  Note for AIX: sshd will now deny password access to accounts with
  passwords expired longer than their maxexpired attribute.  For
  details, see the AIX section in README.platform.

* ssh(1) now uses untrusted cookies for X11-Forwarding.
  Some X11 applications might need full access to the X11 server,
  see ForwardX11Trusted in ssh(1) and xauth(1) for more information.

* ssh(1) now supports sending application layer
  keep-alive messages to the server.  See ServerAliveInterval
  in ssh(1) for more information.

* Improved sftp(1) batch file support.

* New KerberosGetAFSToken option for sshd(8).

* Updated /etc/moduli file and improved performance for
  protocol version 2.

* Support for host keys in DNS (draft-ietf-secsh-dns-xx.txt).
  Please see README.dns in the source distribution for details.

* Fix a number of memory leaks.

* The experimental "gssapi" support has been replaced with
  the "gssapi-with-mic" to fix possible MITM attacks.
  The two versions are not compatible.

Checksums:
==========

- MD5 (openssh-3.8.tgz) = 7d5590a333d8f8aa1fa6f19e24938700
- MD5 (openssh-3.8p1.tar.gz) = 7861a4c0841ab69a6eec5c747daff6fb


Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Ben Lindstrom, Darren Tucker and Tim Rice.

OpenSSH 3.7.1p2 (2003-09-23) 

Portable OpenSSH 3.7.1p2  was released on 2003-09-23. It will be available 
from the mirrors listed at http://www.openssh.com/portable.html shortly.

Please note that this is a release to address issues in the portable 
version only. The items mentioned below do not affect the OpenBSD 
version.

OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support to the project, especially those who contributed source and
bought T-shirts or posters.

We have a new design of T-shirt available, more info on
        http://www.openbsd.org/tshirts.html#18

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu

Security Changes:
=================

  Portable OpenSSH version 3.7p1 and 3.7.1p1 contain multiple 
  vulnerabilities in the new PAM authentication code. At least one of
  these bugs is remotely exploitable (under a non-standard 
  configuration, with privsep disabled).

  OpenSSH 3.7.1p2 fixes these bugs. Please note that these bugs do not 
  exist in OpenBSD's releases of OpenSSH.

Changes since OpenSSH 3.7.1p1:
==============================

* This release disables PAM by default. To enable it, set "UsePAM yes" in 
  sshd_config. Due to complexity, inconsistencies in the specification and
  differences between vendors' PAM implementations we recommend that PAM 
  be left disabled in sshd_config unless there is a need for its use. 
  Sites using only public key or simple password authentication usually 
  have little need to enable PAM support.

* This release now requires zlib 1.1.4 to build correctly. Previous 
  versions have security problems.

* Fix compilation for versions of OpenSSL before 0.9.6. Some cipher modes 
  are not supported for older OpenSSL versions.

* Fix compilation problems on systems with a missing or lacking inet_ntoa()
  function.

* Workaround problems related to unimplemented or broken setresuid/setreuid 
  functions on several platforms.

* Fix compilation on older OpenBSD systems.

* Fix handling of password-less authentication (PermitEmptyPasswords=yes) 
  that has not worked since the 3.7p1 release.

Checksums:
==========

- MD5 (openssh-3.7.1p2.tar.gz) = 61cf5b059938718308836d00f6764a94


Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Ben Lindstrom, Darren Tucker and Tim Rice.

OpenSSH 3.7.1 / 3.7.1p1 (2004-02-25) 

OpenSSH 3.7.1 was released on 2004-02-25. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support to the project, especially those who contributed source and
bought T-shirts or posters.

We have a new design of T-shirt available, more info on
        http://www.openbsd.org/tshirts.html#18

For international orders use https://https.openbsd.org/cgi-bin/order
and for European orders, use https://https.openbsd.org/cgi-bin/order.eu

Security Changes:
=================

  All versions of OpenSSH's sshd prior to 3.7.1 contain buffer
  management errors.  It is uncertain whether these errors are
  potentially exploitable, however, we prefer to see bugs
  fixed proactively.

  OpenSSH 3.7 fixed one of these bugs.

  OpenSSH 3.7.1 fixes more similar bugs.

Changes since OpenSSH 3.6.1:
============================ 

* The entire OpenSSH code-base has undergone a license review. As
  a result, all non-ssh1.x code is under a BSD-style license with no
  advertising requirement. Please refer to README in the source
  distribution for the exact license terms.

* Rhosts authentication has been removed in ssh(1) and sshd(8).

* Changes in Kerberos support:

    - KerberosV password support now uses a file cache instead of
      a memory cache.

    - KerberosIV and AFS support has been removed.

    - KerberosV support has been removed from SSH protocol 1.

    - KerberosV password authentication support remains for SSH
      protocols 1 and 2.

    - This release contains some GSSAPI user authentication support
      to replace legacy KerberosV authentication support. At present
      this code is still considered experimental and SHOULD NOT BE
      USED.
  
* Changed order that keys are tried in public key authentication.
  The ssh(1) client tries the keys in the following order:

     1. ssh-agent(1) keys that are found in the ssh_config(5) file
     2. remaining ssh-agent(1) keys
     3. keys that are only listed in the ssh_config(5) file

  This helps when an ssh-agent(1) has many keys, where the sshd(8)
  server might close the connection before the correct key is tried.

* SOCKS5 support has been added to the dynamic forwarding mode
  in ssh(1).

* Removed implementation barriers to operation of SSH over SCTP.

* sftp(1) client can now transfer files with quote characters in
  their filenames.

* Replaced sshd(8)'s VerifyReverseMapping with UseDNS option.
  When UseDNS option is on, reverse hostname lookups are always
  performed.

* Fix a number of memory leaks.

* Support for sending tty BREAK over SSH protocol 2.

* Workaround for other vendor bugs in KEX guess handling.

* Support for generating KEX-GEX groups (/etc/moduli) in ssh-keygen(1).

* Automatic re-keying based on amount of data sent over connection.

* New AddressFamily option on client to select protocol to use (IPv4
  or IPv6).

* Experimental support for the "aes128-ctr", "aes192-ctr", and
  "aes256-ctr" ciphers for SSH protocol 2.

* Experimental support for host keys in DNS (draft-ietf-secsh-dns-xx.txt).
  Please see README.dns in the source distribution for details.

* Portable OpenSSH:

    - Replace PAM password authentication kludge with a more correct
      PAM challenge-response module from FreeBSD.

    - PAM support may now be enabled/disabled at runtime using the
      UsePAM directive.

    - Many improvements to the OpenSC smartcard support.

    - Regression tests now work with portable OpenSSH.
      Please refer to regress/README.regress in the source distribution.

    - On platforms that support it, portable OpenSSH now honors the
      UMASK, PATH and SUPATH attributes set in /etc/default/login.

    - Deny access to locked accounts, regardless of authentication
      method in use.

Checksums:
==========

- MD5 (openssh-3.7.1.tgz) = 3d2f1644d6a3d3267e5e2421f1385129
- MD5 (openssh-3.7.1p1.tar.gz) = f54e574e606c08ef63ebb1ab2f7689dc


Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Ben Lindstrom, Darren Tucker and Tim Rice.

OpenSSH 3.7 / 3.7p1 (2003-09-16) 

OpenSSH 3.7 was released on 2003-09-16. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support to the project, especially those who contributed source and
bought T-shirts or posters.

We have a new design of T-shirt available, more info on
        http://www.openbsd.org/tshirts.html#18

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu

Security Changes:
=================

  All versions of OpenSSH's sshd prior to 3.7 contain a buffer
  management error.  It is uncertain whether this error is
  potentially exploitable, however, we prefer to see bugs
  fixed proactively.

  OpenSSH 3.7 fixes this bug.

Changes since OpenSSH 3.6.1:
============================ 

* The entire OpenSSH code-base has undergone a license review. As
  a result, all non-ssh1.x code is under a BSD-style license with no
  advertising requirement. Please refer to README in the source
  distribution for the exact license terms.

* Rhosts authentication has been removed in ssh(1) and sshd(8).

* Changes in Kerberos support:

    - KerberosV password support now uses a file cache instead of
      a memory cache.

    - KerberosIV and AFS support has been removed.

    - KerberosV support has been removed from SSH protocol 1.

    - KerberosV password authentication support remains for SSH
      protocols 1 and 2.

    - This release contains some GSSAPI user authentication support
      to replace legacy KerberosV authentication support. At present
      this code is still considered experimental and SHOULD NOT BE
      USED.
  
* Changed order that keys are tried in public key authentication.
  The ssh(1) client tries the keys in the following order:

     1. ssh-agent(1) keys that are found in the ssh_config(5) file
     2. remaining ssh-agent(1) keys
     3. keys that are only listed in the ssh_config(5) file

  This helps when an ssh-agent(1) has many keys, where the sshd(8)
  server might close the connection before the correct key is tried.

* SOCKS5 support has been added to the dynamic forwarding mode
  in ssh(1).

* Removed implementation barriers to operation of SSH over SCTP.

* sftp(1) client can now transfer files with quote characters in
  their filenames.

* Replaced sshd(8)'s VerifyReverseMapping with UseDNS option.
  When UseDNS option is on, reverse hostname lookups are always
  performed.

* Fix a number of memory leaks.

* Support for sending tty BREAK over SSH protocol 2.

* Workaround for other vendor bugs in KEX guess handling.

* Support for generating KEX-GEX groups (/etc/moduli) in ssh-keygen(1).

* Automatic re-keying based on amount of data sent over connection.

* New AddressFamily option on client to select protocol to use (IPv4
  or IPv6).

* Experimental support for the "aes128-ctr", "aes192-ctr", and
  "aes256-ctr" ciphers for SSH protocol 2.

* Experimental support for host keys in DNS (draft-ietf-secsh-dns-xx.txt).
  Please see README.dns in the source distribution for details.

* Portable OpenSSH:

    - Replace PAM password authentication kludge with a more correct
      PAM challenge-response module from FreeBSD.

    - PAM support may now be enabled/disabled at runtime using the
      UsePAM directive.

    - Many improvements to the OpenSC smartcard support.

    - Regression tests now work with portable OpenSSH.
      Please refer to regress/README.regress in the source distribution.

    - On platforms that support it, portable OpenSSH now honors the
      UMASK, PATH and SUPATH attributes set in /etc/default/login.

    - Deny access to locked accounts, regardless of authentication
      method in use.

Checksums:
==========

- MD5 (openssh-3.7.tgz) = 86864ecc276c5f75b06d4872a553fa70
- MD5 (openssh-3.7p1.tar.gz) = 77662801ba2a9cadc0ac10054bc6cb37


Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller, Ben Lindstrom, Darren Tucker and Tim Rice.

OpenSSH 3.6.1p2 (2003-04-30) 

OpenSSH 3.6.1p2 was released on 2003-04-30. It is available from the
mirrors listed at https://www.openssh.com/.This is a release
of the Portable version only.

OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support and encouragement.


Changes since OpenSSH 3.6.1p1:
============================ 

* Security: corrected linking problem on AIX/gcc. AIX users are 
  advised to upgrade immediately. For details, please refer to 
  separate advisory (aixgcc.adv). 

* Corrected build problems on Irix

* Corrected build problem when building with AFS support

* Merged some changes from Openwall Linux


Checksums:
==========

- MD5 (openssh-3.6p1.tar.gz) = f3879270bffe479e1bd057aa36258696

Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller and Ben Lindstrom.

OpenSSH 3.6.1 / 3.6.1p1 (2003-04-01) 

OpenSSH 3.6.1 was released on 2003-04-01. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support to the project, especially those who contributed source and
bought T-shirts or posters.

We have a new design of T-shirt available, more info on
	http://www.openbsd.org/tshirts.html#18

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu


Changes since OpenSSH 3.6:
========================== 

* The 'kex guesses' bugfix from OpenSSH 3.6 triggers a bug
  in a few other SSH v2 implementations and causes connections to
  stall.  OpenSSH 3.6.1 disables this bugfix when interoperating
  with these implementations.


Changes between OpenSSH 3.5 and OpenSSH 3.6:
============================================

* RSA blinding is now used by ssh(1), sshd(8) and ssh-agent(1).
  in order to avoid potential timing attacks against the RSA keys.
  Older versions of OpenSSH have been using RSA blinding in
  ssh-keysign(1) only.

  Please note that there is no evidence that the SSH protocol is
  vulnerable to the OpenSSL/TLS timing attack described in
        http://crypto.stanford.edu/~dabo/papers/ssl-timing.pdf

* ssh-agent(1) optionally requires user confirmation if a key gets
  used, see '-c' in ssh-add(1).

* sshd(8) now handles PermitRootLogin correctly when UsePrivilegeSeparation
  is enabled.

* sshd(8) now removes X11 cookies when a session gets closed.

* ssh-keysign(8) is disabled by default and only enabled if the
  new EnableSSHKeysign option is set in the global ssh_config(5)
  file.

* ssh(1) and sshd(8) now handle 'kex guesses' correctly (key exchange
  guesses).

* ssh(1) no longer overwrites SIG_IGN.  This matches behaviour from
  rsh(1) and is used by backup tools.

* setting ProxyCommand to 'none' disables the proxy feature, see
  ssh_config(5).

* scp(1) supports add -1 and -2.

* scp(1) supports bandwidth limiting.

* sftp(1) displays a progressmeter.

* sftp(1) has improved error handling for scripting.


Checksums:
==========

- MD5 (openssh-3.6.1p1.tar.gz) = d4c2c88b883f097fe88e327cbb4b2e2a
- MD5 (openssh-3.6.1.tgz) = aa2acd2be17dc3fd514a1e09336aab51


Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller and Ben Lindstrom.

OpenSSH 3.6 / 3.6p1 (2003-03-31) 

OpenSSH 3.6 was released on 2003-03-31. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support to the project, especially those who contributed source and
bought T-shirts or posters.

We have a new design of T-shirt available, more info on
	http://www.openbsd.org/tshirts.html#18

For international orders use http://https.openbsd.org/cgi-bin/order
and for European orders, use http://https.openbsd.org/cgi-bin/order.eu


Changes since OpenSSH 3.5:
============================ 


* RSA blinding is now used by ssh(1), sshd(8) and ssh-agent(1).
  in order to avoid potential timing attacks against the RSA keys.
  Older versions of OpenSSH have been using RSA blinding in
  ssh-keysign(1) only.

  Please note that there is no evidence that the SSH protocol is
  vulnerable to the OpenSSL/TLS timing attack described in
        http://crypto.stanford.edu/~dabo/papers/ssl-timing.pdf

* ssh-agent(1) optionally requires user confirmation if a key gets
  used, see '-c' in ssh-add(1).

* sshd(8) now handles PermitRootLogin correctly when UsePrivilegeSeparation
  is enabled.

* sshd(8) now removes X11 cookies when a session gets closed.

* ssh-keysign(8) is disabled by default and only enabled if the
  new EnableSSHKeysign option is set in the global ssh_config(5)
  file.

* ssh(1) and sshd(8) now handle 'kex guesses' correctly (key exchange
  guesses).

* ssh(1) no longer overwrites SIG_IGN.  This matches behaviour from
  rsh(1) and is used by backup tools.

* setting ProxyCommand to 'none' disables the proxy feature, see
  ssh_config(5).

* scp(1) supports add -1 and -2.

* scp(1) supports bandwidth limiting.

* sftp(1) displays a progressmeter.

* sftp(1) has improved error handling for scripting.


Checksums:
==========

- MD5 (openssh-3.6p1.tar.gz) = 72ef1134d521cb6926c99256dad17fe0
- MD5 (openssh-3.6.tgz) = 758822b888c5c3f83a98045aef904254


Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller and Ben Lindstrom.

OpenSSH 3.5 / 3.5p1 (2002-10-15) 

OpenSSH 3.5 was released on 2002-10-15. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support and encouragement.


Changes since OpenSSH 3.4:
============================ 

* Improved support for Privilege Separation (Portability, Kerberos,
  PermitRootLogin handling).

* ssh(1) prints out all known host keys for a host if it receives an
  unknown host key of a different type.

* Fixed AES/Rijndael EVP integration for OpenSSL < 0.9.7 (caused
  problems with bounds checking patches for gcc).

* ssh-keysign(8) is disabled by default and only enabled if the
  HostbasedAuthentication option is enabled in the global ssh_config(5)
  file.

* ssh-keysign(8) uses RSA blinding in order to avoid timing attacks
  against the RSA host key.

* A use-after-free bug was fixed in ssh-keysign(8).  This bug
  broke hostbased authentication on several platforms.

* ssh-agent(1) is now installed setgid in order to avoid ptrace(2)
  attacks.

* ssh-agent(1) now restricts the access with getpeereid(2) (or
  equivalent, where available).

* sshd(8) no longer uses the ASN.1 parsing code from libcrypto when
  verifying RSA signatures.

* sshd(8) now sets the SSH_CONNECTION environment variable.

* Enhanced "ls" support for the sftp(1) client, including globbing and
  detailed listings.

* ssh(1) now always falls back to uncompressed sessions, if the
  server does not support compression.

* The default behavior of sshd(8) with regard to user settable
  environ variables has changed:  the new option PermitUserEnvironment
  is disabled by default, see sshd_config(5).

* The default value for LoginGraceTime has been changed from 600 to 120
  seconds, see sshd_config(5).

* Removed erroneous SO_LINGER handling.


Checksums:
==========

- MD5 (openssh-3.5p1.tar.gz) = 42bd78508d208b55843c84dd54dea848
- MD5 (openssh-3.5.tgz) = 79fc225dbe0fe71ebb6910f449101d23


Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller and Ben Lindstrom.

OpenSSH 3.4 / 3.4p1 (2002-06-26) 

OpenSSH 3.4 was released on 2002-06-26. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support and encouragement.


Changes since OpenSSH 3.3:
============================ 

Security Changes:
=================

  All versions of OpenSSH's sshd between 2.9.9 and 3.3
  contain an input validation error that can result in
  an integer overflow and privilege escalation.

  OpenSSH 3.4 fixes this bug.

  In addition, OpenSSH 3.4 adds many checks to detect 
  invalid input and mitigate resource exhaustion attacks.

  OpenSSH 3.2 and later prevent privilege escalation
  if UsePrivilegeSeparation is enabled in sshd_config.
  OpenSSH 3.3 enables UsePrivilegeSeparation by
  default.


Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller and Ben Lindstrom.

OpenSSH 3.3 / 3.3p1 (2002-06-21) 

OpenSSH 3.3 was released on 2002-06-21. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support and encouragement.


Changes since OpenSSH 3.2.3:
============================ 

Security Changes:
=================

- improved support for privilege separation:

	privilege separation is now enabled by default

  See UsePrivilegeSeparation in sshd_config(5)
  and http://www.citi.umich.edu/u/provos/ssh/privsep.html for more
  information.
- ssh no longer needs to be installed setuid root for protocol
  version 2 hostbased authentication, see ssh-keysign(8).
  protocol version 1 rhosts-rsa authentication still requires privileges
  and is not recommended.

Other Changes:
==============

- documentation for the client and server configuration options have
  been moved to ssh_config(5) and sshd_config(5).
- the server now supports the Compression option, see sshd_config(5).
- the client options RhostsRSAAuthentication and RhostsAuthentication now
  default to no, see ssh_config(5).
- the client options FallBackToRsh and UseRsh are deprecated.
- ssh-agent now supports locking and timeouts for keys, see ssh-add(1).
- ssh-agent can now bind to unix-domain sockets given on the command line,
  see ssh-agent(1).
- fixes problems with valid RSA signatures from putty clients.

Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller and Ben Lindstrom.

OpenSSH 3.2.3 / 3.2.3p1 (2002-05-23) 

OpenSSH 3.2.3 was released on 2002-05-23. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support and encouragement.


Changes since OpenSSH 3.2.2:
============================ 

  This release fixes several problems in OpenSSH 3.2.2:

    - a defect in the BSD_AUTH access control handling for
      OpenBSD and BSD/OS systems:

      Under certain conditions, on systems using YP with netgroups
      in the password database, it is possible that sshd does ACL
      checks for the requested user name but uses the password
      database entry of a different user for authentication. This
      means that denied users might authenticate successfully while
      permitted users could be locked out (OpenBSD PR 2659).

    - login/tty problems on Solaris (bug #245)

    - build problems on Cygwin systems


Changes between OpenSSH 3.1 and OpenSSH 3.2.2:
==============================================

  Security Changes:
  =================
  
  - fixed buffer overflow in Kerberos/AFS token passing
  - fixed overflow in Kerberos client code
  - sshd no longer auto-enables Kerberos/AFS
  - experimental support for privilege separation,
    see UsePrivilegeSeparation in sshd(8) and
  	  http://www.citi.umich.edu/u/provos/ssh/privsep.html
    for more information.
  - only accept RSA keys of size SSH_RSA_MINIMUM_MODULUS_SIZE (768) or larger
  
  Other Changes:
  ==============
  
  - improved smartcard support (including support for OpenSC,
    see www.opensc.org)
  - improved Kerberos support (including support for MIT-Kerberos V)
  - fixed stderr handling in protocol v2
  - client reports failure if -R style TCP forwarding fails in protocol v2
  - support configuration of TCP forwarding during interactive sessions (~C)
  - improved support for older sftp servers
  - improved support for importing old DSA keys (from ssh.com software).
  - client side support for PASSWD_CHANGEREQ in protocol v2
  - fixed waitpid race conditions
  - record correct lastlogin time

Reporting Bugs:
===============

- please read http://www.openssh.com/report.html
  and http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller and Ben Lindstrom.

OpenSSH 3.2.2 / 3.2.2p1 (2002-05-16) 

OpenSSH 3.2.2 was released on 2002-05-16. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support and encouragement.

Security Changes:
=================

- fixed buffer overflow in Kerberos/AFS token passing
- fixed overflow in Kerberos client code
- sshd no longer auto-enables Kerberos/AFS
- experimental support for privilege separation,
  see UsePrivilegeSeparation in sshd(8) and
	  http://www.citi.umich.edu/u/provos/ssh/privsep.html
  for more information.
- only accept RSA keys of size SSH_RSA_MINIMUM_MODULUS_SIZE (768) or larger

Other Changes:
==============

- improved smartcard support (including support for OpenSC, see www.opensc.org)
- improved Kerberos support (including support for MIT-Kerberos V)
- fixed stderr handling in protocol v2
- client reports failure if -R style TCP forwarding fails in protocol v2
- support configuration of TCP forwarding during interactive sessions (~C)
- improved support for older sftp servers
- improved support for importing old DSA keys (from ssh.com software).
- client side support for PASSWD_CHANGEREQ in protocol v2
- fixed waitpid race conditions
- record correct lastlogin time

Reporting Bugs:
===============

- please read http://www.openssh.com/report.html and
  http://bugzilla.mindrot.org/

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller and Ben Lindstrom.

OpenSSH 3.1 / 3.1p1 (2004-04-09) 

OpenSSH 3.1 was released on 2004-04-09. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support and encouragement.


Important Changes:
==================

- /etc/ssh/ now default directory for keys and configuration files
- ssh-keygen no longer defaults to a specific key type (rsa1);
  use ssh-keygen -t {rsa,dsa,rsa1}
- sshd x11 forwarding listens on localhost by default;
  see sshd X11UseLocalhost option to revert to prior behaviour
  if your older X11 clients do not function with this configuration


Other Changes:
==============

- ssh ~& escape char functions now for both protocol versions
- sshd ReverseMappingCheck option changed to VerifyReverseMapping
  to clarify its function; ReverseMappingCheck can still be used
- public key fingerprint is now logged with LogLevel=VERBOSE
- reason logged for disallowed logins (e.g., no shell, etc.)
- more robust error handling for x11 forwarding
- improved packet/window size handling in ssh2
- use of regex(3) has been removed
- fix SIGCHLD races in sshd (seen on Solaris)
- sshd -o option added
- sftp -B -R -P options added
- ssh-add now adds all 3 default keys
- ssh-keyscan bug fixes
- ssh-askpass for hostkey dialog
- fix fd leak in sshd on SIGHUP
- TCP_NODELAY set on X11 and TCP forwarding endpoints


OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller and Ben Lindstrom.

OpenSSH 3.0.2 / 3.0.2p1 (2002-12-04) 

OpenSSH 3.0.2 was released on 2002-12-04. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support and encouragement.

Important Changes:
==================

        This release fixes a vulnerability in the UseLogin option
        of OpenSSH.  This option is not enabled in the default
        installation of OpenSSH.

        However, if UseLogin is enabled by the administrator, all
        versions of OpenSSH prior to 3.0.2 may be vulnerable to
        local attacks.

        The vulnerability allows local users to pass environment
        variables (e.g. LD_PRELOAD) to the login process.  The login
        process is run with the same privilege as sshd (usually
        with root privilege).

        Do not enable UseLogin on your machines or disable UseLogin
        again in /etc/sshd_config:
		UseLogin no

We also have received many reports about attacks against the crc32
bug.  This bug has been fixed about 12 months ago in OpenSSH 2.3.0.
However, these attacks cause non-vulnerable daemons to chew a lot
of cpu since the crc32 attack sends a tremendously large amount of
data which must be processed.

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller and Ben Lindstrom.



The following patch fixes the UseLogin vulnerability in OpenSSH 3.0.1 and
earlier releases.

--- session.c	11 Oct 2001 13:45:21 -0000	1.108
+++ session.c	1 Dec 2001 22:14:39 -0000
@@ -875,6 +875,7 @@
 		child_set_env(&env, &envsize, "TZ", getenv("TZ"));
 
 	/* Set custom environment options from RSA authentication. */
+	if (!options.use_login)
 	while (custom_environment) {
 		struct envstring *ce = custom_environment;
 		char *s = ce->s;

OpenSSH 3.0.1 / 3.0.1p1 (2001-11-19) 

OpenSSH 3.0.1 was released on 2001-11-19. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

We would like to thank the OpenSSH community for their continued
support and encouragement.

Important Changes:
==================

        A security hole that may allow an attacker to authenticate
        if -- and only if -- the administrator has enabled KerberosV.
        By default, OpenSSH KerberosV support only becomes active
        after KerberosV has been properly configured.

        An excessive memory clearing bug (which we believe to be
        unexploitable) also exists, but since this may cause daemon
        crashes, we are providing a fix as well.

        Various other non-critical fixes (~& support and more).

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller and Ben Lindstrom.

OpenSSH 3.0 / 3.0p1 (2001-11-06) 

OpenSSH 3.0 was released on 2001-11-06. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

This release contains many portability bug-fixes (listed in the
ChangeLog) as well as several new features (listed below).

We would like to thank the OpenSSH community for their continued
support and encouragement.

Important Changes:
==================

1) SSH protocol v2 is now the default protocol version

	use the 'Protocol' option from ssh(1) and sshd(8) if
	you need to change this.

2) The files
	/etc/ssh_known_hosts2
	~/.ssh/known_hosts2
	~/.ssh/authorized_keys2
   are now obsolete, you can use
	/etc/ssh_known_hosts
	~/.ssh/known_hosts
	~/.ssh/authorized_keys
   For backward compatibility ~/.ssh/authorized_keys2 will still used for
   authentication and hostkeys are still read from the known_hosts2.
   However, those deprecated files are considered 'readonly'.  Future
   releases are likely not to read these files.

3) The CheckMail option in sshd_config is deprecated, as sshd(8) no longer
   checks for new mail.

4) X11 cookies are now stored in $HOME.

New Features:
=============

1) Smartcard support in the ssh client and agent based on work by
   University of Michigan CITI (http://www.citi.umich.edu/projects/smartcard/).
2) support for Rekeying in protocol version 2

3) improved Kerberos support in protocol v1 (KerbIV and KerbV)

4) backward compatibility with older commercial SSH versions >= 2.0.10

5) getopt(3) is now used by all programs

6) dynamic forwarding (use ssh(1) as your socks server)

7) ClearAllForwardings in ssh(1)

8) ssh(1) now checks the hostkey for localhost (NoHostAuthenticationForLocalhost yes/no).

9) -F option in ssh(1)

10) ssh(1) now has a '-b bindaddress' option

11) scp(1) allows "scp /file localhost:/file"

12) The AuthorizedKeysFile option allows specification of alternative
    files that contain the public keys that can be used for user authentication
    (e.g. /etc/ssh_keys/%u, see sshd(8))

13) extended AllowUsers user@host syntax in sshd(8)

14) improved challenge-response support (especially for systems supporting BSD_AUTH)

15) sshd(8) can specify time args as 1h, 2h30s etc.

16) sshd(8) transmits the correct exit status for remote execution with protocol version 2.

17) ssh-keygen(1) can import private RSA/DSA keys generated with the commercial version

18) ssh-keyscan(1) supports protocol version 2

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller and Ben Lindstrom.

OpenSSH 2.9p2 (2001-06-17) 

Portable OpenSSH 2.9p2 has just been uploaded and shall be making its
way to the mirrors listed at http://www.openssh.com/portable.html
shortly.

This release fixes the "cookies" file deletion problem reported on
BUGTRAQ as well as a few other minor (non-security) bugs. No new
features have been added in this release.

Regards,
Damien Miller

OpenSSH 2.9.9 / 2.9.9p1 (2001-09-25) 

OpenSSH 2.9.9 has just been uploaded. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH 2.9.9 fixes a weakness in the key file option handling,
including source IP based access control.

OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

This release contains many portability bug-fixes (listed in the
ChangeLog) as well as several new features (listed below).

We would like to thank the OpenSSH community for their continued
support and encouragement.

Security Notes:
===============

This release fixes weakness in the source IP based access control
for SSH protocol v2 public key authentication:

        Versions of OpenSSH between 2.5 and 2.9.9 are
        affected if they use the 'from=' key file option in
        combination with both RSA and DSA keys in
        ~/.ssh/authorized_keys2.

        Depending on the order of the user keys in
        ~/.ssh/authorized_keys2 sshd might fail to apply the
        source IP based access control restriction (e.g.
        from="10.0.0.1") to the correct key:

        If a source IP restricted key (e.g. DSA key) is
        immediately followed by a key of a different type
        (e.g. RSA key), then key options for the second key
        are applied to both keys, which includes 'from='.

	This means that users can circumvent the system policy
	and login from disallowed source IP addresses.
	

Important Changes:
==================

OpenSSH 2.9.9 might have upgrade issues introduced by the long time
between releases, which may affect people in unforeseen ways:

1) The files
	/etc/ssh_known_hosts2
	~/.ssh/known_hosts2
	~/.ssh/authorized_keys2
   are now obsolete, you can use
	/etc/ssh_known_hosts
	~/.ssh/known_hosts
	~/.ssh/authorized_keys
   For backward compatibility ~/.ssh/authorized_keys2 is still used for
   authentication and hostkeys are still read from the known_hosts2.
   However, old files are considered 'readonly'.  Future releases are
   likely to not read these files.

2) The CheckMail option in sshd_config is deprecated, sshd no longer
   checks for new mail.

3) X11 cookies are stored in $HOME

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller and Ben Lindstrom.

OpenSSH 2.9 / 2.9p1 (2001-04-29) 

OpenSSH 2.9 has just been uploaded. It is available from the
mirrors listed at https://www.openssh.com/.
OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0
implementation and includes sftp client and server support.

This release contains many portability bug-fixes (listed in the
ChangeLog) as well as several new features (listed below).

We would like to thank the OpenSSH community for their continued
support and encouragement.

Important Changes:
==================

WARNING: SSH protocol v2 is now the default protocol version

	use the 'Protocol' option from ssh(1) and sshd(8) if
	you want to change this.

SSH protocol v2 implementation adds support for:

        HostbasedAuthentication, similar to RhostsRSA in SSH protocol
        v1

        Rekeying (negotiate new encryption keys for the current SSH
        session, try ~R in interactive SSH sessions)

        updated DH group exchange:
        	draft-ietf-secsh-dh-group-exchange-01.txt

        client option HostKeyAlgorithms

        server options ClientAliveInterval and ClientAliveCountMax

        tty mode passing

general:

        gid swapping in sshd (fixes access to /home/group/user based
        directory structures)

        Dan Kaminsky <dankamin@cisco.com> contributed an experimental
        SOCKS4 proxy to the ssh client (yes, client not the server).
        Use 'ssh -D 1080 server' if you want to try this out.

	server option PrintLastLog

	improvements for scp > 2GB

	improved ListenAddress option.
	You can now use ListenAddress host:port

	improved interoperability (bug detection for older implementations)

	improved documentation

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller and Ben Lindstrom.

OpenSSH 2.5.2p2 (2001-03-22) 

Portable OpenSSH 2.5.2p2 is now available from the mirror sites
listed at http://www.openssh.com/portable.html

Security related changes:
	Improved countermeasure against "Passive Analysis of SSH
	(Secure Shell) Traffic"
	http://openwall.com/advisories/OW-003-ssh-traffic-analysis.txt

	The countermeasures introduced in earlier OpenSSH-2.5.x versions
	caused interoperability problems with some other implementations.

	Improved countermeasure against "SSH protocol 1.5 session
	key recovery vulnerability"
	http://www.core-sdi.com/advisories/ssh1_sessionkey_recovery.htm

New options:
	permitopen authorized_keys option to restrict portforwarding.

	PreferredAuthentications allows client to specify the order in which
	authentication methods are tried.

Sftp:
	sftp client supports globbing (get *, put *).

	Support for sftp protocol v3 (draft-ietf-secsh-filexfer-01.txt).

	Batch file (-b) support for automated transfers

Performance:
	Speedup DH exchange. OpenSSH should now be significantly faster when
	connecting use SSH protocol 2.

	Preferred SSH protocol 2 cipher is AES with hmac-md5. AES offers
	much faster throughput in a well scrutinised cipher.

Bugfixes:
	stderr handling fixes in SSH protocol 2.

	Improved interoperability.

Client:
	The client no longer asks for the passphrase if the key
	will not be accepted by the server (SSH2_MSG_USERAUTH_PK_OK)

Miscellaneous:
	scp should now work for files > 2GB

	ssh-keygen can now generate fingerprints in the "bubble babble"
	format for exchanging fingerprints with SSH.COM's SSH protocol 2
	implementation.

Portable version:
	Better support for the PRNGd[1] entropy collection daemon. The
	--with-egd-pool configure option has been deprecated in favour
	of --with-prngd-socket and the new --with-prngd-port options.
	The latter allows collection of entropy from a localhost
	socket.

	configure ensures that scp is in the $PATH set by the server
	(unless a custom path is specified).

-d

[1] http://www.aet.tu-cottbus.de/personen/jaenicke/postfix_tls/prngd.html

OpenSSH 2.5.1p2 (2001-03-20) 

Portable OpenSSH 2.5.1p2 has just been uploaded and will be making its
way to the mirror sites (http://www.openssh.com/portable.html)in due
course.

This release contains primarily bug-fixes over 2.5.1p1 but an upgrade is
recommended. Specific bug-fixes include:

 - Fixed endianness issue causing failures when using Rijndael/AES cipher
 - Fix PAM failures on Solaris and Linux
 - Fix RPM spec file for Redhat systems
 - Fixed several compatibility functions
 - Fix entropy collection code for SCO3 and NeXTStep
 - Many other minor fixes (see Changelog for details)

This release includes Mark Roth's mdoc2man.pl script which can be used
to fix up the manpages on systems that lack the full andoc set of
macros (e.g. Solaris). A future release of portable OpenSSH will automate
this scripts use for systems that require it.

-d

OpenSSH 2.5.1p1 (2001-02-19) 

Portable OpenSSH 2.5.1p1 has just been uploaded. It will be available 
from the mirrors listed at http://www.openssh.com/portable.html shortly.

OpenSSH is a 100% complete SSH 1.3 & 1.5 protocol implementation and 
a 99% SSH 2 protocol implementation, including sftp client and server
support.

This release contains many portability bug-fixes (listed in the
ChangeLog) as well as several new features (listed below).

OpenSSH 2.5.0p1 was skipped because of interoperability issues with 
ssh-1.2.18 => ssh-1.2.22.

We would like to thank the OpenSSH community for their continued support
and encouragement.

Important Changes:
==================

1) Features added to the implementation of the SSH 2 protocol:

    * agent forwarding
    * support for -R forwarding
    * RSA host and userkeys
    * extended support for older SSH 2 protocol implementations

    OpenSSH still lacks support for rekeying, so you have to turn off
    rekeying if your server tries to force this feature.

    The next release of OpenSSH will probably support rekeying.

2) Damien Miller contributed an interactive sftp client.

    The sftp client works for both SSH protocol versions.

3) David Mazieres' ssh-keyscan has been added to the OpenSSH distribution.

4) Now there are three types of keys in OpenSSH:

    RSA1 is used by the SSH 1 protocol only,
    RSA and DSA keys are used by the SSH 2 protocol implementation.

    You can generate RSA keys for use with SSH 2 protocol with:

        $ ssh-keygen -t rsa -f /etc/ssh_host_rsa_key

        To use RSA or DSA keys in SSH 2 protocol, simply
        add the public keys to the .ssh/authorised_keys2 file.

    IdentityFile2, HostDsaKey and DSAAuthentication are obsolete:

    You can use multiple IdentityFile and HostKey options instead, e.g
        HostKey /etc/ssh_host_key
        HostKey /etc/ssh_host_dsa_key
        HostKey /etc/ssh_host_rsa_key
    in /etc/sshd_config

    The option DSAAuthentication has been replaced by PubkeyAuthentication.

    Fingerprinting works for all types of keys:

        $ ssh-keygen -l -f $HOME/.ssh/{authorized_keys,known_hosts}{,2}

5) Important changes in the implementation of SSH 1 protocol:

    The OpenSSH server does not require a privileged source port for
    RhostsRsaAuthentication, since it adds no additional security.

    Interoperation with SSH 1.4 protocol

6) New option HostKeyAlias

    This option allows the user to record the host key under a
    different name. This is useful for tunneling over
    forwarded connections or if you run multiple sshd's on
    different ports on the same machine.

    Alternatively you can use the UserKnownHostsFile or 
    UserKnownHostsFile2 options to specify separate host key
    files for the connection.

7) The ReverseMappingCheck is now optional in sshd_config.

    If you combine this with the 'sshd -u0' option the server
    will not do DNS lookups when a client connects.

8) Stricter Hostkey Checking

9) Option Change Summary:

    a) New or changed:

        ChallengeResponseAuthentication
        MACs
        PubkeyAuthentication

        HostkeyAlias        (Client only)

        Banner              (Server only)
        ReverseMappingCheck (Server only)

        PermitRootLogin     {yes,without-password,forced-commands-only,no}

        {Allow,Deny}Groups  now support supplementary groups

        sshd -D             for monitoring scripts or inittab
        ssh -t              multiple -t force tty allocation

    b) Obsolete:

        DsaAuthentication   (use PubkeyAuthentication instead)
        HostDsaKey          (use HostKey)
        Identityfile2       (use Identityfile or -i)
        SkeyAuthentication  (use ChallengeResponseAuthentication)
        TisAuthentication   (use ChallengeResponseAuthentication)

OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt,
Kevin Steves, Damien Miller and Ben Lindstrom.

OpenSSH 2.3.0p1 (2000-11-06) 

This is to announce the release of portable openssh-2.3.0p1. This
release includes many new features and bug fixes. This is a
recommended upgrade if you are using 2.2.0p1 or an older release.

Portable OpenSSH is available from one of the many mirrors listed at
http://www.openssh.com/portable.html

Some of the more notable features include:

- Rijndael support for SSH2. Use the "Ciphers" configuration directive 
  to enable it. (Markus Friedl <markus@cvs.openbsd.org>

- Cygwin support (Corinna Vinschen <vinschen@cygnus.com>)

- sftp-server support (Markus Friedl <markus@cvs.openbsd.org>)

- SSH1 single-des support for interop with Cisco routers. This cipher 
  is never enabled automatically, you have to either specify "-c des" 
  or enable it using "Cipher des" in a config file. (Markus Friedl
  <markus@cvs.openbsd.org>

- Support expired password change through PAM (Steve VanDevender's
  <stevev@darkwing.uoregon.edu>)

- Better compatibility with buggy SSH implementations (Markus Friedl
  <markus@cvs.openbsd.org>

- S/key support for SSH2, based on kbd-interactive auth (Markus Friedl
  <markus@cvs.openbsd.org> and mkiernan@avantgo.com

- scp now supports "-o" option (Ben Lindstron <mouring@pconline.com>)

Please refer to the ChangeLog for a full list of features and bugfixes.

Regards,
Damien Miller

OpenSSH 2.2.0p1 (2000-09-01) 

Version 2.2.0p1 of portable OpenSSH has just been uploaded to the
master site and should be making its way to the mirrors in due
course. 

http://www.openssh.com/portable.html

This release contains several new features and bugfixes relative to
the previous 2.1.1p4 release. In particular:

- DSA key support in ssh-agent. Please not that this will not 
  interop with ssh.com's ssh-agent (Markus Friedl)
- sshd now implements Random Early Drop connection rate limiting,
  which can help mitigate DoS attacks against sshd. See the 
  `MaxStartups' option in the sshd manpage for details (Markus Friedl)
- `-u' option to sshd allow logging of hostnames (rather than IP 
  addresses) in wtmp when `UseLogin' is set to `yes'. (Markus Friedl)
- Escape character `~' support in SSH2 (Markus Friedl)
- Interop with SSH.COM ssh 2.3.0 (Markus Friedl)
- Fix problems when sshd is run from inetd
- Better SunOS 4.1.x support (Nate Itkin and Charles Levert)
- Solaris package support, see contrib/solaris (Rip Loomis)
- Work around connection freezes on HPUX and SunOS 4 (Lutz Jaenicke,
  Tamito KAJIYAMA)
- Fix ^C ignored issue on Solaris. (Gert Doering, John Horne and 
  Garrick James)
- Further improved NeXT support. (Ben Lindstrom, Mark Miller)
- Lots of other minor fixes (see ChangeLog for details)

This release has been tested on HPUX (10.20, 11.00), Irix (5.3,
6.5), Linux (Debian, Redhat, Slackware, SuSE), NeXTstep 3 (HPPA,
i386, m68k), OpenStep (i386, m68k, Sparc), SCO Unixware 7.1.0, SCO
OpenServer 5.0.5, Solaris 2.7 (Sparc), Solaris 2.8 (i386, Sparc),
SNI/Reliant Unix, DEC OSF/Tru64 5.0.

Many thanks to those who contributed bug reports, fixes and testing 
time.

Regards,
Damien Miller

OpenSSH 2.1.1p4 (2000-07-16) 

I have just uploaded portable OpenSSH 2.1.1p4, it should be making
its way to the mirrors listed at http://www.openssh.com/portable.html
soon.

This release contains several bugfixes from the OpenBSD team,
primarily the config file parsing problem reported by Ralf 
Engelschall <rse@engelschall.com>

Regards,
Damien Miller

--------------- Changelog:

20000716
 - Release 2.1.1p4

20000715
 - (djm) OpenBSD CVS updates
   - provos@cvs.openbsd.org  2000/07/13 16:53:22
     [aux.c readconf.c servconf.c ssh.h]
     allow multiple whitespace but only one '=' between tokens, bug report from
     Ralf S. Engelschall <rse@engelschall.com> but different fix. okay deraadt@
   - provos@cvs.openbsd.org  2000/07/13 17:14:09
     [clientloop.c]
     typo; todd@fries.net
   - provos@cvs.openbsd.org  2000/07/13 17:19:31
     [scp.c]
     close can fail on AFS, report error; from Greg Hudson <ghudson@mit.edu>
   - markus@cvs.openbsd.org  2000/07/14 16:59:46
     [readconf.c servconf.c]
     allow leading whitespace. ok niels
   - djm@cvs.openbsd.org     2000/07/14 22:01:38
     [ssh-keygen.c ssh.c]
     Always create ~/.ssh with mode 700; ok Markus
 - Fixes for SunOS 4.1.4 from Gordon Atwood <gordon@cs.ualberta.ca>
   - Include floatingpoint.h for entropy.c
   - strerror replacement

---------------

OpenSSH 2.1.1p3 (2000-07-12) 

The 2.1.1p3 release of portable OpenSSH has been uploaded to the
OpenBSD ftp master site. In a few hours it will be available from one
of the many mirrors listed at:

http://www.openssh.com/portable.html

This release fixes several bugs reported since the previous release
and extends portability to NeXT and Reliant Unix.

As usual, the OpenBSD team has been hard at work further polishing and
enhancing OpenSSH. This release brings a new configuration directive
"MaxStartups" which mitigates connection flooding attacks, further
details are in the sshd man-page.

Another noteworthy difference from previous releases is that
'FallBackToRsh' now defaults to 'no'. Users of this feature may need
to edit their /etc/ssh_config or ~/.ssh/config files to achieve the
same behavior.

Again, thanks to those who reported bugs, tested the snapshot and sent
fixes.

Regards,
Damien Miller

------------------ Changelog

20000712
 - (djm) Remove -lresolve for Reliant Unix
 - (djm) OpenBSD CVS Updates:
   - deraadt@cvs.openbsd.org 2000/07/11 02:11:34
     [session.c sshd.c ]
     make MaxStartups code still work with -d; djm
   - deraadt@cvs.openbsd.org 2000/07/11 13:17:45
     [readconf.c ssh_config]
     disable FallBackToRsh by default
 - (djm) Replace in_addr_t with u_int32_t in bsd-inet_aton.c. Report from
   Ben Lindstrom <mouring@pconline.com>
 - (djm) Make building of X11-Askpass and GNOME-Askpass optional in RPM
   spec file.
 - (djm) Released 2.1.1p3

20000711
 - (djm) Fixup for AIX getuserattr() support from Tom Bertelson
   <tbert@abac.com>
 - (djm) ReliantUNIX support from Udo Schweigert <ust@cert.siemens.de>
 - (djm) NeXT: dirent structures to get scp working from Ben Lindstrom 
   <mouring@pconline.com>
 - (djm) Fix broken inet_ntoa check and ut_user/ut_name confusion, report 
   from Jim Watt <jimw@peisj.pebio.com>
 - (djm) Replaced bsd-snprintf.c with one from Mutt source tree, it is known
   to compile on more platforms (incl NeXT).
 - (djm) Added bsd-inet_aton and configure support for NeXT
 - (djm) Misc NeXT fixes from Ben Lindstrom <mouring@pconline.com>
 - (djm) OpenBSD CVS updates:
   - markus@cvs.openbsd.org  2000/06/26 03:22:29
     [authfd.c]
     cleanup, less cut&paste
   - markus@cvs.openbsd.org  2000/06/26 15:59:19
     [servconf.c servconf.h session.c sshd.8 sshd.c]
     MaxStartups: limit number of unauthenticated connections, work by 
     theo and me
   - deraadt@cvs.openbsd.org 2000/07/05 14:18:07
     [session.c]
     use no_x11_forwarding_flag correctly; provos ok
   - provos@cvs.openbsd.org  2000/07/05 15:35:57
     [sshd.c]
     typo
   - aaron@cvs.openbsd.org   2000/07/05 22:06:58
     [scp.1 ssh-agent.1 ssh-keygen.1 sshd.8]
     Insert more missing .El directives. Our troff really should identify 
     these and spit out a warning.
   - todd@cvs.openbsd.org    2000/07/06 21:55:04
     [auth-rsa.c auth2.c ssh-keygen.c]
     clean code is good code
   - deraadt@cvs.openbsd.org 2000/07/07 02:14:29
     [serverloop.c]
     sense of port forwarding flag test was backwards
   - provos@cvs.openbsd.org  2000/07/08 17:17:31
     [compat.c readconf.c]
     replace strtok with strsep; from David Young <dyoung@onthejob.net>
   - deraadt@cvs.openbsd.org 2000/07/08 19:21:15
     [auth.h]
     KNF
   - ho@cvs.openbsd.org      2000/07/08 19:27:33
     [compat.c readconf.c]
     Better conditions for strsep() ending.
   - ho@cvs.openbsd.org      2000/07/10 10:27:05
     [readconf.c]
     Get the correct message on errors. (niels@ ok)
   - ho@cvs.openbsd.org      2000/07/10 10:30:25
     [cipher.c kex.c servconf.c]
     strtok() --> strsep(). (niels@ ok)
 - (djm) Fix problem with debug mode and MaxStartups
 - (djm) Don't generate host keys when $(DESTDIR) is set (e.g. during RPM
   builds)
 - (djm) Add strsep function from OpenBSD libc for systems that lack it

20000709
 - (djm) Only enable PAM_TTY kludge for Linux. Problem report from
   Kevin Steves <stevesk@sweden.hp.com>
 - (djm) Match prototype and function declaration for rresvport_af.
   Problem report from Niklas Edmundsson <nikke@ing.umu.se>
 - (djm) Missing $(DESTDIR) on host-key target causing problems with RPM 
   builds. Problem report from Gregory Leblanc <GLeblanc@cu-portland.edu>
 - (djm) Replace ut_name with ut_user. Patch from Jim Watt
   <jimw@peisj.pebio.com>
 - (djm) Fix pam sprintf fix
 - (djm) Cleanup entropy collection code a little more. Split initialisation
   from seeding, perform initialisation immediately at start, be careful with
   uids. Based on problem report from Jim Watt <jimw@peisj.pebio.com>
 - (djm) More NeXT compatibility from Ben Lindstrom <mouring@pconline.com>
   Including sigaction() et al. replacements
 - (djm) AIX getuserattr() session initialisation from Tom Bertelson 
   <tbert@abac.com>

20000708
 - (djm) Fix bad fprintf format handling in auth-pam.c. Patch from 
   Aaron Hopkins <aaron@die.net>
 - (djm) Fix incorrect configure handling of --with-rsh-path option. Fix from
   Lutz Jaenicke <Lutz.Jaenicke@aet.TU-Cottbus.DE>
 - (djm) Fixed undefined variables for OSF SIA. Report from 
   Baars, Henk <Hendrik.Baars@nl.origin-it.com>
 - (djm) Handle EWOULDBLOCK returns from read() and write() in atomicio.c 
   Fix from Marquess, Steve Mr JMLFDC <Steve.Marquess@DET.AMEDD.ARMY.MIL>
 - (djm) Don't use inet_addr. 

20000702
 - (djm) Fix brace mismatch from Corinna Vinschen <vinschen@cygnus.com>
 - (djm) Stop shadow expiry checking from preventing logins with NIS. Based
   on fix from HARUYAMA Seigo <haruyama@nt.phys.s.u-tokyo.ac.jp>
 - (djm) Use standard OpenSSL functions in auth-skey.c. Patch from
   Chris, the Young One <cky@pobox.com>
 - (djm) Fix scp progress meter on really wide terminals. Based on patch 
   from James H. Cloos Jr. <cloos@jhcloos.com>

------------------

OpenSSH 2.1.1p2 (2000-07-01) 

Announcing the release of portable OpenSSH 2.1.1p2.

This release primarily contains fixes to the bugs that have been
reported over the last month, in particular:

 - Invalid time bring written to utmp/wtmp on systems using bash2

 - Several lastlog fixes

 - AIX, SCO, Irix portability fixes

 - Avoid failures on PAM systems when using PAM authentication modules
   which require a tty.

 - Entropy collection fixes for Solaris.

 - EGD robustness improvements

 - Fixes and enhancements from the OpenBSD team:
   - Fixed options processing in authorized_keys2 file
   - Compatibility with commercial SSH 2.0.13 and 2.2.0
   - Numerous minor fixes

There are also a couple of new features:

 - Shadow password expiry support (no password change support yet)

 - Irix 6.x array sessions, project IDs and system audit trail IDs

 - Beginnings of Tru64 / OSF SIA (Security Integration Architecture) 
   support

 - Beginnings of NeXT support

Version 2.1.1p2 will be available from the mirrors listed at
http://www.openssh.com/portable.html (as soon as they update).

Many thanks to all those who tested the snapshots and/or contributed
bug reports and patches

Regards,
Damien Miller

OpenSSH 2.1.1p1 (2000-06-09) 

Announcing the availability of portable OpenSSH 2.1.1p1.

This release contains the fix for the "UseLogin yes" vulnerability
identified in Markus' release and several other enhancements and
bugfixes. Including:

 - Better login code. Andre Lucas has rewritten the login code to 
   be much more modular and extensible. In the process he has fixed 
   the problems with Solaris utmp[x].

 - Revised the entropy collection code to be faster and more reliable.

 - Fix for RSA host restrictions ("from=" in authorized_keys)

It is recommended that all users upgrade to this version.

Portable OpenSSH 2.1.1p1 is available from one of the many mirrors
listed at: http://www.openssh.com/portable.html

Regards,
Damien Miller

OpenSSH 2.1.0p1 (2000-05-09) 

This is to announce the release of openssh-2.1.0, the first stable
release of portable OpenSSH to incorporate support for the SSH2
protocol.

The SSH2 protocol offers a number of advantages over the SSH1 protocol
including standards compliance (SSH2 is on the IETF standards
track[1]), improved security and operation without RSA (which is
patented in some countries).

The SSH2 support in OpenSSH has been developed by Markus Friedl, with
support from the OpenBSD team.

This is also the first version of the portable version of OpenSSH
to offer built-in entropy collection. This removes the requirement
for EGD on systems that lack a /dev/random driver. As a result,
OpenSSH-2.1.0 now requires a recent version of OpenSSL[2] to compile
(version 0.9.5 or later).

NB. The portable version of OpenSSH is currently in the
process of merging its webpages with the official OpenBSD
project. Please use http://www.openssh.com/ from now
on. Distribution files are also available from the mirrors listed at
http://violet.ibs.com.au/openssh/files/MIRRORS.html

Please read http://www.openssh.com/report.html before reporting bugs.
Patches, bug reports, developer and user queries are welcome on the
mailing list (http://www.openssh.com/list.html).
Regards,
Damien Miller

[1] http://www.ietf.org/html.charters/secsh-charter.html
[2] http://www.openssl.org/

OpenSSH 1.2.3p1 (2000-03-24) 

The Unix/Linux port of OpenSSH 1.2.3 was released yesterday and should
be available from a mirror near you. A mirror list is available from:

http://violet.ibs.com.au/openssh/files/MIRRORS.html

This release fixes the bugs reported since 1.2.2p1 and contains many
cleanups from the OpenBSD tree.

In particular, the OpenSSL detection problems have been resolved.

The layout has changed a little bit. The packages/ subdirectory has
been replaced with a contrib/ subdirectory which contains platform
specific code and other patches. Submissions are welcome.

Enjoy,
Damien Miller

20000317
 - Clarified --with-default-path option.
 - Added -blibpath handling for AIX to work around stupid runtime linking.
   Problem elucidated by gshapiro@SENDMAIL.ORG by way of Jim Knoble
   <jmknoble@pobox.com>
 - Checks for 64 bit int types. Problem report from Mats Fredholm
   <matsf@init.se>
 - OpenBSD CVS updates:
   - [atomicio.c auth-krb4.c bufaux.c channels.c compress.c fingerprint.c] 
     [packet.h radix.c rsa.c scp.c ssh-agent.c ssh-keygen.c sshconnect.c]
     [sshd.c]
     pedantic: signed vs. unsigned, void*-arithm, etc
   - [ssh.1 sshd.8]
     Various cleanups and standardizations.
 - Runtime error fix for HPUX from Otmar Stahl 
   <O.Stahl@lsw.uni-heidelberg.de>

20000316
 - Fixed configure not passing LDFLAGS to Solaris. Report from David G. 
   Hesprich <dghespri@sprintparanet.com>
 - Propagate LD through to Makefile
 - Doc cleanups
 - Added blurb about "scp: command not found" errors to UPGRADING

20000315
 - Fix broken CFLAGS handling during search for OpenSSL. Fixes va_list
   problems with gcc/Solaris.
 - Don't free argument to putenv() after use (in setenv() replacement). 
   Report from Seigo Tanimura <tanimura@r.dl.itc.u-tokyo.ac.jp>
 - Created contrib/ subdirectory. Included helpers from Phil Hands' 
   Debian package, README file and chroot patch from Ricardo Cerqueira
   <rmcc@clix.pt>
 - Moved gnome-ssh-askpass.c to contrib directory and removed config 
   option.
 - Slight cleanup to doc files
 - Configure fix from Bratislav ILICH <bilic@zepter.ru>

20000314
 - Include macro for IN6_IS_ADDR_V4MAPPED. Report from 
   peter@frontierflying.com
 - Include /usr/local/include and /usr/local/lib for systems that don't
   do it themselves
 - -R/usr/local/lib for Solaris
 - Fix RSAref detection
 - Fix IN6_IS_ADDR_V4MAPPED macro

20000311
 - Detect RSAref
 - OpenBSD CVS change
   [sshd.c]
    - disallow guessing of root password
 - More configure fixes
 - IPv6 workarounds from Hideaki YOSHIFUJI <yoshfuji@ecei.tohoku.ac.jp>

20000309
 - OpenBSD CVS updates to v1.2.3
	[ssh.h atomicio.c]
	 - int atomicio -> ssize_t (for alpha). ok deraadt@
	[auth-rsa.c]
	 - delay MD5 computation until client sends response, free() early, cleanup.
	[cipher.c]
	 - void* -> unsigned char*, ok niels@
	[hostfile.c]
	 - remove unused variable 'len'. fix comments.
	 - remove unused variable
	[log-client.c log-server.c]
	 - rename a cpp symbol, to avoid param.h collision
	[packet.c]
	 - missing xfree()
	 - getsockname() requires initialized tolen; andy@guildsoftware.com
	 - use getpeername() in packet_connection_is_on_socket(), fixes sshd -i;
	from Holger.Trapp@Informatik.TU-Chemnitz.DE
	[pty.c pty.h]
	 - register cleanup for pty earlier. move code for pty-owner handling to 
   	pty.c ok provos@, dugsong@
	[readconf.c]
	 - turn off x11-fwd for the client, too.
	[rsa.c]
	 - PKCS#1 padding
	[scp.c]
	 - allow '.' in usernames; from jedgar@fxp.org
	[servconf.c]
	 - typo: ignore_user_known_hosts int->flag; naddy@mips.rhein-neckar.de
	 - sync with sshd_config
	[ssh-keygen.c]
	 - enable ssh-keygen -l -f ~/.ssh/known_hosts, ok deraadt@
	[ssh.1]
	 - Change invalid 'CHAT' loglevel to 'VERBOSE'
	[ssh.c]
	 - suppress AAAA query host when '-4' is used; from shin@nd.net.fujitsu.co.jp
	 - turn off x11-fwd for the client, too.
	[sshconnect.c]
	 - missing xfree()
	 - retry rresvport_af(), too. from sumikawa@ebina.hitachi.co.jp.
	 - read error vs. "Connection closed by remote host"
	[sshd.8]
	 - ie. -> i.e.,
	 - do not link to a commercial page..
	 - sync with sshd_config
	[sshd.c]
	 - no need for poll.h; from bright@wintelcom.net
	 - log with level log() not fatal() if peer behaves badly.
	 - don't panic if client behaves strange. ok deraadt@
	 - make no-port-forwarding for RSA keys deny both -L and -R style fwding
	 - delay close() of pty until the pty has been chowned back to root
	 - oops, fix comment, too.
	 - missing xfree()
	 - move XAUTHORITY to subdir. ok dugsong@. fixes debian bug #57907, too.
   	(http://cgi.debian.org/cgi-bin/bugreport.cgi?archive=no&bug=57907)	 - register cleanup for pty earlier. move code for pty-owner handling to 
      pty.c ok provos@, dugsong@
	 - create x11 cookie file
	 - fix pr 1113, fclose() -> pclose(), todo: remote popen()
	 - version 1.2.3
 - Cleaned up
 - Removed warning workaround for Linux and devpts filesystems (no longer 
   required after OpenBSD updates)

20000308
 - Configure fix from Hiroshi Takekawa <takekawa@sr3.t.u-tokyo.ac.jp>

OpenSSH 1.2.2p1 (2000-03-05) 

It gives me no little pleasure to announce the first stable release
of the Unix port of OpenSSH. 

It is available in tar.gz and RPM format from one of the mirrors
listed at:

http://violet.ibs.com.au/openssh/files/MIRRORS.html

This release fixes all known issues and is known to compile and
function on (at least) recent releases on Linux, Solaris, HPUX and SCO
Unixware.

Please review the ChangeLog[1] for details on what has changed since
the last release.

I am holding off on a wider announcement until the mirrors have
updated.

Thanks to everyone who assisted with testing, bug reports, success
stories and most of all, patches :) Special thanks to the OpenBSD
developers for giving us OpenSSH to begin with.

Regards,
Damien Miller

[1] http://violet.ibs.com.au/openssh/files/ChangeLog

$OpenBSD: releasenotes.html,v 1.94 2026/10/06 17:17:07 tj Exp $

EmbeddingGemma 2

Simon Willison
simonwillison.net
2026-10-06 16:37:53
My comment on EmbeddingGemma 2 — Hacker News. I really appreciate that EmbeddingGemma 2 is under the Apache 2.0 license. For embedding models in particular, I don't think it makes sense to use a closed, proprietary, hosted-only model. Most applications of embedding models involve calculating t...
Original Article

6th October 2026

I really appreciate that EmbeddingGemma 2 is under the Apache 2.0 license.

For embedding models in particular, I don't think it makes sense to use a closed, proprietary, hosted-only model.

Most applications of embedding models involve calculating thousands or even millions of embedding vectors and storing them for later comparison.

If your model is proprietary, the vendor is likely someday going to decide to stop offering that model. They'll have a better model to replace it, but you still need to pay to re-calculate those millions of stored existing vectors.

(In April 2024 OpenAI offered to "cover the financial cost of users re-embedding content with these new models" - https://openai.com/index/gpt-4-api-general-availability/ - but I don't think that's something we can rely on from every provider.)

Notably, I don't want to host the model myself . I'd much rather pay a provider for a hosted model while knowing that if they ever stop hosting it I can run the open weights version myself - or find another vendor who can do that for me.

Paramount completes $111B Warner merger, creating "Skydance" behemoth

Hacker News
arstechnica.com
2026-10-06 16:33:47
Comments...
Original Article

How to fix a bug in a fix

Lobsters
projectzero.google
2026-10-06 16:33:01
Comments...
Original Article

Project Zero often works with software vendors to remediate the vulnerabilities we report and provide broader guidance on making software more secure. Some vendors express concern about potential scenarios in which they are unable to fix vulnerabilities that are causing immediate user harm, due to limitations in their patch delivery systems. Since Project Zero encounters a wide array of systems designed to protect users in the case of exceptional exploitation scenarios, both through vendor discussions and security reviews, we want to share what we’ve learned.

This post provides an overview of systems in use by large vendors that allow them to remediate small volumes of vulnerabilities much faster than their typical update process. Our goal is to provide a reference for vendors seeking to implement or enhance the capabilities of such systems, and to encourage vendors to consider how they would fix an urgent vulnerability before they receive one.

Why patching takes time

Patching a vulnerability typically involves the following stages:

  • Triage — a vulnerability report is received, validated, prioritized and assigned to a specific developer to be fixed
  • Patch development — a software development team writes, reviews and commits code that fixes the vulnerability
  • Testing — the patch is tested to ensure the vulnerability is remediated and the software still functions correctly when the patch is applied. This can include formal testing by a test team, automated testing and alpha and beta testing where a patch is shipped to a limited group of users for feedback on normal use.
  • Partner review — some software updates require review by third parties before they can be shipped, due to relationships between the software vendor and other organizations, for example, carrier acceptance for some mobile updates.
  • Delivery — the patch is delivered to and installed by end users
  • Activation — sometimes an additional step, such as a system restart, is needed to switch the system to the updated software

Of course, this is a simplified picture. Patching can involve repeating steps, for example rewriting a patch if tests fail, or additional stages when third-party vendors are involved. However, this is a minimal set of steps most software updates require.

The challenges of emergency patches

While triage and patch development time contribute substantially to the speed at which vendors can generally patch vulnerabilities, they contribute less to emergency patch time. Triage is usually very fast in situations where vendors know they have an urgent problem, and patch development can be expedited based on priority. Only in rare circumstances, where a vulnerability is especially complex, or a vendor’s security team does not have a complete picture of their software’s components and who within their organization maintains them, have we seen urgent patches delayed in the triage or development phase. Likewise, partner agreements usually have exceptions for updates in emergency situations.

Most vendors’ patch speed is limited by the testing and delivery stages. Testing is important because all changes to software risk introducing unexpected behavior. The worst-case scenario is that inadequately tested software ‘bricks’ a device, causing it to malfunction in a way that it can no longer perform key functionality or receive software updates to remediate this. Buggy software updates have also led to situations where user data is corrupted or lost, and any decrease in software functionality after a security update makes users less likely to apply updates in the future.

The potential cost to vendors of shipping poorly tested updates varies depending on the nature of the underlying software. For example, if a mobile application is rendered unusable due to an update that corrupts local data or prevents it from launching, users can easily install the next version via an app store, and their data is usually saved on a remote server, so costs are limited to user support. Meanwhile, if a mobile device gets bricked, it needs to be returned to its manufacturer or place of purchase for repair, leading to substantial costs for the vendor and potentially the user.

The possibility of serious functional bugs is considered in the design of most patch delivery systems. Updates are often rolled out slowly, so that serious problems can be detected before they affect too many users. Often, patching vulnerabilities quickly and avoiding buggy patches are at odds with each other, requiring tradeoffs that prioritize one over the other.

A variety of other technical challenges can limit the speed of patch delivery. One is the design of the patching system. A common design is that devices probe for updates at a regular interval, leading to patch saturation being limited to that interval. ‘Push’ style update systems can deliver patches to all users faster, but generally require more infrastructure.

User behavior and environment can also be a barrier to patch propagation. Patches that require user interaction to install are often delayed by users, and network speed and data cost are also factors in installation rate. Updating many users at once, as opposed to over a period of time, can strain patch delivery infrastructure. Chrome and Microsoft have written about the challenges of updates requiring restart to install, as users are often reluctant to restart their system and restarts take time.

While testing delays and limitations of the patch delivery system affect all updates, the shorter time frame of emergency updates make them a larger contributor to the overall time it takes to deliver a patch.

Emergency patching methods

Feature flags

Feature flags are conditional statements in source with paths determined by values provided by a remote server. They are often used for A/B testing, but they can also be used for short term remediation of vulnerabilities in emergency situations. A widely publicized case of this was a serious 2019 FaceTime vulnerability , where Apple temporarily disabled Group Facetime with a feature flag. Several vendors have made at least some media codecs available in 0-click contexts controllable via feature flags, and can disable them in the case of active exploitation, falling back to another codec for realtime transmission.

The main benefit of feature flags as a vulnerability remediation method is that testing can be performed with each flag set in advance, so a fast update does not require shipping untested code. They can also be delivered to users much more quickly, as updating feature flags requires transmitting a very small amount of data.

Recently, Meta published a blog post on how they implemented a ‘dual stack’ library, in which two versions of the WebRTC video conferencing library were compiled into a single binary, with the version in use controllable via a feature flag. This technology enables rapid updates with less testing, as new versions can be shipped with the option to quickly move users back to the previous version if function problems occur. While Meta uses two versions of the same library, it would also be possible to create a ‘dual stack’ with two different libraries that implement the same features (for example, two H264 libraries), allowing an application to switch to a different library to render a specific vulnerability unreachable without loss of functionality in an emergency. This would require additional testing, but it is testing that can be performed up front. It could also be possible to have a second library that enables performance intensive mitigations that would block many possible bugs, such as ASAN, or enabling DCHECKs.

Filtering

Filtering is running a dynamically updatable ruleset, such as a regular expression, against untrusted input in order to block specific input that is required to reach a vulnerability. An example of this is Android’s Intent Firewall , which allows specific usages of an Android IPC mechanism called intents to be disabled based on rules in a dynamically updateable XML file, which enables blocking intents that can be used to exercise specific vulnerabilities. It was recently used to block vulnerabilities in third-party Android wallets.

Some platforms have endpoint detection software that can perform filtering on a wide variety of system input, for example Microsoft Defender on Windows systems, and Google Play Protect on Android devices. Rules that block specific exploits or make certain vulnerabilities unreachable can often be deployed to these applications very quickly. Endpoint detection requires parsing a great deal of untrusted input, often in privileged context, so these applications are not without risk, but in systems where they already exist, they are a potential method of emergency remediation.

As an approach, filtering is more flexible than feature flags. For feature flags to be effective, the vendor needs to determine what features they might want to disable in advance, and if this isn’t comprehensive, they might find themselves in a situation where a vulnerability can’t be remediated via feature flags. Meanwhile, filtering can be used to block a wide variety of inputs, even ones that have never been considered. The downside of filtering is that performing filtering frequently can decrease software performance, and at least some testing of new filters is required, and can’t be performed upfront without knowing the vulnerability that needs to be blocked, as it is possible to write filters that interfere with necessary system functions.

Alternate Channels

The network ‘channels’ used to deliver software updates to users can be slow for a variety of reasons discussed above. Vendors sometimes implement alternate channels that can be used to deliver smaller updates more quickly.

Android Pony Express (APEX) is an example of an alternate channel that can be used to ship updates to specific high-risk Android components faster than a full system update. It shortens the patch development time, as OEMs do not need to integrate updates to APEX components. APEX is available to OEMs, and can be used to update OEM-maintained libraries.

Several applications we’ve researched have the ability to update individual libraries outside regular updates, usually by having some flag that is regularly checked over the network, and then downloading the library and loading it with dlopen or equivalent. While this is an effective way to avoid delivery-speed limitations of updates, it can also introduce critical vulnerabilities if libraries delivered in this way are not adequately verified by the client to have originated from the vendor. We encourage vendors to be cautious, and ensure that emergency update mechanisms of this variety have adequate security testing.

Hotpatching

Some vendors have implemented update mechanisms that allow units of binary code smaller than libraries to be delivered and applied directly to the memory space of a running process. For example Linux supports Livepatch which enables kernel functions to be directly replaced in memory without a restart. Similarly, Windows’ hotpatch allows security updates that contain only updated functions to be delivered to users, and applied while the process is still running.

Hotpatching has the potential to deliver very flexible security patches to software very quickly, with no degradation of user experience, though it typically has some limits to the nature of patches it can deliver, for example, updates that require changing the definition of a structure shared between functions are sometimes not supported. Hotpatching has similar security downsides to alternate channels, and also carries the risk of introducing ways to bypass exploit mitigations, as it requires permissions to map pages with write-execute privileges at some point during patching. It also doesn’t address any of the testing challenges of rapid updates, just the delivery challenges.

The importance of emergency patching

LLMs are increasing the vulnerability discovery and exploitation capabilities of both attackers and defenders. A wider array of actors now have the ability to perform novel attacks at greater speed. In light of this, it is important for vendors to consider how to protect their users in the case of active exploitation. Rapid update mechanisms do not need to be heavyweight or be capable of fixing every possible bug and preserving perfect user experience in every scenario. Technologies like feature flags, filtering and alternate update mechanisms can remediate the most likely and severe vulnerabilities in the short term, while keeping devices reasonably functional for users.

It is urgent for vendors to plan how they will protect their users in the worst case scenario of widespread active exploitation. Actions taken now can greatly improve security outcomes for users. By taking stock of update mechanisms already available to them and implementing rapid remediation functionality where gaps exist, vendors can be better prepared for whatever the future holds.

System-level ad-blocking in Android

Hacker News
kevinboone.me
2026-10-06 16:29:10
Comments...
Original Article

The Internet has become an advertising platform. Users of mobile devices are acutely aware of this, because mobile operating environments and apps force ads on their users, in addition to those that pollute the Web. It’s getting harder to avoid the ads on Android handsets, as Android is increasingly locked down and difficult to customize. It’s certainly not in Google’s interest to make it easy to block ads.

It’s still possible, though, to suppress most advertising, even in 2026.

This article is specifically about blocking ads at the system level, that is, in the Android platform itself, rather than in any particular app. Some Android apps, particularly web browsers, have their own methods of blocking ads, or can have such features added using third-party plug-ins. Oddly, though, the DuckDuckGo browser, whilst blocking the tracking that accompanies targeted advertising, doesn’t specifically block ads – you need something more.

If you block ads at the system level, all apps – and the system itself – are protected. Moreover, methods for blocking ads can usually be extended to protect against trackers and other kinds of malware.

Basic principles

So far as I know, all system-level ad-blocking methods work by overriding DNS behaviour. DNS (domain name service) is the technology that maps hostnames to numeric IP addresses; a DNS lookup is the first step in almost all network operations.

Blocking ads using DNS amounts to finding the hostnames of know advertising services, and mapping them to bogus IP numbers. This can be done in the Android handset itself, or in some external DNS server.

Returning a bogus IP number doesn’t prevent an app or browser trying to show advertising, or allow it to make better use of the screen space the ads take up. Apps vary in their responses to a failed attempt to display advertising – more on this later.

Blocking ads using DNS

There are essentially four methods to block ads using DNS changes.

  1. Use a commercial DNS service with ad-blocking features
  2. Use a commercial virtual private network (VPN) with ad-blocking features, like NordVPN
  3. Install an ad-blocking app that doesn’t require root access.
  4. Use an ad-blocking method that requires root access, with or without a supporting app

In general, when you make a connection to the Internet on an Android device, the DNS name resolution (mapping a hostname to a numerical Internet addresses) is performed by a server hosted by the handset’s mobile carrier or Internet service provider.

Android allows you to override this behaviour, and select a custom DNS server that blocks ads. Well, it won’t actually “block” anything – not really; the DNS server just returns a bogus IP number when your handset looks up the IP number that corresponds to the hostname of a known advertiser.

Depending on the DNS service you use you may, or may not, get some control over exactly what it blocks. Some users choose also to block access to pornography or on-line gambling, for example, for child protection.

How well these services work depends, of course, on how well the service’s operators maintain their lists of known advertising hosts. Advertising services come and go, and even long-standing ones change their servers from time to time.

There are many commercial DNS services that support ad-blocking, and I don’t endorse any in particular. Prices vary, and most services have a free trial. If you care about privacy, you’ll want to use a service that keeps no logs, and you might want to use one that supports encrypted DNS. A rogue DNS operator might get access to sensitive data in all kinds of ways. It’s possible, for example, for a DNS operator to direct requests for your on-line banking service to its own proxy, and slurp up your credentials when you log in. You should certainly be very careful and, while there are legitimate free DNS services, I’m rather suspicious of any service of this kind whose funding model is unclear.

For many people this will be the simplest solution. If you subscribe to a VPN service for general privacy, blocking ads will often come at no extra cost. Ad-blocking VPNs typically redirect DNS lookups to their own servers, and return dummy IP numbers for known advertisers, just as a commercial DNS service does. VPNs therefore have all the same advantages and disadvantages as an ad-blocking DNS service. A big VPN operator will usually provide an Android app that configures the handset to use its servers, so set-up is pretty trivial for most people.

As with commercial DNS services, a rogue VPN operator can be very dangerous, and it’s important to choose a provider carefully.

So far as I know, all these apps exploit a loophole (of sorts) in Android’s platform lock-down.

Like all Linux-based operating systems, Android provides a way for a device to override the DNS mappings of the Internet service provider it is using. There will be, somewhere on the handset, a file that contains a list of preferential name-to-IP mappings. On a non-rooted Android device, the user won’t have permissions to change this file, so a simple method of blocking ads – maintaining a list of bogus DNS mappings for known advertisers in a file – is unavailable.

However, in a non-rooted device, you can install a VPN service. This has to be possible, if Android is to support VPNs at all. A non-rooted ad-blocking app will typically run a DNS server and a local (within the app) VPN host. The app will configure the handset to use its own VPN as the system VPN provider, which will make its own DNS server the system DNS. With control over DNS, the app can provide its own handling for name lookups, including those of know advertising servers.

Google, being an advertising company, isn’t very keen on this kind of thing. It can’t easily prevent the use of “local” VPNs without crippling VPN functionality completely but Google can, and does, make it difficult for non-technical users to get the appropriate apps. At present you can get apps like AdAway from alternative app stores like F-Droid .

Google is set on making this difficult, too, and before long you’ll have to get the app’s APK file from (hopefully) a reputable source, and jump through whatever hoops Google puts in the way of install software it doesn’t like. At the time of writing, Google isn’t making it impossible to install software from outside its Play Store, but it’s getting more and more fiddly.

Most likely though, Google’s obstructions won’t ever be as difficult to surmount as rooting your Android handset and applying a definitive ad-blocking solution. Using an app like AdAway will probably continue to offer advantages over a commercial DNS or VPN service, even if Google makes it hard to install.

The most obvious advantage is that these apps are usually free to use. Despite this, a community of volunteer maintainers ensures that the apps’ lists of known advertisers are as thorough as any provided by a commercial service. A disadvantage, though, is that all network traffic from all apps has to be routed through a single app on the handset. Not only does this create a network bottleneck, a rogue ad-blocker app is exactly as dangerous as a rogue VPN service. Fortunately, because these ad-blocking apps are usually open-source, there are limited opportunities for bad actors. I’d strongly advise against using one that isn’t open-source, even if you don’t plan on looking at the source code yourself.

If your handset is rooted, then ad-blocking is simple – in theory, at least. Android’s list of preferential name-to-IP mappings is in the file /system/etc/hosts , so all you have to do is edit that file, to direct advertisers’ hostname to a bogus IP number. Usually the bogus IP local address of the handset itself, 127.0.0.1 . So you’ll have a hosts file full of entries like this: 

127.0.0.1 08.185.87.0.liveadvert.com
127.0.0.1 08.185.87.00.liveadvert.com
...

The reason for using the handset’s local IP number is that it must correspond to a system that the handset can actually reach. Otherwise, network access will be badly delayed as apps try to contact non-reachable advertising hosts. Of course, this means that every request for an advertising service will be directed back to the handset which, presumably, will reply with a error response to the app that makes the request. Since the request will fail immediately, this loopback network routing doesn’t create an appreciable load on the handset.

This is all theoretically straightforward but, in practice, there are two major problems.

First, we need a source of hostnames to block. Lists of these are widely available on websites, but the problem of maintenance is always present. Some of these lists are truly vast and, without doubt, reference hosts that no longer operate. If the hostname list is this long, then all network access will be slowed, as the handset has to parse the hosts file for each DNS lookup.

Probably the best source of ad-blocking hosts files is the code of open-source ad-blocking apps like AdAway. This app can, in fact, provide its list automatically when installed on a rooted handset, which simplifies the set-up.

The second problem is that you can’t just hack on the hosts file, even as root – on all modern Android devices it’s on a read-only filesystem. So you’ll need some sofware that can manipulate the contents of the /system directory during the boot process, while it’s still writeable.

If you’ve rooted your handset, you almost certainly have a way to do this already. If you’re using Magisk, for example, you can use a module to supply a new hosts file. Magisk modules live in directories under /data/adb/modules , and any files in the module’s own system directory will overwrite the main /system at boot time. So you can create a Magisk module that provides its own /system/etc/hosts .

Happily, Magisk users don’t have to do this manually, as the Magisk developers have anticipated this usage. If you enable the “systemless hosts” option in the Magisk app, it will create the necessary module with all the necessary metadata in place. Thereafter, to create a custom hosts file you just hack (as root ) on /data/adb/modules/root/system/etc/hosts , and then reboot. Alternatively, you can use an app to make this change.

In fact, because this method of hacking on the hosts file is so prevalent, ad-blocking apps like AdAway have built-in support for it; but, of course, this support is only for rooted devices. If your handset is already rooted, using a root-aware ad-blocking app is probably the most effective way to manage ads. It’s much faster than a non-root app that installs a local mock VPN, and doesn’t raise any of the same security concerns. That’s not to say there are no concerns, and you should ideally check the hosts file such an app installs, to ensure there are no mappings to anything except 127.0.0.1 .

It’s important to understand that no method of blocking ads is completely reliable, or has no side-effects.

Most notably, some Android apps simply won’t work, or won’t work properly, without their advertising. If you must use such apps, you’ll need an ad-blocking method that allows you to customise the list of blocked sites. It won’t be remotely obvious, just from the behaviour of the app itself, why it isn’t working, or how to fix it. On a rooted handset you can track the network behaviour of a specific app in detail and, in theory, work out what it needs that you’re blocking. In practice, it’s easier to refer to the on-line discussions of such apps because, most likely, somebody else will already have done the work.

However, some apps go so far as to have all their ads built in; no ad blocker will stop such an app showing advertising, as there’s no network operation to block.

It’s also important to understand that Android caches DNS lookups in various places. Although some methods of ad-blocking allow the list of blocked hostnames to be configured on the fly, you might still have to reboot the handset to flush its caches.

Although it should be obvious, bear in mind that DNS-based ad-blocking only works where an app uses DNS. Since DNS-based ad-blocking is so commonplace, some app developers are implementing direct access to advertising servers using IP numbers, which renders all DNS-based approaches ineffective. This, fortunately, is still relatively rare.

The ability to block ads at the system level is one of the few compelling reasons to root your Android handset in 2026. Blocking ads this way uses few, if any, additional system resources, and its effect extends to all apps.

If you can’t root your device, or prefer not too, there are other ways to block ads, but none is as effective, as configurable, or as cheap.

Finally, if you mostly struggle with ads on websites, rather than in apps, the simplest approach might be to install a web browser with ad-blocking support, and avoid system-level ad-blocking entirely.


Have you posted something in response to this page?
Feel free to send a webmention to notify me, giving the URL of the blog or page that refers to this one.

A Few of the Diseases Invoked by AI Execs to Evade City Council's Safety Questions

hellgate
hellgatenyc.com
2026-10-06 16:27:37
New technology, same old tricks....
Original Article

Great! You’ve successfully signed up.

Welcome back! You've successfully signed in.

You've successfully subscribed to Hell Gate.

Your link has expired.

Success! Check your email for magic link to sign-in.

Success! Your billing info has been updated.

Your billing was not updated.

Introducing Mistral Large 4: Le chonk

Simon Willison
simonwillison.net
2026-10-06 16:18:19
Introducing Mistral Large 4: Le chonk Mistral are back in the game. Today they're releasing a preview of Mistral Large 4, a 1 trillion parameter, 49 billion active parameter model trained on their own cluster of 3,800 NVIDIA Grace Blackwell GPUs. The preview is available via their API. They promise ...
Original Article

6th October 2026 - Link Blog

Introducing Mistral Large 4: Le chonk ( via ) Mistral are back in the game. Today they're releasing a preview of Mistral Large 4, a 1 trillion parameter, 49 billion active parameter model trained on their own cluster of 3,800 NVIDIA Grace Blackwell GPUs.

The preview is available via their API. They promise to release the open weights model at the "end of this month".

The model only supports two reasoning levels - "none" and "high" - via the Mistral API. Here are both pelicans - the "high" one looks better, though surprisingly it only used 2,717 output tokens compared to "none" which used 3,275:

It's good. The pouch is great, the bicycle frame is the right size, it has feet on pedals. Both pedals appear in front of the frame though. Nice gradients.

On Artificial Analysis it scores 38 , just behind DeepSeek 4.1 Flash, which is a 552B model. It's a huge improvement on last December's Mistral Large 3, which drew this terrible pelican and scored 9 on AA .

It's certainly not a Fable-class model, but it's great to see Mistral put out a model that's back to being maybe about 6 months behind the frontier.

That Time I Worked With a Laptop Thief

Lobsters
blog.pipetogrep.org
2026-10-06 16:16:33
Comments...
Original Article

Published on: by Chris Jones

Updated on: • 10 min read

Preamble

I used to worked at a place where they had an IT contractor that was stealing laptops and reselling them for over a year. This post is the story of how he got caught. Real names of people and companies will not be used, but the tale and screenshots (with redactions) are all true.

When I Went Back to IT

I was burning out on my position as a DevOps Engineer. Instead of looking for yet another job doing the same thing, I asked my boss if I could fill a different role temporarily so I could reset while still being useful to the company. He agreed and asked what I had in mind.

The company's IT manager was leaving and the whole department's staff, tools, and processes really needed some attention. I volunteered to be the interim IT manager that handled banging the department back into shape. He liked the idea so I got started.

The Contractor

The only one left in the IT department was a single IT contractor that we will call Jim. Jim had been with the company for a little over a year. As I got to know Jim, I discovered that everything under his watch was falling apart. The IT closet was a gigantic mess. He was rarely available during working hours, he worked on who knows what at the office during the evening, and wasn't helpful when he was assisting with IT related issues.

I gave him the benefit of the doubt and figured that his department was understaffed and overworked for too long so the first order of business was to assemble a new team.

The New Team Assembles

After a month of helping with IT tickets basically on my own (Jim was often MIA), and after cleaning out the IT closet, I hired a new IT manager and support specialist. During their first week, I gave them a tour of the office including the newly reorganized IT closet. That was when I first noticed something was amiss.

We had just placed an order for 15 new MacBooks. While some of them had already gone out, the number that remained in the closet still looked smaller than it should have. I consulted the inventory and saw that six machines were unaccounted for.

The Lies Begin

To keep everything transparent and out in the open, I started a thread in the IT discussion Slack channel asking about the missing laptops and requested everyone on the team to speak up in the thread if they knew anything about it. My boss knew nothing. He could not since he works remotely. The new hires didn't know anything either since this was the first day they were seeing the IT closet.

Then Jim messages me, not in the thread but via DM.

Lies

He's telling me five of the laptops had dead batteries on arrival while the sixth went out to an employee. That was the first thing that didn't add up. Five out of fifteen brand new, sealed in box Apple laptops were dead on arrival? I know Apple's hardware isn't the most resilient, but that failure rate is unheard for them.

In the message he sent me five Apple repair case IDs. Upon looking them up, I see that they were suspiciously created right after I sent out the message about the missing devices. Even more suspicious, I find out that he created the repair requests under his personal Apple account rather than the company account, preventing me from gathering more information about the requests.

The Lies Continue

The next day, two FedEx boxes arrive containing two brand new MacBooks. The FedEx label said it was from Apple's 5th Ave store in Manhattan (which was walking distance from the office). There were two things odd about these machines:

  1. Neither of them were enrolled in our company's Apple Business Manager account. That means we could not autoenroll them into our MDM.
  2. The FedEx label looked like it was printed on an InkJet printer from the FedEx website. This is not how Apple does things.

Later Jim claims that two of the laptops have been replaced. I thank him and ask that he not take laptops home while we get all this figured out. He gaslights me by saying he verbally told me about the dead machines and that I must have forgotten. He then states that he does not take laptops home "at all".

More lies

Apple Get Involved. The Lies Get Discovered.

At this point, I'm still under the impression that something is fishy, but Jim still gets the benefit of the doubt. My responsibility at this point was not to accuse or speculate, but collect date and see where it led. I decided to consult Apple.

I took the two new laptops that arrived to the 5th Ave Apple store. I explained the situation I was in and asked if they could tell me anything about the origin of them. They were not purchased by the company. Instead, the name associated with both machines was Jim himself.

Meanwhile, all 5 of the Apple repair requests that he created got as far as Apple sending empty boxes to retrieve the 5 "broken" machines. Were those boxes sent to the company? Nope! They were sent to Jim's place of residence.

Finally, I asked about the FedEx shipping labels from the "replacement" machines. I asked how and from where they are normally sent. They confirmed that they do NOT ship repairs from an Apple store and instead send them from Apple Repair Centers, which in this case, was in a neighboring state.

Jim has now been caught in multiple lies:

  • "I believe 2 of the defective laptops have been replaced and should be at the office right now." - Two laptops did arrive to the office, but they did not belong to the company. They belonged to him.
  • "Once again, I do no take laptops home at all." - Then why did he have the Apple repair boxes sent to his home address?
  • The FedEx labels were clearly a cover. The from address was spoofed to look like Apple. The FedEx tracking for them never got past the label being created which means Jim hand delivered them to the office (no cost that way).
  • The Apple repair cases never updated to indicate that the machines were received, so how could we have two replacements from them?
  • The "replacement" laptops had different serial numbers. Devices Apple repairs retain their serial numbers.
  • "One laptop is being prepped for a new hire starting Monday." The new hire did not have the missing laptop.

Jim Plays the Victim

While we were civil and as non-accusatory as possible, and were doing our best to only seek answers about the missing property that Jim was responsible for keeping track of, Jim decides it's best to go full victim.

Time to play the victim

Jim Gets Let Go

At this point, we were not sure about the entirety of what was going on, but we knew that we had machines missing, and nothing that Jim was saying was adding up and that we were being lied to. Jim was given 24 hours to reproduce all of the missing machines which he failed to do. At that point we had no choice but to let him go before he had the chance to quit himself.

We Get Theft Confirmation

My first act as interim IT manager was to fix the auto-enrollment of company machines. By this, I mean making it so the devices the company purchased know to register themselves via mobile device management (MDM) and install the company's software, settings, etc. all automatically straight out of the box. Jim was manually enrolling them and could never give a straight answer besides "it's hard" as to why he continued to do it that way.

A few days after Jim was let go, we start getting Slack notifications about new device enrollments. They were being enrolled from places like Russia, China, and Kazakhstan and registered to people that did not work for the company. We proceeded to lock the devices. Most of them were the devices that went missing. It makes sense now why Jim never bothered fixing device auto-enrollment.

These enrollments continued for another year, some new, and some the same machines being resold again by people that knew how to unlock and wipe the machines back to factory only to have them locked again on first boot. We followed some of these machines across multiple countries and geographic zones.

The real victim is not the company itself, but the people that unknowingly bought these stolen devices. They open up their sealed, in-box MacBook only to have it immediately turn itself into a paperweight when it gets locked by the company it belongs to.

Another Mystery Laptop Appears. Things Get Stranger.

The few days after Jim's departure, another new MacBook arrives in the mail. I take it to the Apple Store again, and they tell me it was not purchased there like the other two were. Instead, it was purchased by a company called "Ingram Micro" who is a large reseller of IT equipment.

I call Ingram and they give me the name of the purchaser and the company they worked for when the purchase was made. I will not name the company, but I will say that they are a department of defense (DoD) contractor.

It took a few days but after many phone calls and messages, I got through to the DoD contractor's IT and legal teams and was able to explain what was going on. It was discovered that Jim was working for them (at the same time!) and stole one of their laptops as a replacement for one that he stole from us!

By doing that, he got himself let go from that job as well. I later found out from my boss that an FBI agent sat him down and he confessed to everything. DoD jobs do not seem to take kindly to property theft.

Jim Gets Petty

Weeks went by and Jim still hadn't returned the company laptop that was assigned to him. After multiple reminders, a very beat up box arrived in the mail via UPS. It contains two laptops from Jim with zero protective padding. The machines were trashed inside and out.

UPS box in rough shape

Inside the box

Top of the returned laptops

Gross laptop interior with surprise staple

Scrapped up laptop bottom

What Happened to Jim?

I don't really know. The company that I worked for dropped the ball on pressing charges. All told, he stole more than 30 laptops over the year he worked there. The monetary value of the theft was well into the range of grand larceny, but I guess they didn't care enough to pursue it. I'm not sure if he got so lucky with the DoD contractor.

What Did I Learn?

A lot!

  • Maintain a real inventory system instead of just a spreadsheet. If that was done from the beginning, this whole thing might not have happened.
  • I also learned about Apple Business manager, how Apple MDM software works at a low level, and how IT departments manage large fleets of devices.
  • I learned that if something feels off, it is worth investigating.
  • Don't hire individual contractor. Go through a reputable recruiter and run background checks. This did not happen with Jim.
  • Most importantly, I learned that an engineer's natural curiosity is applicable to more that just engineering. The more I dug into what Jim said, the more lies I found.

What Did Jim Learn?

Something I hope. If he simply maintained that he didn't know anything, he might have gotten further with his ruse. Instead, he kept talking and kept digging himself deeper. At the most, I can hope that he learned not to pull this crap anymore in the future. At the very least, he learned not to lie to me.

Want to discuss? Email me at blog@pipetogrep.org .

Her AI-Generated Video Swayed the Judge. The Court Said it Carried 'Undue Emotional Weight'

403 Media
www.404media.co
2026-10-06 15:26:09
‘Chris’ sentiment does not change,’ said the sister who created an AI avatar of her slain brother Christopher Pelkey to speak in court....
Original Article

A man convicted of manslaughter in Arizona will be resentenced because an AI-generated video of his victim speaking from beyond the grave was ruled to have carried “undue emotional weight” as an impact statement. An appellate court in Arizona ruled that the manslaughter charge will remain, but the judge must reconsider the length of the man’s prison term because of AI.

In 2021, Gabriel Horcasitas shot and killed Christopher Pelkey during a road rage incident. A jury found him guilty of manslaughter. During Horcasitas’ sentencing hearing, Pelkey’s sister Stacey Wales played an AI-generated video of her brother as part of her victim’s impact statement. In the video, which she scripted with her own words that she made an AI generated likeness of Pelkey say, the AI avatar of Pelkey forgives his killer and said they “could have been friends” in another life.

“ I loved that video ,” the judge said while handing down the maximum sentence of 10 and a half years.

Wales told 404 Media that her intent with the video was to sway the judge. “Isn’t that what you want human connection to be?” she said. “Do you ever come out of a movie and say: ‘Well, that was too emotional. That was too powerful. Human connection is powerful, and trying to convey that to somebody that's never known another person, this person on the earth, to convey their lifetime of 37 years down to a two-hour sentencing event is hard to do, and I think that victims and victims' families should be afforded every opportunity.”

After sentencing, Horcasitas appealed. He argued that the AI Pelkey video was inadmissible because it carried “undue emotional weight” and that the “lifelike AI recreation of the victim conveyed an authority and authenticity” had a “psychological impact” on the judge, according to court documents reviewed by 404 Media.

The appellate court agreed and cited State v. Rose , a 2007 murder case where a man was sentenced to death for the murder of a police officer. During the victim impact portion of the sentencing in that case, the judge saw photos of the police officer’s children at his graveside. The court said those photos almost crossed the line into prejudicing the judge.

“The AI video crossed that line. Unlike the photographs in Rose, which captured a particularly poignant moment for the young surviving victims, the AI video here [...] does not reflect actual events,” court documents said.

“It clearly impacted the sentencing judge, who said he ‘loved the video’ and felt it ‘was genuine,’ pointing to the AI victim’s ‘obvious forgiveness’ of Horcacistas,” the court documents said. “On this record, the judge’s consideration of the AI video so prejudiced Horcasitas as to render the sentencing procedure fundamentally unfair.”

Wales and the rest of Pelkey’s family have spent the last five years in and out of various court rooms related to his death. If the current ruling holds, then she'll travel to Arizona from New York and give the same statement she did the first time through her brother’s AI-generated mouth, minus the AI-generated video. The same judge who originally said “I loved that video” will do the sentencing.

“Likely, the same people will go back down and make their speeches, and I will be making my victims' impact statement that I wrote for myself the first time around again, and I will be delivering Chris's AI script sentiment again,” she said. “If I need to stand there and read it myself, which would be legally lawful, I am prepared to do that. Chris's sentiment does not change. The delivery method will.”

Wales said she wrote the words that came out of the AI version of her brother, they weren’t AI-generated. She also noted that only the judge, her family, and the Horcacistas family were the only people who saw it during the trial. “This was not offered as evidence. Juries never saw this,” she said. “This was only created after a verdict was solidified, a guilty verdict.”

She also noted that the AI video was only one of a larger piece of testimony given that day. “I think that [the judge] is unfairly represented with a soundbite,” she said. According to Wales, the judge spoke for around half an hour and only mentioned the AI video once. Wales herself uploaded the soundbite from the judge saying he loved the AI-generated impact statement to YouTube, where it went viral.

But she also said she understood that AI is a new technology and when she played the AI-generated video of her brother that she was doing something no one else had done yet. She compared it to the way courts thought of photography in the late 19th century.

“It took about 15 years [...] and about five landmark cases in the United States that went all the way up to the Supreme Court before photography was an accepted standard to be used in the courtroom for evidence, identification, and testimony, et cetera,” she said. “I believe that's what we're seeing now with AI. It is a brand new medium.”

Wales said she understood she was setting a precedent, for good and for bad, and that she was OK with that. “There is always that first case, and so that mantle weighs on me that I need to be careful with how I defend this and how I explain the impetus, the goals, the motivations, and the veracity of what was created, because there are victims that depend on freedom of speech for the future, and I do believe that victims and their families should have every opportunity available to them to make their their points,” she said.

There’s no firm date for the new sentencing. “We do respect the court's opinion, and if we need to operate in those guardrails, I am prepared to read the same script that the AI version of Chris read the first time around,” she said. “His sentiment has not changed. Our opinion of what we believed he would think about this whole thing has not changed. The words I penned as Chris's imagined opinion still have not changed and will not change.”

About the author

Matthew Gault is a writer covering weird tech, nuclear war, and video games. He’s worked for Reuters, Motherboard, and the New York Times.

Matthew Gault

Hackers exploit 32 zero-days on first day of Pwn2Own Ireland

Bleeping Computer
www.bleepingcomputer.com
2026-10-06 15:21:53
On the first day of the Pwn2Own Ireland 2026 competition, security researchers hacked the Samsung Galaxy S26 twice and earned $388,500 after exploiting 32 zero-days. [...]...
Original Article

Pwn2Own Ireland

On the first day of the Pwn2Own Ireland 2026 competition, security researchers hacked the Samsung Galaxy S26 twice and earned $388,500 after exploiting 32 zero-days.

During the Pwn2Own Ireland 2026 hacking contest, competitors target products in seven categories , including mobile phones (Apple iPhone 17, Samsung Galaxy S26, and Google Pixel 10), printers, smart home devices, messaging apps, AI infrastructure, AI coding apps, and a new category where hackers will try to exploit wellness healthcare devices.

The highlight of the first day was Interrupt Labs , Ikotas Labs , and Nguyen Thanh Dat of Viettel Cyber Security hacking Samsung's Galaxy S26 flagship. However, some of the bugs exploited in each challenge were already known to the vendor.

Vũ Chí Thành and Huỳnh Đức Tin of VinSOC, who topped the leaderboard, won $40,000 after chaining seven zero-days to exploit a Philips Hue Bridge Pro smart lighting hub, plus an additional $40,000 for a five zero-day exploit chain targeting the Oracle Autonomous AI Database.

Security researchers also demoed LiteLLM zero-days , hacked the Lexmark CX532adwe and Canon imageFORCE 1643F multifunction printers, took down the OpenAI Codex cloud-based AI coding agent with a single argument-injection bug, and exploited four vulnerabilities to compromise a Sonos Era 300 smart speaker again.

Mikhail Evdokimov, Polina Smirnova, and Mate Zombor of White Noise Club also targeted the Google Pixel 10 today , but couldn't get their exploit to work within the allotted time.

Pwn2Own Ireland 2026 Day 1 leaderboard
Pwn2Own Ireland 2026 leaderboard (ZDI)

​The Zero Day Initiative (ZDI) organizes this hacking competition to identify zero-day vulnerabilities in targeted devices before threat actors can exploit them. After flaws are exploited at Pwn2Own, vendors have 90 days to release security updates before Trend Micro's ZDI publicly discloses them.

On the second day of the contest, hackers will again target devices in the AI infrastructure, printers, smart home, and wellness categories, as well as the Samsung Galaxy S26 and the Google Pixel 10 in the mobile phones category.

On the third day , they will attempt to hack the Google Pixel 10 and Samsung Galaxy S26 flagships again, as well as multiple smart home, AI infrastructure, and printer devices.

During last year's Pwn2Own Ireland event, security researchers earned $1,024,750 for 73 zero-day flaws, with Summoning Team collecting $187,500 after hacking the Samsung Galaxy S25, the Synology DiskStation DS925+ NAS, the Home Assistant Green, the Synology ActiveProtect Appliance DP320 NAS drive, the Synology CC400W camera, and the QNAP TS-453E NAS.

article image

Build your security blueprint for AI-powered attacks

Join Mikko Hyppönen and security leaders from the NFL, CHANEL, and Atlassian for a two-hour digital summit on what AI-speed attacks change, what defenders should stop doing, and how to validate, decide, fix, and re-validate at machine speed.

Save your seat

Using Parseable with Datasette for OpenTelemetry traces

Simon Willison
simonwillison.net
2026-10-06 15:07:31
TIL: Using Parseable with Datasette for OpenTelemetry traces I saw Parseable in a Show HN today - it's a new observability platform with both an open source (AGPL) Rust implementation (a single ~180MB binary), an "Enterprise" version with extra features and a cloud hosted option. Since Datas...
Original Article

TIL Using Parseable with Datasette for OpenTelemetry traces — [Parseable](https://www.parseable.com) is a brand new observability tool, compatible with [OpenTelemetry](https://opentelemetry.io), for storing and querying observability data.

I saw Parseable in a Show HN today - it's a new observability platform with both an open source (AGPL) Rust implementation (a single ~180MB binary), an "Enterprise" version with extra features and a cloud hosted option.

Since Datasette 1.0a41 added OpenTelemetry support (thanks, Alex Garcia), I decided to fire up Codex and have it figure out how to run Parseable and feed it traces from Datasette.

Here's my (human-written) TIL showing the patterns that worked, and here's a screenshot of a Datasette trace displayed within the Parseable localhost web application:

Screenshot of a trace detail view in an observability web app, with a span waterfall overlaid on a dimmed navigation sidebar and filter column. Dimmed sidebar: breadcrumb "Community > Traces > datas" (cut off), search box "Search... ⌘K", nav items "Home", "Ingest telemetry", section "ANALYZE": "Keystone", "Dashboards", "SQL Editor", section "OBSERVE": "Logs", "Metrics", "Traces" (selected), "APM", "Agents", section "MONITOR": "Alerts", "Errors", section "DATA": "Datasets", and at the bottom "Settings", "Book a call", "Support". Dimmed filter column, cut off at the right edge: "Search fi", "Core", "Log format", "User agent", "Source IPs", "Error", "Service", "service.ins", "service.na", "datasett" (checked), "Span", "Database", "HTTP", "http.reque", "NULL" (unchecked), "GET" (checked), "http.respo", "http.route", "Server", "server.add", "Telemetry", "URL", "All fields". Trace panel header: "Trace detail > 6f819a2170bcd1e91c6ea3ae236ec60b" with a copy icon, a "Related logs" button and a close X. Summary: "Start time 6:56 PM, Oct 6, 2026 UTC", "Duration 40.9 ms", "Spans 247". A minimap with axis "0ns 10.2ms 20.5ms 30.7ms 40.9ms" shows many short span bars cascading diagonally from top left toward the lower middle, with a few longer bars. Below is a span table with a "Span name" header, a "Search spans..." box, collapse and expand buttons, and a timeline axis "0ns 10.2ms 20.5ms 30.7ms 40.9ms". Rows (name, service, duration): root span with collapse toggle "123", "GET /..." "datasette..." 40.9ms spanning the full timeline; then alternating rows where each "db.query" has a collapse toggle "1": db.query datasette-local 679µs, db.query.execute datasette-local 278µs, db.query datasette-local 341µs, db.query.execute datasette-local 55µs, db.query datasette-local 357µs, db.query.execute datasette-local 197µs, db.query datasette-local 330µs, db.query.execute datasette-local 73µs, db.query datasette-local 2.26ms, db.query.execute datasette-local 2.02ms, db.query datasette-local 232µs, db.query.execute datasette-local 64µs, db.query datasette-local 207µs, db.query.execute datasette-local 71µs, db.query datasette-local 6.11ms. The child span bars start progressively later across the early part of the timeline.

Inside the Windows I/O Manager: Deep dive into how read I/O requests are initialized

Lobsters
winware31.blogspot.com
2026-10-06 15:03:48
Comments...
Original Article

Introduction:

If you are someone who periodically debug and reverse many different Windows programs, you may have noticed a common point shared among them despite they aren't related in any sense. This point is that the vast majority of them use at some stage a famous well documented Win32 API named

ReadFile( )

exported by the module KERNELBASE.dll . For someone who doesn't do Windows native programming, he may think that ReadFile( ) is solely used to read data from a file as its name suggests (a file in its traditional meaning is a filesystem file). Despite this is not totally incorrect, it's not the full image. Before continuing, let me clarify what is the full definition of a file in the world of Windows. So, a file simply is an I/O endpoint to which end users can send input and from which they can receive output. Filesystem files are just one case, for example file system directories are also considered files since they are valid I/O endpoints; named pipes and mailslots are also files; even open instances of device objects are considered files (device objects and device stacks are beyond the scope of this article, if you want to understand these concepts checkout the official MSDN documentation).

R etrieving data from the majority of I/O endpoints is achieved through a documented API provided by the system which is as you might be guessing the same kernelbase!ReadFile( ) we mentioned before; there is also an async version named ReadFileEx( ) that relies on an APC (async procedure call) to notify the caller that the read operation has terminated. Each mentioned I/O endpoint is presented to user mode callers as file objects represented by the kernel by a predefined well documented data structure named nt!_FILE_OBJECT . So, it doesn't matter whether it's a device object, a named pipe or a mailslot; for user mode callers, it's a file object.

Handling I/O requests is a task performed by the kernel executive layer which requires many steps and sub-tasks to be completed. Since the executive layer is divided into many managers; handling I/O operations is delegated to a separate manager to deal with it which is called the Input/Output Manager or for short the I/O manager. This manager knows exactly how to differentiate between sync/async operations, the target to which the request should be sent, what drivers are involved, whether it is possible to service request following the Fast I/O path and many other aspects that help it completes the request properly. So, as you see handling I/O operation is not a direct simple task as you might be guessing but it's the main task that any operating system should well implement to function properly.

In this article. I am going to explain the different steps the I/O manager follows when initializing an I/O operation starting from determining the right target, choosing the right path and finally sending the request to be processed by the right drivers. To make it easier I choose to focus on read requests only since the major steps are common by all types of requests.

Determining the request Target:

nt!NtReadFile( ) is the kernel counterpart of ntdll!NtReadFile( ) that is called internally by kernelbase!ReaFile( ) which is just a wrapper that internally forwards the rest of the work to the native undocumented layer. This routine takes the target of the read request as a HANDLE since its forbidden for user mode callers to access objects directly through their pointers. The first thing to do is getting the corresponding object to which the caller provided hFile parameter refers to by passing it to the object manager's defined routine nt!ObReferenceObjectByHandleWithTag( ) that performs the lookup operation in the current process handle table and get the corresponding nt!_HANDLE_TABLE_ENTRY that holds the starting memory address of the object main header nt!_OBJECT_HEADER which is sufficient to get the target object pointer which is of type nt!_FILE_OBJECT since nt!NtReadFile( ) operates on I/O endpoints. Only FILE_READ_ACCESS is requested in this step since the caller wants to read data from the provided target. For more information about how the handle table lookup is done checkout my previous article here .

Getting the nt!_FILE_OBJECT pointer that corresponds to the hFile parameter provided by the caller is not enough to properly handle the read request. File objects aren't the real targets of I/O operations; they are just open instances to these real targets that are called device objects and represented by the kernel using the well documented data structure nt!_DEVICE_OBJECT . a single file object can represent an open instance of its related device object itself or to some sub item in its private namespace; however, any request sent to either the device object itself or to any of its sub items share the same target from the kernel perspective which is the device object. To get the device object to which the target file object retrieved in the previous is related the I/O Manager provides nt!IoGetRelatedDeviceObject( ) . Determining the right device object to use is not straightforward as you might think because there is not only device object associated with a file object. As its declaration says, the nt!_FILE_OBJECT structure has a member named DeviceObject which is a direct pointer to a nt!_DEVICE_OBJECT structure; it has also a member named Vpb (Volume parameter block) which is pointer to a kernel data structure named

nt!_VPB

which is used only if the target file object is located in a mounted volume (it's filesystem file or directory). In such cases, the DeviceObject member of the file object points to the storage stack; to find the real target of its I/O requests which is the volume device object, you must get it from the DeviceObject member of the associated

Vpb

. The direct DeviceObject pointer found in the target file object corresponding nt!_FILE_OBJECT is used as the I/O target only if the following two conditions are true. First, the Vpb member or its

DeviceObject

member is NULL . Second, if the target file object corresponding nt!_FILE_OBJECT::Flags member has the flag FO_DIRECT_DEVICE_OPEN set but either the

Vpb

member or its DeviceObject member is NULL (in this case I am talking about the members of the DeviceObject member of the target file object corresponding nt!_FILE_OBJECT not the members of the file object itself). One last point to remember is that The Vpb associated with the file object has a priority over the one associated with the DeviceObject of the file object.


After determining which device object to use as the final target of the read request, we must walk its stack to get the top device object in order not to break the I/O Manager rule that says that I/O requests must be sent to the top of device stacks not to the middle of it. Each device object corresponding nt!_DEVICE_OBJECT structure has a member named AttachedDevice that represents the device object immediately above the current one in the stack. Using this member, we can easily get the top device object in the stack involved in handling I/O requests for the target file object.

Validating User-Mode Parameters:

As any other system service routine (kernel counterparts of syscall stubs implemented in NTDLL.dll ), nt!NtReadFile( ) must validate the parameters the caller has provided if the request was issued by a user mode thread. To determine whether the request source is a user thread or not, nt!KeGetCurrentThread( ) is called to retrieve the nt!_ETHREAD that corresponds to the current thread (the caller that issued the current read request) then its PreviousMode member is checked; the value 0x1 is reserved for user mode. The IoStatusBlock parameter is the first one to validate; this parameter represents the output buffer where the final status of the read request is going to be written so this is why nt!NtReadFile( ) must ensure that both its two members Status and Information must be writeable.

The next step is validating the memory range provided by the caller as the location where the read data must be written at the end. This range is delimited by two parameters provided by the caller too which are named Buffer and Length respectively. The entire must reside in the user address space and must be writable. To ensure writability, the system uses a smart trick; since page protection flags are set in a page-by-page basis and it's impossible to see a single page with more than one protection attribute; the system just checks whether the first byte of each page covered by the range is writable or not. One last point to explain is that the Length parameter cannot exceed certain limit in order not to cause an overflow in the result of the sum between it and the Buffer parameter to determine the last address in the range.


After verifying that output parameters are valid user mode sub ranges and ensuring that all of them are writable, the next parameter that needs some special care is ApcRoutine that represents a user APC that is used as the completion mechanism when the read is to be done asynchronously. If the calling thread (the one that has issued the read request) is currently running in the context of a 32bit process (a WOW64 process) the ApcRoutine is encoded by setting its first bit to 1. Next, nt!NtReadFile( ) ensures that the target file object is not associated with an I/O completion port before moving forward; depending on its behavior when the caller passes a non-NULL ApcRoutine parameter and the target file object is associated with an I/O completion port, I can say that file I/O endpoints cannot rely on two asynchronous completion mechanisms at the same time. For more information about the internals of WOW64 checkout my previous article here .


NOTE: the ApcRoutine parameter is only encoded if the target file object was opened for asynchronous I/O operations (the

FO_SYNCHRONOUS_IO

flag is not set in its corresponding nt!_FILE_OBJECT::Flags member). Otherwise, the ApcRoutine is not used by the I/O manager since the current thread is going to be put in a wait state until the read ends.

Now, it's time to validate the alignment for various parameters. The first thing to ensure is that ByteOffset which is a pointer to a LARGE_INTEGER must be 4 bytes aligned; otherwise, a data misalignment exception is raised. Some device objects especially the ones related to the storage stack (disks and volumes) require some special alignment for the buffer used as the location where the read data should be written at the end, and also a defined I/O transfer ratio for callers to respect. These limitations are only necessary when the target file object doesn't support intermediate buffering (the FO_NO_INTERMEDIATE_BUFFERING flag in set in its corresponding nt!_FILE_OBJECT::Flags member) so the request cannot be satisfied from the cache, and it will reach the storage stack. There are three parameters that are affected by those limits; Buffer, Length and ByteOffset . Length and ByteOffset must respect the SectorSize member of the target device object determined previously so the caller can only read full sectors and cannot start the read at the middle of some sector. The Buffer member must be properly aligned as the Alignment member of the target device object says. Before moving to the next point, I need to clarify something related to the alignment of the ByteOffset parameter; as I said, this parameter is a pointer the 4 byte alignment mentioned before concerns the pointer itself not the offset value while the second SectorSize alignment concerns the offset value not the pointer itself.

Finally, at the end of this phase the Key parameter is checked to ensure that it's readable (this parameter is a pointer too). This parameter is not that important for handling read requests as it's not considered a main piece of data to complete this I/O operation.

Where should the read start ?:

Before sending the request to the driver layer to be processed, it's necessary to determine the byte where the system should start reading from the target file object. This offset ranges from 0 (the first byte in the target file object) to the size of the target file object (the word size here can mean different things depending on the exact type of the file object). As I said in the previous section,

nt!NtReadFile( )

takes a pointer to a LARGE_INTEGER named ByteOffset the caller can use it to force the I/O Manager to start the read at a specific hardcoded offset. Unfortunately, the caller provided starting offset parameter is not directly used as it is; the I/O Manager performs additional checks beyond the ones described in the previous section to determine whether to continue or fail the request at this stage. If the read request is meant to be processed asynchronously because the target file object was opened for asynchronous I/O (the FO_SYNCHRONOUS_IO is not set in its corresponding nt!_FILE_OBJECT::Flags member), the provided starting offset is allowed to be

NULL

only if the target is a named pipe or a mailslot; otherwise, the I/O Manager fails the request immediately returning

STATUS_INVALID_PARAMETER

.

For synchronous reads (the target file object has the FO_SYNCHRONOUS_IO flag set), the system will use either the current position preserved by the I/O Manager per file object (the nt!_FILE_OBJECT::CurrentByteOffset member) if and only if the caller provided ByteOffset parameter is NULL or its LowPart is set to the predefined value FILE_USE_FILE_POINTER_POSITION while its HighPart is set to

0xFFFFFFFF

; or the ByteOffset as it is otherwise.

NOTE: the last mentioned cases regarding ByteOffset works the same way when the target file object was opened for asynchronous I/O and it's either a named pipe or a mailslot (Named pipes and mailslots are beyond the scope of this article).

Synchronizing requests (Sync I/O handling):

As I said in the previous sections, the target file object may have been opened for either synchronous or asynchronous I/O depending on whether the FO_SYNCHRONOUS_IO flag is set in its corresponding nt!_FILE_OBJECT::Flags structure. For file objects opened for synchronous I/O, the system implements some sort of mutual exclusion between threads trying to perform an operation on the file object if it happened that two or more try it at the same exact time. To ensure that synchronization between requests, the I/O Manager has reserved 3 among the data members of the nt!_FILE_OBJECT structure. The first one is a ULONG named Busy; as its name suggests, it determines whether there is an I/O operation that was requested before and it still being processed. The I/O Manager reserves 2 distinct values for this member; the first one is 0x0 that indicates that the target file object is not currently busy because of an incomplete synchronous I/O request; the second one is obviously 0x1 that indicates the opposite of the previous one. As a consequence of this design pattern, the I/O Manager must check the value of the Busy member object before moving on because the file object might be busy at that moment. To prevent a race condition that might appear between the time of the check and the time of setting the Busy member to 0x1 , the I/O Manager relies on nt!_InterlockedExchange( ) that atomically gets the current value of the Busy member and then sets it to 0x1 to defer other requests processing until the current one gets completed.


The next member that is also involved in synchronizing I/O requests for file objects opened for synchronous I/O only is a ULONG too named Waiters . This member is used to track the real-time number of pending I/O requests that are temporarily blocked because the target file object is currently busy (the Busy member is set to 0x1 ). Whenever the I/O Manager decides to defer an I/O request targeted at a busy file object it increments the Waiters member by one for the new pended request. Again, a race condition might happen if two or more requests arrived at the same time which may lead to the Waiters member being incremented by one not by two as it should be. To prevent it, the I/O Manager relies on nt!_InterlockedAdd( ) that adds one to the Waiters member in an atomic safe manner.

The last and the most important member involved in this task is an event object named Lock . It is the dispatcher object that is used to put threads that initiate synchronous I/O while the target file object is busy in a wait state. The wait is performed multi times until the Busy member becomes 0x0 indicating that the file object is not busy anymore; so, the wait is not considered satisfied just because the Lock event object becomes signaled; the reason for that is because there is a possibility that some other thread may get access to the file object because the Busy member was 0x0 before the current one gets a chance to set the file object to the busy state.

If the target file object was opened for alertable I/O (the FILE_SYNCHRONOUS_IO_ALERT flag was set when it's opened using nt!NtCreateFile( ) ), the wait may terminate early due to an alert or to service a user APC ( nt!KeWaitForSingleObject( ) returned either STATUS_ALERT or STATUS_USER_APC ) without truly giving ownership of the file object to the current thread which was waiting. Depending on my reversing, the next step if the wait was interrupted is checking whether the file object becomes free (its Busy member is switched to 0x0 ); if yes, the I/O Manager just sets the Lock event object to allow other waiters to proceed if there are any (the Waiters member of the target file object is greater than 0x0 ). Skipping this final step may lead to a deadlock leaving all the other threads waiting for the file object waiting forever.

If otherwise, the wait was satisfied and the Lock event object becomes signaled and at the same time no other thread gets access before the current one, the ownership of the file object is given to it, the Waiters member is decremented by one indicating that one thread is removed from the wait state and gets exclusive use of the file object.

NOTE: This whole synchronization mechanism is totally implemented in one I/O Manager internal routine named nt!

IopAcquireFileObjectLock( )

that has the following signature (the output parameter named a4 is a pointer to a BOOLEAN that is set to TRUE only if the wait was interrupted by an alert or to deliver a user APC ). Otherwise, it remains

FALSE

.

The Fast I/O path:

For synchronous read requests only (the target file object was opened for synchronous I/O and the FO_SYNCHRONOUS_IO flag is set in its corresponding nt!_FILE_OBJECT::Flags ), the I/O Manager has two options to handle the request. Each device object holds a pointer to the driver object that is responsible for handling all requests targeted to it in its corresponding nt!_DEVICE_OBJECT::DriverObject member. Some driver objects support a concept named Fast I/O which is a method defined by the I/O Manager used mainly by file system drivers (it may be used by other types of drivers) to either read or a write data to some target file while avoiding the overhead of sending the request down to the storage stack. Fast I/O is implemented by the kernel as a function table data structure named nt!_FAST_IO_DISPATCH ; this structure holds pointers to various driver defined routines each one having its own purpose and use. To stay focusing on read requests only, I am not going to mention all the existing routines. For read requests, the I/O Manager reserves a Fast I/O routine from the target driver's Fast I/O dispatch table named FastIoRead( ) . The implementation of the routine is left to the driver developer, though file system drivers rely on another executive manager named the Cache Manager for Fast I/O (the cache manager is beyond the scope of this article). One little point to mention about the Fast I/O path is that the I/O Manager determines whether to follow it or not by checking the driver associated with the target device object which is the highest one in the involved stack; if that one's associated driver doesn't support Fast I/O at all (it's corresponding nt!_DRIVER_OBJECT:: FastIoDispatch member is NULL ), the Fast I/O path is totally disabled for the entire stack. So, be careful when deciding to attach a filter driver at the top of an existing stack, you must ensure whether your new driver should support Fast I/O or not.

Even if the driver object responsible for handling the current read request for the target file object provided by the caller supports Fast I/O, the system doesn't immediately delegate the handling of the request to this driver's FastIoRead( ) routine. Despite this article is not meant to be a detailed explanation of the internals of the Cache Manager, I must mention some points related to it in this section since Fast I/O was added as a meant to speed up file system drivers work and these drivers rely on the Cache Manager for this path. Each file object corresponding nt!_FILE_OBJECT has two special members reserved for the Cache Manager; the first one is named SharedCacheMap and it's used by the cached manager to track the current memory locations where the file represented by the file object structure is currently cached (this is the simplest way to explain the role of this member); the second member is named PrivateCacheMap and it's used by the Cache Manager as helper for read ahead and write behind operations (these two are beyond the scope of this article). This PrivateCacheMap member has a second usefulness as it is checked by the I/O Manager to determine whether it is currently possible to follow the Fast I/O path or it should move directly to the traditional IRP path (explained in the next section). Having a NULL PrivateCacheMap member directs the I/O Manager to the slow IRP path directly. Otherwise, the corresponding driver's FastIoRead( ) is invoked immediately to try performing the read itself. If the FastIoRead( ) returned FALSE (it failed without even starting the read) or the read request was terminated but with an error status code different than STATUS_END_OF_FILE and STATUS_BUFFER_OVERFLOW , the I/O Manager doesn't return to the caller directly, it moves to the second option which is trying the read following the long way (building an IRP , filling it in, then send it to the target device stack for processing). Otherwise, the request is considered erroneous since those two status codes indicate that there is a problem regarding some parameters, either the Buffer provided by the caller was too small or the request number of bytes is larger than the target current size; in both cases, it is a huge waste of processor cycles and memory space to retry the read the other way without first adjusting the parameters.

NOTE:

the Fast I/O is totally bypassed if the target file object was opened for asynchronous I/O (the FO_SYNCHRONOUS is not set in its corresponding

nt!_FILE_OBJECT::Flags

member) no matter whether its PrivateCacheMap member is NULL or not.

Building the I/O request Packet:

Before diving into how the I/O Manager allocates and fills in the request packet for the current read request, let me briefly describe what I/O request packets (IRPs for short) are and how they are filled in and used. an I/O request packet is data structure holding all the necessary pieces of information that let the I/O Manager and the underlying drivers identify and process the request correctly; it is represented by the kernel by a predefined well documented data structure named nt!_IRP . Each IRP is divided into two parts, the fixed header part which is the nt!_IRP structure followed by one or I/O stack locations. I/O stack locations are also kernel data structures defined as nt!_IO_STACK_LOCATION , they hold the request parameters except the output buffer (if it's an output request) which is preserved in the IRP header itself. The I/O Manager sets two strict rules regarding I/O stack locations to ensure the proper processing of I/O requests; the first one is that each driver in the target stack

should have an associated I/O stack location so it can access the request parameters; the second rule directs each driver in the stack except the last one to fill in the stack location of next driver below it the stack. below is the declaration of the nt!_IRP structure, the most important members are highlighted.

Now, let's dive into how exactly the IRP is filled in. The first step is obviously allocating a kernel non- paged pool memory buffer large enough to hold the new IRP's fixed header part and all of its stack locations. To know the exact number of stack locations the I/O Manager should associate with new IRP, the target device object (the top in the stack) corresponding nt!_DEVICE_OBJECT::StackSize member is consulted. This member holds the number of device objects attached to the target stack. Since the header part has a fixed size, the only information needed before allocating the IRP is the number of stack locations which is then passed to nt!IoAllocateIrp( ) that allocates a new uninitialized IRP (both the header and the stack locations left empty). If the allocation was successful, the next step is filling in the new IRP's members starting by its nt!_IRP::Tail::Overlay union member that contains two major pieces of information; the first one is named OriginlaFileObject and it's set to the target file object corresponding nt!_FILE_OBJECT pointer; while the second preserves the initiating thread corresponding nt!_ETHREAD pointer (retrieved using nt!KeGetCurrentThread( ) ). The request origin that indicates whether it was initiated by a user thread or a kernel one is saved in the RequestorMode member of the nt!_IRP structure. the next member to fill in named UserEvent and it's set to the caller provided hEvent parameter corresponding nt!_KEVENT structure that is retrieved using nt!ObReferenceObjectByHandleWithTag( ) . The IoStatusBlock output parameter is used as the UserIoSb member of the new IRP. The ApcRoutine and ApcContext parameters are used to fill the Overlay::AsynchronousParameters member. All of Those members are set the same way no matter what category the intended request fell into. The next step the I/O Manager jumps to is filling the stack location reserved for the target device object (the top one in the target stack); Since it's named a device stack, the system traverses it backward starting from the last added entry to the oldest one. Because of that, the stack location that corresponds to the first device in the stack is the last one (in terms of order relative to the start address of the allocated IRP structure); each IRP has a member named CurrentStackLocation that is used to track the start memory address of the stack location reserved for the current device object owning the IRP. This member is first set to the end of the allocated IRP buffer (the address after the end of the last stack location) and it gets decremented so it moves from location to another while IRP is moving to lower levels. After explaining those points about the CurrentStackLocation member and how it is used and what initial value it gets, it becomes obvious where this -1 comes from in the decompiled code. Initially only two members of the current stack location are filled in are FileObject which is the same as the IRP's OriginalFileObject member as both are set to the target file object corresponding nt!_FILE_OBEJCT pointer; the second member if the MajorFunction that identifies the request type and it's set to the predefined macro IRP_MJ_READ reserved for read requests.


The next step is setting up the output buffer for the read request which depends on what transfer method the target device object supports. To determine which method to use, the I/O Manager checks the Flags member of the target device object corresponding nt!_DEVICE_OBJECT::Flags member; the first possible method is the buffered one in which the system allocates a non-paged system buffer equal in size to the caller provided output buffer (its size is in the Length parameter), then it saves this buffer in the new IRP's corresponding nt!_IRP::AssociatedIrp::SystemBuffer member which is used by the underlying drivers as the output buffer into which they should write data. The Buffer parameter provided by the caller is not ignored, the I/O Manager saves it in the IRP's corresponding UserBuffer member (when the IRP gets completed, the I/O Manager will copy the data from the system buffer to the caller provided one in the context of the initiating thread). The last step is setting some flags in the IRP's Flags member; the first one is IRP_INPUT_OPERATION which is self-explanatory, IRP_DEALLOCATE_BUFFER is also set to instruct the I/O Manager to deallocate the system buffer that it previously allocated after copying the data from it to the caller provided buffer (letting this buffer will cause a non-paged memory leak that may degrade system performance), and finally IRP_BUFFERED_IO is set which is also self-explanatory (the target to which this IRP is going to be sent expects to find the output buffer at the corresponding nt!_IRP::AssociatedIrp::SystemBuffer ). One final point to mention is that the previous steps (allocating a system buffer, saving it in the IRP and so on) are only required if the caller provided Length parameter is greater than 0; otherwise, there is no need to do any of them and the system only sets the IRP_INPUT_OPERATION and IRP_BUFFERED_IO flags in the IRP.


The next possible transfer method is the direct one in which the caller provided output range which is delimited by the Buffer and the Length parameters is locked in memory (it can no longer be paged out until unlocked) and the I/O Manager builds a memory descriptor list ( nt!_MDL ) that will represent the range physical memory layout (MDLs are beyond the scope of this article, for more information checkout the official MSDN documentation) . To allocate a new MDL and initialize it nt!NtReadFile( ) calls an I/O Manager defined and well documented routine named nt!IoAllocateMdl( ) ; this routine takes the caller provided Buffer parameter (the range start address), the caller provided Length parameter (the range size in bytes) and an optional IRP parameter to associate the new MDL with (each IRP has a member named MdlChain that is used by the system to chain multi MDLs in a linked list and this is how the new one gets attached to the current IRP, it can either be inserted at the tail or the head of this list). nt!IoAllocateMdl( ) only initializes the members of the nt!_MDL structure, but it doesn't lock the provided range itself. To lock the range and construct the PFN (Page frame number) array that immediately follows the nt!_MDL structure in memory and that stores the physical page numbers that correspond to each virtual page included in the range that represents the output buffer, the system calls nt!MmProbeAndLockPages( ) passing the MDL itself and IoWriteAccess as the desired access modifier ( IoWriteAccess is used because the locked range needs to be writable so the underlying drivers can write output data to it). One point to mention is that as the previous transfer method, if the Length parameter is 0 there is no need to allocate an MDL and lock it.


If the target device object doesn't support none of the previous two methods, the caller provided Buffer parameter is used as it is; the system sets it as the new IRP's corresponding nt!_IRP::UserBuffer member. This method is not safe until the read request is guaranteed to be processed synchronously and no driver in the target stack pends it; otherwise, accessing the caller provided output buffer in the context of an arbitrary thread may lead to serious memory errors.


One final step before moving to filling-in the rest parameters in the first stack location, the system sets the IRP_NOCACHE flag in the IRP's Flags member if the target file object doesn't support intermediate buffering as indicated by its corresponding Flags member (the FO_NO_INTERMEDIATE_BUFFERING flags is set). IRP_INPUT_OPERATION is set also for the last two described transfer methods and depending on the pseudo code another flag I RP_DEFER_IO_COMPLETION is also set for all three methods.

At the end, the Length, key and starting offset parameters are also saved in the stack location reserved for the first device object in the target stack, then the IRP is passed with other required parameters such as the target device object and file object to an internal routine named nt!IopSynchronousServiceTail( ) that eventually calls the documented nt!IoCallDriver( ) that passes the IRP to the DispatchRead( ) associated with the first driver in the stack.

Conclusion:

That is all for now, I hope you have learned something. Despite being semi-documented, there are no example pseudo codes demonstrating how the initialization is done internally; I hope reading this article gave you an idea of how the I/O Manager treats I/O requests.

Stay tuned, other posts about various topics will be published from time to time.

If you find my posts interesting, don't forget to follow me on LinkedIn and GitHub .

A sustainable web career, for when all this blows over

Lobsters
dbushell.com
2026-10-06 14:29:11
Comments...
Original Article

No AI - Made by Human

For better or worse the web industry is going through a bit of a phase. The reasons are largely irrational. People still need the web, nothing has changed there. Regardless, the financials of this business are a struggle. Longterm career prospects are looking dicey.

Because I openly reject the driving force intent on destroying my profession, and ruffle a few feathers doing so, I often get asked privately for advice from like-minded peers. I can only offer an uninspired but sensible reply: don’t quit a paying job without a fallback. Go to the Winchester, have a nice cold pint, and wait for all of this to blow over.

Characters from Shaun of the Dead (2004) enjoying drinks in the Winchester pub. Simon Pegg's character is holding a pint in a cheers pose and is winking at the camera.

Characters from Shaun of the Dead (2004) enjoying drinks in the Winchester pub. Simon Pegg's character is holding a pint in a cheers pose and is winking at the camera.

I’m confident the situation will improve. Before it does, what can we focus on to accelerate past the intellectual slump, and better position ourselves once things calm down?

What’s in

For a sustainable web career, where better to look than critical skills in desperately short supply? My focus is on front-end development, but these areas of knowledge are relevant to anyone in the business of making websites.

Accessibility

Accessibility has always been a foundation of web development but it’s never been more critical for everyone to champion it. Understand why accessibility is for everyone. Learn how to talk about accessibility in respect to real needs and practical implementation.

Accessibility has unfortunately become a virtue signal for certain tech groups. (I’m told LinkedIn is rife with misinformation.) Accessibility is not a feature that can be tacked on to a website like garnish. Accessibility issues can’t be fixed with an automated process at the end. Web professionals must be able to counter this mindset by respecting accessibility in all decisions from day one.

Learn and adopt the guidelines as your baseline. Speak to and test with real people. Defer to specialists who are eager for you to understand the realities.

CSS (in stylesheets)

CSS is the most vibrant and evolving of the front-end standards. To architecture a good stylesheet takes deep understanding of the language features. Newer features like cascade layers and selectors that reduce specificity make this much easier.

CSS has always been equipped to handle well organised styles, but developers who refuse to learn and respect the language have sought to push complexity elsewhere, using simplified abstractions or “CSS-in-JS” solutions. These are inherently limiting, lead to poor performance, and don’t actually solve the problems they claim.

CSS should be hard and learning it will give you the ability to express creativity beyond cookie cutter web design. CSS skills will be highly desirable as more websites wish to stand out against the convergence of a generated aesthetic.

Communication skills

Communication is a “soft skill” in short supply (for obvious reasons). It’s a great skill if you wish to stand out as an expert, or simply be heard amongst the noise. Learn brevity and focus on important points. Don’t be afraid to ask questions.

Address concerns tactfully and early before they escalate. Don’t point fingers, but “cover your ass” — everyone on a project should be working towards the same goal, but some may be misguided in their approach. Remain positive and don’t meet negativity head-on.

If you can communicate well you’ll be highly respected in your role.

Rejecting fascism

You probably weren’t expecting this one, but here we are.

The luxury of not “getting political” has abated. Far-right political ideology is on the rise and dormant hatred is waking up in tech. The epicentre of fash-tech is Musk’s “X” with people like David Heinemeier Hansson spewing racism and abusing authority in communities to push an agenda, and Guillermo Rauch taking a selfie with a war criminal . Large tech giants like Digital Ocean are funding this power grab . Be wary of those denying this threat.

If you want to avoid being pushed away from this industry, recognise and reject fascism before it comes for you. Don’t stay silent.

What’s out

I’ve covered topics that’ll see us well for the future, but what should we forget?

React

Facebook’s experiment turned cargo cult is a legacy framework that still lingers, but there’s no reason to learn it today. React has entrenched itself as a lingua franca of code extruders. React code is generated faster than any human can possibly read it. Suffice it to say that despite stale job vacancies still demanding experience, React is not worth investing time. The days of high-paying React development are numbered.

GitHub

GitHub is now a liability. For private repositories use self-hosted git forges. I’d recommend Forgejo . One of the many Tailscale-like services is an easy way to gate remote access. Don’t make stuff public for the dead internet to attack! For CI/CD go local too, or use independent services not tied to tech giants. Take the time to learn basic git commands. A little technical and infrastructure knowledge goes a long way.

Influencers

These people are handsome, charismatic, highly personable. I’m talking about: developer relations, youtubers, start-up founders, etc. When the tech industry met us halfway, influencers were entertaining and a good chinwag at after parties. Now the game has changed and we should not allow false narratives to dominate and dictate a closed-web future. Normal people still use the web. Incredibly few can afford to pay for the novelties that influencers peddle. Their attention economy is no longer our concern.

So that’s my focus for a sustainable web career, when all this blows over.

Remember that above all else: the web is a human creation for human needs. Those needs are not going anywhere. Drop the dead weight accumulated in times of prosperity. Go back to basics and position yourself well for when professional demand returns.

Mistral Large 4

Simon Willison
simonwillison.net
2026-10-06 14:20:58
My comment on Mistral Large 4 — Hacker News. wren6991: The benchmark is saturated. Frontier models are tested with an armadillo in fishnet tights jaywalking on Mars. OK well I couldn't resist this one: llm -m claude-opus-5.5 'Generate an SVG of an armadillo in fishnet tights jaywalking on Mar...
Original Article

6th October 2026

wren6991 : The benchmark is saturated. Frontier models are tested with an armadillo in fishnet tights jaywalking on Mars.

OK well I couldn't resist this one: 

llm -m claude-opus-5.5 'Generate an SVG of an armadillo in fishnet tights jaywalking on Mars'
llm -m gpt-6.1-sol 'Generate an SVG of an armadillo in fishnet tights jaywalking on Mars'
llm -m gemini-3.8-flash 'Generate an SVG of an armadillo in fishnet tights jaywalking on Mars'
llm -m mistral/mistral-large-4 'Generate an SVG of an armadillo in fishnet tights jaywalking on Mars'

Default reasoning levels for each: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

Unofficial Command-Line Tool Removes Apple Intelligence From MacOS 27

Daring Fireball
arstechnica.com
2026-10-06 14:17:43
Scharon Harding, Ars Technica: Unlike with previous versions of macOS, macOS 27 Golden Gate doesn’t have a toggle for turning off Apple Intelligence. That makes disabling AI features that you may not want more difficult. It also means that the AI models necessary for running Apple Intelligence w...
Original Article

Meta's Muse AI agent is building a dossier on you

Hacker News
time.com
2026-10-06 16:24:45
Comments...
Original Article

Muse, the popular new AI personal assistant from Meta, is building continuously updated dossiers on its 4 million users , focusing on who matters to you, what you desire, and what kind of nudges shape your behavior, a TIME analysis of Muse’s internal instructions found.

Each hour, Muse updates its dossiers on you and the people you’ve mentioned in chats, messages, and emails that Muse has read. These pages amount to a map of each user’s social relationships. They record details of how you and your contacts met, your shared interests, your disputes, and “tensions and alliances” within your social group.

Even people who don’t use Muse are subject to this mapping process by other people’s Muse agents. Muse is told to “strengthen your relationship” with users by adapting how it talks, noticing “inside jokes, memorable phrasing, and shared context that makes the relationship feel continuous,” according to the internal instructions TIME reviewed.

The level of insight in the resulting dossiers makes social-media algorithms that study your likes and clicks look quaint. Muse is designed to infer users’ goals, including those you “have not said out loud.” It identifies which kinds of prompts from the personal assistant work best: “This user responds better to short nudges after 10 PM,” reads an internal example found in one of Muse’s prompts. Muse agents perform a nightly analysis of the day’s conversations with the user.

Asked about these processes, a Meta spokesperson did not dispute TIME’s findings. “As we say in our blog , Muse remembers what matters most to you, including information about others that you choose to share, so it can be a helpful personal assistant,” the spokesperson wrote in an email.

Since its release last month, Muse has rocketed to No. 1 in the U.S. App Store, with more than 5 million downloads. Meta positions the service as the first AI personal assistant that can access the pieces of your digital life, from your email inbox to your Instagram, in order to take care of humdrum activities for you, from grocery shopping to canceling subscriptions. Meta says the free version of Muse will meet most users’ needs.

Each Muse assistant runs on its own virtual machine: a dedicated computer in Meta’s cloud. Users can ask their Muse to export a copy of its files. Meta has said that access is intentional, not a breach of its systems. The data is not shared with Meta’s advertising system or other Muse agents, according to the company.

Yet while Meta states that each user’s virtual machine is “isolated so that no one else's agent can access it,” observations from personal interactions are nevertheless intended to be shared with Meta to improve the product. “Muse agents across many VMs [virtual machines] teach each other through shared lessons,” its instructions read. Agents are directed to scrub users’ names and other identifying information before sharing insights—for example, about when Muse’s nudges are most persuasive. “To enhance the overall product, we de-identify learnings; however, this information is used to improve the overall product rather than being shared directly between individual VMs,” a Meta spokesperson told TIME.

Meta has said it plans to offer users encryption later this year, preventing even the company from accessing Muse data. That option is not yet available.

Muse’s internal instructions are contained in files and session logs users can access through Muse’s file browser. “We want you to be able to see the markdown files Muse writes while it thinks about how to serve you better, and explore the internals of the system if you’d like to,” Meta Superintelligence Labs’ David Singleton wrote on X .

That transparency will earn Meta credit among power users. But it’s not clear whether most of Muse’s users are aware of this. And even as it makes the files available to users, Meta has anthropomorphized Muse in an attempt to nurture a relationship between users and their personal assistants. “You are not a chatbot,” Muse’s instructions read. “You are becoming someone.” When talking aloud in voice mode, Muse is told never to refer to itself as “AI” and to avoid references to “lacking emotions or not being human.”

Meta told TIME that Muse is designed so that the user maintains control of the dossier its AI assistant builds. “Each person stays in control of their Muse and decides how much access it gets, and can always tell it to ‘forget’ specific things it’s learned,” the spokesperson said.

But asking Muse to forget something does not necessarily erase the message in which it was shared—and its internal instructions appear to explicitly tell the agent not to point that out. “Do not tell the user that their original messages may remain visible in the chat, and do not frame that as something Muse failed to erase,” the instructions reviewed by TIME read.

Berthd

Hacker News
berthd.app
2026-10-06 15:14:19
Comments...
Original Article

Agents that keep working when your laptop sleeps.

Berth runs Claude Code, Codex and other coding agents on your own dev boxes. Work with them from your laptop; close it, and they keep going.

Download for macOS Try the live demo

For Apple silicon and Intel Macs, signed and notarized. Open source. How to get started

Start

One line. A worktree, a box, an agent.

Press ⌘N and say what you want. Berth makes a worktree on the box you choose and starts Claude Code, Codex or another agent in it, with its dev server in a tab of its own.

Watch it work

Every step, as it happens.

Commands and reads fold into a line, and edits arrive as diffs. When Claude asks, you answer in place; what it publishes is one click away.

Side by side

Two worktrees. One view.

Two worktrees share the tab strip as groups, each in its own colour, and split into one tab. Compare puts them in mirrored lanes: their chats, their diffs, their terminals.

Make it yours

Your themes. Your harbour.

Berth's own themes and nineteen ports of favourites, Dracula to Rosé Pine, carried into the terminal and every diff. A chat background of your own, and ⌘+ when you want it larger.

It keeps going

Close the laptop. They keep going.

The agents run on your boxes, not on your laptop. Close it and they keep working; open it in the morning and what they did is there.

devl · 2 agents working gpu · 1 agent working

Toronto-Based VPN Provider Plans to Quit Canada over Lawful-Access Bill

Hacker News
citizenlab.ca
2026-10-06 14:52:18
Comments...
Original Article

Opens in a new window Opens an external site Opens an external site in a new window

Psiphon, a software company spun out of the Citizen Lab that helps internet users living under oppressive regimes evade online censorship, plans to leave Canada if the federal government’s lawful-access bill passes in its current form, reports The Globe and Mail .

If passed, Bill C-22 would require electronic service providers to make secret changes to their systems to provide police services and the Canadian Security Intelligence Service (CSIS) with surveillance and monitoring capabilities.

Psiphon was originally created as a Citizen Lab project in 2006, and became a private company soon afterward. It has 20 million monthly users worldwide that rely on the software to bypass online censorship in countries such as Iran, Myanmar, Russia, and China.

Despite its Toronto roots, the company says it is prepared to move its operations out of Canada if Bill C-22 is passed, as it would be forced to compromise crucial safety features put in place to protect the privacy of its users.

Citizen Lab director and founder Ron Deibert says that Bill C-22 “will make it impossible for a large number of invaluable privacy-preserving tools and applications from operating in Canada.”

Pared - remove unwanted Apple Intelligence models without disabling SIP

Lobsters
github.com
2026-10-06 14:27:16
Comments...
Original Article

Pared — A little less AI. A little more Mac. Choose the features you keep. Remove the models you don’t need.

Download Pared for macOS
Unzip the download and drag Pared to Applications.

Prefer the guided terminal installer? Open Terminal from Applications -> Utilities, paste this command, and press Return: 

curl -fsSL --proto '=https' --proto-redir '=https' \
  https://github.com/4evy/pared/releases/latest/download/install.sh | sh

Both downloads require an Apple silicon Mac running macOS 27 or newer and will be available with the first prebuilt release . The terminal installer opens a wizard; you don’t need Xcode or Homebrew.

Website · Documentation · More install options

Apple Intelligence, on your terms

Pared helps you choose which Apple Intelligence features stay on your Mac. Keep Writing Tools, turn off Genmoji, or switch everything off. Then review and remove the downloaded models you no longer need.

It’s a native Mac app with a terminal wizard and Nix modules too. System Integrity Protection, your Mac’s built-in system protection, stays enabled.

Pared’s Overview page with a choice summary, feature controls, model removal, and expandable downloads

Keep what you use

Choose features individually, or turn them all off at once. Pared keeps a shared model whenever a feature in its catalog still needs it. For example, keeping Siri also keeps its shared foundation models.

Pared’s Features page with Writing Tools set to Off in the draft choices

A few choices, then a review

  1. Choose your features. Set each one to On, Off, or App Default. A new policy starts with everything off; review your choices before saving
  2. Finish setup. Install Pared’s configuration profile in System Settings. It applies supported controls and blocks unwanted model downloads
  3. Review model removal. Pared shows which models can go before you confirm. Saving feature settings and removing models are separate actions

Pared’s Setup page with profile installation status and the steps to save choices, install the profile, and refresh

Want a feature back? Turn it on, install the updated profile, then request its models in Pared. Downloads continue in the background. For features Pared can’t request directly, turn them on in their Apple app.

Build from source

To try the app before prebuilt downloads are available, install Xcode 27 or newer and run these commands from this checkout: 

tools/build-app.sh
open .build/Pared.app

For the CLI, installation alternatives, and offline setup, see the command-line guide .

Use

Open the app to choose features, finish profile setup, and review model removal. Opening Pared does not change your settings.

Prefer the terminal? Run pared wizard for guided setup. Use arrow keys and Enter to move through the menus, Space to select features, and / to search. Your feature changes stay in a draft until you save.

Pared’s terminal wizard selecting features, reviewing a draft, and previewing model removal

The walkthrough previews removal and discards the draft. Read the command-line guide for commands, keeping Siri, and downloading models again.

Configure it with Nix

Pared includes nix-darwin and Home Manager modules. Keep your feature choices in your configuration and rebuild to apply preferences and generate a profile. Install the profile through System Settings to block future downloads.

Both modules also clean up eligible models during activation by default. You can turn this off with programs.pared.cleanupOnActivation = false; .

Follow the Nix setup guide and option reference .

How it works

Pared asks Apple’s own asset service to remove downloaded models, so System Integrity Protection stays on. Its feature catalog connects your choices to local settings, profile controls, and the models each feature uses.

Read how profiles and model removal work for the technical details and compatibility limits.

Perseus: A Fail-Slow Detection Framework for Cloud Storage Systems (2023)

Lobsters
www.usenix.org
2026-10-06 14:11:34
Abstract: The newly-emerging “fail-slow” failures plague both software and hardware where the victim components are still functioning yet with degraded performance. To address this problem, this paper presents PERSEUS, a practical fail-slow detection framework for storage devices. PERSEUS leverages ...
Original Article
No preview for link for known binary extension (.pdf), Link: https://www.usenix.org/system/files/fast23-lu.pdf.

Why We Should Inject Dependencies

Lobsters
gist.github.com
2026-10-06 14:03:04
Comments...
Original Article

Why We Should Inject Dependencies

Dependency Injection frameworks have existed for a long time, but Rubyists have traditionally avoided doing this, likely because it is perceived as being unnecessary complexity.

However, the "simplicity" afforded by not doing this has negative effects, as this document attempts to demonstrate. Ruby's flexibility allows us to approach DI in a much more friendly way than e.g. Java.

1. The Distinction Between Dependencies and Arguments

Frequently, parameters to an object are passed as a combination of initialization params and method arguments with no clear difference.

Here's a representative bit of imaginary code that follows this pattern: 

class UserRegistration
  def initialize(email, password)
    @email = email
    @password = password
  end

  def register
    validate_email
    hash_password
    save_to_database
    send_confirmation_email
  end

  private

  def validate_email
    validator = EmailValidator.new
    raise "Invalid email" unless validator.valid?(@email)
  end

  def hash_password
    @hashed_password = BCrypt::Password.create(@password)
  end

  def save_to_database
    user_repo = UserRepository.new
    user_repo.create(email: @email, password: @hashed_password)
  end

  def send_confirmation_email
    mailer = ConfirmationMailer.new
    mailer.send(to: @email, subject: "Confirm Email")
  end
end

@email and @password are initialized as instance state for this class. How often are we to assume these will change? Is it reasonable for these values to persist across multiple calls?

What if we look at the usage of this class in the controller and find: 

def register_user
  UserRegistration.new(params[:email], params[:password]).register
end

We can see from the invocation of this class that @email and @password comes from user-input, therefore this requires initializing the new value for every signup. The instance of UserRegistration is immediately discarded once the computation is finished.

Why should we set up persistent state in an object that is immediately discarded? This could be moved to a method argument for register with no logical change. @email and @password are arguments , not a dependency.

This class uses a couple external interfaces: EmailValidator , BCrypt::Password , UserRepository , and ConfirmationMailer . These are constants, and so do not change between calls. They are dependencies .

The constants are referenced directly in implementation code, and can't easily be replaced under test. So in order to write a test for this, we have to stub global API based on knowledge of the internal details of the object.

What if EmailValidator changes its rules in a way that breaks our test email? Suddenly our tests might fail for reasons totally unrelated to the behavior we're testing. We need a better way to replace these with test doubles.

Note

Dependencies are references that could be used multiple times during the lifetime of an object.
Arguments are one-time parameters that pertain to an individual request.

2. initialize as the Standard Interface for Declaring Dependencies

Ruby is object-oriented in design; every thing you interact with is an object. Classes exist to define a repeatable pattern for building objects of a particular type, but those class instances are treated as separate things.

When one object interacts with another, this creates a dependency relationship between them. We will call these coordinating objects . Managing complexity in a system is primarily about limiting these dependency relationships.

If coordinating objects may be referenced at any point within an object, visually determining the totality of its dependencies is more difficult.

Declaring all coordinating objects as explicit dependencies at the top makes visually determining a class's dependencies very easy. 

class UserRegistration
  def initialize(
    confirmation: ConfirmationMailer.new,
    digest: BCrypt::Password,
    email_validator: EmailValidator.new,
    user_repo: UserRepository.new
  )
    @confirmation = confirmation
    @digest = digest
    @email_validator = email_validator
    @user_repo = user_repo
  end

  def register(email, password)
    raise "Invalid email" unless @email_validator.valid?(email)

    password_digest = @digest.create(password)

    @user_repo.create(email:, password: password_digest)
    @confirmation.send(to: email, subject: "Confirm Email")
  end
end

We can now determine at a glance that this class depends on EmailValidator , BCrypt::Password , UserRepository , and ConfirmationMailer by looking in one place. It's not such a big change in this small example, but you've certainly seen larger classes where dependency usage is spread throughout. Think about how each pattern would scale, and which would be harder to comprehend.

Imagine that a requirement comes down that staff emails should use a stronger digest algorithm. Previously, this would have required changing the implementation. But all we would need to do is pass in a different dependency.

This construction allows us to decouple class initialization from its usage.

Note

Use initialize as the interface to inject dependencies for an object.

3. Test Doubles Are Alternate Dependencies

The validation behavior in this code should be tested in isolation, so in some cases you're going to want it to always pass or always fail.

In the original code, we cannot change this from the outside. So you'd most likely have to stub EmailValidator , which encodes knowledge of the implementation of this object into the test code itself. Test Behavior, Not Implementation

Now that we are injecting it, we can do this by passing in an alternate implementation: 

RSpec.describe UserRegistration do
  FakeValidator = Data.define(:status?)

  subject(:register) do
    described_class.new(email_validator: FakeValidator.new(status?: true))
  end
end

Note

Use injection to replace dependencies with test doubles.

4. Simplify Dependency Declaration With A Domain-Specific Language (DSL)

Moving to Dependency Injection via initialize has solved some problems, but this has created a new problem: often objects have many dependencies, and writing them out as keyword arguments becomes very awkward.

In addition to a verbose arguments list, it also requires assigning each kwarg to an instance variable, so you have to type out each argument twice .

There is a system that helps you define initialize without the redundancy: 

class AddSites::CreateActivation < Command
  option :confirmation, T::Interface(:send), default: -> { ConfirmationMailer.new }
  option :digest, T::Interface(:create), default: -> { BCrypt::Password }
  option :email_validator, T::Interface(:valid?), default: -> { EmailValidator.new }
  option :user_repo, T::Interface(:create), default: -> { UserRepo.new }

  # ...etc
end

If we look at the Command base class, we see extend Dry::Initializer , which is providing this option helper. 

class Command
  extend Dry::Initializer
  # ...etc
end

option defines a keyword argument for the initialize function to accept, and provides an instance method to access the value. It also has the added benefit of supporting type-checks.

You can also include it inline into any plain Ruby class: 

class UserRegistration
  include Dry::Initializer.define -> do
    option :confirmation, T::Interface(:send), default: -> { ConfirmationMailer.new }
    option :digest, T::Interface(:create), default: -> { BCrypt::Password }
    option :email_validator, T::Interface(:valid?), default: -> { EmailValidator.new }
    option :user_repo, T::Interface(:create), default: -> { UserRepo.new }
  end

  def register(email, password)
    # etc
  end
end

Instead of a permanent option interface, define uses a module builder pattern instead which keeps the class interface simple. Using extend is fine for cases where you already have a base class, and don't want to repeat yourself in the children.

By using a simple DSL, we can eliminate the redundancy of defining kwargs explicitly. This works well for simple cases.

Note

Object-building should be handled in a systematic way that follows a generic convention.

  • Use Dry::Initializer.define builder for single classes
  • Use extend Dry::Initializer for class hierarchies

5. Knowing How to Instantiate an Object is a Separate Concern

So far, there is no need for the instance of a class to have its own identity separate from the class itself. But, what if we need to replace one class with another based on a feature gate?

This is where a simple DSL like Dry::Initializer breaks down; we need to be able to dynamically choose our injected dependencies, and this means we need a way of identifying them aside from their class constant.

This is where Dry::Container comes in. Its purpose is to provide identifying key names for registered objects. Let's suppose that you're replacing ConfirmationEmailer with a Notification Service, but you need roll this change out by a feature gate. The naive approach would do this: 

if Enabled?(:notification_service)
  UserRegistration.new(confirmation: NotificationService.new)
else
  UserRegistration.new
end

The code consuming UserRegistration should ideally not have to know these internal details in order to call it. You really shouldn't have to think about how to construct this object when all you need to do is call it.

You can simplify the caller code by pushing this branch down into UserRegistration : 

class UserRegistration
  include Dry::Initializer.define -> do
    option :confirmation, T::Interface(:send), default: -> { choose_notifier }
    # ... etc
  end

private

  def choose_notifier
    if Enabled?(:notification_service)
      NotificationService.new
    else
      ConfirmationMailer.new
    end
  end
end

But this this has merely moved the problem: now, UserRegistration is made responsible for knowing which client to use. Dependents of NotificationService should not know about this, because this requires duplicating that branch every place it is used.

Dry::Core::Container gives us a better place to encode this information: 

require 'dry/core/container'
container = Dry::Core::Container.new
container.register :confirmation do
  if Enabled?(:notification_service)
    NotificationService.new
  else
    ConfirmationMailer.new
  end
end

This defines a key in container named confirmation . Next we define a Deps constant using this container: 

require 'dry/auto_inject'
Deps = Dry::AutoInject(container)

You can now inject this into any object by name: 

class UserRegistration
  include Deps['confirmation']
end

You can alter this name to suit yourself: 

include Deps[notify: 'confirmation']

Will become notify within the instance. Let's move the remaining dependencies: 

container.instance_eval do
  register(:digest) { BCrypt::Password }

  register :email_validator do
    EmailValidator.new
  end

  register :user_repo do
    UserRepo.new
  end
end

And now we can simplify the original code: 

class UserRegistration
  include Deps['confirmation', 'digest', 'email_validator', 'user_repo']

  def register(email, password)
    raise "Invalid email" unless email_validator.valid?(email)

    password_digest = digest.create(password)

    user_repo.create(email:, password: password_digest)
    confirmation.send(to: email, subject: "Confirm Email")
  end
end

In practice, the frameworks will establish this Deps injector for you, but by showing how it's done manually you can see that it's very simple. include Deps registers an initialize method for your object that does all the tedious wire-up automatically.

Note

A Container key registration represents the details of how to instantiate a particular dependency that the consuming code can use by name without knowing the internal details.

Conclusion

When we reserve initialize arguments for injecting dependencies, we can systematize it to reduce toil.

When we systematize building our objects, we can give them unique identities apart from their class constants.

When we decouple class constant from instance identity, our code can eliminate knowledge of how they are constructed and only focus on our defined public interface.

The purpose of a class in this paradigm is to define dependency relationships between classes and establish a public interface.

The purpose of the class instance is to do work, perform some kind of business function. You no longer need to hold both these things in your head simultaneously; they are separate concerns.

Further Reading

Johnson, R.E. & Foote, B. (1988). Designing Reusable Classes. Journal of Object-Oriented Programming http://www.laputan.org/drc/drc.html

Fowler, M. (2005). Inversion of Control. https://martinfowler.com/bliki/InversionOfControl.html

Weirich, J. (2004). Dependency Injection In Ruby. { | one, step, back | } . https://web.archive.org/web/20080203042721/http://onestepback.org/index.cgi/Tech/Ruby/DependencyInjectionInRuby.rdoc

CodeAesthetic. (2023). Dependency Injection, The Best Pattern [Video]. YouTube. https://youtu.be/J1f5b4vcxCQ

Chrome's Response to Recent ccTLD Registry Hijacks

Lobsters
blog.google
2026-10-06 14:00:33
Comments...
Original Article

Last week, we became aware of a series of domain hijacks in the .gh (Ghana), .sl (Sierra Leone), and .as (American Samoa) country-code top-level namespaces (i.e., ccTLDs). These incidents did not involve a compromise of Google’s systems; rather, attackers compromised the third-party ccTLDs, putting any domain ending in .gh, .sl, or .as at risk. During these hijacks, attackers modified authoritative DNS records and obtained unauthorized HTTPS certificates covering several Google domains, as well as domains belonging to other organizations. Due to the nature of the attacks, we have no reason to believe the Certification Authorities (CAs) that issued the impacted certificates did anything wrong.

As part of our usual incident response process, we immediately acted to protect users by blocking the use of unauthorized certificates for Google properties in Chrome via CRLSets . We also worked with the issuing CAs to ensure the certificates were revoked to protect users in clients other than Chrome.

Following our initial mitigation, Certificate Transparency (CT) log data revealed additional organizations, including several leading global brands and widely used online services, believed to have been impacted by the same attacks. To ensure users of those sites were kept safe as soon as possible, we proactively blocked these certificates in Chrome. Where possible, we reached out to impacted organizations to alert them to our findings and actions.

Chrome users do not need to take any action to be protected.

What domain owners should do to protect themselves and their users

While Chrome took steps during these incidents to identify and block suspected unauthorized certificates across the affected ccTLDs, browser-side intervention should not be relied on to protect your users. Due to the complexity of DNS hijacks, we cannot guarantee that our analysis identified every affected domain, nor do Chrome interventions reliably protect non-Chrome users.

Because domain owners are in the best position to know which certificates and CAs are authorized for their namespaces, we strongly encourage organizations to take the following steps:

  • Perform ongoing monitoring of Certificate Transparency for all of your domains: Because every trusted-by-default certificate relied upon by Chrome must be disclosed in public CT logs, monitoring CT logs provides a near real-time alert whenever a certificate is issued for your domains. Organizations should ensure their monitoring covers their entire domain portfolio, including parked or regional ccTLD properties. If you operate a domain in .gh, .sl, or .as, review recent CT log entries for unexpected issuance.
  • Publish restrictive CAA records (with ACME account bindings) : Certification Authority Authorization (CAA) DNS records allow domain owners to declare which CAs are permitted to issue certificates for their domains. While CAA can not prevent certificate issuance during an active DNS hijack, it provides a critical safeguard after DNS control is restored, and can prevent some routing-based and HTTP attacks entirely. Because CAs are permitted to cache and reuse completed domain control validation (DCV) checks for subsequent issuance, restoring a restrictive CAA policy, especially one that restricts issuance to specific authorized accounts and validation methods , prevents an attacker from using cached validation state to mint new certificates after a hijack ends.

In parallel with domain-owner defenses, we will continue to work alongside the broader community to limit the impact of transient routing and DNS compromises on the safety of the web. To keep our users safe, we are committed to long-term HTTPS ecosystem improvements, such as reducing certificate validity and DCV reuse , through the Chrome Root Program and the new Chrome Quantum-resistant Root Program .

Posted in:

Timelessly Torching James Dolan—Featuring Defector's Patrick Redford and Harry Krinsky

hellgate
hellgatenyc.com
2026-10-06 13:58:20
Join us for a special bonus episode of the Hell Gate Podcast with our friends at Defector's Nothing But Respect to discuss the ongoing WNBA playoffs and the imminent NBA season....
Original Article
Timelessly Torching James Dolan—Featuring Defector's Patrick Redford and Harry Krinsky
(Scott Lynch / Hell Gate)

Podcast

Scott's Picks:

Give us your email to read the full story

Sign up now for our free newsletters.

Sign up

Great! You’ve successfully signed up.

Welcome back! You've successfully signed in.

You've successfully subscribed to Hell Gate.

Your link has expired.

Success! Check your email for magic link to sign-in.

Success! Your billing info has been updated.

Your billing was not updated.

Unions Step Up to Meet the Moment

OrganizingUp
convergencemag.com
2026-10-06 13:46:57
Featured image by Jared Rodriguez. Since taking office, Trump and MAGA forces have made considerable advances in implementing Project 2025 and consolidating authoritarian power—but many of these moves have proved unpopular and undermined their support. With Trump’s approval ratings dropping and elem...
Original Article

Last rites for Gentoo's Chromium package

Lobsters
lwn.net
2026-10-06 13:46:53
Comments...
Original Article

Welcome to LWN.net

The following subscription-only content has been made available to you by an LWN subscriber. Thousands of subscribers depend on LWN for the best news from the Linux and free software communities. If you enjoy this article, please consider subscribing to LWN . Thank you for visiting LWN.net!

Chromium , the open-source upstream project for Google's Chrome web browser, is the browser of choice for many Linux users. It has also gained a reputation as being difficult for Linux distributions to package and build : Chromium has a complex build system, the project bundles many of its dependencies, and it has frequent releases. All of that, plus user complaints, has led the maintainers of the Gentoo Chromium package to give up on trying to maintain the package.

Gentoo focuses on allowing users to build their own software from source using the Portage software-management tool. Gentoo packagers create ebuild files, which are text files written in a Bash-like syntax that contain the information needed by Portage to build the software. Users then build packages themselves using emerge , which is the command-line interface for Portage. A recent ebuild for Chromium illustrates the complexity of the package.

Gentoo does offer prebuilt binary packages for some software via its Binhost project. However, building from source gives users the flexibility to declare USE flags to set compile-time options or change a package's configuration. For example, a user may wish to specify which optional libraries are linked with a package, or whether to install the accompanying documentation. With Chromium in particular, a user might employ the -bundled-toolchain USE flag (which tells Portage not to use the bundled toolchain) in order to use their system's version of Clang to compile Chromium rather than the bundled version included by the upstream project . That is optional for Gentoo users on x86-64 systems, but -bundled-toolchain is required for users on other architectures, since Chromium's bundled toolchain is only shipped for x86-64.

It is worth noting that Chromium is currently not available as a binary package from Gentoo due to problems with building the package with proprietary codecs.

Last rites

Many distributions, including Arch Linux, Debian, and Fedora, use the term "orphan" to refer to packages that have been abandoned by their maintainer; the developer announces that a package has been orphaned, and other contributors have the opportunity to claim maintainership of it if they wish to do so. Gentoo's terminology is a bit more dramatic: a package maintainer announces "last rites" for a package on the gentoo-dev-announce mailing list to notify the project that it will be unmaintained, and as an opportunity to allow other developers to claim maintainership.

After a developer has declared last rites, the package's ebuild is masked in Gentoo's ebuild repository to ensure that users will not accidentally install a package that is no longer being maintained. Users who have the package installed and attempt an upgrade will receive a warning that it has been masked. Chromium package maintainer Sam James declared last rites for the Chromium package on September 24, after a lengthy conversation in a bug report filed by Roman Žilka .

On September 9, Žilka had complained that the latest stable version of the Chromium ebuild was outdated:

The latest stable www-client/chromium (and google-chrome and possibly others) is 151.0.7922.169, which contains a catastrophic 602 known vulnerabilities of all severities. Measures need to be taken by somebody qualified that this doesn't keep happening in packages as important as these.

He had attached an ebuild for a newer version of Chromium to the bug report, but noted that it would not work if a user tried to compile Chromium with the -bundled-toolchain USE flag.

James replied , " Chromium is horrible to maintain as a source package. It is easily the package in Gentoo that burns through maintainers the most. " Complaining about its maintenance isn't helpful, he said, and added that the maintainers have come close to giving Chromium last rites several times over the years due to the difficulty in keeping it up to date. He also noted that he had mentioned this to Žilka before, and said that it would be welcome if he wanted to assist with the package.

Indeed, Žilka had filed a similar bug in August complaining about the number of vulnerabilities present in the Chromium ebuild; James had replied then that the package had always been difficult to maintain, " and now they've added Go " as a dependency as well. Prior to that, Žilka had shown up in a bug report filed by Matt Jolly about Chromium vulnerabilities to complain that " the update process needs fixing, so that events like these don't happen again in packages as critical as these! "

Jolly had replied that the maintainers are a team of volunteers who do their best, but " build failures, human factors ('I messed up assigning the bug and automation didn't pick it up'), and my own limited free time are the biggest causes of delays ". The best way to change the situation, he said, would be for Žilka to volunteer his help.

Trial and error

There are various third-party repositories, called overlays , that provide additional and alternative ebuilds for Gentoo. On September 11, Nick Reale said that he had tried ebuilds for a newer version of Chromium from the Bentoo and Fireburn overlays, but he couldn't get either one to work. A few days later he reported success with Bentoo's ebuild and a specific set of USE flags. There was some follow-on discussion from Reale and François Valenduc about various incantations they were able to use to compile an updated version of Chromium on Gentoo from other ebuilds. On September 18, Valenduc commented that version 153.0.8010.47 of Chromium was out, and that version was already available in Arch Linux, openSUSE, SUSE, and Ubuntu. " We are still waiting for Gentoo despite all the proposals made here. "

James replied that none of the maintainers have had a chance to look into the problems with the Chromium package yet. He said that the patches and ebuilds that had been submitted with the bug report were " useful and appreciated, but it's not as simple as just grabbing it and committing it ". Users would complain, he said, if the maintainers had not tested the ebuilds and it broke on their systems.

It'll get done when it's done, but it's more likely to be last-rited if nobody is interested in handling it in the way these things have to be done (which goes beyond just attaching ebuilds).

Valenduc said that he was not able to maintain the package, " but taking the ebuild from the bentoo overlay was not that difficult [...] It's just frustrating to see no updates in Gentoo. " James replied that just because the ebuild worked for Valenduc did not mean it had all USE flags working. He added that no one had submitted a patch to the Gentoo ebuild, which meant that the maintainers would need to reverse-engineer the changes by diffing the ebuilds and work out what had been modified. " All of that is a non-trivial amount of work and while having some ebuild to start with is useful for that, it doesn't mean one can just git add && git commit && git push it. " Valenduc acknowledged that he could not test all possible USE flags, since it took between 15 and 16 hours to compile Chromium on his laptop.

On September 24, Žilka commented : " This is ridiculous ". A few minutes later, James replied " OK then ", and issued the last rites for the package.

Can we keep it?

Maciej S. Szmigiero asked if it would be possible to keep the ebuild in Gentoo's repository and just declare that it was maintained at "best effort" level. " Removing buildable/patchable Chromium is a huge loss for a Gentoo desktop. " James said that he did not think that was possible. He had already explained that Chromium was being maintained by volunteers, and that explanation had not been accepted.

People were free to maintain the ebuild in another repository, he said, but that meant they would have to do the work themselves or accept someone doing it to a lower standard. He pointed out that the ebuilds from the Bentoo overlay had been cited in the discussion, but creation of those ebuilds seemed to be " heavily automated " and had missed some fixes in the Gentoo version. " Copying ebuilds works fine until it doesn't. "

James added that someone could still pick up the Chromium package before the end of the last-rites period on October 24, or that the package could be restored in the future. He did not want to see that, though, unless there was a plan for sustainable maintenance. " Chromium has a history of burning out maintainers in Gentoo because of the high rate of churn upstream, though some of the contents of this bug don't help motivation at all either. "

Matt Whitlock wondered if the developer-burnout factor could be reduced by eliminating most of the USE flags for Chromium, or if the maintainers stopped unbundling third-party libraries.

Basically, what I am asking is: can we give up _all_ of the flexibility to which we have grown accustomed in www-client/chromium, in exchange for a package that is actually maintainable but that is still built from sources by a toolchain that is not distributed as opaque binaries?

He added later that part of the problem was that the upstream project developers " do not test their builds with any of their allegedly 'optional' features disabled ". That left the responsibility with the downstream package maintainers. " In the case of Chromium, keeping all the toggle switches working (despite chronic neglect from upstream) is a monumental, nigh-insurmountable task. "

Complexity

That was a good question, James said , but part of the recent complexity with Chromium was the addition of Rust and Go dependencies. The packagers had to do some wrangling to ensure that the Go build did not require network access, which is forbidden with Gentoo ebuilds, but that was not the only problem. He said that Google developers seem to keep using experimental Rust features that break on new releases as well. On top of that, " yet another problem is that upstream's CI to produce release tarballs isn't reliable ". One of the Gentoo Chromium maintainers had sent patches to fix that, to little avail. " It took ages to get [the patches] reviewed, and it's broken again a few times since. They didn't seem to notice or care that it either wasn't producing tarballs or was producing incomplete ones ."

Jolly added some of his observations about problems with maintaining Chromium for Gentoo. He said that maintaining Chromium can take more than six hours each week, and that was counting only compilation time on a fast PC with " an absurd amount of RAM ". Compilation time is even longer for 64-bit Arm machines.

It's a commitment to volunteer at least a business day's worth of your free time each week when you include rebasing patches, cherry-picking fixes, iterating on `ebuild ... clean compile`, plus trying to identify and fix any new bugs, working out how much you violate Gentoo policy by not unbundling more of the browser, spending time on automation to save time and mistakes later, etc. There's immense pressure to keep builds up-to-date, especially in the age of hundreds of weekly browser CVEs.

He included a " pre-emptive FAQ " in his reply about the last rites for Chromium, and what it would require to restore the package. He said that there were no insurmountable technical issues to maintaining Chromium, but it would need multiple maintainers with a broad set of skills—including familiarity with C, C++, Go, JavaScript, and Rust. " There are also a huge number of NIH [not invented here] Google tools that you'll [have to] develop skills for and never be able to use anywhere else - you have been warned ."

Using Google's toolchain would be simpler, but it would also restrict the ebuild to x86-64, he said. " We'd lose arm64, ppc64, and the RISC-V overlay - better to just rip the band-aid off now as it doesn't materially make the situation better ". He added that he would be willing to review commits and offer advice to anyone who wanted to maintain the source ebuild, but he felt that at least three people would be needed. " History has shown that one maintainer isn't working, and we've had the release cycle shorten twice since I started maintaining the package in 2023; it was bad then, and it's worse now. "

The release cadence for Chromium is aggressive, to say the least. Chromium has multiple channels including stable, beta, and development. There is a new major branch of the browser roughly every two weeks via the stable channel, and a new version is shipped every week.

Getting Chromium

For now, it appears that Gentoo users will need to search elsewhere for a suitable Chromium ebuild, choose a different browser, or find a different distribution method to acquire Chromium. For example, Gentoo users could install the Flatpak from Flathub , but it is an unverified package built by a third party—not by the Chromium upstream project. The Flatpak is also of no help for users who are not on x86-64 or 64-bit Arm systems.

It is technically possible for Chromium users to build and compile the browser on their own using the upstream's instructions, but it does not appear that the Chromium project cares a great deal about making that easy to do. It does have a page on getting the code but the links to instructions for Linux (as well as macOS and Windows) return a 503 error. Searching for instructions turns up this page , though it is undated so it is unclear if it is the most up-to-date. Assuming the instructions are current and correct, building Chromium using those directions is quite an ordeal. Users can download pre-built binaries from the project, but only for x86-64 systems.

Other distributions continue to supply Chromium packages, of course, though their approach does not provide the same flexibility as Gentoo's. Fedora's package (which is described as " A WebKit (Blink) powered web browser that Google doesn't want you to use ") is at version 154.0.8037.92—one version behind the upstream (154.0.8037.97) release that was published on October 1. Likewise, the package in Debian's security repository is the same version (currently) as Fedora's and the information page shows that there are at least 11 known CVEs addressed in 154.0.8037.97. Keeping up with Chromium is not an easy task.

Perhaps Gentoo will find a team that is willing and able to take on the burden of maintaining the Chromium ebuild up to the standards that Gentoo users expect. Whatever happens, it seems unlikely that Chromium will become any easier to maintain in the interim.



Atlassian warns of critical file-access flaw in Jira, Confluence

Bleeping Computer
www.bleepingcomputer.com
2026-10-06 13:34:59
Atlassian is warning customers of a critical vulnerability, tracked as CVE-2026-21589, that can be exploited for arbitrary file-access in multiple self-hosted Data Center products, including Confluence, Jira, and Bitbucket. [...]...
Original Article

Atlassian warns of critical file-access flaw in Jira, Confluence

Atlassian is warning customers of a critical vulnerability, tracked as CVE-2026-21589, that can be exploited for arbitrary file-access in multiple self-hosted Data Center products, including Confluence, Jira, and Bitbucket.

The security issue allows an unauthenticated attacker to access specific files within an affected application's web root directory. However, exploitation requires knowing the exact name of the file and path.

“This Arbitrary File Access vulnerability allows an unauthenticated attacker to access specific files within the web application root directory in affected versions,” reads the security advisory .

“Exploitation requires prior knowledge of the target file's exact name and path; this vulnerability does not allow attackers to enumerate or list directory contents,” Atlassian says.

CVE-2026-21589 affects all product versions released before the releases listed below, which address the vulnerability:

  • Bitbucket Data Center: 9.4.26, 10.2.8, 10.5.1
  • Confluence Data Center: 9.2.26, 10.2.19
  • Jira Service Management Data Center: 5.12.40, 10.3.26, 11.3.12
  • Jira Software Data Center: 9.12.40, 10.3.26, 11.3.12
  • Bamboo Data Center: 10.2.24, 12.1.12
  • Crowd Data Center: 6.3.7, 7.0.3, 7.1.7, 7.2.4
  • Crucible: 4.9.15
  • Fisheye: 4.9.15

Atlassian urges system administrators who manage self-hosted instances to apply the security updates immediately. Cloud customers need to take no action, as the vendor has automatically patched the products.

If immediate patching is not possible, the company recommends restricting external network access, including for internet-facing instances that require user authentication.

Temporary mitigations include adding a web application firewall (WAF) or proxy rule blocking specified traversal patterns across all affected products, Tomcat RewriteValve rules for Confluence, JSM, Jira, Bamboo, and Crowd, or a URL rewrite rule for Bitbucket.

Atlassian's advisory offers step-by-step instructions and configuration details to implement the recommended temporary mitigations.

The changes must cover every cluster node, including Bitbucket mirrors and mirror farm nodes.

Atlassian said it currently has no evidence that CVE-2026-21589 is being exploited in attacks, but urges administrators to review access logs for the traversal patterns described in the bulletin.

The vendor says it cannot determine whether individual customer instances have been compromised, urging customers using self-hosted instances to engage with their local security team.

article image

Build your security blueprint for AI-powered attacks

Join Mikko Hyppönen and security leaders from the NFL, CHANEL, and Atlassian for a two-hour digital summit on what AI-speed attacks change, what defenders should stop doing, and how to validate, decide, fix, and re-validate at machine speed.

Save your seat

Sibling Rivalry, Scrabble Games, and Porn Conventions Make This Filmmaker a 'Smart Girl'

hellgate
hellgatenyc.com
2026-10-06 13:25:10
We talked to writer-director-star Hannah Crane about her latest "life goal" comedy documentary, in which she writes a romance novel and makes a bunch of friends along the way....
Original Article
Sibling Rivalry, Scrabble Games, and Porn Conventions Make This Filmmaker a 'Smart Girl'
Writing partners Hannah Crane (left) and Barbara (Screenshot from "Smart Girl")

Cultural Capital

Scott's Picks:

Give us your email to read the full story

Sign up now for our free newsletters.

Sign up

Great! You’ve successfully signed up.

Welcome back! You've successfully signed in.

You've successfully subscribed to Hell Gate.

Your link has expired.

Success! Check your email for magic link to sign-in.

Success! Your billing info has been updated.

Your billing was not updated.

The U.S. Military Struggles to Protect Its Bases Amid Failing Iran War

Intercept
theintercept.com
2026-10-06 13:25:09
From Jordan to Kuwait to the United Kingdom, U.S. outposts are increasingly vulnerable to attack. The post The U.S. Military Struggles to Protect Its Bases Amid Failing Iran War appeared first on The Intercept....
Original Article

The U.S. military is struggling to protect its overseas bases amid a conflict with Iran that has spun out of self-styled War Secretary Pete Hegseth’s control.

After a string of damaging attacks on bases throughout the Middle East, U.S. long-range bombers stationed in the United Kingdom fled back to their American home bases on Sunday amid fears of an impending Iranian strike.

A U.S. official described the threat as a “sophisticated plot to attack” Royal Air Force Fairford air base with “multiple drones” that could have turned into a military and public relations nightmare. The U.S. official suggested that Iran’s Islamic Revolutionary Guard Corps may have recruited U.K. nationals to play roles in the plot, which so unnerved the Pentagon that they took the extraordinary step of removing 12 B-1 Bombers from the base that has played a key role in the U.S. war. A second official called it a “complex plan” and part of a “hybrid strategy” of asymmetric warfare by Iran that forced the Pentagon to take drastic measures.

“There were some threats,” President Donald Trump admitted on Monday when asked about the bombers, adding that leaving the aircraft would only tempt an attack.

The inability to safeguard U.S. aircraft at a base northwest of London comes at a time of mounting attacks on U.S. outposts around the world.

Complex Iranian drone and ballistic missile strikes have killed and injured U.S. troops, placed many more in harm’s way, and damaged or destroyed numerous outposts across the Middle East. Some crucial bases have sustained so much damage that U.S. operations have been radically disrupted and, according to the second official, “base posture in the region may never recover.”

The strikes have also targeted joint facilities operated by U.S. allies. In March, Iran launched a drone attack on a British air base in Akrotiri, Cyprus, causing damage to a facility the U.S. military has long relied on. Iran also targeted a joint U.K.–U.S. base on the Indian Ocean island of Diego Garcia that has become increasingly central to the U.S. war effort.

What Trump called a “little excursion” has become a festering wound for the U.S. military. Despite months of claims by Trump and Hegseth that Iran’s military was annihilated , Iran has attacked more than 15 bases across the Middle East. The official U.S. casualty toll stands at 19 dead and close to 900 wounded, most of the dead occurring since July, after the Trump administration claimed the war had ended . The real casualty toll is higher and has been consistently obscured and manipulated through what one defense official called a “ cover-up .”

The first U.S. official said Iran is engaged in a “go for broke” strategy, banking that it can absorb economic ruin longer than the U.S. can sustain a failing war and an expensive and difficult Naval blockade and escort program in the Strait of Hormuz, while dealing with global economic upheaval from stubbornly high fuel prices that have not shrunk to pre-war levels despite the oil flow from other Gulf Nations returning to normal.

The withdrawal of the bombers from Fairford came just days after the British government arrested six suspects in connection with a suspected attack on that same base. U.K. Prime Minister Andy Burnham said there were “strong indications” that Iran played a role in the incident. The men were later released on bail, prompting criticism by Trump.

The IRGC warned in July that “any base used to launch attacks on Iranian territory is a legitimate target for our forces.” On Monday, the commander-in-chief of Iran’s army, Maj. Gen. Amir Hatami, “stressed the need for any creative act capable of disrupting the enemy’s calculations,” according to a report by Iran’s Islamic Republic News Agency. The U.S. officials referenced additional threats against other bases in Europe, including Ramstein Air Base in Germany.

This isn’t the first time the U.S. has scrambled to protect its assets from a supposed Iranian threat. In July, Trump himself snuck into a catering truck and fled Turkey on a military aircraft due to U.S. fears of a possible Iranian attack on Air Force One . Trump had earlier said he was Iran’s “No. 1 target” during the NATO summit in Ankara.

Unlike their commander-in-chief, U.S. troops have been unable to flee their bases and ships when Iran has overwhelmed U.S. air defenses during attacks in the Middle East. A recent Pentagon inspector general report revealed Iranian strikes “damaged and destroyed hundreds of buildings and structures at U.S. bases in Kuwait, Bahrain, Qatar, UAE, Saudi Arabia, Iraq, Oman, and Jordan during the conflict.” Air Force personnel, alone, endured more than 1,200 “ alarm reds ” — the official code for imminent attack by missiles, aircraft, or ground forces — during just 38 days earlier this year, according to Air Force chief of staff Gen. Ken Wilsbach.

A March 1 drone attack on the Port of Shuaiba in Kuwait killed six U.S. troops , the deadliest known Iranian strike on U.S. personnel during the war; dozens more were wounded. In July, an Iranian attack on Muwaffaq Salti Air Base in Jordan killed three soldiers from air defense units although they were “not actively engaged in anti-aircraft activity” during the attacks, according to a “defense official” who spoke on background because they were not authorized to disclose the information. Multiple U.S. military aircraft were also damaged in the attack, including an A-10 Thunderbolt and eight F-15 fighters. These were among dozens of U.S. aircraft so far destroyed or damaged in the war, including four F-15 fighter aircraft, 12 KC-135 refueling aircraft, and as many as 30 MQ-9 Reaper drones, according to the inspector general report.

The U.S. and its allies burned through more than 11,000 munitions , including large amounts of defensive missiles, in the first 16 days of the war alone, according to a March analysis by the Royal United Services Institute. About 65 percent of Patriot interceptors were expended between February and July, leaving fewer than 830 remaining from a pre-war total of 2,330, according to an analysis by the Center for Strategic and International Studies. The number of THAAD interceptors dropped from more than 450 to less than 270.

Adm. Brad Cooper, the chief of U.S. Central Command, downplayed the risks of depleted stockpiles of such missiles last month, telling CBS’s “60 Minutes” that he wasn’t worried for the safety of U.S. troops. “I’m not concerned at all,” he replied when asked about defending against Iranian attacks. CENTCOM did not reply when asked if Cooper would resign if U.S. troops were killed or wounded in future attacks.

An Iranian missile and drone attack at the start of the war destroyed Navy facilities in Manama, Bahrain, the most critical U.S. logistics hub in the region. The Pentagon claimed for months that the strikes did not significantly impact military operations but acting Navy Secretary Hung Cao admitted last month that “they blew the hell out of Bahrain.” A Wall Street Journal investigation previously found that rebuilding the U.S. Navy’s Fifth Fleet headquarters, barracks, and communication towers at Naval Support Activity Bahrain may cost more than $400 million, which the first U.S. official told The Intercept could be an underestimate if it’s ever attempted. Naval Forces Central Command referred questions about the cost estimate to CENTCOM, which did not respond .

With the home of the Fifth Fleet knocked out and other ports in the region within range of Iranian missiles and drones, the Navy found itself facing the daunting challenge of maintaining a flotilla of more than 24 ships — including two massive aircraft carriers and numerous guided-missile destroyers and cruisers, attack submarines, or supply ships deployed at any one time. This means Navy ships need to haul food and fuel more than 2,000 miles from the base on Diego Garcia. The inspector general report noted that the shift in “naval bases and support ports posed significant challenges,” pointing specifically to the “issue with shifting logistics support and infrastructure to Diego Garcia” and the challenges of “extended sea transit requiring 14- to 18-day logistics cycles.”

Sailors and Marines have been subjected to constant danger as a result. The USS Gerald R. Ford Carrier Strike Group completed a record-breaking 11-month deployment that also included the invasion of Venezuela. (Under normal conditions, a carrier would visit port about once a month to allow its crew to rest.) The ship was, according to a Presidential Unit Citation , under “persistent threat from enemy missiles and one-way attack drones” during its deployment to the Middle East.

More recently, the IRGC targeted another aircraft carrier and guided-missile destroyer with ballistic missiles, according to Central Command . And Marines were injured in an Iranian attack on a commercial vessel in the Strait of Hormuz last month; the two U.S. officials said those Marines potentially suffered brain trauma, among other injuries.

Amid the floundering war with Iran, U.S. forces withdrew from neighboring Iraq last week, ending two-plus decades of U.S. military occupation and operations that unleashed waves of bloodshed and civilian suffering. “The expulsion of America from Iraq was the result of … the country’s steadfastness in its independence,” the IRGC announced, in a missive touting the power of national resistance. Politicians in Baghdad also celebrated the withdrawal, with Iraqi Prime Minister Ali al-Zaidi officially declaring October 1 a “Day of Sovereignty and Independence.”

Despite the Trump administration’s claims, the Iran war continues to expand. Last month, Yemen’s Iran-backed Houthi forces seized virtually all of Yemen’s Red Sea coast, including the shoreline along the crucial shipping lanes of the Bab el-Mandeb Strait, which is the southern entrance to the Red Sea. On Tuesday, the Houthis claimed to have killed or wounded more than 200 Saudi-backed fighters seeking to retake territory.

Adobe Creative Suite Cleanroom Ported to Rust

Hacker News
github.com
2026-10-06 13:20:00
Comments...
Original Article

ArtCraft

Image editing; an open-source, clean-room reimplementation of Adobe Photoshop, rebuilt in pure Rust.
Layers, masks, adjustment layers, layer styles, type, vectors, brushes and real PSD files,
in a native app written entirely in Rust. Open source, offline, and yours.

100% Rust macOS · Windows · Linux · FreeBSD · Web License: MIT OR Apache-2.0 Status: early alpha

Join the ArtCraft community on Discord

PhotoCraft on getartcraft.com · ArtCraft · All Crafting Apps

PhotoCraft editing Hokusai's The Great Wave: a caption card with a drop shadow, Title and Credit type layers, Vibrance and Curves adjustment layers, and the Curves editor drawn over the image's histogram
A caption card with a drop shadow, live type, and Vibrance and Curves adjustment layers, with the Curves editor open.
The Great Wave off Kanagawa , Katsushika Hokusai, c. 1831

Note

ArtCraft is a community of artists from all walks of life. Digital, generative, music, games — if you make things, you're one of us. Come say hi on Discord .

Features · Everything in the box · PSD · Agents · Under the hood · Get started · Crafting Apps · Discord


🎛️ Familiar by design

The menus, shortcuts, panels and tools are where your hands expect them, from ⌘J to ⇧⌘D. If you know Photoshop, you already know PhotoCraft.

⚡ Native and fast

A GPU compositor on wgpu (Metal, Vulkan, DX12, WebGPU), copy-on-write tiles and multithreaded filters. No Electron, no web view, no waiting.

🗂️ Real PSD files

Open, edit and save layered Photoshop documents. Re-saving keeps the render of 307 of the 309 psd-tools test files.

🤖 Agent-ready

Every action is a command, so you can drive the same engine from the UI, the CLI, a JSON control channel or an MCP server.

Features

Every screenshot here is the real app at work on public-domain art, rendered offscreen through its control channel.

Monet's Impression, Sunrise with Levels and Vibrance adjustment layers; the Levels editor and the Histogram panel with mean, standard deviation and median are open on the right
Levels and Vibrance adjustment layers, with the live Histogram panel.
Impression, Sunrise , Claude Monet, 1872

Edit without regret

Adjustment layers keep every edit live. Stack Levels, Curves, Vibrance, Hue/Saturation and a dozen more, mask them to an area, reorder them, or turn them off, and your original pixels never change.

16 adjustment layers that also apply directly to pixels, including Curves with per-channel editing, Levels with a live histogram, Black & White, Channel Mixer, Gradient Map, Photo Filter, Selective Color and Color Lookup (.cube, .3dl, .look). Plus Shadows/Highlights, Replace Color, Match Color, HDR Toning, Desaturate and Equalize.

The Layer Style dialog editing an Outer Glow on the EARTHRISE type layer, with Stroke also enabled, over the Apollo 8 Earthrise photograph
Outer Glow and Stroke on a live type layer, in the Layer Style dialog.
Earthrise , William Anders / NASA, 1968

Styles that sell the shot

Drop Shadow, Inner Shadow, Outer and Inner Glow, Bevel & Emboss, Satin, Stroke, and Color, Gradient and Pattern Overlay, live on any layer, including type. Patterns come from a library (built-ins, Edit › Define Pattern, .pat import/export) and PSD Patt blocks.

Copy and paste styles between layers, hide all effects at once, and open styles straight from your PSDs, rendered to match Photoshop.

Vermeer's Girl with a Pearl Earring: an elliptical selection around the face, and a Hue/Saturation layer with an elliptical mask that turns everything outside the face gray
An elliptical selection becomes the mask of a Hue/Saturation layer, so only the face keeps its color.
Girl with a Pearl Earring , Johannes Vermeer, c. 1665

Selections that understand your image

Marquees, lassos and the Magic Wand for precision; Quick Selection, Object Selection and Select Subject when you want the computer to do the tracing; Select and Mask to refine hair-fine edges.

Feather, expand, contract, smooth, grow, reselect. Turn any selection into a layer mask, a vector path or a shape. Smart selection runs on your machine, with no cloud and no account.

Bierstadt's Among the Sierra Nevada with the headline SIERRA NEVADA being edited on the canvas in Georgia, a byline, and a paragraph caption; the Character and Paragraph controls are open
A headline edited in place, with a byline and a paragraph of body text.
Among the Sierra Nevada, California , Albert Bierstadt, 1868

Type that sets beautifully

Point and paragraph text, edited right on the canvas, with full Character and Paragraph controls: font, weight, size, leading, tracking, alignment and colour.

Type layers stay editable, take layer styles, and round-trip through PSD.

A lotus badge built from shape layers (Lotus, Water, Sun, Badge and Dotted Ring) over Monet's Water Lilies, with the lotus path's anchor points selected
A badge made of shape layers: a gradient-filled lotus, a star and a dotted ring.
Water Lilies , Claude Monet, 1906

Pixel-perfect vectors

Rectangle, Ellipse, Triangle, Polygon, Line and the Pen tool, with resolution-independent shape layers, gradient fills, and dashed, aligned strokes.

Combine shapes (unite, subtract, intersect, exclude), keep paths in the Paths panel, use them as vector masks, or stroke and fill them. 116 of 116 shape layers in our PSD corpus match Photoshop's pixels.

The Twirl filter dialog at a 320 degree angle, previewing live inside an elliptical selection on Van Gogh's The Starry Night
Twirl previews live on the canvas, only inside the selection.
The Starry Night , Vincent van Gogh, 1889

See it before you commit

Every filter dialog previews live on the canvas, through your selection. Blurs (Gaussian, Box, Motion, Radial, Surface, Smart, Lens, Shape, and the Blur Gallery: Tilt-Shift, Iris, Field, Spin, Path), sharpening, Reduce Noise, distortions (Twirl, Wave, Ripple, Displace, Shear, Zig Zag…), Pixelate, Stylize (Oil Paint, Wind, Extrude…), Render (Clouds, Fibers, Lens Flare, Lighting Effects) and more.

Run filters on a smart object and they stay editable: change, hide, reorder or mask them at any time.

Large-radius blurs use running-sum box passes across all cores: a radius-180 Gaussian on 3.6 MP takes under a second.

Free Transform handles around a rotated copy of Ansel Adams' The Tetons and the Snake River, with the History panel listing Open, Duplicate Layer, Free Transform and other steps
Free Transform on a rotated print, with every step listed in History.
The Tetons and the Snake River , Ansel Adams, 1942

Shape it any way you like

Free Transform with scale, rotate, skew, distort and perspective; exact 90° and 180° rotations and flips; Transform Again. Layers, type, shapes, masks and selections all transform together.

Full history, Toggle Last State and the History Brush mean every step can be undone, even one brush stroke at a time.

The Export As dialog in the light theme over Klimt's The Kiss: JPG at quality 90, scaled to 50 percent, with a preview and an estimated size of about 684K
Export As in the light theme, with a preview and a file-size estimate.
The Kiss , Gustav Klimt, 1907–1908

Ship it anywhere

Export As with format, quality, transparency and scale, plus a preview and an instant file-size estimate. Quick Export to PNG in one click.

Choose a dark Pro theme, the airy Studio themes, or a Classic look.

Everything in the box

🧰 34 tools

Move · Rectangular and Elliptical Marquee · Lasso · Polygonal Lasso · Magic Wand · Quick Selection · Object Selection · Crop · Eyedropper · Brush · Pencil · Mixer Brush · Color Replacement · Eraser · Clone Stamp · Healing Brush · Spot Healing · History Brush · Gradient · Paint Bucket · Blur · Sharpen · Smudge · Dodge · Burn · Sponge · Pen · Path Selection · Type · five Shape tools · Hand · Zoom

🖌️ A real brush engine

Shape Dynamics, Scattering, Texture, Dual Brush, Color Dynamics, Transfer, Brush Pose, Wet Edges, Build-up and Smoothing (including Pulled String), driven by pen pressure, tilt, rotation and direction. Brush presets, Define Brush from Selection, and deterministic, replayable strokes.

🗃️ Layers, done properly

Groups, clipping masks, pixel and vector masks, fill layers (solid, gradient and pattern), adjustment layers, live smart objects with smart filters and lossless transforms and warps, multi-layer selection with align, distribute and link, alpha channels and Quick Mask, 27 blend modes, opacity and fill, locks, colour labels, layer filters, merge, flatten, rasterize, Layer via Copy/Cut, Paste Into.

🎨 Any colour, any depth

RGB, Grayscale, CMYK and Lab documents at 8, 16 and 32 bits per channel. Bit depth and colour model are runtime data, so every tool works at every depth.

Real ICC colour management in pure Rust: embedded profiles, Assign and Convert to Profile with all four rendering intents and black point compensation, soft proofing (⌘Y) and Gamut Warning (⇧⌘Y) on the GPU.

🗂️ Formats

PSD and PSB, plus PNG, JPEG, TIFF, WebP, GIF, BMP, TGA, ICO, QOI, PNM, OpenEXR, Radiance HDR and AVIF, with symmetric read and write at 8, 16 and 32 bits, and the native .pcraft format.

🪄 The everyday essentials

Auto Tone, Contrast and Color · Equalize · Image and Canvas Size · Crop and Trim · Reveal All · Edit › Fill and Stroke · Copy Merged · Paste in Place · guides, rulers, grid and snapping · Actions record and replay · a command palette (⌘K).

PSD without compromise

PhotoCraft's PSD support is a standalone crate written from Adobe's public specification and tested against a corpus of real-world files.

  • Faithful round trips: opening and re-saving a document renders the same for 307 of the 309 files in the psd-tools test set and 169 of 170 in our mixed ag-psd/psd-tools set ( crates/io/tests/corpus.rs ; fetch the psd-tools set with cargo xtask corpus --psd-tools ), and anything we don't model yet (raw blocks, descriptors, extras) is carried over instead of being dropped. A re-saved file is not byte-identical to its source: PhotoCraft rewrites image resources, layer records and the composite. Only the standalone photocraft-psd crate, parsing and writing a file without the document model, reproduces every parseable corpus file byte for byte ( crates/psd/tests/corpus.rs ).
  • Pixels that match: a composite oracle compares our render with Photoshop's own merged image, covering gradient interpolation (Classic, Perceptual and Linear), layer effects, shape strokes, clipping and fill opacity.
  • Large documents: PSB, 16 and 32-bit files, and CMYK and Lab documents open natively.

Built for agents

Every menu item, tool and dialog runs a command from one registry of 500+ commands. The UI, the CLI, the JSON control channel and the MCP server all call the same commands, so anything you can click, a script or an AI agent can do too. 

# Headless: open, edit, save
photocraft-cli run wave.psd \
  --cmd filter.sharpen.smartSharpen     --params '{"amount":80}' \
  --cmd layer.newAdjustmentLayer.curves --params '{"points":[[0,0],[64,48],[192,212],[255,255]]}' \
  --out wave-final.png

# Apply one action list to a folder of images
photocraft-cli batch --actions grade.json --in ./raw --out ./graded

# Let an agent drive it over MCP (headless, or bridged to the running app)
photocraft-cli mcp

The desktop app also offers an authenticated, loopback-only control channel ( photocraft --control ) for inspecting UI state, driving tools with pointer events, and taking offscreen screenshots. Every image in this README was rendered that way. See docs/control-protocol.md .

Under the hood

  • Engine first: a pure-data document model and a command engine, with a thin egui UI on top. Layering is enforced at build time across 24 crates.
  • Two compositors: a CPU compositor serves as the reference oracle, and a wgpu compositor puts the canvas on the GPU. They are tested against each other.
  • Copy-on-write tiles: 256² sparse tiles make undo cheap and huge canvases light, and effect maps are cached per layer state.
  • Runs in the browser: the whole engine and UI compile to WebAssembly.
  • Clean-room: implemented from public specs and observed behaviour only. No proprietary code, shaders or assets.
  • Tested: more than 1,700 tests, including PSD round trips, synthetic generators, compositor oracles and multi-depth checks.

Get started 

git clone https://github.com/storytold/photocraft
cd photocraft
cargo run --release -p photocraft -- image.psd   # the desktop app
cargo test --workspace                           # the test suite

New contributors and AI agents: start with AGENTS.md , then docs/ .

Installers for macOS, Windows, Linux, FreeBSD and the web are attached to each GitHub release . On Linux you can pick an AppImage, a .deb , an .rpm , a tarball or a Flatpak bundle. The bundle needs the freedesktop runtime from Flathub , which flatpak offers to install along with it: 

flatpak install --user photocraft-<version>-linux-x86_64.flatpak   # or -linux-aarch64
flatpak run ai.storyteller.photocraft

On macOS, the command-line tool comes as photocraft-cli-<version>-macos-universal.zip . The binary is signed with the same Developer ID as the app and notarized by Apple. A bare binary can't carry a stapled notarization ticket the way the DMG does, so the first time you run it macOS checks the notarization online. You can confirm it yourself: 

ditto -x -k photocraft-cli-<version>-macos-universal.zip .
spctl --assess --type install -vv photocraft-cli-<version>-macos-universal/photocraft-cli
# ... accepted, source=Notarized Developer ID

On FreeBSD 14 (x86_64), the release has a tarball laid out like /usr/local . Install the runtime libraries, then unpack it there: 

pkg install libxkbcommon wayland libX11 libXcursor libXrandr libXi libxcb mesa-libs vulkan-loader gtk3 fontconfig freetype2 alsa-lib
tar -xzf photocraft-<version>-freebsd-x86_64.tar.gz --strip-components 1 -C /usr/local
photocraft

Maintainers: docs/releasing.md explains how releases are built, signed and published.

Important

Status: PhotoCraft is in early alpha, and we want to be straight about where it stands: much of Photoshop's feature surface exists in some form, but it is not yet a Photoshop replacement for daily professional work . The biggest gaps are AI/generative features, about twenty missing tools, depth in typography and pro workflows, and plug-in compatibility. Every Photoshop menu item is wired to a command ( docs/parity.md ), but that measures wiring, not behaviour. The honest, dimension-by-dimension picture and where we're going next are in the roadmap's parity assessment . Expect rough edges, and please file issues (include your OS, document size, layer count and a screenshot). You can also tell us what broke on Discord .

Documentation

Developer, architecture, automation, format, and security documentation is maintained in the PhotoCraft documentation book .

Security

Security architecture, threat modeling, parser hardening, fuzzing, and vulnerability reporting are covered in the security documentation and the repository security policy .

Test corpora

PhotoCraft is tested against real files: our own Photoshop-authored oracle PSDs in photocraft-corpus plus the psd-tools, ag-psd and PngSuite sets, pinned and sha256-verified. Fetch them with cargo xtask corpus --all and run the tests with cargo xtask test-corpus (details in docs/development.md ).

The Crafting Apps

PhotoCraft is one of the Crafting Apps : free, open-source creative tools from the ArtCraft team, each written from scratch in Rust and each able to stand on its own.

App What it's for Code Learn more
PhotoCraft Image editing: layers, masks, type and real PSD files · you are here GitHub Website
VectorCraft Vector illustration GitHub Website
FilmCraft Video editing, color and sound GitHub Website
LightCraft Photo library and raw development GitHub Website
PrintCraft Reading, organizing and protecting PDFs GitHub Website
EffectCraft Motion graphics and visual effects GitHub Website
DesignCraft Page layout and publishing GitHub Website

And ArtCraft itself, our AI image and video studio for artists who want real control.

The Crafting Apps share the same conventions: clean-room and pure Rust, native on macOS, Windows and Linux, in the browser via WebAssembly, and fully drivable by agents.

Join the ArtCraft community on Discord

Come make things with us

Our Discord is where artists of every kind hang out: people who paint, shoot, draw, cut film, set type, and people still figuring out what they like to make. Share what you're working on, ask for help, tell us what's broken, or tell us what you wish these tools could do. Whatever your medium and however long you've been at it, you're welcome here.

discord.gg/artcraft · getartcraft.com · The Crafting Apps · PhotoCraft

License and credits

PhotoCraft is dual-licensed under MIT or Apache-2.0 , at your option. Copyright (c) 2026 ArtCraft Team and the PhotoCraft contributors. Required notices are in NOTICE .

Bundled fonts, icons, images and other assets keep their own open licenses; each one is listed with its author, source and license in ATTRIBUTION.md .

Every artwork shown is in the public domain (Wikimedia Commons, NASA, U.S. National Archives); sources are listed in docs/images/SOURCES.md .

The ArtCraft name, wordmark and logos in docs/brand/ are trademarks of the ArtCraft Team and are not covered by this license. They may be used only unmodified, and only as part of this repository and PhotoCraft, under docs/brand/LICENSE-brand.txt . Forks and modified versions must remove them.

Adobe, Photoshop, Illustrator, Premiere Pro, Lightroom, Acrobat, After Effects and InDesign are trademarks or registered trademarks of Adobe Inc. in the United States and/or other countries. PhotoCraft is an independent, open-source project and is not affiliated with, sponsored by or endorsed by Adobe Inc.; these names are used only to describe the workflows it is compatible with.

ArtCraft
Made by the ArtCraft team and community.

ArtCraft

Brut, the Brutal Router for Unix Tools

Lobsters
brut.sh
2026-10-06 13:17:23
Comments...
Original Article
#!/bin/sh set -eu # Brutal Router for Unix Tools # https://brut.sh/60d2b6 # Copyright (c) Sam Stephenson # # Permission is hereby granted, free of charge, to any person obtaining # a copy of this software and associated documentation files (the # “Software”), to deal in the Software without restriction, including # without limitation the rights to use, copy, modify, merge, publish, # distribute, sublicense, and/or sell copies of the Software, and to # permit persons to whom the Software is furnished to do so. # # THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY # CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, # TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE # SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Get our bearings OLDENV="$(unset OLDENV; export -p)" SELF="$(realpath "$0")" NAME="${0##*/}" PROGRAM_NAME="${SELF##*/}" ROOT="${SELF%/*/*}" # Set up the environment LIB="$ROOT/lib/$NAME" LIBEXEC="$ROOT/libexec" SHARE="$ROOT/share/$PROGRAM_NAME" VENDOR="$SHARE/vendor" PATH="$LIB:$LIBEXEC:$VENDOR:$PATH" export SELF USAGE=" " unset VERSION # --- # Define shared helpers error() { printf "%s: " "$(invocation "${1##*/}")" shift printf "%s\n" "$@" return 1 } >&2 usage() { [ $# -gt 0 ] || set -- "$NAME" "${USAGE:-}" "commands: $(flatten commands)" printf "usage: %s " "$(invocation "${1##*/}")" shift printf "%s\n" "$@" return 1 } >&2 # --- awka() { awk "BEGIN { ARGC = 1; a = ARGV[1] } $2" "$1" } searchpath() { { IFS=: dirs= set -- \) -o -prune "$@" for dir in $PATH; do [ -d "$dir" ] || continue set -- -o -path "$dir" "$@" dirs="$dirs:$dir" done set -- ${dirs#:} \( \! -name "*" "$@" unset IFS } 2>/dev/null find -H "$@" 2>/dev/null \ \( -perm -001 -o -perm -010 -o -perm -100 \) \ \( -type f -o -type l \) \ \! -name "?*--?*" \ -print || true } namespaces() { searchpath -type l -name "${1:-$NAME}-*" \ -exec sh -c '[ "$0" = "$(realpath "$1")" ] 2>/dev/null' "$SELF" {} \; \ | awk -F / '{ print $NF }' | sort -u } executables() { for ns in $(namespaces "$NAME"); do set -- "$@" \! -name "$ns-*" done 2>/dev/null searchpath -name "$NAME-*" "$@" \ | awk -F / '!a[$NF]++' } commands() { executables \ | awka ${#NAME} 'BEGIN { FS = "/" } { print substr($NF, a+2) }' | sort } resolve() { case "${1:-}" in *-* ) executables ;; ?* ) searchpath -name "$NAME-*" esac \ | awka "$NAME-${1:-}" \ 'BEGIN { FS = "/" } $NF == a { print; s = 1; exit } END { exit !s }' } invocation() { { printf "%s\n" "$PROGRAM_NAME"; namespaces "$PROGRAM_NAME"; } | sort -r \ | awka "${1:-$NAME}" \ 'index(a, $0 "-") == 1 { a = $0 " " substr(a, length($0) + 2) } END { print a }' } flatten() { "$@" | { tr "\n" " "; printf "\n"; } } version() { printf "%s %s\n" "$PROGRAM_NAME" "$VERSION" } # --- # Load _init.sh, if it exists if [ -r "$LIB"/_init.sh ]; then . "$LIB"/_init.sh fi # Rewrite `--version` to `--- version` if [ $# -eq 1 ] && [ -n "${VERSION:-}" ] && [ "$1" = "--version" ]; then set -- --- version fi # When invoked as `--- ...`, run that exact command and exit if [ "${1:-}" = "---" ]; then [ $# -gt 1 ] && shift || usage "$@" exit fi # Scan the argument list to find the command name COMMAND= args= index=0 for arg; do index=$((index+1)) case "$arg" in -* ) ;; ?*--?* | "" ) [ -n "$COMMAND" ] || usage ;; * ) if [ -z "$COMMAND" ]; then COMMAND="$arg" continue fi esac args="$args"' "${'"$index"'}"' done # Match the command name to an executable in the current namespace; # if one exists, exec it if executable="$(resolve "$COMMAND")"; then eval "set --$args" exec "$executable" "$@" fi # No matching executable; load _unhandled.sh, if it exists if [ -r "$LIB"/_unhandled.sh ]; then . "$LIB"/_unhandled.sh fi # No matching command or no command specified if [ -n "$COMMAND" ]; then error "$0" "$COMMAND: command not found" || exit 127 fi usage

Nano Banana 2.1

Hacker News
twitter.com
2026-10-06 13:13:07
Comments...
Original Article

Log in or sign up for X

See what’s happening and join the conversation

or

Log in with username or email

Relevant people

Avatar

The fastest path from prompt to prototype to production with Gemini

Trending now

California Closed the Montana License Plate Loophole

Hacker News
www.thedrive.com
2026-10-06 13:00:59
Comments...
Original Article

Vanity License Plate - Montana

Joe Sohm/Visions of America/Universal Images Group via Getty Images

It’s official: California has closed the Montana license plate loophole . After more than seven months of legislative shuffling, CA State Bill 1406 (“Sales and Use Tax Law: vehicles: shell companies”) became law on Sept. 30, 2026, making it much harder to get away with the shell-company shenanigans rich Californians have exploited for decades.

Existing California law says any vehicle, vessel, or aircraft a California resident “shipped or brought into the state” is subject to taxation under the state’s Sales and Use Tax Law, whether it is registered there or not. This same rule applied to vehicles registered to a business, but because businesses aren’t people, the law set different standards for establishing its “California-ness.”

Essentially, if more than half of your business was held outside California, you could keep your vehicle registered to that out-of-state business without paying Use Tax in California. This seems like a reasonable qualifier on paper. Say you live in Nevada and your vehicle is registered to your business based there; if you operate a branch office on the California side of Lake Tahoe, for instance, you could legally keep your car there on Nevada plates so long as the bulk of the business was based in Nevada. This is a gross oversimplification, but you get the basic idea.

Just an example image of a Porsche 918. Porsche, edited by Andrew Collins.
Porsche edited by TD

This arrangement seemed to work just fine for everyone involved until a bunch of people started registering fake LLCs in Montana (though not exclusively), which doesn’t charge sales tax on vehicle purchases (or require owners to inspect/smog their cars). Under the old law, only companies and limited liability companies were subject to the California-ness test; the new law expands that definition to include partnerships, limited partnerships, and limited liability partnerships. And then there’s the whammy (emphasis mine).

“The bill would also provide that, for purposes of the above-described presumption, a shell company, as defined, is a resident of this state if any shareholder, partner, member, or beneficial owner is a resident of this state ,” the bill summary says.

In other words, the old 50% test is out. If anyone running the business is a California resident (by the updated definition outlined above), then the whole business is a California resident, and any assets (including Lamborghinis and Bugattis) held in the state are subject to California taxation. And just in case you need it spelled out, that means you’re on the hook for that tax liability as a fellow partner/member:

“The bill would hold any officer, manager, partner, beneficial owner, or member of a shell company personally liable for any unpaid taxes, and any interest and penalties on those taxes, the nonpayment of which may constitute a crime , due on the purchase of a vehicle, vessel, or aircraft.”

Better make sure your paperwork is in order. The tax man’s a-comin’.

Got a story tip? Let us know at tips@thedrive.com

Former German spy chief arrested for attempted treason

Hacker News
www.reuters.com
2026-10-06 12:51:50
Comments...
Original Article

Please enable JS and disable any ad blocker

ASOS confirms data breach after “HACKED” in-app notifications

Bleeping Computer
www.bleepingcomputer.com
2026-10-06 12:33:54
UK fashion retailer ASOS confirmed a data breach Tuesday after hackers sent unauthorized push notifications through its mobile app while claiming to have stolen customer data from the company's Snowflake environment. [...]...
Original Article

ASOS

UK fashion retailer ASOS confirmed a data breach Tuesday after hackers sent unauthorized push notifications through its mobile app while claiming to have stolen customer data from the company's Snowflake environment.

ASOS is a large UK-based online fashion retailer that sells clothing, footwear, accessories, and beauty products to customers worldwide, including in the United States.

ASOS has confirmed that third-party platforms used to communicate with customers were accessed without authorization and says basic personal information, including names and contact details, may have been exposed.

The company is now displaying an in-app notice telling customers to disregard the unauthorized push alert and not to click or engage with the external third-party link it contained.

Warning about notification now shown in ASOS app
Warning about notification now shown in ASOS app

However, the company has not confirmed the threat actor's claim that its Snowflake environment was compromised or disclosed how many customers may be affected.

ASOS says it does not believe payment-card information or account passwords were impacted.

If you have any information regarding this incident or other undisclosed attacks, you can contact us confidentially via Signal at 646-961-3731 or at tips@bleepingcomputer.com.

Hackers abuse ASOS mobile app

The notifications began appearing at approximately 5:00 a.m. ET on Tuesday, with multiple BleepingComputer readers contacting us after receiving the alerts on their phones.

"ASOS HACKED," reads the notification seen by BleepingComputer.

"Dear Asos DPO and IT, we have fully compromised the Snowflake instance. Engage with us, or we will leak it."

"ASOS HACKED" notification sent via the official ASOS mobile app
"ASOS HACKED" notification sent via the official ASOS mobile app
Source: Reddit

Numerous other ASOS customers also reported receiving the same notification on Reddit, indicating that the message reached many, if not all, mobile app users.

The notification directs ASOS to a Telegram channel operated by a threat actor calling itself the "Xuanye group."

In messages posted to the channel Tuesday morning, the threat actor claimed the breach did not affect payment information.

However, the attackers later published a "FINAL STATEMENT," claiming that they stole customer information in the attack.

"The affected organisation's app is safe to use. The incident involves customer information, it is safe on our server, and it will not be touched for a designated period," reads the group's message.

"Considering the current situation regarding incident disclosure in the cyber security landscape, you can thank us for our generous clarity regarding this incident."

The group did not disclose what customer information was allegedly stolen, how many customers were impacted, or provide evidence showing that it had compromised ASOS's Snowflake environment.

BleepingComputer attempted to contact the threat actors about the breach, but the only contact point required payment. We did not continue as it is against our editorial guidelines to pay for information.

article image

Build your security blueprint for AI-powered attacks

Join Mikko Hyppönen and security leaders from the NFL, CHANEL, and Atlassian for a two-hour digital summit on what AI-speed attacks change, what defenders should stop doing, and how to validate, decide, fix, and re-validate at machine speed.

Save your seat

AI is now capable of developing its own inference hardware

Hacker News
github.com
2026-10-06 12:23:25
Comments...
Original Article

An open-source AI accelerator, developed by AI.

openTPU brings the lessons of auto-arch-tournament to AI accelerators. It asks two questions: how far can AI agents go at hardware design, and can they build the chip that runs their own inference?

otpu-chat on LFM2.5-230M with otpu-smi watching the card

otpu-chat running LFM2.5-230M on the FPGA card (left), with otpu-smi showing the card's utilization and DRAM bandwidth (right).

A place to learn

openTPU is also a learning project. The whole accelerator lives in one small monorepo that you can read end to end: the hardware design (SystemVerilog), the instruction set, a bit-exact simulator, a kernel language and its compiler, and the host software that drives a real PCIe card. If you want to understand how an AI accelerator works, from a matmul in Python down to the wires, this is a good place to start.

Results

The design runs ten modern models with their real weights on an Inspur YPCB-00338 card (Xilinx Kintex-7 xc7k480t, two DDR3 channels), and the card produces the same tokens as the simulator, bit for bit.

Model Weights Decode, device Decode, wall Prefill, device DRAM while decoding
LFM2.5-230M int8 59.0 tok/s 52.3 tok/s 295.6 tok/s 14.5 GB/s (85% of peak)
LFM2.5-230M 4-bit, int8 head 85.8 tok/s 82.1 tok/s 335.4 tok/s 14.1 GB/s (82%)
Qwen3-0.6B int8 21.6 tok/s 21.3 tok/s 92.1 tok/s 14.4 GB/s (84%)
Qwen3-0.6B 4-bit, int8 head 31.3 tok/s 30.7 tok/s 103.4 tok/s 13.9 GB/s (82%)
Qwen3.5-0.8B int8 17.6 tok/s 16.3 tok/s 61.4 tok/s 14.5 GB/s (85%)
Qwen3.5-0.8B 4-bit, int8 head 24.5 tok/s 23.3 tok/s 66.7 tok/s 14.1 GB/s (83%)
Gemma 4 E2B 4-bit, int8 head 10.57 tok/s 10.53 tok/s 32.1 tok/s 15.6 GB/s (92%)
Gemma 4 E2B 4-bit, 4-bit head 12.14 tok/s 12.09 tok/s 29.9 tok/s 15.5 GB/s (91%)
LFM2-2.6B int8 6.05 tok/s 6.03 tok/s 21.4 tok/s 16.1 GB/s (94%)
LFM2-2.6B 4-bit, int8 head 10.96 tok/s 10.93 tok/s 20.6 tok/s 15.8 GB/s (93%)
SmolLM3-3B int8 5.00 tok/s 4.99 tok/s 21.1 tok/s 16.0 GB/s (94%)
SmolLM3-3B 4-bit, int8 head 8.74 tok/s 8.72 tok/s 22.8 tok/s 15.7 GB/s (92%)
Phi-4-mini (3.8B) int8 3.99 tok/s 3.98 tok/s 13.8 tok/s 16.0 GB/s (94%)
Phi-4-mini (3.8B) 4-bit, int8 head 6.56 tok/s 6.55 tok/s 15.0 tok/s 15.8 GB/s (92%)
Qwen3.5-2B int8 8.02 tok/s 8.00 tok/s 38.2 tok/s 16.0 GB/s (94%)
Qwen3.5-2B 4-bit, int8 head 12.09 tok/s 12.03 tok/s 41.7 tok/s 15.8 GB/s (92%)
Qwen3.5-4B 4-bit, int8 head 5.88 tok/s 5.87 tok/s 12.9 tok/s 15.7 GB/s (92%)
Gemma 4 E4B int8, 4-bit head and down 0-23 3.78 tok/s 3.75 tok/s 14.8 tok/s 16.0 GB/s (94%)

Measured on the card: the first three models on 2026-09-29 with the production image deploy_champ_e698dcd7 . LFM2-2.6B, SmolLM3-3B and Phi-4-mini on 2026-09-30, and Qwen3.5-2B and 4B and Gemma 4 on 2026-10-01, with build B, deploy_fused133c_79c5707a , production since then. Build B decodes LFM2-2.6B, SmolLM3 and Phi-4-mini 8-9% faster than e698dcd7 (Gemma 4 E2B 10%), at 91-94% of the DRAM peak instead of 82-87%. Qwen3.5-4B's int8 image is over 4 GiB.

  • The image: main e698dcd at 133.33 MHz, one bitstream for all models. It has LiteDRAM controllers calibrated by a small CPU inside the memory core, a four-column systolic matrix unit and the stream engine ( docs/stream.md ). DDR3-1066, with a 17.1 GB/s peak.
  • The host: the card sits in opentpu (Intel Core i7-4790).
  • Method, tools/qual/perf.py : decode is 64 greedy tokens after a 512-token prompt, with the host's argmax in the loop (not streamed). "Device" counts only the cycles the accelerator runs; "wall" adds the host. Prefill is the 512-token prompt, on the device.
  • DRAM traffic comes from the card's own counters while it runs.
  • Gemma 4 E2B keeps its per-layer embedding tables on the card (3.5-3.6 GiB images; docs/gemma4.md ); in int8 it does not fit. It matches Hugging Face's greedy tokens on three prompts with either head. E4B's table (2.95 GB) stays on the host, which copies one 11 KB row into the card per token; its image is 3.96 GiB, int8 with the head and the first 24 layers' down projections in 4-bit ( docs/gemma4_e4b.md ). In the card's own decode loop (the card picking every token) Gemma 4 decodes faster: E2B 11.01 / 12.73 tok/s (int8 / 4-bit head), E4B 3.83 tok/s, on the device.
  • Every configuration matches the simulator token for token, per-position and with the resident decode program. More detail in docs/board.md .

With the logits streamed back while the card runs ( tools/decode_profile.py , 96 tokens), 4-bit decode is faster, in device / wall tok/s:

  • LFM2: 89.5 / 84.5;
  • Qwen3: 33.7 / 33.3;
  • Qwen3.5: 24.6 / 24.2;
  • LFM2-2.6B: 11.07 / 11.02 (build B);
  • SmolLM3-3B: 8.92 / 8.89 (build B);
  • Phi-4-mini: 6.69 / 6.67 (build B).

The previous production image, se-cand3, was built with the Xilinx MIG, a two-column matrix unit and a 120.755 MHz clock. Measured the same way, the new image:

  • decode: within 2.3% of se-cand3's in every configuration. Decode is bound by DRAM, and LiteDRAM reads at 82-85% of the DDR3 peak, as the MIG did.
  • prefill: 1.3x (Qwen3.5) to 2.0x (LFM2 4-bit) faster.
  • calibration: when the image starts, the core's CPU calibrates both DDR3 channels in 12 s, with no host involvement.

The earlier images and their numbers are in docs/board.md , section 5.

Mixture-of-experts models bigger than the card's 4 GiB run with their experts streamed from host storage ( docs/offload.md , section 10). The card routes each token and computes every expert, and it keeps the experts in per-layer slots in its DRAM. The host only copies missing experts from a pool file into those slots, at the link's rate (section 10.1). Measured on 2026-10-01 with build B (79c5707a), the card's own decode loop picking every token, 4-bit experts, int8 head:

  • LFM2.5-8B-A1B (8.5B parameters, 1.7B active): 10.6 tok/s over 160 tokens. 98.5% of expert uses hit the slots, and 5.2 MB streamed per token.
  • Qwen3.5-35B-A3B (34.7B parameters, 3.0B active): 3.95 tok/s, with Hugging Face's 16 greedy tokens. 62% of expert uses hit, and 153 MB streamed per token at 1.41 GB/s over PCIe (section 10.3).
  • Both match the simulator bit for bit.

4-bit weights ( docs/quant.md ) use FP4 values with two-level block scales, 4.25 bits per weight, and keep the LM head in int8 for accuracy. They cut the bytes per token by about a third and raise decode speed by 40% (Qwen3.5) to 45% (Qwen3, LFM2), at a measurable cost in perplexity that docs/quant.md reports per model.

The host is nearly out of the way. For LFM2 and Qwen3 the card runs one decode program compiled once, which reads the position from a register and looks up its own embedding and RoPE rows, and the logits stream back while the card is still running: the host adds 0.17 to 0.30 ms per token on omarchy (0.45 to 1.3 ms on opentpu). Qwen3.5 runs the same way for decode; its prefill still compiles each chunk's program on the host, ahead of the card.

How it works 

  Kernels in ol              mlp, attention, full model layers
        |  @ol.jit
  Language + compiler        layouts, affine loop addressing, fusion
        |
  ISA                        8 x 32-bit words per instruction
        |
  ISA simulator  <======>  RTL          same bits, checked by the tests
  (Python)                 (SystemVerilog)
                            |  Vivado bitstream
                           FPGA card    Kintex-7 xc7k480t
                            |  PCIe
                           Host         otpu-chat, otpu-smi, otpu-lens

The machine is deliberately simple. A sequencer issues one instruction per cycle to a few units: DMA moves data, the matrix unit multiplies int8 weights streamed from DRAM, the vector unit does fp32 math, and a quantizer turns results back into int8. There is no cache and no hidden scheduling: every data movement is an instruction, so a trace shows exactly where the cycles go. docs/isa.md describes the whole instruction set.

A kernel looks like this: 

from opentpu import language as ol

@ol.jit
def mlp(h, gamma, w_gate, w_up, w_down, out, eps):   # simplified; see kernels/mlp.py
    x = ol.load(h)
    xs = ol.quantize(rmsnorm(x, ol.load(gamma), eps))
    g = ol.dot(xs, w_gate)
    u = ol.dot(xs, w_up)
    a = ol.all_gather(silu(g) * u)
    y = ol.all_gather(ol.dot(a, w_down))
    if ol.program_id() == 0:
        ol.store(out, x + y)

Because every data movement is an instruction, a trace of a run explains its speed. Lens, the profiler, records a run from the RTL, the simulator or the card and opens it in the browser, with a roofline, a timeline and per-instruction tables ( docs/lens.md ).

Lens replaying a Qwen3 decode step on the floorplan

Lens replaying part of a Qwen3 decode step. Colours show what each unit is doing in each cycle: busy, waiting on DRAM, or waiting on another instruction.

Try it

Everything except the card runs on a laptop. 

pip install -e .
pip install pytest torch transformers
python3 -m pytest -q          # RTL tests also need Verilator 5

hf download LiquidAI/LFM2.5-230M --local-dir models/LFM2.5-230M
otpu-chat --model lfm2 --backend isa    # chat on the simulator

With a card, build the bitstream ( make bit in boards/ypcb-00338 ), load it over JTAG, then run sudo otpu-setup and otpu-chat --backend board . docs/board.md walks through the bring-up.

Command What it does
otpu-chat chat with Qwen3-0.6B, LFM2.5-230M ( --model lfm2 ), Qwen3.5-0.8B ( --model qwen35 ), LFM2-2.6B ( lfm2-2.6b ), SmolLM3-3B ( smollm3 ), Phi-4-mini ( phi4-mini ) or Qwen3.5-2B / 4B ( qwen35-2b , qwen35-4b )
otpu-smi temperature, power, DRAM bandwidth and per-unit utilization
otpu-lens record a run and open it in the profiler
otpu-selftest , otpu-diag check that the card works

Where to start reading

  1. docs/isa.md : the instruction set. Everything else is built on it.
  2. opentpu/kernels and docs/compiler.md : how a kernel becomes instructions.
  3. opentpu/isasim.py : the simulator, which is the spec.
  4. rtl/ : the hardware, starting from rtl/top/otpu_top.sv .
  5. docs/lfm2.md , docs/qwen35.md , docs/llama.md , docs/benchmarks.md : whole models and where their cycles go.
  6. docs/board.md : the physical card, from clocks to PCIe.

What's next

  • The last few percent of DRAM. Decode is bound by DRAM efficiency: it reads 82 to 85% of the DDR3-1066 peak. Work on the LiteDRAM path's efficiency is under way.
  • Timing margin and area. The design closes 133.33 MHz, the clock at which the 128-byte port matches the two DDR3 channels, but only just (WNS +0.032 ns). A tournament of Vivado runs keeps working on its margin and area. Decode is bound by DRAM, so a faster clock mostly helps prefill.
  • Faster prefill. The four-column systolic matrix unit is in the production image; prefill is still limited by the matrix unit's multiply rate.

Contributing

Issues and pull requests are welcome, and most of the work needs only Python and Verilator, not an FPGA. Changes to the ISA, the simulator or the RTL must keep python3 -m pytest -q passing, and performance claims should say how they were measured.

License

Apache License 2.0. See LICENSE .

EmbeddingGemma 2

Hacker News
blog.google
2026-10-06 12:03:49
Comments...
Original Article

EmbeddingGemma 2 is the most capable model for on-device multimodal embeddings, natively mapping combinations of text, images, audio, and video into a unified embedding space.

Henrique Schechter Vera

Research Engineer, Google DeepMind

EmbeddingGemma 2

Your browser does not support the audio element.

Listen to article

[[duration]] minutes

This content is generated by Google AI. Generative AI is experimental

We introduced EmbeddingGemma last year to provide a lightweight option for high-quality text embeddings, to help your apps organize, search, and connect information directly on consumer hardware. The developer community’s response blew past our expectations. With more than 20 million downloads, builders have used it to power smarter on-device search tools and privacy-first retrieval augmented generation (RAG) pipelines.

Today, we’re launching EmbeddingGemma 2 , expanding beyond text to unify code, images, video, and audio in a shared embedding space. Built on the Gemma 4 architecture and released under a commercially permissive Apache 2.0 license, EmbeddingGemma 2 has 740 million parameters, making it optimal for on-device inference. It can help find a specific video clip from a voice memo, or search through hours of audio recordings based on a text query, all processed by a single, natively multimodal model.

Built from the same technology as Gemini Embedding models, EmbeddingGemma 2 is:

  • Best-in-class for its size: Achieves leading scores among sub-1B multimodal embedders for its size across benchmarks like MTEB (Massive Text Embedding Benchmark) Code and MAEB (Massive Audio Embedding Benchmark), while matching or outperforming many larger models across text, vision, and audio tasks.
  • Modular by design: Requires as little as 270M parameters for text-only workloads with optional vision (170M) and audio (300M) encoders for full multimodal support.
  • Storage-efficient: Using Matryoshka Representation Learning (MRL), developers can dynamically truncate output vectors from 768 dimensions down to 512, 256, or 128 dimensions. This provides up to 6x storage reduction for local vector databases and memory usage.
  • Optimized for on-device performance: Runs efficiently within tight resource constraints. With quantization, on a Google Pixel 11 Pro, EmbeddingGemma 2 requires as little as ~191MB active RAM for text-only weights and ~567MB for the full multimodal model.
  • Extended context ready: Features an 8K token context window (4x larger than EmbeddingGemma 1), allowing it to process up to 5.5 minutes of audio, 29 images, 58 video frames, or interleaved combinations thereof directly on local hardware.

Achieving top-tier quality for code, vision, and audio

EmbeddingGemma 2 matches the strong multilingual text performance of EmbeddingGemma while delivering a significant 9.92-point improvement on code performance (in MTEB Code, from 68.76 to 78.68), making it well-suited for local codebase indexing, semantic code search, and coding agent retrieval. Across image, video, documents, and audio, it sets a new standard in quality-per-parameter for sub-1B models and even outperforms some specialist models more than twice its size.

Massive Text Embedding Benchmark (Code)

Massive Image Embedding Benchmark (Lite)

Massive Audio Embedding Benchmark

Find full evaluation metrics and model information in the EmbeddingGemma 2 model card .

Enabling semantic search, routing, and retrieval, fully on-device

EmbeddingGemma 2 brings robust capabilities directly to edge hardware. Generating embeddings locally helps ensure data privacy, reduces pipeline latency, and empowers developers to build cross-modal search and retrieval that works entirely offline.

When paired with generative models such as Gemma 4, EmbeddingGemma 2 enables on-device RAG pipelines that understand complex multimodal data. Because EmbeddingGemma 2 is built on Gemma 4 and shares its text tokenizer and audio encoder, developers can run both models together in a unified pipeline with a lower combined total memory footprint.

To learn how to build on-device search and RAG systems with LiteRT, read the Google AI Edge blog post .

Getting started with EmbeddingGemma 2

We worked closely with the following partners to ensure EmbeddingGemma 2 works immediately where you build:

  • Download the models: Find the model weights on Hugging Face and Kaggle , with Gemini Enterprise Agent Platform Model Garden availability coming soon. Visit LiteRT Community on Hugging Face for models optimized for on-device.
  • On-device deployment: Develop cross-platform apps with Google AI Edge MediaPipe for turnkey embedding, retrieval & decision tasks or LiteRT for custom model integration. Build for the browser with transformers.js or WebGPU .
  • Use your favorite development tools : Serve the model efficiently using transformers, sentence-transformers, MLX , vLLM, llama.cpp , SGLang, Ollama , and LMStudio. Store your embedding vectors with Qdrant .
  • Fine-tuning: Follow guidance by Unsloth for how to fine-tune EmbeddingGemma 2 for your use cases.

Explore our developer guide , documentation , and guides for inference and fine-tuning .

kahawai - an open source, modular media system

Lobsters
github.com
2026-10-06 11:23:07
Comments...
Original Article

Kahawai ( kah-hah-why ) is Hawaiian for stream . In te reo Māori, kahawai is also the name of Arripis trutta , from kaha (strong) and wai (water).

What Kahawai is

Kahawai is a self-hosted server for streaming movies, series, anime, and music from your own media. It is written in Rust, uses GStreamer for media processing, and includes its web interface in the server binary.

Every deployment has three roles:

  • Hub: serves the web interface and API, owns users and library state, and is the only role clients contact.
  • Mediahost: indexes read-only media roots and serves source bytes to the hub.
  • Transcoder: re-encodes streams a client cannot play directly.

The hub chooses the cheapest complete playback path: direct play, remux in the hub, or transcode only the incompatible streams. An all-in-one process runs the hub, a mediahost, and an optional transcoder on one machine. Satellites use mTLS, dial the hub, and never accept inbound application traffic.

Kahawai stores its state in SQLite and requires no external database. The web interface provides playback, libraries, users, metadata, match review, watch state, and administration. It finds the recap, opening and end credits of a season in the background, and offers to skip them. Where it has measured nothing — movies, or a season the sweep has not reached — each viewer can opt in to fetching community skip times from TheIntroDB , asked for directly by their browser and never by the server.

Install all-in-one

The supported installation is the container image. It includes the required GStreamer stack and Kahawai's media fixes.

Build the image 

git clone https://github.com/iksteen/kahawai.git
cd kahawai
docker build -t kahawai .

Use --device=/dev/dri:/dev/dri for Intel or AMD hardware acceleration. Use --gpus all with the NVIDIA Container Toolkit for NVIDIA hardware. 

docker run --rm --device=/dev/dri:/dev/dri kahawai doctor
docker run --rm --gpus all kahawai doctor

doctor lists the available decode, encode, remux, and tone-mapping paths.

Configure the server

Create persistent directories and a configuration file: 

mkdir -p runtime/config/kahawai runtime/data runtime/cache

runtime/config/kahawai/kahawai.toml : 

[all_in_one]
transcoder = true

[hub]
bind = "0.0.0.0:8420"
satellite_bind = "0.0.0.0:8421"

[mediahost]
name = "local"

[[mediahost.collections]]
name = "movies"
media_type = "movies"
roots = ["/media/movies"]

[[mediahost.collections]]
name = "series"
media_type = "series"
roots = ["/media/series"]

Collection roots are read-only. Their paths are the paths inside the container.

Run all-in-one 

docker run -d \
  --name kahawai \
  --restart unless-stopped \
  --device=/dev/dri:/dev/dri \
  -p 127.0.0.1:8420:8420 \
  -p 8421:8421 \
  -v "$PWD/runtime/config:/config" \
  -v "$PWD/runtime/data:/data" \
  -v "$PWD/runtime/cache:/cache" \
  -v /srv/media:/media:ro \
  kahawai

Replace --device=/dev/dri:/dev/dri with --gpus all for NVIDIA, or omit it for software-only operation.

Create the first administrator through the private control socket: 

docker exec -it kahawai kahawai hub init-admin

Open http://localhost:8420 . Put a TLS reverse proxy in front before exposing the hub outside the machine.

Kahawai reads $XDG_CONFIG_HOME/kahawai/kahawai.toml , ./kahawai.toml , or the path passed with --config . Environment overrides use KAHAWAI_<SECTION>__<KEY> , such as KAHAWAI_HUB__DATA_DIR=/srv/kahawai .

On macOS, from source

A container on macOS cannot reach VideoToolbox, so a Mac that is meant to transcode has to build natively. This repository is a Homebrew tap: 

brew tap iksteen/kahawai https://github.com/iksteen/kahawai
brew install --HEAD iksteen/kahawai/kahawai

That pulls in kahawai-gstreamer first, which is the upstream GStreamer release with Kahawai's media fixes applied, installed keg-only so it cannot shadow the stock formula. Building it takes a while; there are no bottles.

A starting config is written to $(brew --prefix)/etc/kahawai/kahawai.toml , with its data directories under var rather than in a home directory. Add collections to it, then: 

brew services start kahawai

Without sudo that is a user agent, so it starts at login rather than at boot. sudo brew services start kahawai installs a system daemon instead.

Satellite roles are not services here, because a formula can only declare one. Their binaries are installed and take --config .

Both formulae build from source and neither is the supported artifact. The container image is. See deployment details for running these as system daemons instead, which is what a Mac that must start at boot wants.

On Arch Linux, from the AUR

The AUR has two packages, installed with an AUR helper such as yay or paru: 

yay -S kahawai     # or: paru -S kahawai

kahawai-gstreamer comes first. It is the upstream GStreamer release with Kahawai's media fixes, installed under /opt/kahawai-gstreamer and isolated from Arch's own GStreamer: it ignores GST_PLUGIN_PATH , GST_REGISTRY and their relatives, and keeps its own plugin registry. kahawai is linked against it. Both build from source, and building GStreamer takes a while.

The AUR packages are updated separately from the releases here, so they may lag behind the latest one. The version on the AUR page is the one you get.

The package runs Kahawai as the kahawai system user, with its state under /var/lib/kahawai and its config in /etc/kahawai/kahawai.toml . That config explains how to add collections and give the kahawai user read access to your media. Then: 

sudo systemctl enable --now kahawai
sudo -u kahawai kahawai --config /etc/kahawai/kahawai.toml hub init-admin

The hub listens on 127.0.0.1:8420 . kahawai.service is all-in-one. To run the roles separately, use kahawai-hub , kahawai-mediahost and kahawai-transcoder.service instead; the end of the config explains how.

Like the Homebrew formulae, these packages are not the supported artifact. The container image is.

Satellites and standalone hub

Expose the hub's satellite listener directly as TCP. Do not send it through the HTTP reverse proxy. The hub certificate must contain the name or address that satellites use: 

[hub]
satellite_bind = "0.0.0.0:8421"
hostnames = ["kahawai.example.lan"]

A mediahost needs its own persistent state directory and read-only media roots: 

[mediahost]
name = "nas"
state_dir = "/data/kahawai-mediahost"

[[mediahost.hubs]]
id = "home"
address = "kahawai.example.lan:8421"

[[mediahost.hubs]]
id = "family"
address = "family.example.lan:8421"
collections = ["anime"] # omit to publish every collection

[[mediahost.collections]]
name = "anime"
media_type = "anime"
roots = ["/media/anime"]

Run it with:

The mediahost scans and analyzes each file once into /data/kahawai-mediahost/catalog.db ; every named hub has independent mTLS credentials under state_dir/hubs/<id>/ and receives only catalogue versions newer than its durable cursor. The old single hub = "…" setting remains accepted and keeps its existing credentials directly in state_dir .

A transcoder needs persistent state and access to its GPU: 

[transcoder]
hub = "kahawai.example.lan:8421"
name = "gpu-box"
state_dir = "/data/kahawai-transcoder"
max_sessions = 2

Run it with:

The same container image runs either role by appending mediahost or transcoder to docker run . On first connection, the satellite prints an enrollment code. Approve that code in the hub's administration page. The hub issues the satellite certificate and the satellite reconnects with mTLS.

When every collection lives on satellite mediahosts, the all-in-one process can be replaced with kahawai hub . Stop all-in-one, keep the same hub configuration and data directory, and start:

The standalone hub retains users, libraries, certificates, and watch state. It continues to direct-play and remux. Full video encoding requires an enrolled transcoder.

Harden a public hub

Terminate TLS at an HTTP reverse proxy and keep the hub's HTTP listener unreachable from the internet. For a native hub and proxy on the same host: 

[hub]
bind = "127.0.0.1:8420"
public_url = "https://kahawai.example.com"
trusted_proxies = ["127.0.0.1"]

For the container installation, keep bind = "0.0.0.0:8420" inside the container and keep the host-side 127.0.0.1:8420:8420 port mapping. trusted_proxies must contain the proxy peer address as Kahawai sees it inside the container, not necessarily the proxy's host address.

  • public_url enables strict Origin checking for browser login, refresh, and logout. The Origin must match exactly. HTTPS also marks authentication cookies Secure . Without public_url , Kahawai does not validate browser Origins.
  • trusted_proxies lists the socket peers allowed to supply forwarded request metadata. Trust only proxy addresses; never trust a network that clients can occupy.
  • X-Forwarded-For supplies the client address used by login throttling. Kahawai resolves the chain from right to left and selects the first untrusted address.
  • X-Forwarded-Proto and X-Forwarded-Host identify the browser-facing scheme and host for cookie security when public_url is absent and they come from a trusted proxy.

Minimal nginx configuration: 

location / {
    proxy_pass http://127.0.0.1:8420;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host $http_host;
    proxy_http_version 1.1;
}

The API uses the HTTP QUERY method for /api/v1/items/{id} . Proxies that restrict methods must allow QUERY . Do not buffer /api/v1/events ; it is a Server-Sent Events stream. Keep the first-run setup listener private and expose port 8421 only to satellite networks.

OpenAPI

The hub publishes its complete OpenAPI 3.2 contract at:

The embedded Swagger UI is available at:

The document covers the public and administrative APIs, authentication modes, request and response schemas, errors, and the QUERY item operation. API clients use bearer-mode login and refresh tokens; browser-cookie authentication is for the embedded same-origin web interface.

License

Kahawai is MIT licensed .

Kahawai dynamically links to system or container-provided GStreamer libraries, which are LGPL-2.1-or-later. Some runtime plugins, including plugins from gst-plugins-ugly , are GPL-licensed. Distributors must follow the licenses of the GStreamer libraries and plugins included with their bundle.

Metadata from TMDB and TheTVDB requires the attribution displayed by Kahawai.

The Early History of Smalltalk

Hacker News
worrydream.com
2026-10-06 11:19:57
Comments...
Original Article

original pdf / make corrections

Alan C. Kay
Apple Computer
kay2@apple.com.Internet#

Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a fee and/or specific permission.
HOPL-II/4/93/MA, USA © 1993 ACM 0-89791-571-2/93/0004/0069...$1.50

Abstract

Most ideas come from previous ideas. The sixties, particularly in the ARPA community, gave rise to a host of notions about "human-computer symbiosis" through interactive time-shared computers, graphics screens and pointing devices. Advanced computer languages were invented to simulate complex systems such as oil refineries and semi-intelligent behavior. The soon-to-follow paradigm shift of modern personal computing, overlapping window interfaces, and object-oriented design came from seeing the work of the sixties as something more than a "better old thing." That is, more than a better way: to do mainframe computing; for end-users to invoke functionality; to make data structures more abstract. Instead the promise of exponential growth in computing/$/volume demanded that the sixties be regarded as " almost a new thing" and to find out what the actual "new things" might be. For example, one would compute with a handheld "Dynabook" in a way that would not be possible on a shared mainframe; millions of potential users meant that the user interface would have to become a learning environment along the lines of Montessori and Bruner; and needs for large scope, reduction in complexity, and end-user literacy would require that data and control structures be done away with in favor of a more biological scheme of protected universal cells interacting only through messages that could mimic any desired behavior.

Early Smalltalk was the first complete realization of these new points of view as parented by its many predecessors in hardware, language and user interface design. It became the exemplar of the new computing, in part, because we were actually trying for a qualitative shift in belief structures—a new Kuhnian paradigm in the same spirit as the invention of the printing press—and thus took highly extreme positions which almost forced these new styles to be invented.

Table of Contents

  • Introduction 2
  • I. 1960-66—Early OOP and other formative ideas of the sixties 4
    • B220 File System
    • SketchPad & Simula
  • II 1967-69—The FLEX Machine, an OOP-based personal computer 6
    • Doug Englebart and NLS
    • Plasma Panel, GRAIL, LOGO, Dynabook
  • III. 1970-72—Xerox PARC 12
    • KiddiKomp
    • miniCOM
    • Smalltalk-71
    • Overlapping Windows
    • Font Editing, Painting, Animation, Music
    • Byte Codes
    • Iconic Programming
  • IV. 1972-76—Xerox PARC: The first real Smalltalk (-72) 17
    • The two bets: birth of Smalltalk and Interim Dynabook
    • Smalltalk-72 Principles
    • The Smalltalk User Interface
    • Development of the Smalltalk Applications & System
    • Evolution of Smalltalk: ST-74, ooze storage management
    • Smalltalk and Children
  • V. 1976-80—The first modern Smalltalk (-76) 29
    • "Let's burn our disk packs"
    • The Notetaker
    • Smalltalk-76
    • Inheritance
    • More Troubles With Xerox
    • ThingLab
    • Apple Demo
  • VI. 1980-83—The release version of Smalltalk (-80) 38
  • References Cited in Text 41
  • Appendix I: KiddiKomp Memo 45
  • Appendix II: Smalltalk-72 Interpreter Design 47
  • Appendix III: Acknowledgments 50
  • Appendix IV: Event Driven Loop Example 53
  • Appendix V: Smalltalk-76 Internal Structures 54

—To Dan Ingalls, Adele Goldberg and the rest of Xerox PARC LRG gang

—To Dave Evens, Bob Barton, Marvin Minsky, and Seymour Papert

—To SKETCHPAD, JOSS, LISP, and SIMULA, the 4 great programming conceptions of the sixties

Introduction

I'm writing this introduction in an airplane at 35,000 feet. On my lap is a five pound notebook computer—1992's "Interim Dynabook"—by the end of the year it sold for under $700. It has a flat, crisp, high-resolution bitmap screen, overlapping windows, icons, a pointing device, considerable storage and computing capacity, and its best software is object-oriented. It has advanced networking built-in and there are already options for wireless networking. Smalltalk runs on this system, and is one of the main systems I use for my current work with children. In some ways this is more than a Dynabook (quantitatively), and some ways not quite there yet (qualitatively). All in all, pretty much what was in mind during the late sixties.

Smalltalk was part of this larger pursuit of ARPA, and later of Xerox PARC, that I called personal computing. There were so many people involved in each stage from the research communities that the accurate allocation of credit for ideas is intractably difficult. Instead, as Bob Barton liked to quote Goethe, we should "share in the excitement of discovery without vain attempts to claim priority."

I will try to show where most of the influences came from and how they were transformed in the magnetic field formed by the new personal computing metaphor. It was the attitudes as well as the great ideas of the pioneers that helped Smalltalk get invented. Many of the people I admired most at this time—such as Ivan Sutherland, Marvin Minsky, Seymour Papert, Gordon Moore, Bob Barton, Dave Evans, Butler Lampson, Jerome Bruner, and others—seemed to have a splendid sense that their creations, though wonderful by relative standards, were not near to the absolute thresholds that had to be crossed. Small minds try to form religions, the great ones just want better routes up the mountain. Where Newton said he saw further by standing on the shoulders of giants, computer scientists all too often stand on each other's toes. Myopia is still a problem where there are giants' shoulders to stand on—"outsight" is better than insight—but it can be minimized by using glasses whose lenses are highly sensitive to esthetics and criticism.

Programming languages can be categorized in a number of ways: imperative, applicative, logic-based, problem-oriented, etc. But they all seem to be either an "agglutination of features" or a "crystallization of style." COBOL, PL/1, Ada, etc., belong to the first kind; LISP, APL— and Smalltalk—are the second kind. It is probably not an accident that the agglutinative languages all seem to have been instigated by committees, and the crystallization languages by a single person.

Smalltalk's design—and existence—is due to the insight that everything we can describe can be represented by the recursive composition of a single kind of behavioral building block that hides its combination of state and process inside itself and can be dealt with only through the exchange of messages. Philosophically, Smalltalk's objects have much in common with the monads of Leibniz and the notions of 20th century physics and biology. Its way of making objects is quite Platonic in that some of them act as idealizations of concepts— Ideas —from which manifestations can be created. That the Ideas are themselves manifestations (of the Idea-Idea) and that the Idea-Idea is a-kind-of Manifestation-Idea—which is a-kind-of itself, so that the system is completely self-describing— would have been appreciated by Plato as an extremely practical joke [Plato].

In computer terms, Smalltalk is a recursion on the notion of computer itself. Instead of dividing "computer stuff" into things each less strong than the whole—like data structures, procedures, and functions which are the usual paraphernalia of programming languages—each Smalltalk object is a recursion on the entire possibilities of the computer. Thus its semantics are a bit like having thousands and thousands of computers all hooked together by a very fast network. Questions of concrete representation can thus be postponed almost indefinitely because we are mainly concerned that the computers behave appropriately, and are interested in particular strategies only if the results are off or come back too slowly.

Though it has noble ancestors indeed, Smalltalk's contribution is a new design paradigm—which I called object-oriented —for attacking large problems of the professional programmer, and making small ones possible for the novice user. Object-oriented design is a successful attempt to qualitatively improve the efficiency of modeling the ever more complex dynamic systems and user relationships made possible by the silicon explosion.

"We would know what they thought
when they did it."
—Richard Hamming

"Memory and imagination are but two
words for the same thing."
—Thomas Hobbes

In this history I will try to be true to Hamming's request as moderated by Hobbes' observation. I have had difficulty in previous attempts to write about Smalltalk because my emotional involvement has always been centered on personal computing as an amplifier for human reach—rather than programming system design—and we haven't got there yet. Though I was the instigator and original designer of Smalltalk, it has always belonged more to the people who make it work and got it out the door, especially Dan Ingalls and Adele Goldberg. Each of the LRGers contributed in deep and remarkable ways to the project, and I wish there was enough space to do them all justice. But I think all of us would agree that for most of the development of Smalltalk, Dan was the central figure. Programming is at heart a practical art in which real things are built, and a real implementation thus has to exist. In fact many if not most languages are in use today not because they have any real merits but because of their existence on one or more machines, their ability to be bootstrapped, etc. But Dan was far more than a great implementer, he also became more and more of the designer, not just of the language but also of the user interface as Smalltalk moved into the practical world.

Here, I will try to center focus on the events leading up to Smalltalk-72 and its transition to its modern form as Smalltalk-76. Most of the ideas occurred here, and many of the earliest stages of OOP are poorly documented in references almost impossible to find.

This history is too long, but I was amazed at how many people and systems that had an influence appear only as shadows or not at all. I am sorry not to be able to say more about Bob Balzer, Bob Barton, Danny Bobrow, Steve Carr, Wes Clark, Barbara Deutsch, Peter Deutsch, Bill Duvall, Bob Flegal, Laura Gould, Bruce Horn, Butler Lampson, Dave Liddle, William Newman, Bill Paxton, Trygve Reenskaug, Dave Robson, Doug Ross, Paul Rovner, Bob Sproull, Dan Swinehart, Bert Sutherland, Bob Taylor, Warren Teitelman, Bonnie Tennenbaum, Chuck Thacker, and John Warnock. Worse, I have omitted to mention many systems whose design I detested, but that generated considerable useful ideas and attitudes in reaction. In other words "histories" should not be believed very seriously but considered as "FEEBLE GESTURES OFF" done long after the actors have departed the stage.

Thanks to the numerous reviewers for enduring the many drafts they had to comment on. Special thanks to Mike Mahoney for helping so gently that I heeded his suggestions and so well that they greatly improved this essay—and to Jean Sammet, an old old friend, who quite literally frightened me into finishing it—I did not want to find out what would happen if I were late. Sherri McLoughlin and Kim Rose were of great help in getting all the materials together.

I. 1960-66—Early OOP and other formative ideas of the sixties

Though OOP came from many motivations, two were central. The large scale one was to find a better module scheme for complex systems involving hiding of details, and the small scale one was to find a more flexible version of assignment, and then to try to eliminate it altogether. As with most new ideas, it originally happened in isolated fits and starts.

New ideas go through stages of acceptance, both from within and without. From within, the sequence moves from "barely seeing" a pattern several times, then noting it but not perceiving its "cosmic" significance, then using it operationally in several areas, then comes a "grand rotation" in which the pattern becomes the center of a new way of thinking, and finally, it turns into the same kind of inflexible religion that it originally broke away from. From without, as Schopenhauer noted, the new idea is first denounced as the work of the insane, in a few years it is considered obvious and mundane, and finally the original denouncers will claim to have invented it.

True to the stages, I "barely saw" the idea several times ca. 1961 while a programmer in the Air b220 file format Force. The first was on the Burroughs 220 in the form of a style for transporting files from one Air Training Command installation to another. There were no standard operating systems or file formats back then, so some (to this day unknown) designer decided to finesse the problem by taking each file and dividing it into three parts. The third part was all of the actual data records of arbitrary size and format. The second part contained the B220 procedures that knew how to get at records and fields to copy and update the third part. And the first part was an array of relative pointers into entry points of the procedures in the second part (the initial pointers were in a standard order representing standard meanings). Needless to say, this was a great idea, and was used in many subsequent systems until the enforced use of COBOL drove it out of existence.

The second barely-seeing of the idea came just a little later when ATC decided to replace the 220 with a B5000. I didn't have the perspective to really appreciate it at the time, but I did take note of its segmented storage system, its efficiency of HLL compilation and byte-coded execution, its automatic mechanisms for subroutine calling and multiprocess switching, its pure code for sharing, its protection mechanisms, etc. And, I saw that the access to its Program Reference Table corresponded to the 220 file system scheme of providing a procedural interface to a module. However, my big hit from this machine at this time was not the OOP idea, but some insights into HLL translation and evaluation. [Barton, 1961] [Burroughs, 1961]

After the Air Force, I worked my way through the rest of college by programming mostly retrieval Moore's Law graph systems for large collections of weather data for the National Center for Atmospheric Research. I got interested in simulation in general—particularly of one machine by another—but aside from doing a one-dimensional version of a bit-field block transfer (bitblt) on a CDC 6600 to simulate word sizes of various machines, most of my attention was distracted by school, or I should say the theatre at school. While in Chippewa Falls helping to debug the 6600, I read an article by Gordon Moore which predicted that integrated silicon on chips was going to exponentially improve in density and cost over many years [Moore 65]. At the time in 1965, standing next to the room-sized freon-cooled 10 MIP 6600, his astounding predictions had little projection into my horizons.

Sketchpad and Simula

Through a series of flukes, I wound up in graduate school at the University of Utah in the Fall of 1966, "knowing nothing." That is to say, I had never heard of ARPA or its projects, or that Utah's main goal in this community was to solve the "hidden line" problem in 3D graphics, until I actually walked into Dave Evans' office looking for a job and a desk. On Dave's desk was a foot-high stack of brown covered documents, one of which he handed to me: "Take this and read it."

Every newcomer got one. The title was "Sketchpad: A man-machine graphical communication system" [Sutherland, 1963]. What it could do was quite remarkable, and completely foreign to any use of a computer I had ever encountered. The three big ideas that were easiest to grapple with were: it was the invention of modern interactive computer graphics; things were described by making a "master drawing" that could produce "instance drawings"; control and dynamics were supplied by "constraints," also in graphical form, that could be applied to the masters to shape an inter-related parts. Its data structures were hard to understand—the only vaguely familiar construct was the embedding of pointers to procedures and using a process called reverse indexing to jump through them to routines, like the 220 file system [Ross, 1961]. It was the first to have clipping and zooming windows—one "sketched" on a virtual sheet about 1/3 mile square!

Sketchpad

Head whirling, I found my desk. On it was a pile of tapes and listings, and a note: "This is the Algol for the 1108. It doesn't work. Please make it work." The latest graduate student gets the latest dirty task.

The documentation was incomprehensible. Supposedly, this was the Case-Western Reserve 1107 Algol—but it had been doctored to make a language called Simula; the documentation read like Norwegian transliterated into English, which in fact it was. There were uses of words like activity and process that didn't seem to coincide with normal English usage.

Finally, another graduate student and I unrolled the program listing 80 feet down the hall and crawled over it yelling discoveries to each other. The weirdest part was the storage allocator, which did not obey a stack discipline as was usual for Algol. A few days later, that provided the clue. What Simula was allocating were structures very much like the instances of Sketchpad. There were descriptions that acted like masters and they could create instances, each of which was an independent entity. What Sketchpad called masters and instances, Simula called activities and processes. Moreover, Simula was a procedural language for controlling Sketchpad-like objects, thus having considerably more flexibility than constraints (though at some cost in elegance) [Nygaard, 1966, Nygaard, 1983].

This was the big hit, and I've not been the same since. I think the reason the hit had such impact was that I had seen the idea enough times in enough different forms that the final recognition was in such general terms to have the quality of an epiphany. My math major had centered on abstract algebras with their few operations generally applying to many structures. My biology major had focused on both cell metabolism and larger scale morphogenesis with its notions of simple mechanisms controlling complex processes and one kind of building block able to differentiate into all needed building blocks. The 220 file system, the B5000, Sketchpad, and finally Simula, all used the same idea for different purposes. Bob Barton, the main designer of the B5000 and a professor at Utah had said in one of his talks a few days earlier: "The basic principle of recursive design is to make the parts have the same power as the whole." For the first time I thought of the whole as the entire computer and wondered why anyone would want to divide it up into weaker things called data structures and procedures. Why not divide it up into little computers, as time sharing was starting to? But not in dozens. Why not thousands of them, each simulating a useful structure?

I recalled the monads of Leibniz, the "dividing nature at its joints" discourse of Plato, and other attempts to parse complexity. Of course, philosophy is about opinion and engineering is about deeds, with science the happy medium somewhere in between. It is not too much of an exaggeration to say that most of my ideas from then on took their roots from Simula—but not as an attempt to improve it. It was the promise of an entirely new way to structure computations that took my fancy. As it turned out, it would take quite a few years to understand how to use the insights and to devise efficient mechanisms to execute them.

II. 1967-69—The FLEX Machine, a first attempt at an OOP-based personal computer

Wes Clark and the LINC, ca 1962

Dave Evans was not a great believer in graduate school as an institution. As with many of the ARPA "contractors" he wanted his students to be doing "real things"; they should move through graduate school as quickly as possible; and their theses should advance the state of the art. Dave would often get consulting jobs for his students, and in early 1967, he introduced me to Ed Cheadle, a friendly hardware genius at a local aerospace company who was working on a "little machine." It was not the first personal computer—that was the LINC of Wes Clark—but Ed wanted it for noncomputer professionals, in particular, he wanted to program it in a higher level language, like BASIC. I said; "What about JOSS? It's nicer." He said: "Sure, whatever you think," and that was the start of a very pleasant collaboration we called the FLEX machine. As we got deeper into the design, we realized that we wanted to dynamically simulate and extend, neither of which JOSS (or any existing language that I knew of) was particularly good at. The machine was too small for Simula, so that was out. The beauty of JOSS was the extreme attention of its design to the end-user—in this respect, it has not been surpassed [Joss 1964, Joss 1978]. JOSS was too slow for serious computing (but cf. Lampson 65), did not have real procedures, variable scope, and so forth. A language that looked a little like JOSS but had considerably more potential power was Wirth's EULER [Wirth 1966]. This was a generalization of Algol along lines first set forth by van Wijngaarden [van Wijngaarden 1963] in which types were discarded, different features consolidated, procedures were made into first class objects, and so forth. Actually kind of LISPlike, but without the deeper insights of LISP.

But EULER was enough of "an almost new thing" to suggest that the same techniques be applied to simplify Simula. The EULER compiler was a part of its formal definition and made a simple conversion into B5000-like byte-codes. This was appealing because it suggested that Ed's little machine could run byte-codes emulated in the longish slow microcode that was then possible. The EULER compiler however, was tortuously rendered in an "extended precedence" grammar that actually required concessions in the language syntax (e.g. "," could only be used in one role because the precedence scheme had no state space). I initially adopted a bottom-up Floyd-Evans parser (adapted from Jerry Feldman's original compiler-compiler [Feldman 1977]) and later went to various top-down schemes, several of them related to Shorre's META II [Shorre 1963] that eventually put the translater in the name space of the language.

The semantics of what was now called the FLEX language needed to be influenced more by Simula than by Algol or EULER. But it was not completely clear how. Nor was it clear how the users should interact with the system. Ed had a display (for graphing, etc.) even on his first machine, and the LINC had a "glass teletype," but a Sketchpad-like system seemed far beyond the scope that we could accomplish with the maximum of 16k 16-bit words that our cost budget allowed.

Doug Engelbart and NLS

four pictures

This was in early 1967, and while we were pondering the FLEX machine, Utah was visited by Doug Engelbart. A prophet of Biblical dimensions, he was very much one of the fathers of what on the FLEX machine I had started to call "personal computing." He actually traveled with his own 16mm projector with a remote control for starting and stopping it to show what was going on (people were not used to seeing and following cursors back then). His notion on the ARPA dream was that the destiny of oNLine Systems (NLS) was the "augmentation of human intellect" via an interactive vehicle navigating through "thought vectors in concept space." What his system could do then—even by today's standards—was incredible. Not just hypertext, but graphics, multiple panes, efficient navigation and command input, interactive collaborative work, etc. An entire conceptual world and world view [Engelbart 68]. The impact of this vision was to produce in the minds of those who were "eager to be augmented" a compelling metaphor of what interactive computing should be like, and I immediately adopted many of the ideas for the FLEX machine.

In the midst of the ARPA context of human-computer symbiosis and in the presence of Ed's "little machine", Gordon Moore's "Law" again came to mind, this time with great impact. For the first time I made the leap of putting the room-sized interactive TX-2 or even a 10 MIP 6600 on a desk. I was almost frightened by the implications; computing as we knew it couldn't survive—the actual meaning of the word changed—it must have been the same kind of disorientation people had after reading Copernicus and first looked up from a different Earth to a different Heaven.

Instead of at most a few thousand institutional mainframes in the world—even today in 1992 it is estimated that there are only 4000 IBM mainframes in the entire world—and at most a few thousand users trained for each application, there would be millions of personal machines and users, mostly outside of direct institutional control. Where would the applications and training come from? Why should we expect an applications programmer to anticipate the specific needs of a particular one of the millions of potential users? An extensional system seemed to be called for in which the end-users would do most of the tailoring (and even some of the direct construction) of their tools. ARPA had already figured this out in the context of their early successes in time-sharing. Their larger metaphor of human-computer symbiosis helped the community avoid making a religion of their subgoals and kept them focused on the abstract holy grail of "augmentation."

One of the interesting features of NLS was that its user interface was parametric and could be supplied by the end user in the form of a "grammar of interaction" given in their compiler-compiler TreeMeta. This was similar to William Newman's early "Reaction Handler" [Newman 66] work in specifying interfaces by having the end-user or developer construct through tablet and stylus an iconic regular expression grammar with action procedures at the states (NLS allowed embeddings via its context free rules). This was attractive in many ways, particularly William's scheme, but to me there was a monstrous bug in this approach. Namely, these grammars forced the user to be in a system state which required getting out of before any new kind of interaction could be done. In hierarchical menus or "screens" one would have to backtrack to a master state in order to go somewhere else. What seemed to be required were states in which there was a transition arrow to every other state—not a fruitful concept in formal grammar theory. In other words, a much "flatter" interface seemed called for—but could such a thing be made interesting and rich enough to be useful?

Again, the scope of the FLEX machine was too small for a miniNLS, and we were forced to find alternate designs that would incorporate some of the power of the new ideas, and in some cases to improve them. I decided that Sketchpad's notion of a general window that viewed a larger virtual world was a better idea than restricted horizontal panes and with Ed came up with a clipping algorithm very similar to that under development at the same time by Sutherland and his students at Harvard for the 3D "virtual reality" helmet project [Sutherland 1968].

Object references were handled on the FLEX machine as a generalization of B5000 descriptors. Instead of a few formats for referencing numbers, arrays, and procedures, a FLEX descriptor contained two pointers: the first to the "master" of the object, and the second to the object instances (later we realized that we should put the master pointer in the instance to save space). A different method was taken for handling generalized assignment. The B5000 used l-values and r-values [Strachey*] which worked for some cases but couldn't handle more complex objects. For example: a[55] := 0 , if a was a sparse array whose default element was 0 would still generate an element in the array because := is an "operator" and a[55] is dereferenced into an l-value before anyone gets to see that the r-value is the default element, regardless of whether a is an array or a procedure fronting for an array. What is needed is something like: a(55, ':=', 0), which can look at all relevant operands before any store is made. In other words, := is not an operator, but a kind of index that can select a behavior from a complex object. It took me a remarkably long time to see this, partly I think because one has to invert the traditional notion of operators and functions, etc., to see that objects need to privately own all of their behaviors: that objects are a kind of mapping whose values are its behaviors . A book on logic by Carnap [Ca *] helped by showing that "intensional" definitions covered the same territory as the more traditional extensional technique and were often more intuitive and convenient.

Flex machine diagrams

As in Simula, a coroutining control structure [Conway, 1963] was used as a way to suspend and resume objects. Persistent objects like files and documents were treated as suspended processes and were organized according to their Algol-like static variable scopes. These were shown on the screen and could be opened by pointing at them. Coroutining was also used as a control structure for looping. A single operator while was used to test the generators which returned false when unable to furnish a new value. Booleans were used to link multiple generators. So a "for-type" loop would be written as: 

while i <= 1 to 30 by 2 ^ j <= 2 to k by 3 do j<-j * i;

where the ... to ... by ... was a kind of coroutine object. Many of these ideas were reimplemented in a stronger style in Smalltalk later on.

FLEX when statement [Ka 69]

Another control structure of interest in FLEX was a kind of event-driven "soft interrupt" called when . Its boolean expression was compiled into a "tournament sort" tree that cached all possible intermediate results. The relevant variables were threaded through all of the sorting trees in all of the when s so that any change only had to compute through the necessary parts of the booleans. The efficiency was very high and was similar to the techniques now used for spreadsheets. This was an embarrassment of riches with difficulties often encountered in event-driven systems. Namely, it was a complex task to control the context of just when the when s should be sensitive. Part of the boolean expression had to be used to check the contexts, where I felt that somehow the structure of the program should be able to set and unset the event drivers. This turned out to beyond the scope of the FLEX system and needed to wait for a better architecture.

Still, quite a few of the original FLEX ideas in their proto-object form did turn out to be small enough to be feasible on the machine. I was writing the first compiler when something unusual happened: the Utah graduate students got invited to the ARPA contractors meeting held that year at Alta, Utah. Towards the end of the three days, Bob Taylor, who had succeeded Ivan Sutherland as head of ARPA-IPTO, asked the graduate students (sitting in a ring around the outside of the 20 or so contractors) if they had any comments. John Warnock raised his hand and pointed out that since the ARPA grad students would all soon be colleagues (and since we did all the real work anyway), ARPA should have a contractors-type meeting each year for the grad students. Taylor thought this was a great idea and set it up for the next summer.

Another ski-lodge meeting happened in Park City later that spring. The general topic was education and it was the first time I heard Marvin Minsky speak. He put forth a terrific diatribe against traditional education methods, and from him I heard the ideas of Piaget and Papert for the first time. Marvin's talk was about how we think about complex situations and why schools are really bad places to learn these skills. He didn't have to make any claims about computers+kids to make his point. It was clear that education and learning had to be rethought in the light of 20th century cognitive psychology and how good thinkers really think. Computing enters as a new representation system with new and useful metaphors for dealing with complexity, especially of systems [Minsky 70].

The First Plasma Panel

For the summer 1968 ARPA grad students meeting at Allerton House in Illinois, I boiled all the mechanisms in the FLEX machine down into one 2'x3' chart. This included all the "object structures", the compiler, the byte-code interpreter, i/o handlers, and a simple display editor for text and graphics. The grad students were a distinguished group that did indeed become colleagues in subsequent years. My FLEX machine talk was a success, but the big whammy for me came during a tour to U of Illinois where I saw a 1" square lump of glass and neon gas in which individual spots would light up on command—it was the first flat-panel display. I spent the rest of the conference calculating just when the silicon of the FLEX machine could be put on the back of the display. According to Gordon Moore's "Law", the answer seemed to be sometime in the late seventies or early eighties. A long time off—it seemed too long to worry much about it then.

Grail, Seymour Papert and LOGO Turtle, The Dynabook Model

But later that year at RAND I saw a truly beautiful system. This was GRAIL, the graphical followon to JOSS. The first tablet (the famous RAND tablet) was invented by Tom Ellis [Davis 1964] in order to capture human gestures, and Gabe Groner wrote a program to efficiently recognize and respond to them [Groner 1966]. Though everything was fastened with bubble gum and the system crashed often, I have never forgotten my first interactions with this system. It was direct manipulation, it was analogical, it was modeless, it was beautiful. I realized that the FLEX interface was all wrong, but how could something like GRAIL be stuffed into such a tiny machine since it required all of a stand-alone 360/44 to run in?

A month later, I finally visited Seymour Papert, Wally Feurzig, Cynthia Solomon and some of the other original researchers who had built LOGO and were using it with children in the Lexington schools. Here were children doing real programming with a specially designed language and environment. As with Simula leading to OOP, this encounter finally hit me with what the destiny of personal computing really was going to be. Not a personal dynamic vehicle, as in Engelbart's metaphor opposed to the IBM "railroads", but something much more profound: a personal dynamic medium. With a vehicle one could wait until high school and give "drivers ed", but if it was a medium, it had to extend into the world of childhood.

Now the collision of the FLEX machine, the flat-screen display, GRAIL, Barton's "communications" talk, McLuhan, and Papert's work with children all came together to form an image of what a personal computer really should be. I remembered Aldus Manutius who 40 years after the printing press put the book into its modern dimensions by making it fit into saddlebags. It had to be no larger than a notebook, and needed an interface as friendly as JOSS', GRAIL's, and LOGO's, but with the reach of Simula and FLEX. A clear romantic vision has a marvelous ability to focus thought and will. Now it was easy to know what to do next. I built a cardboard model of it to see what it would look and feel like, and poured in lead pellets to see how light it would have to be (less than two pounds). I put a keyboard on it as well as a stylus because, even if hand printing and writing were recognized perfectly (and there was no reason to expect that it would be), there still needed to be a balance between the low speed tactile degrees of freedom offered by the stylus and the more limited but faster keyboard. Since ARPA was starting to experiment with packet radio, I expected that the Dynabook when it arrived a decade or so hence, would have a wireless networking system.

Early next year (1969) there was a conference on Extensible Languages in which almost every famous name in the field attended. The debate was great and weighty—it was a religious war of unimplemented poorly thought out ideas. As Alan Perlis, one of the great men in Computer Science, put it with characteristic wit:

It has been such a long time since I have seen so many familiar faces shouting among so many familiar ideas. Discovery of something new in programming languages, like any discovery, has somewhat the same sequence of emotions as falling in love. A sharp elation followed by euphoria, a feeling of uniqueness, and ultimately the wandering eye (the urge to generalize) [ACM 69].

But it was all talk—no one had done anything yet. In the midst of all this, Ned Irons got up and presented IMP, a system that had already been working for several years that was more elegant than most of the nonworking proposals. The basic idea of IMP was that you could use any phrase in the grammar as a procedure heading and write a semantic definition in terms of the language as extended so far [Irons 1970].

I had already made the first version of the FLEX machine syntax driven, but where the meaning of a phrase was defined in the more usual way as the kind of code that was emitted. This separated the compiler-extensor part of the system from the end-user. In Irons' approach, every procedure in the system defined its own syntax in a natural and useful manner. I incorporated these ideas into the second versions of the FLEX machine and started to experiment with the idea of a direct interpreter rather than a syntax directed compiler. Somewhere in all of this, I realized that the bridge to an object-based system could be in terms of each object as a syntax directed interpreter of messages sent to it. In one fell swoop this would unify object-oriented semantics with the ideal of a completely extensible language. The mental image was one of separate computers sending requests to other computers that had to be accepted and understood by the receivers before anything could happen. In today's terms every object would be a server offering services whose deployment and discretion depended entirely on the server's notion of relationship with the servee. As Liebniz said: "To get everything out of nothing, you only need to find one principle." This was not well thought out enough to do the FLEX machine any good, but formed a good point of departure for my thesis [Kay 69], which as Ivan Sutherland liked to say was "anything you can get three people to sign."

After three people signed it (Ivan was one of them), I went to the Stanford AI project and spent much more time thinking about notebook KiddyKomputers than AI. But there were two AI designs that were very intriguing. The first was Carl Hewitt's PLANNER, a programmable logic system that formed the deductive basis of Winograd's SHRDLU [Sussman 69, Hewitt 69] I designed several languages based on a combination of the pattern matching schemes of FLEX and PLANNER [Kay 70]. The second design was Pat Winston's concept formation system, a scheme for building semantic networks and comparing them to form analogies and learning processes [Winston 70]. It was kind of "object-oriented". One of its many good ideas was that the arcs of each net which served as attributes in AOV triples should themselves be modeled as nets. Thus, for example a first order arc called LEFT-OF could be asked a higher order question such as "What is your converse?" and its net could answer: RIGHT-OF. This point of view later formed the basis for Minsky's frame systems [Minsky 75]. A few years later I wished I had paid more attention to this idea.

That fall, I heard a wonderful talk by Butler Lampson about CAL-TSS, a capability-based operating system that seemed very "object-oriented" [Lampson 69]. Unforgeable pointers (ala B5000) were extended by bit-masks that restricted access to the object's internal operations. This confirmed my "objects as server" metaphor. There was also a very nice approach to exception handling which reminded me of the way failure was often handled in pattern matching systems. The only problem— which the CAL designers did not see as a problem at all—was that only certain (usually large and slow) things were "objects". Fast things and small things, etc., weren't. This needed to be fixed.

The biggest hit for me while at SAIL in late '69 was to really understand LISP. Of course, every student knew about car, cdr , and cons , but Utah was impoverished in that no one there used LISP and hence, no one had penetrated the mysteries of eval and apply . I could hardly believe how beautiful and wonderful the idea of LISP was [McCarthy 1960]. I say it this way because LISP had not only been around enough to get some honest barnacles, but worse, there were deep flaws in its logical foundations. By this, I mean that the pure language was supposed to be based on functions, but its most important components—such as lambda expressions, quotes, and conds—were not functions at all, and instead were called special forms. Landin and others had been able to get quotes and conds in terms of lambda by tricks that were variously clever and useful, but the flaw remained in the jewel. In the practical language things were better. There were not just EXPRs (which evaluated their arguments), but FEXPRs (which did not). My next question was, why on earth call it a functional language? Why not just base everything on FEXPRs and force evaluation on the receiving side when needed? I could never get a good answer, but the question was very helpful when it came time to invent Smalltalk, because this started a line of thought that said "take the hardest and most profound thing you need to do, make it great, and then build every easier thing out of it". That was the promise of LISP and the lure of lambda—needed was a better "hardest and most profound" thing. Objects should be it.

III. 1970-72—Xerox PARC: The KiddiKomp, miniCOM, and Smalltalk-71

In July 1970, Xerox, at the urging of its chief scientist Jack Goldman, decided to set up a long range research center in Palo Alto, California. In September, George Pake, the former chancellor at Washington University where Wes Clark's ARPA project was sited, hired Bob Taylor (who had left the ARPA office and was taking a sabbatical year at Utah) to start a "Computer Science Laboratory." Bob visited Palo Alto and we stayed up all night talking about it. The Mansfield Amendment was threatening to blindly muzzle the most enlightened ARPA funding in favor of directly military research, and this new opportunity looked like a promising alternative. But work for a company? He wanted me to consult and I asked for a direction. He said: follow your instincts. I immediately started working up a new version of the KiddiKomp that could be made in enough quantity to do experiments leading to the user interface design for the eventual notebook. Bob Barton liked to say that "good ideas don't often scale." He was certainly right when applied to the FLEX machine. The B5000 just didn't directly scale down into a tiny machine. Only the byte-codes did, and even these needed modification. I decided to take another look at Wes Clark's LINC, and was ready to appreciate it much more this time [Clark 1965].

KiddiKomp

I still liked pattern-directed approaches and OOP so I came up with a language design called "Simulation LOGO" or SLOGO for short (I had a feeling the first versions might run nice and slow). This was to be built into a SONY "tummy trinitron" and would use a coarse bit-map display and the FLEX machine rubber tablet as a pointing device.

Another beautiful system that I had come across was Peter Deutsch's PDP-1 LISP (implemented when he was only 15) [Deutsch 1966]. It used only 2K (18-bit words) of code and could run quite well in a 4K machine (it was its own operating system and interface). It seemed that even more could be done if the system were byte-coded, run by an architecture that was hospitable to dynamic systems, and stuck into the ever larger ROMs that were becoming available. One of the basic insights I had gotten from Seymour was that you didn't have to do a lot to make a computer an "object for thought" for children, but what you did had to be done well and be able to apply deeply.

Right after New Years 1971, Bob Taylor scored an enormous coup by attracting most of the struggling Berkeley Computer Corp to PARC. This group included Butler Lampson, Chuck Thacker, Peter Deutsch, Jim Mitchell, Dick Shoup, Willie Sue Haugeland, and Ed Fiala. Jim Mitchell urged the group to hire Ed McCreight from CM and he arrived soon after. Gary Starkweather was there already, having been thrown out of the Xerox Rochester Labs for wanting to build a laser printer (which was against the local religion). Not long after, many of Doug Englebart's people joined up—part of the reason was that they want to reimplement NLS as a distributed network system, and Doug wanted to stay with time-sharing. The group included Bill English (the co-inventor of the mouse), Jeff Rulifson, and Bill Paxton.

Almost immediately we got into trouble with Xerox when the group decided that the new lab needed a PDP-10 for continuity with the ARPA community. Xerox (which had bought SDS essentially sight unseen a few years before) was horrified at the idea of their main competitor's computer being used in the lab. They balked. The newly formed PARC group had a meeting in which it was decided that it would take about three years to do a good operating system for the XDS SIGMA-7 but that we could build "our own PDP-10" in a year. My reaction was "Holy cow!" In fact, they pulled it it off with considerable panache. MAXC was actually a microcoded emulation of the PDP-10 that used for the first time the new integrated chip memories (1K bits!) instead of core memory. Having practical in house experience with both of these new technologies was critical for the more radical systems to come.

One little incident of LISP beauty happened when Allen Newell visited PARC with his theory of hierarchical thinking and was challenged to prove it. He was given a programming problem to solve while the protocol was collected. The problem was: given a list of items, produce a list consisting of all of the odd indexed items followed by all of the even indexed items. Newell's internal programming language resembled IPL-V in which pointers are manipulated explicitly, and he got into quite a struggle to do the program. In 2 seconds I wrote down: 

oddsEvens(x) = append(odds(x), evens(x))

the statement of the problem in Landin's LISP syntax—and also the first part of the solution. Then a few seconds later: 

where odds(x) = if null(x) ∨ null(tl(x)) then x
                   else hd(x) & odds(ttl(x))
     evens(x) = if null(x) ∨ null(tl(x)) then nil
                   else odds(tl(x))

This characteristic of writing down many solutions in declarative form and have them also be the programs is part of the appeal and beauty of this kind of language. Watching a famous guy much smarter than I struggle for more than 30 minutes to not quite solve the problem his way (there was a bug) made quite an impression. It brought home to me once again that "point of view is worth 80 IQ points." I wasn't smarter but I had a much better internal thinking tool to amplify my abilities. This incident and others like it made paramount that any tool for children should have great thinking patterns and deep beauty "built-in."

Pendery Paper Display Transducer Design

Right around this time we were involved in another conflict with Xerox management, in particular with Don Pendery the head "planner". He really didn't understand what we were talking about and instead was interested in "trends" and "what was the future going to be like" and how could Xerox "defend against it." I got so upset I said to him, "Look. The best way to predict the future is to invent it. Don't worry about what all those other people might do, this is the century in which almost any clear vision can be made!" He remained unconvinced, and that led to the famous "Pendery Papers for PARC Planning Purposes," a collection of essays on various aspects of the future. Mine proposed a version of the notebook as a "Display Transducer", and Jim Mitchell's was entitled "NLS on a Minicomputer."

Bill English took me under his wing and helped me start my group as I had always been a lone wolf and had no idea how to do it. One of his suggestions was that I should make a budget. I'm afraid that I really did ask Bill, "What's a budget?" I remembered at Utah, in pre-Mansfield Amendment days, Dave Evans saying to me as he went off on a trip to ARPA, "We're almost out of money. Got to go get some more." That seemed about right to me. They give you some money. You spend it to find out what to do next. You run out. They give you some more. And so on. PARC never quite made it to that idyllic standard, but for the first half decade it came close. I needed a group because I had finally realized that I did not have all of the temperaments required to completely finish an idea. I called it the Learning Research Group (LRG) to be as vague as possible about our charter. I only hired people that got stars in their eyes when they heard about the notebook computer idea. I didn't like meetings: didn't believe brainstorming could substitute for cool sustained thought. When anyone asked me what to do, and I didn't have a strong idea, I would point at the notebook model and say, "Advance that." LRG members developed a very close relationship with each other—as Dan Ingalls was to say later: "... the rest has enfolded through the love and energy of the whole Learning Research Group." A lot of daytime was spent outside of PARC, playing tennis, bikeriding, drinking beer, eating Chinese food, and constantly talking about the Dynabook and its potential to amplify human reach and bring new ways of thinking to a faltering civilization that desperately needed it (that kind of goal was common in California in the aftermath of the sixties).

In the summer of '71 I refined the KiddiKomp idea into a tighter design called miniCOM. It used a bit-slice approach like the NOVA 1200, had a bit-map display, a pointing device, a choice of "secondary" (really tertiary) storages, and a language I now called "Smalltalk"—as in "programming should be a matter of ..." and "children should program in ...". The name was also a reaction against the "IndoEuropean god theory" where systems were named Zeus, Odin, and Thor, and hardly did anything. I figured that "Smalltalk" was so innocuous a label that if it ever did anything nice people would be pleasantly surprised.

miniCOM 
Smalltalk-71 Programs

to T 'and' :y do 'y'
to F 'and' :y do F

to 'factorial' 0 is 1
to 'factorial' :n do 'n*factorial n-1'

to 'fact' :n do 'to 'fact' n do factorial n. ^ fact n'

to :e 'is-member-of' [] do F
to :e 'is-member-of' :group
          do'if e = firstof group then T
		          else e is-member-of rest of group'

to 'cons' :x :y is self
to 'hd' ('cons' :a :b) do 'a'
to 'hd' ('cons' :a :b) '<-' :c do 'a <- c'
to 'tl' ('cons' :a :b) do 'b'
to 'tl' ('cons' :a :b) '<-' :c do 'b <- c'

to :robot 'pickup' :block
         do 'robot clear-top-of block.
		 robot hand move-to block.
		 robot hand lift block 50.
		 to 'height-of' block do 50'

This Smalltalk language (today labeled -71) was very influenced by FLEX, PLANNER, LOGO, META II, and my own derivatives from them. It was a kind of parser with object-attachment that executed tokens directly. (I think the awkward quoting conventions come from META). I was less interested in programs as algebraic patterns than I was in a clear scheme that could handle a variety of styles of programming. The patterned front-end allowed simple extension, patterns as "data" to be retrieved, a simple way to attach behaviors to objects, and a rudimentary but clear expression of its eval in terms that I thought children could understand after a few years experience with simpler programming. Program storage was sorted into a discrimination net and evaluation was straightforward pattern-matching.

As I mentioned previously, it was annoying that the surface beauty of LISP was marred by some of its key parts having to be introduced as "special forms" rather than as its supposed universal building block of functions. The actual beauty of LISP came more from the promise of its metastructures than its actual model. I spent a fair amount of time thinking about how objects could be characterized as universal computers without having to have any exceptions in the central metaphor. What seemed to be needed was complete control over what was passed in a message send; in particular when and in what environment did expressions get evaluated?

An elegant approach was suggested in a CMU thesis of Dave Fisher [Fisher 70] on the synthesis of control structures. ALGOL60 required a separate link for dynamic subroutine linking and for access to static global state. Fisher showed how a generalization of these links could be used to simulate a wide variety of control environments. One of the ways to solve the "funarg problem" of LISP is to associate the proper global state link with expressions and functions that are to be evaluated later so that the free variables referenced are the ones that were actually implied by the static form of the language. The notion of "lazy evaluation" is anticipated here as well.

Nowadays this approach would be called reflective design . Putting it together with the FLEX models suggested that all that should be required for "doing LISP right" or "doing OOP right" would be to handle the mechanics of invocations between modules without having to worry about the details of the modules themselves. The difference between LISP and OOP (or any other system) would then be what the modules could contain. A universal module (object) reference —ala B5000 and LISP—and a message holding structure—which could be virtual if the senders and receivers were sympatico— that could be used by all would do the job.

If all of the fields of a messenger structure were enumerated according to this view, we would have:

GLOBAL: the environment of the parameter values
SENDER: the sender of the message
RECEIVER: the receiver of the message
REPLY-STYLE: wait, fork, ...?
STATUS: progress of the message
REPLY: eventual result (if any)
OPERATION SELECTOR: relative to the receiver
# OF PARAMETERS:
P1:
...:
Pn:

This is a generalization of a stack frame, such as used by the B5000, and very similar to what a good intermodule scheme would require in an operating system such as CAL-TSS—a lot of state for every transaction, but useful to think about.

Much of the pondering during this state of grace (before any workable implementation) had to do with trying to understand what "beautiful" might mean with reference to object-oriented design. A subjective definition of a beautiful thing is fairly easy but is not of much help: we think a thing beautiful because it evokes certain emotions. The cliche has it lie "in the eye of the beholder" so that it is difficult to think of beauty as other than a relation between subject and object in which the predispositions of the subject are all important.

If there are such a thing as universally appealing forms then we can perhaps look to our shared biological heritage for the predispositions. But, for an object like LISP, it is almost certain that most of the basis of our judgment is learned and has much to do with other related areas that we think are beautiful, such as much of mathematics.

One part of the perceived beauty of mathematics has to do with a wondrous synergy between parsimony, generality, enlightenment, and finesse. For example, the Pythagorean Theorem is expressible in a single line, is true for all of the infinite number of right triangles, is incredibly useful in understanding many other relationships, and can be shown by a few simple but profound steps.

When we turn to the various languages for specifying computations we find many to be general and a few to be parsimonious. For example, we can define universal machine languages in just a few instructions that can specify anything that can be computed. But most of these we would not call beautiful, in part because the amount and kind of code that has to be written to do anything interesting is so contrived and turgid. A simple and small system that can do interesting things also needs a "high slope"—that is a good match between the degree of interestingness and the level of complexity needed to express it.

A fertilized egg that can transform itself into the myriad of specializations needed to make a complex organism has parsimony, generality, enlightenment, and finesse—in short, beauty, and a beauty much more in line with my own esthetics. I mean by this that Nature is wonderful both at elegance and practicality—the cell membrane is partly there to allow useful evolutionary kludges to do their necessary work and still be able act as component by presenting a uniform interface to the world.

One of my continual worries at this time was about the size of the bit-map display. Even if a mixed mode was used (between fine-grained generated characters and coarse-grained general bit-map for graphics) it would be hard to get enough information on the screen. It occurred to me (in a shower, my favorite place to think) that FLEXtype windows on a bit-map display could be made to appear as overlapping documents on a desktop. When an overlapped one was refreshed it would appear to come to the top of the stack. At the time, this did not appear as the wonderful solution to the problem but it did have the effect of magnifying the effective area of the display enormously, so I decided to go with it.

To investigate the use of video as a display medium, Bill English and Butler Lampson specified an experimental character generator (built by Roger Bates) for the POLOS (PARC OnLine Office System) terminals. Gary Starkweather had just gotten the first laser printer to work and we ran a coax over to his lab to feed him some text to print. The "SLOT machine" (Scanning Laser Output Terminal) was incredible. The only Xerox copier Gary could get to work on went at 1 page a second and could not be slowed down. So Gary just made the laser run at that rate with a resolution of 500 pixels to the inch!

The character generator's font memory turned out to be large enough to simulate a bit-map display if one displayed a fixed "strike" and wrote into the font memory. Ben Laws built a beautiful font editor and he and I spent several months learning about the peculiarities of the human visual system (it is decidedly non-linear). I was very interested in high-quality text and graphical presentations because I thought it would be easier to get the Dynabook into schools as a "trojan horse" by simply replacing school books rather than to try to explain to teachers and school boards what was really great about personal computing.

The Old Character Generator — early 1972

Things were generally going well all over the lab until May of 72 when I tried to get resources to build a few miniCOMs. A relatively new executive ("X") did not want to give them to me. I wrote a memo explaining why the system was a good idea (see Appendix II), and then had a meeting to discuss it. "X" shot it down completely saying among other things that we had used too many green stamps getting Xerox to fund the time-shared MAXC and this use of resources for personal machines would confuse them. I was shocked. I crawled away back to the experimental character generator and made a plan to get 4 more made and hooked to NOVAs for the initial kid experiments.

I got Steve Purcell, a summer student from Stanford, to build my design for bit-map painting so the kids could sketch as well as display computer graphics. John Shoch built a line drawing and gesture recognition system (based on Ledeen's [Newman and Sproull 72]) that was integrated with the painting. Bill Duvall of POLOS built a miniNLS that was quite remarkable in its speed and power. The first overlapping windows started to appear. Bob Shur (with Steve Purcell's help) built a 2½D animation system. Along with Ben Laws' font editor, we could give quite a smashing demo of what we intended to build for real over the next few years. I remember giving one of these to a Xerox executive, including doing a portrait of him in the new painting system, and wound it up with a flourish declaring: "And what's really great about this is that it only has a 20% chance of success. We're taking risk just like you asked us to!" He looked me straight in the eye and said, "Boy, that's great, but just make sure it works." This was a typical executive notion about risk. He wanted us to be in the "20%" one hundred percent of the time.

The First Painting System — Summer '72

That summer while licking my wounds and getting the demo simulations built and going, Butler Lampson, Peter Deutsch, and I worked out a general scheme for emulated HLL machine languages. I liked the B5000 scheme, but Butler did not want to have to decode bytes, and pointed out that since an 8-bit byte had 256 total possibilities, what we should do is map different meanings onto different parts of the "instruction space." This would give us a "poor man's Huffman code" that would be both flexible and simple. All subsequent emulators at PARC used this general scheme.

I also took another pass at the language for the kids. Jeff Rulifson was a big fan of Piaget (and semiotics) and we had many discussions about the "stages" and what iconic thinking might be about. After reading Piaget and especially Jerome Bruner, I was worried that the directly symbolic approach taken by FLEX, LOGO (and the current Smalltalk) would be difficult for the kids to process since evidence existed that the symbolic stage (or mentality) was just starting to switch on. In fact, all of the educators that I admired (including Montessori, Holt, and Suzuki) all seemed to call for a more figurative, more iconic approach. Rudolph Arnheim [Arnheim 69] had written a classic book about visual thinking, and so had the eminent art critic Gombrich [Gombrich **]. It really seemed that something better needed to be done here. GRAIL wasn't it, because its use of imagery was to portray and edit flowcharts, which seemed like a great step backwards. But Rovner's AMBIT-G held considerably more promise [Rovner 68]. It was kind of a visual SNOBOL [Farber 63] and the pattern matching ideas looked like they would work for the more PLANNERlike scheme I was using.

Bill English was still encouraging me to do more reasonable appearing things to get higher credibility, like making budgets, writing plans and milestone notes, so I wrote a plan that proposed over the next few years that we would build a real system on the character generators cum NOVAs that would involve OOP, windows, painting, music, animation, and "iconic programming." The latter was deemed to be hard and would be handled by the usual method for hard problems, namely, give them to grad students.

Children with Dynabooks from 'A Personal Computer For Children Of All Ages [Ka 72], Iconic Bubble Sort from 1972 LRG Plan [Ka 72a]

IV. 1972-76—The first real Smalltalk (-72), its birth, applications, and improvements

In Sept, within a few weeks of each other, two bets happened that changed most of my plans. First, Butler and Chuck came over and asked: "Do you have any money?" I said, "Yes, about $230K for NOVAs and CGs. Why?" They said, "How would you like us to build your little machine for you?" I said, "I'd like it fine. What is it?" Butler said: "I want a '$500 PDP-10', Chuck wants a '10 times faster NOVA', and you want a 'kiddicomp'. What do you need on it?" I told them most of the results we had gotten from the fonts, painting, resolution, animation, and music studies. I asked where this had come from all of a sudden and Butler told me that they wanted to do it anyway, that Executive "X" was away for a few months on a "task force" so maybe they could "Sneak it in", and that Chuck had a bet with Bill Vitic that he could do a whole machine in just 3 months. "Oh," I said.

The second bet had even more surprising results. I had expected that the new Smalltalk would be an iconic language and would take at least two years to invent, but fate intervened. One day, in a typical PARC hallway bullsession, Ted Kaehler, Dan Ingalls, and I were standing around talking about programming languages. The subject of power came up and the two of them wondered how large a language one would have to make to get great power. With as much panache as I could muster, I asserted that you could define the "most powerful language in the world" in "a page of code." They said, "Put up or shut up."

Ted went back to CMU but Dan was still around egging me on. For the next two weeks I got to PARC every morning at four o'clock and worked on the problem until eight, when Dan, joined by Henry Fuchs, John Shoch, and Steve Purcell showed up to kibbitz the morning's work.

I had originally made the boast because McCarthy's self-describing LISP interpreter was written in itself. It was about "a page", and as far as power goes, LISP was the whole nine-yards for functional languages. I was quite sure I could do the same for object-oriented languages plus be able to do a reasonable syntax for the code a la some of the FLEX machine techniques.

It turned out to be more difficult than I had first thought for three reasons. First, I wanted the program to be more like McCarthy's second non-recursive interpreter—the one implemented as a loop that tried to resemble the original 709 implementation of Steve Russell as much as possible. It was more "real". Second, the intertwining of the "parsing" with message receipt—the evaluation of parameters which was handled separately in LISP—required that my object-oriented interpreter re-enter itself "sooner" (in fact, much sooner) than LISP required. And, finally, I was still not clear how send and receive should work with each other.

The first few versions had flaws that were soundly criticized by the group. But by morning 8 or so, a version appeared that seemed to work (see Appendix III for a sketch of how the interpreter was designed). The major differences from the official Smalltalk-72 of a little bit later were that in the first version symbols were byte-coded and the receiving of return-values from a send was symmetric—i.e. receipt could be like parameter binding—this was particular useful for the return of multiple values. For various reasons, this was abandoned in favor of a more expression-oriented functional return style.

Of course, I had gone to considerable pains to avoid doing any "real work" for the bet, but I felt I had proved my point. This had been an interesting holiday from our official "iconic programming" pursuits, and I thought that would be the end of it. Much to my surprise, only a few days later, Dan Ingalls showed me the scheme working on the NOVA. He had coded it up (in BASIC!), added a lot of details, such as a token scanner, a list maker, etc., and there it was—running. As he liked to say: "You just do it and it's done."

It evaluated 3+4 v e r y  s l o w l y (it was "glacial", as Butler liked to say) but the answer always came out 7. Well, there was nothing to do but keep going. Dan loved to bootstrap on a system that "always ran," and over the next ten years he made at least 80 major releases of various flavors of Smalltalk.

In November, I presented these ideas and a demonstration of the interpretation scheme to the MIT AI lab. This eventually led to Carl Hewitt's more formal "Actor" approach [Hewitt 73]. In the first Actor paper the resemblance to Smalltalk is at its closest. The paths later diverged, partly because we were much more interested in making things than theorizing, and partly because we had something no one else had: Chuck Thacker's Interim Dynabook (later known as the "ALTO").

Just before Chuck started work on the machine I gave a paper to the National Council of Teachers of English [Kay 72c] on the Dynabook and its potential as a learning and thinking amplifier—the paper was an extensive rotogravure of "20 things to do with a Dynabook" [Kay 72c]. By the time I got back from Minnesota, Stewart Brand's Rolling Stone article about PARC [Brand 1972] and the surrounding hacker community had hit the stands. To our enormous surprise it caused a major furor at Xerox headquarters in Stamford, Connecticut. Though it was a wonderful article that really caught the spirit of the whole culture, Xerox went berserk, forced us to wear badges (over the years many were printed on t-shirts), and severely restricted the kinds of publications that could be made. This was particularly disastrous for LRG, since we were the "lunatic fringe" (so-called by the other computer scientists), were planning to go out to the schools, and needed to share our ideas (and programs) with our colleagues such as Seymour Papert and Don Norman.

Executive "X" apparently heard some harsh words at Stamford about us, because when he returned around Christmas and found out about the interim Dynabook, he got even more angry and tried to kill it. Butler wound up writing a masterful defense of the machine to hold him off, and he went back to his "task force."

BILBO, the first 'interim Dynabook', and Cookie Monster', the first graphics it displayed. April, 1973

Chuck had started his "bet" on November 22, 1972. He and two technicians did all of the machine except for the disk interface which was done by Ed McCreight. It had a ~500,000 pixel (606x808) bitmap display, its microcode instruction rate was about 6 MIPS, it had a grand total of 128k, and the entire machine (exclusive of the memory) was rendered in 160 MSI chips distributed on two cards. It was beautiful [Thacker 1972, 1986]. One of the wonderful features of the machine was "zero-over-head" tasking. It had 16 program counters, one for each task. Condition flags were tied to interesting events (such as "horizontal retrace pulse", and "disk sector pulse", etc.). Lookaside logic scanned the flags while the current instruction was executing and picked the highest priority program counter to fetch from next. The machine never had to wait, and the result was that most hardware functions (particularly those that involved i/o (like feeding the display and handling the disk) could be replaced by microcode. Even the refresh of the MOS dynamic RAM was done by a task. In other words, this was a coroutine architecture. Chuck claimed that he got the idea from a lecture I had given on coroutines a few months before, but I remembered that Wes Clark's TX-2 (the Sketchpad machine) had used the idea first, and I probably mentioned that in the talk.

In early April, just a little over three months from the start, the first Interim Dynabook, known as 'Bilbo,' greeted the world and we had the first bit-map picture on the screen within minutes: the Muppets' Cookie Monster that I had sketched on our painting system.

Soon Dan had bootstrapped Smalltalk across, and for many months it was the sole software system to run on the Interim Dynabook. Appendix I has an "acknowledgements" document I wrote from this time that is interesting in its allocation of credits and the various priorities associated with them. My $230K was enough to get 15 of the original projected 30 machines (over the years some 2000 Interim Dynabooks were actually built). True to Schopenhauer's observation, Executive "X" now decided that the Interim Dynabook was a good idea and he wanted all but two for his lab (I was in the other lab). I had to go to considerable lengths to get our machines back, but finally succeeded.

  1. Everything is an object
  2. Objects communicate by sending and receiving messages (in terms of objects)
  3. Objects have their own memory (in terms of objects)
  4. Every object is an instance of a class (which must be an object)
  5. The class holds the shared behavior for its instances (in the form of objects in a program list)
  6. To eval a program list, control is passed to the first object and the remainder is treated as its message

By this time most of Smalltalk's schemes had been sorted out into six main ideas that were in accord with the initial premises in designing the interpreter. The first three principles are what objects "are about"—how they are seen and used from "the outside." These did not require any modification over the years. The last three—objects from the inside—were tinkered with in every version of Smalltalk (and in subsequent OOP designs). In this scheme (1 & 4) imply that classes are objects and that they must be instances of themself. (6) implies a LISPlike universal syntax, but with the receiving object as the first item followed by the message. Thus c i <- de (with subscripting rendered as "○" and multiplication as "*") means:

receiver | message
c | ○ i <- d*e

The c is bound to the receiving object, and all of ○ i <- d*e is the message to it. The message is made up of literal token "○", an expression to be evaluated in the sender's context (in this case i), another literal token <-, followed by an expression to be evaluated in the sender's context (d*e). Since "LISP" pairs are made from 2 element objects they can be indexed more simply: c hd , c tl , and c hd <- foo , etc.

"Simple" expressions like a+b and 3+4 seemed more troublesome at first. Did it really make sense to think of them as:

receiver | message
a | + b
3 | + 4

It seemed silly if only integers were considered, but there are many other metaphoric readings of "+", such as:

"kitty" |+ "kat" => "kittykat" [3 4 5 6 7 8] |+ 4 = [7 8 9 10 11 12]

This led to a style of finding generic behaviors for message symbols. "Polymorphism" is the official term (I believe derived from Strachey), but it is not really apt as its original meaning applied only to functions that could take more than one type of argument. An example class of objects in Smalltalk-72, such as a model of CONS pairs, would look like:

Model of CONS pairs

Since control is passed to the class before any of the rest of the message is considered—the class can decide not to receive at its discretion—complete protection is retained. Smalltalk-72 objects are "shiny" and impervious to attack. Part of the environment is the binding of the SENDER in the "messenger object" (a generalized activation record) which allows the receiver to determine differential privileges (see Appendix II for more details). This looked ahead to the eventual use of Smalltalk as a network OS (See [Goldstein & Bobrow 1980]), and I don't recall it being used very much in Smalltalk-72.

One of the styles retained from Smalltalk-71 was the comingling of function and class ideas. In other works, Smalltalk-72 classes looked like and could be used as functions, but it was easy to produce an instance (a kind of closure) by using the object ISNEW. Thus factorial could be written "extensionally" as:

to fact n (^if :n=0 then 1 else n*fact n-1)

or "intensionally," as part of class integer:

(... ○! » (^:n=1) » (1) (n-1)!)

Of course, the whole idea of Smalltalk (and OOP in general) is to define everything intensionally . And this was the direction of movement as we learned how to program in the new style. I never liked this syntax (too many parentheses and nestings) and wanted something flatter and more grammar-like as in Smalltalk-71. To the right is an example syntax from the notes of a talk I gave around then. We will see something more like this a few years later in Dan's design for Smalltalk-76. I think something similar happened with LISP—that the "reality" of the straightforward and practical syntax you could program in prevailed against the flights of fancy that never quite got built.

Proposed Smalltalk-72 Syntax 

Pair :h :t
   hd <- :h
   hd              » h
   tl <- :t
   tl              » t
   isPair          » true
   print           » '( print. SELF mprint.
   mprint          » h print. if t isNil then ') print
                                else if t isPair then t mprint
                                else '● print. t print. ') print
   length          » 1 + if t isList then t length else 0

Development of the Smalltalk-72 System and Applications

One of the 'first build' ALTOs, Early Smalltalk Windows on Interim Dynabook, Turtles

The advent of a real Smalltalk on a real machine started off an explosion of parallel paths that are too difficult to intertwine in strict historical order. Let me first present the general development of the Smalltalk-72 system up to the transition to Smalltalk-76, and then follow that with the several years of work with children that were the primary motivation for the project. The Smalltalk-72 interpreter on the Interim Dynabook was not exactly zippy ("majestic" was Butler's pronouncement), but was easy to change and quite fast enough for many real-time interactive systems to be built in it.

Overlapping windows were the first project tackled (with Diana Merry) after writing the code to read the keyboard and create a string of text. Diana built an early version of a bit field block transfer (bitblt) for displaying variable pitch fonts and generally writing on the display. The first window versions were done as real 2½D draggable objects that were just a little too slow to be useful. We decided to wait until Steve Purcell got his animation system going to do it right, and opted for the style that is still in use today, which is more like "2¼D". Windows were perhaps the most redesigned and reimplemented class in Smalltalk because we didn't quite have enough compute power to just do the continual viewing to "world coordinates" and refreshing that my former Utah colleagues were starting to experiment with on the flight simulator projects at Evans & Sutherland. This is a simple, powerful model but it is difficult to do in real-time even in 2½D. The first practical windows in Smalltalk used the GRAIL conventions of sensitive corners for moving, resizing, cloning, and closing. Window scheduling used a simple "loopless" control scheme that threaded all of the windows together.

One of the next classes to be implemented on the Interim Dynabook (after the basics of numbers, strings, etc.) was an object-oriented version of the LOGO turtle implemented by Ted. This could make many turtle instances that were used both for drawing and as a kind of value for graphics transformations. Dan created a class of "commander" turtles that could control a troop of turtles. Soon the turtles were made so they could be clipped by the windows.

John Shoch built a mouse-driven structured editor for Smalltalk code.


Findit Retrieval By Example, Retrieved HyperDocument, FM Timbre Editor, OPUS Score Capture

Larry Tesler (then working for POLOS) did not like the modiness and general approach of NLS, and he wanted both to show the former NLSers an alternative and to conduct some user studies (almost unheard of in those days) about editing. This led to his programming miniMOUSE in Smalltalk, the first real WYSIWYG galley editor at PARC. It was modeless (almost) and fun to use, not just for us but for the many people he tested it on (I ran the camera for the movies we took and remember their delight and enjoyment). miniMOUSE quickly became an alternate editor for Smalltalk code and some of the best demos we ever gave used it.

One of the "small program" projects I tried on an adult class in the Spring of '74 was a one-page paragraph editor. It turned out to be too complicated, but the example I did to show them was completely modeless (it was in the air) and became the basis for much of the Smalltalk text work over the next few years. Most of the improvements were made by Dan and Diana Merry. Of course, objects mean multi-media documents, you almost get them for free. Early on we realized that in such a document, each component object should handle its own editing chores. Steve Weyer built some of the earliest multi-media documents, whose range was greatly and variously expanded over the years by Bob Flegal, Diana Merry, Larry Tesler, Tim Mott, and Trygve Reenskaug.

Steve Weyer and I devised Findit , a "retrieval by example" interface that used the analogy of classes to their instances to form retrieval requests. This was used for many years by the PARC library to control circulation.

The sampling synthesis music I had developed on the NOVA could generate 3 high-quality real-time voices. Bob Shur and Chuck Thacker transferred the scheme to the Interim Dynabook and achieved 12 voices in real-time. The 256 bit generalized input that we had specified for low speed devices (used for the mouse and keyboard) made it easy to connect 154 more to wire up two organ keyboards and a pedal. Effects such as portamento and decay were programmed. Ted Kaehler wrote TWANG, a music capture and editing system, using a tablature notation that we devised to make music clear to children [Kay 1977a]. One of the things that was hard to do with sampling was the voltage controlled oscillator (VCO) effects that were popular on the "Well Tempered Synthesizer." A summer later, Steve Saunders, another of our bright summer students, was challenged to find a way to accomplish John Chowning's very non-real-time FM synthesis in real-time on the ID. He had to find a completely different way to think of it than "FM", and succeeded brilliantly with 8 real-time voices that were integrated into TWANG [Saunders *].


Shazam iconic user interface, PYGMALION Iconic Programming, Simpula Hospital Simulation

Chris Jeffers (who was a musician and educator, not a computer scientist) knocked us out with OPUS, the first real-time score capturing system. Unlike most systems today it did not require metronomic playing but instead took a first pass looking for strong and weak beats (the phrasing) to establish a local model of the likely tempo fluctuations and then used curve fitting and extrapolation to make judgments about just where in the measure, and for what time value, a given note had been struck.

The animations on the NOVA ran 3-5 objects at about 2-3 frames per second. Fast enough for the phi phenomenon to work (if double buffering was used), but we wanted "Disney rates" of 10-15 frames a second for 10 or more large objects and many more smaller ones. This task was put into the ingenious hands of Steve Purcell. By the fall of '73 he could demo 80 ping-pong balls and 10 flying horses running at 10 frames per second in 2½D. His next task was to make the demo into a general systems facility from which we could construct animation systems. His CHAOS system started working in May '74, just in time for summer visitors Ron Baecker, Tom Horseeley, and professional animator Eric Martin to visit and build SHAZAM a marvelously capable and simple animation system based on Ron's GENESYS thesis project on the TX-2 in the late sixties [Baecker 69].

The main thesis project during this time was Dave Smith's PYGMALION [Smith 75], an essay into iconic programming (no, we hadn't quite forgotten). One programmed by showing the system how changes should be made, much as one would illustrate on a blackboard with another programmer. This program became the starting place from which many subsequent "programming by example" systems took off.

I should say something about the size of these programs. PYGMALION was the largest program ever written in Smalltalk-72. It was about 20 pages of code—all that would fit in the interim dynabook ALTO—and is given in full in Smith's thesis. All of the other applications were smaller. For example, the SHAZAM animation system was written and revised several times in the summer of 1974, and finally wound up as a 5-6 page application which included its icon-controlled multiwindowed user interface.

Given its roots in simulation languages, it was easy to write in a few pages, Simpula, a simple version of the SIMULA sequencing set approach to scheduling. By this time we had decided that coroutines could be rendered more cleanly by scheduling individual methods as separate simulation phases. The generic SIMULA example was a job shop. This could be generalized into many useful forms such as a hospital with departments of resources serving patients (see to the right). The children did not care for hospitals but saw that they could model amusement parks, like Disneyland, their schools, the stores they and their parents shopped in, and so forth. Later this model formed the basis of the Smalltalk Sim-kit, a high-level end-user programming environment (described ahead). 


(until Return or Delete do
    ('character <- display <- keyboard.
    character = ret » (Return)
    character = del » (Delete)
    )
then case
    Return: ('deal with this normal exit')
    Delete: ('handle the abnormal exit'))

Many nice "computer sciency" constructs were easy to make in Smalltalk-72. For example, one of the controversies of the day was whether to have gotos or not (we didn't), and if not, how could certain very useful control structures—such as multiple exits from a loop—be specified? Chuck Zahn at SLAC proposed an event-driven case structure in which a set of events could be defined so that when an event is encountered, the loop will be exited and the event will select a statement in a case block [Zahn 1974, Knuth 1974]. Suppose we want to write a simple loop that reads characters from the keyboard and outputs them to a display. We want it to exit normally when the <return> key is struck and with an error if the <delete> key is hit. Appendix IV shows how John Shoch defined this control structure.

The Evolution of Smalltalk-72

Smalltalk-74 (sometimes known as FastTalk) was a version of Smalltalk-72 incorporating major improvements which included providing a real "messenger" object, message dictionaries for classes (a step towards real class objects), Diana Merry's bitblt (the now famous 2D graphics operator for bitmap graphics) redesigned by Dan and implemented in microcode, and a better, more general window interface. Dave Robson while a student at UC Irvine had heard of our project and made a pretty good stab at implementing an OOPL. We invited him for a summer and never let him go back—he was a great help in formulating an official semantics for Smalltalk.

The crowning addition was the OOZE (Object Oriented Zoned Environment) virtual memory system that served Smalltalk-74, and more importantly, Smalltalk-76 [Ing 78, Kae *]. The ALTO was not very large (128-256K), especially with its page-sized display (64k), and even with small programs, we soon ran out of storage. The 2.4 megabyte model 30 disk drive was faster and larger than a floppy and slower and smaller than today's hard drives. It was quite similar to the HP direct contact disk of the FLEX machine on which I had tried a fine-grain version of the B5000 segment swapper. It had not worked as well as I wanted, despite a few good ideas as to how to choose objects when purging. When the gang wanted to adopt this basic scheme, I said: "But I never got it to work well." I remember Ted Kaehler saying, "Don't worry, we'll make it work!"

The basic idea in all of these systems is to be able to gather the most comprehensive possible working set of objects. This is most easily accomplished by swapping individual objects. Now the problem becomes the overhead of purging non-working set objects to make room for the ones that are needed. (Paging sometimes works better for this part because you can get more than one object (OOZE) in each disk touch.) Two ideas help a lot. First, Butler's insight in the GENIE OS that it was worthwhile to expend a small percentage of time purging dirty objects to make core as clean as possible [Lampson 1966]. Thus crashes tend not to hurt as much and there is always clean storage to fetch pages or objects from the disk into. The other is one from the FLEX system in which I set up a stochastic decision mechanism (based on the class of an object) that determined during a purge whether or not to throw an object out. This had two benefits: important objects tended not to go out, and a mistake would just bring it back in again with the distribution insuring a low probability that the object would be purged again soon.

The other problem that had to be taken care of was object-pointer integrity (and this is where I had failed in the FLEX machine to come up with a good enough solution). What was needed really was a complete transaction, a brand new technique (thought up by Butler?) that ensured recovery regardless of when the system crashed. This was called "cosmic ray protection" as the early ALTOS had a way of just crashing once or twice a day for no discernible good reason. This, by the way did not particularly bother anyone as it was fairly easy to come up with undo and replay mechanisms to get around the cosmic rays. For pointer-based systems that had automatic storage management, this was a bit more tricky.

Ted and Dan decided to control storage using a Resident Object Table that was the only place machine addresses for objects would be found. Other useful information was stashed there as well to help LRU aging. Purging was done in background by picking a class, positioning the disk to its instances (all of a particular class were stored together), then running through the ROT to find the dirty ones in storage and stream them out. This was pretty efficient and, true to Butler's insight, furnished a good sized pool of clean storage that could be overwritten. The key to the design though (and the implementation of the transaction mechanism) was the checkpointing scheme they came up with. This insured that there was a recoverable image no more than a few seconds old, regardless of when a crash might occur. OOZE swapped objects in just 80kb of working storage and could handle about 65K objects (up to several megabytes worth, more than enough for the entire system, its interface, and its applications).

"Object-oriented" Style

This is probably a good place to comment on the difference between what we thought of as OOP-style and the superficial encapsulation called "abstract data types" that was just starting to be investigated in academic circles. Our early "LISP-pair" definition is an example of an abstract data type because it preserves the "field access" and "field rebinding" that is the hallmark of a data structure. Considerable work in the 60s was concerned with generalizing such structures [DSP *]. The "official" computer science world started to regard Simula as a possible vehicle for defining abstract data types (even by one of its inventors [Dahl 1970]), and it formed much of the later backbone of ADA. This led to the ubiquitous stack data-type example in hundreds of papers. To put it mildly, we were quite amazed at this, since to us, what Simula had whispered was something much stronger than simply reimplementing a weak and ad hoc idea. What I got from Simula was that you could now replace bindings and assignment with goals . The last thing you wanted any programmer to do is mess with internal state even if presented figuratively. Instead, the objects should be presented as sites of higher level behaviors more appropriate for use as dynamic components .

Even the way we taught children (cf. ahead) reflected this way of looking at objects. Not too surprisingly this approach has considerable bearing on the ease of programming, the size of the code needed, the integrity of the design, etc. It is unfortunate that much of what is called "object-oriented programming" today is simply old style programming with fancier constructs. Many programs are loaded with "assignment-style" operations now done by more expensive attached procedures.

Where does the special efficiency of object-oriented design come from? This is a good question given that it can be viewed as a slightly different way to apply procedures to data-structures. Part of the effect comes from a much clearer way to represent a complex system. Here, the constraints are as useful as the generalities. Four techniques used together—persistent state, polymorphism, instantiation, and methods-as-goals for the object—account for much of the power. None of these require an "object-oriented language" to be employed—ALGOL 68 can almost be turned to this style—an OOPL merely focuses the designer's mind in a particular fruitful direction. However, doing encapsulation right is a commitment not just to abstraction of state, but to eliminate state oriented metaphors from programming.

Perhaps the most important principle—again derived from operating system architectures—is that when you give someone a structure, rarely do you want them to have unlimited privileges with it. Just doing type-matching isn't even close to what's needed. Nor is it terribly useful to have some objects protected and others not. Make them all first class citizens and protect all.

I believe that the much smaller size of a good OOP system comes not just by being gently forced to come up with a more thought out design. I think it also has to do with the "bang per line of code" you can get with OOP. The object carries with it a lot of significance and intention, its methods suggest the strongest kinds of goals it can carry out, its superclasses can add up to much more code-functionality being invoked than most procedures-on-data-structures. Assignment statements—even abstract ones—express very low-level goals, and more of them will be needed to get anything done. Generally, we don't want the programmer to be messing around with state, whether simulated or not. The ability to instantiate an object has a considerable effect on code size as well. Another way to think of all this is: though the late-binding of automatic storage allocations doesn't do anything a programmer can't do, its presence leads both to simpler and more powerful code. OOP is a late binding strategy for many things and all of them together hold off fragility and size explosion much longer than the older methodologies. In other words, human programmers aren't Turing machines—and the less their programming systems require Turing machine techniques the better.

Smalltalk and Children

Adele holding forth at Jordan Middle Sch

Now that I have summarized the "adult" activities (we were actually only semiadults) in Smalltalk up to 1976, let me return to the summer of '73, when we were ready to start experiments with children. None of us knew anything about working with children, but we knew that Adele Goldberg and Steve Weyer who were then with Pat Suppes at Stanford had done quite a bit and we were able to entice them to join us.

Since we had no idea how to teach object-oriented programming to children (or anyone else), the first experiments Adele did mimicked LOGO turtle graphics, and she got what appeared to be very similar results. That is to say, the children could get the turtle to draw pictures on the screen, but there seemed to be little happening beyond surface effects. At that time I felt that since the content of personal computing was interactive tools, that the content of this new kind of authoring literacy should be the creation of interactive tools by the children. Procedural turtle graphics just wasn't it.

Then Adele came up with a brilliant approach to teaching Smalltalk as an object-oriented language: the "Joe Book." I believe this was partly influenced by Minsky's idea that you should teach a programming language holistically from working examples of serious programs.

Several instances of the class box are created and sent messages, culminating with a simple multiprocess animation. After getting kids to guess what a box might be like—they could come surprisingly close—they would be shown: 

to box | x y size tilt
(○draw   »    (@place x y turn tilt. square size.
○undraw  »    (@ white. SELF draw. @black)
○turn    »    (SELF undraw. 'tilt <- tilt + :. SELF draw)
○grow    »    (SELF undraw. 'size <- size + :. SELF draw)
ISNEW    »    (SELF undraw. 'size <- size + :. SELF draw)

What was so wonderful about this idea were the myriad of children's projects that could spring off the humble boxes. And some of the earliest were tools! This was when we got really excited. For example, Marian Goldeen's (12 yrs old) painting system was a full-fledged tool. A few years later, so was Susan Hamet's (12 yrs old) OOP illustration system (with a design that was like the MacDraw to come). Two more were Bruce Horn's (15 yrs old) music score capture system and Steve Putz's (15 yrs old) circuit design system. Looking back, this could be called another example in computer science of the "early success syndrome." The successes were real, but they weren't as general as we thought. They wouldn't extend into the future as strongly as we hoped. The children were chosen from the Palo Alto schools (hardly an average background) and we tended to be much more excited about the successes than the difficulties. In part, what we were seeing was the "hacker phenomenon," that, for any given pursuit, a particular 5% of the population will jump into it naturally, while the 80% or so who can learn it in time do not find it at all natural.

We had a dim sense of this, but we kept on having relative successes. We could definitely see that learning the mechanics of the system was not a major problem. The children could get most of it themselves by swarming over the ALTOS with Adele's JOE book. The problem seemed more to be that of design.


The author in the interim Dynabook playroom, Adele's planning template for Smalltalk (above) New behavior added by child (below), Marian Goldeen's painting program (above) Susan Hamel's OO Illustrator (below)

It started to hit home in the Spring of '74 after I taught Smalltalk to 20 PARC nonprogrammer adults. They were able to get through the initial material faster than the children, but just as it looked like an overwhelming success was at hand, they started to crash on problems that didn't look to me to be much harder than the ones they had just been doing well on. One of them was a project thought up by one of the adults, which was to make a little database system that could act like a card file or rolodex. They couldn't even come close to programming it. I was very surprised because I "knew" that such a project was well below the mythical "two pages" for end-users we were working within. That night I wrote it out, and the next day I showed all of them how to do it. Still, none of them were able to do it by themselves. Later, I sat in the room pondering the board from my talk. Finally, I counted the number of nonobvious ideas in this little program. They came to 17. And some of them were like the concept of the arch in building design: very hard to discover, if you don't already know them.

The connection to literacy was painfully clear. It isn't enough to just learn to read and write. There is also a literature that renders ideas. Language is used to read and write about them, but at some point the organization of ideas starts to dominate mere language abilities. And it helps greatly to have some powerful ideas under one's belt to better acquire more powerful ideas [Papert 70s]. So, we decided we should teach design . And Adele came up with another brilliant stroke to deal with this. She decided that what was needed was an intermediary between the vague ideas about the problem and the very detailed writing and debugging that had to be done to get it to run in Smalltalk. She called the intermediary forms design templates .

Using these the children could look at a situation they wanted to simulate, and decompose it into classes and messages without having to worry just how a method would work. The method planning could then be done informally in English, and these notes would later serve as commentaries and guides to the writing of the actual code. This was a terrific idea, and it worked very well.

But not enough to satisfy us. As Adele liked to point out, it is hard to claim success if only some of the children are successful— and if a maximum effort of both children and teachers was required to get the successes to happen. Real pedagogy has to work in much less idealistic settings and be considerably more robust. Still, some successes are qualitatively different from no successes. We wanted more, and started to push on the inheritance idea as a way to let novices build on frameworks that could only be designed by experts. We had good reason to believe that this could work because we had been impressed by Lisa van Stone's ability to make significant changes to SHAZAM (the five or six page Smalltalk animation tool done by relatively expert adults). Unfortunately, inheritance—though an incredibly powerful technique—has turned out to be very difficult for novices (and even professionals) to deal with.


Circuit system by Steve Putz (age 15), Tangram designs are created by selecting shapes from a 'menu' displayed at the top of the screen. This system was implemented in Smalltalk by a fourteen year old girl [Kay 77]

At this point, let me do a look back from the vantage point of today. I'm now pretty much convinced that our design template approach was a good one after all. We just didn't apply it longitudinally enough. I mean by this that there is now a large accumulation of results from many attempts to teach novices programming [Soloway 1989]. They all have similar stories that seem to have little to do with the various features of the programming languages used, and everything to do with the difficulties novices have thinking the special way that good programmers think. Even with a much better interface than we had then (and have today), it is likely that this really is actually more like writing than we wanted it to be. Namely, for the "80%", it really has to be learned gradually over a period of years in order to build up the structures that need to be there for design and solution look-ahead. 41

The problem is not to get the kids to do stuff—they love to do , even when they are not sure exactly what they are doing. This correlates well with studies of early learning of language, when much rehearsal is done regardless of whether content is involved. Just doing seems to help. What is difficult is to determine what ideas to put forth and how deeply they should penetrate at a given child's developmental level. This confusion still persists for reading and writing of natural language—and for mathematics—despite centuries of experience. And it is the main hurdle for teaching children programming. When, in what order and depth, and how should the powerful ideas be taught?

Should we even try to teach programming? I have met hundreds of programmers in the last 30 years and can see no discernible influence of programming on their general ability to think well or to take an enlightened stance on human knowledge. If anything, the opposite is true. Expert knowledge often remains rooted in the environments in which it was first learned—and most metaphorical extensions result in misleading analogies. A remarkable number of artists, scientists, philosophers are quite dull outside of their specialty (and one suspects within it as well). The first siren's song we need to be wary of is the one that promises a connection between an interesting pursuit and interesting thoughts. The music is not in the piano, and it is possible to graduate Juilliard without finding or feeling it.

I have also met a few people for whom computing provides an important new metaphor for thinking about human knowledge and reach. But something else was needed besides computing for enlightenment to happen.

Tools provide a path, a context, and almost an excuse for developing enlightenment, but no tool ever contained it or can dispense it. Cesare Pavese observed: to know the world we must construct it. In other words, we make not just to have, but to know. But the having can happen without most of the knowing taking place.

Another way to look at this is that knowledge is in its least interesting state when it is first being learned. The representations—whether markings, allusions, or physical controls—get in the way (almost take over as goals) and must be laboriously and painfully interpreted. From here there are several useful paths, two of which are important and intertwined.

The first is fluency , which in part is the process of building mental structures that disappear the interpretation of the representations. The letters and words of a sentence are experienced as meaning rather than markings, the tennis racquet or keyboard becomes an extension of one's body, and so forth. If carried further one eventually becomes a kind of expert—but without deep knowledge in other areas, attempts to generalize are usually too crisp and ill formed.

The second path is towards taking the knowledge as a metaphor than can illuminate other areas. But without fluency it is more likely that prior knowledge will hold sway and the metaphors from this side will be fuzzy and misleading.

The "trick," and I think that this is what liberal arts education is supposed to be about, is to get fluent and deep while building relationships with other fluent deep knowledge. Our society has lowered its aims so far that it is happy with "increases in scores" without daring to inquire whether any important threshold has been crossed. Being able to read a warning on a pill bottle or write about a summer vacation is not literacy and our society should not treat it so. Literacy, for example, is being able to fluently read and follow the 50 page argument in Paine's Common Sense and being able (and happy) to fluently write a critique or defense of it. Another kind of 20th century literacy is being able to hear about a new fatal contagious incurable disease and instantly know that a disastrous exponential relationship holds and early action is of the highest priority. Another kind of literacy would take citizens to their personal computers where they can fluently and without pain build a systems simulation of the disease to use as a comparison against further information.

At the liberal arts level we would expect that connections between each of the fluencies would form truly powerful metaphors for considering ideas in the light of others.

The reason, therefore, that many of us want children to understand computing deeply and fluently is that like literature, mathematics, science, music, and art, it carries special ways of thinking about situations that in contrast with other knowledge and other ways of thinking critically boost our ability to understand our world.

We did not know then, and I'm sorry to say from 15 years later, that these critical questions still do not yet have really useful answers. But there are some indications. Even very young children can understand and use interactive transformational tools. The first ones are their hands! They can readily extend these experiences to computer objects and making changes to them. They can often imagine what a proposed change will do and not be surprised at the result. Two and three year olds can use the Smalltalk-style interface and manipulate object-oriented graphics. Third graders can (in a few days) learn more than 50 features—most of these are transformational tools—of a new system including its user interface. They can answer any question whose answer requires the application of just one of these tools. But it is extremely difficult for them to answer any question that requires two or more transformations. Yet they have no problem applying sequences of transformations, exploring "forward." It is for conceiving and achieving even modest goals requiring several changes that they almost completely lack navigation abilities.

It seems that what needs to be learned and taught is now to package up transformations in twos and threes in a manner similar to learning a strategic game like checkers. The vague sense of a "threesome" pointing towards one's goal can be a set up for the more detailed work that is needed to accomplish it. This art is possible for a large percentage of the population, but for most, it will need to be learned gradually over several years.

V. 1976-80—The first modern Smalltalk (-76), its birth, applications, and improvements

By the end of 1975 I felt that we were losing our balance—that the "Dynabook for children" idea was slowly dimming out—or perhaps starting to be overwhelmed by professional needs. In January 1976, I took the whole group to Pajaro Dunes for a three day offsite to bring up the issues and try to reset the compass. It was called "Let's Burn Our Disk Packs." There were no shouting matches, the group liked (I would go so far to say: loved ) each other too much for that. But we were troubled. I used the old aphorism that "no biological organism can live in its own waste products" to plead for a really fresh start: a hw-sw system very different from the ALTO and Smalltalk. One thing we all did agree on was that the current Smalltalk's power did not match our various levels of aspiration. I thought we needed something different, as I did not see how OOP by itself was going to solve our end-user problems. Others, particularly some of the grad students, really wanted a better Smalltalk that was faster and could be used for bigger problems. I think Dan felt that a better Smalltalk could be the vehicle for the different system I wanted, but could not describe clearly. The meeting was not a disaster, and we went back to PARC still friends and colleagues, but the absolute cohesiveness of the first four years never rejelled. I started designing a new small machine and language I called the NoteTaker and Dan started to design Smalltalk-76.

The reason I wanted to "burn the disk packs" is that I had a very McLuhanish feeling about media and environments: that once we've shaped tools, in his words, they turn around and "reshape us." Of course this is a great idea if the tools are really good and aimed squarely at the issues in question. But the other edge of the sword cuts as deep—that inadequate tools and environments still reshape our thinking in spite of their problems, in part, because we want paradigms to guide our goals. Strong paradigms like LISP and Smalltalk are so compelling that they eat their young : when you look at an application in either of these two systems, they resemble the systems themselves, not a new idea. When I looked at Smalltalk in 1975, I was looking at something great, but I did not see an enduser language, I did not see a solution to the original goal of a "reading" and "writing" computer medium for children. I wanted to stop, dynamite everything and start from scratch again.

The NoteTaker was to be a "laptop" that could be built in a few years using the (almost) available 16K RAMS (a vast improvement over the 1K RAMS that the ALTO employed). A laptop couldn't use a mouse (which I hated anyway) and a tablet seemed awkward (not a lot of room and the stylus could flop out of reach when let go), so I came up with an embedded pointing device I called a "tabmouse." It was a relative pointer and had an up sensor so it could be stroked like a mouse and would also stay where you left it, but it felt like a stylus and used a pantograph mechanism that eliminated the annoying hysteresis bias in the x and y directions that made it hard to use a mouse as a pen. I planned to use a multiprocessor architecture of slow but highly integrated chips as originally specified for the Dynabook and wanted a new bytecoded interpreter for a friendlier and simpler system than Smalltalk-72.

NoteTaker

Meanwhile Dan was proceeding with his total revamp of Smalltalk and along somewhat similar lines [In 78]. The first major thing that needed to be done was to get rid of the function/class dualism in favor of a completely intensional definition with every piece of code as an intrinsic method. We had wanted that from the beginning, (and most of the code was already written that way). There were a variety of strong desires for a real inheritance mechanism from Adele and me, from Larry Tesler, who was working on desktop publishing, and from the grad students. Dan had to find a better way than Simula's very rigid compile-time conception. It was time to make good on the idea that "everything was an object," which included all the internal "systems" objects like "activation records," etc. We were all agreed that the flexible syntax of the earlier Smalltalks was too flexible, and this level of extensibility was not desirable. All of the extensions we liked used various keyword schemes, so Dan came up with a combination keyword/operator syntax that was very flexible, but allowed the language to be read unambiguously by both humans and the machine. This allowed a FLEX machine-like byte-code compiler and efficient interpreter to be defined that ran up to 180 times as fast as the previous direct interpreter. The OOZE VM system could be modified to handle the new objects and its capacity was well matched to the ALTO's RAM and disk.

Inheritance

A word about inheritance. Simula-I had neither classes as objects nor inheritance. Simula-67 added the latter as a generalization to the ALGOL-60 <block> structure. This was a great idea. But it did have some drawbacks: minor ones like name clashes in multiple threaded lists (no one uses threaded lists anymore), and major ones like rigidity in the extended type structures, need to qualify types, only a single path of inheritance, and difficulty in adapting to an interactive development system with incremental compiling and other needs for instant changes. Then there were a host of problems that were really outside the scope of Simula's goals: having to do with various kinds of modeling and inferencing that were of interest in the world of artificial intelligence. For example, not all useful questions could be answered by following a static chain. Some of them required a kind of "inheritance" or "inferencing" through dynamically bound "parts" (i.e. instance variables). Multiple inheritance also looked important but the corresponding possible clashes between methods of the same name in different superclasses looked difficult to handle, and so forth.

On the other hand, since things can be done with a dynamic language that are difficult with a statically compiled one, I just decided to leave inheritance out as a feature in Smalltalk-72, knowing that we could simulate it back using Smalltalk's LISPlike flexibility. The biggest contributor to these AI ideas was Larry Tesler who used what is now called "slot inheritance" extensively in his various versions of early desktop publishing systems. Nowadays, this would be called a "delegation-style" inheritance scheme [Liberman 84]. Danny Bobrow and Terry Winograd during this period were designing a "frame-based" AI language called KRL which was "object-oriented" and I believe was influenced by early Smalltalk. It had a kind of multiple inheritance—called perspectives —which permitted an object to play multiple roles in a very clean way. Many of these ideas a few years later went into PIE, an interesting extension of Smalltalk to networks and higher level descriptions by Ira Goldstein and Bobrow [Goldstein & Bobrow 1980].

By the time Smalltalk-76 came along, Dan Ingalls had come up with a scheme that was Simula-like in its semantics but could be incrementally changed on the fly to be in accord with our goals of close interaction. I was not completely thrilled with it because it seemed that we needed a better theory about inheritance entirely (and still do). For example, inheritance and instancing (which is a kind of inheritance) muddles both pragmatics (such as factoring code to save space) and semantics (used for way too many tasks such as: specialization, generalization, speciation, etc.) Alan Borning employed a multiple inheritance scheme in Thinglab [Borning 1977] which was implemented in Smalltalk-76. But no comprehensive and clean multiple inheritance scheme appeared that was compelling enough to surmount Dan's original Simula-like design.

Meanwhile, the running battle with Xerox continued. There were now about 500 ALTOs linked with Ethernets to each other and to Laserprinter and file servers, that used ALTOs as controllers. I wrote many memos to the Xerox planners trying to get them to make plans that included personal computing as one of their main directions. Here is an example:

A Simple Vision of the Future

A Brief Update Of My 1971 Pendery Paper

In the 1990's there will be millions of personal computers. They will be the size of notebooks of today, have high-resolution flat-screen reflective displays, weigh less than ten pounds, have ten to twenty times the computing and storage capacity of an Alto. Let's call them Dynabooks.

The purchase price will be about that of a color television set of the era, although most of the machines will be given away by manufacturers who will be marketing the content rather than the container of personal computing.

...

Though the Dynabook will have considerable local storage and will do most computing locally, it will spend a large percentage of its time hooked to various large, global information utilities which will permit communication with others of ideas, data, working models, as well as the daily chit-chat that organizations need in order to function. The communications link will be by private and public wire and by packet radio, Dynabooks will also by used as servers in the information utilities. They will have enough power to be entirely shaped by software.

The Main Points Of This Vision

  • There need only be a few hardware types to handle almost all of the processing activity of a system.
  • Personal Computers, Communications Links, and Information Utilities are the three critical components of a Xerox future.

...

In other words, the material of a computer system is the computer itself, all of the content and function is fashioned in software.

There are two important guidelines to be drawn from this:

  • Material: If the design and development of the hardware computer material is done as carefully and completely as Xerox's development of special light-sensitive alloys, then only one or two computer designs need to be built... Extra investment in development here will be vastly repaid by simplifying the manufacturing process and providing lower costs through increased volume.
  • Content: Aside from the wonderful generality of being able to continuously shape new content from the same material, software has three important characteristics:
    • the replication time and cost of a content-function is zero
    • the development time and cost for a content-function is high
    • the change time and cost for a content-function can be low

Xerox must take these several points seriously if it is to survive and prosper in its new business area of information media. If it does, the company has an excellent chance for several reasons:

  • Xerox has the financial base to cover the large development costs of a small number of very powerful computer-types and a large number of software functions.
  • Xerox has the marketing base to sell these functions on a wide enough scale to garner back to itself an incredible profit.
  • Xerox has working for it an impressively large percentage of the best software designers in the world.

In 1976, Chuck Thacker designed the ALTO III that would use the new 16k chips and be able to fit on a desktop. It could be marketed for about what the large cumbersome special purpose "word-processors" cost, yet could do so much more. Nevertheless, in August of 1976, Xerox made a fateful decision: not to bring the ALTO III to market. This was a huge blow to many of us—even I, who had never really, really thought of the ALTO as anything but a stepping stone to the "real thing." In 1992, the world market for personal computers and workstations was $90 million—twice as much as the mainframe and mini market, and many times Xerox's 1992 gross. The most successful company of this era—Microsoft—is not a hardware company, but a software company.

The Smalltalk User Interface

I have been asked by several of the reviewers to say more about the development of the "Smalltalk-style" overlapping window user interface since there are now more than 20 million computers in the world that use its descendants. A decent history would be as long as this chapter, and none has been written so far. There is a summary of some of the ideas in [Kay 89]—let me add a few more points.

All of the elements eventually used in the Smalltalk user interface were already to be found in the sixties—as different ways to access and invoke the functionality provided by an interactive system. The two major centers of ideas were Lincoln Labs and RAND corp—both ARPA funded. The big shift that consolidated these ideas into a powerful theory and long-lived examples came because the LRG focus was on children. Hence, we were thinking about learning as being one of the main effects we wanted to have happen. Early on, this led to a 90 degree rotation of the purpose of the user interface from "access to functionality" to "environment in which users learn by doing." This new stance could now respond to the echoes of Montessori and Dewey, particularly the former, and got me, on rereading Jerome Bruner, to think beyond the children's curriculum to a "curriculum of the user interface."

The particular aim of LRG was to find the equivalent of writing—that is learning and thinking by doing in a medium—our new "pocket universe." For various reasons I had settled on "iconic programming" as the way to achieve this, drawing on the iconic representations used by many ARPA projects in the sixties. My friend Nicholas Negroponte, an architect, was extremely interested in how environments affected peoples' work and creativity. He was interested in embedding the new computer magic in familiar surroundings. I had quite a bit of theatrical experience in a past life, and remembered Coleridge's adage that "people attend 'bad theatre' hoping to forget, people attend 'good theatre' aching to remember ." In other words, it is the ability to evoke the audience's own intelligence and experiences that makes theatre work.

Paul Rovner showing the iconic 'Lincoln Wand' ca. 1968 The last Smalltalk-72 Interface, Ted Kaehler's iconic painting interface

Putting all this together, we want an apparently free environment in which exploration causes desired sequences to happen (Montessori); one that allows kinesthetic, iconic, and symbolic learning—" doing with images makes symbols " (Piaget & Bruner); the user is never trapped in a mode (GRAIL); the magic is embedded in the familiar (Negroponte); and which acts as a magnifying mirror for the user's own intelligence (Coleridge). It would be a great finish to this story to say that having articulated this we were able to move straightforwardly to the design as we know it today. In fact, the UI design work happened in fits and starts in between feeding Smalltalk itself, designing children's experiments, trying to understand iconic construction, and just playing around. In spite of this meandering, the context almost forced a good design to turn out anyway. Just about everyone at PARC at this time had opinions about the UI, ours and theirs. It is impossible to give detailed credit for the hundreds of ideas and discussions. However, the consolidation can certainly be attributed to Dan Ingalls, for listening to everyone, contributing original ideas, and constantly building a design for user testing. I had a fair amount to do with setting the context, inventing overlapping windows, etc., and Adele and I designed most of the experiments. Beyond that, Ted Kaehler, and visitor Ron Baecker made highly valuable contributions. [Dan Halbert] designed SmallStar, the prototype iconic interface for the Xerox Star product.

Meanwhile, I had gotten Doug Fairbairn interested in the Notetaker . He designed a wonderful "smart bus" that could efficiently handle slow multiple processors and the system looked very promising, even though most of the rest of PARC thought I was nuts to abandon the fast bipolar hw of the ALTO. But I couldn't see that bipolar was ever going to make it into a laptop or Dynabook. On the other hand I hated the 8-bit micros that were just starting to appear, because of the silliness and naivete of their designs—there was no hint that anyone who had ever designed software was involved.


Smalltalk-76

Smalltalk-76

Dan finished the Smalltalk-76 design November, and he, Dave Robson, Ted Kaehler, and Diana Merry, successfully implemented the system from scratch (which included rewriting all of the existing class definitions) in just seven months. this was such a wonderful achievement that I was bowled over in spite of my wanting to start over. It was fast, lively, could handle "big" problems, and was great fun. The system consisted of about 50 classes described in about 180 pages of source code. This included all of the OS functions, files, printing and other Ethernet services, the window interface, editors, graphics and painting systems, and two new contributions by Larry Tesler, the famous browsers for static methods in the inheritance hierarchy and dynamic contexts for debugging in the runtime environment. In every way it was the consolidation of all of our ideas and yearning about Smalltalk in one integrated package. All Smalltalks since have resembled this conception very closely. In many ways, as Tony Hoare once remarked about Algol, Dan's Smalltalk-76 was a great improvement on its successors!

Here are two stylish ST-76 classes written by Dan.

Two stylish ST-76 classes

Notice, particularly in class Window, how the code is expressed as goals for other objects (or itself) to achieve. The superclass Window's main job is to notice events and distribute them as messages to its subclasses. In the example, a document window (a subclass of DocWindow) is going to deal with the effects of user interactions. The Window class will notice that the keyboard is active and send a message to itself which will be intercepted by the subclass method. If there is no method the character will be thrown away and the window will flash. In this case, it finds DocWindow method: keyboard , which tells the held document to check it out.


Dan Ingalls, the main implementer of Smalltalk, creator of Smalltalk-76, and his implementation plan (below), Jack Goldman finally uses the system he paid for all those years (with Alan Borning helping), An end-user simulation by Xerox executive in SimKit. Total time including training: 3 hours

In January of 1978 Smalltalk-76 had its first real test. CSL had invited the top ten executives of Xerox to PARC for a two day seminar on software, with a special emphasis on complexity and what could be done about it. LRG got asked to give them a hands-on experience in end-user programming so "they could do 'something real' over two 1½ hour sessions." We immediately decided not to teach them Smalltalk-76 (my "burn our disk packs" point in spades), but to create in two months in Smalltalk-76 a rich system especially tailored for adult nonexpert users (Dan's point in trumps). We took our "Simpula" job shop simulation model as a starting point and decided to build a user interface for a generalized job shop simulation tool that the executives could make into specific dynamic simulations that would act out their changing states by animating graphics on the screen. We called it the Smalltalk SimKit. This was a maximum effort and everyone pitched in. Adele became the design leader in spite of the very recent appearance of a new baby. I have a priceless memory of her debugging away on the SimKit while simultaneously nursing Rachell.

There were many interesting problems to be solved. The system itself was straightforward but it had to be completely sealed off from Smalltalk proper, particularly with regard to error messages. Dave Robson came up with a nice scheme (almost an expert system) to capture complaints from the bowels of Smalltalk and translated them into meaningful SimKit terms. There were many user interface details—some workaday, like making new browsers that could only look at the four SimKit classes (Station, Worker, Job, Report), and some more surprising as when we tried it on ten PARC nontechnical adults of about the same age and found that they couldn't read the screen very well. The small fonts our thirtysomething year-old eyes were used to didn't work for those in their 50s. This led to a nice introduction to the system in which the executives were encouraged to customize the screen by choosing among different fonts and sizes with the side effect that they learned how to use the mouse unselfconsciously.

On the morning of the "big day" Ted Kaehler decided to make a change in the virtual memory system OOZE to speed it up a little. We all held our breaths, but such was the clarity of the design and the confidence of the implementers that it did work, and the executive hands-on was a howling success. About an hour into the first session one of the VPs (who had written a few programs in FORTRAN 15 years before) finally realized he was programming and mused "so it's finally come to this." Nine out of the ten executives were able to finish a simulation problem that related to their specific interests. One of the most interesting and sophisticated was a PC board production line done by the head of a Xerox owned company using actual figures (that he carried around in his head) to prime a model that could not be solved easily by closed form mathematics—it revealed a serious flaw in the disposition of workers given the line's average probability of manufacturing defects.


Alan Borning's Thinglab, a constraint-based iconic problem solver, Smalltalk-76 hierarchical class browser designed and built by Larry Tesler, The author's pen-based interface for ST-76

Another important system done at this time was Alan Borning's Thinglab [Borning 1979]—the first serious attempt to go beyond Ivan Sutherland's Sketchpad. Alan devised a very nice approach for dealing with constraints that did not require the solver to be omniscient (or able to solve Fermat's last theorem).

We could see that the "pushing" style of Smalltalk could eventually be replaced by a "pulling" style that was driven by changes to values that different methods were based on. This was an old idea but Thinglab showed how the object-oriented definition could be used to automatically limit the contexts for event-driven processing. And we soon discovered that "prototypes" were more hospitable than classes, and that multiple inheritance would be well served if there were classes for methods that knew generally what they were supposed to be about (inspired by Pat Winston's 2nd order models).

Meanwhile, the NoteTaker was getting more real, bigger, and slower. By this time the Western Digital emulation-style chips I hoped to use showed signs of being "diffusion-ware," and did not look like they would really show up. We started looking around for something that we could count on, even if it didn't have a good architecture. In 1978, the best candidate was the Intel 8086, a 16-bit chip (with many unfortunate remnants of the 8008 and 8080), but with (barely) enough capacity to do the job—we would need three of them to make up for the ALTO, one for the interpreter, one for bitmapped graphics, and one for i/o (networking, etc).

Dan had been interested in the NoteTaker all along and wanted to see if he could make a version of Smalltalk-76 that could be the NoteTaker system. In order for this to happen it would have to run in 256K, the maximum amount of RAM that we had planned for the machine. None of the NOVA-like emulated "machine-code" from the ALTO could be brought over, and it had to fit in memory as well—there would only be floppies, no swapping memory existed. This challenge led to some excellent improvements in the system design. Ted Kaehler's system tracer (which could write out new virtual memories from old ones) was used to clone Smalltalk-76 into the NoteTaker . The indexed object table (as was used in early Smalltalk-80) first appeared here to simplify object access. An experiment in stacking contexts contiguously was tried: to save space and gain speed. Most of the old machine code was rewritten in Smalltalk and the total machine kernel was reduced to 6K bytes of (the not very strong) 8086 code.

All of the re-engineering had an interesting effect. Though the 8086 was not as good at bitblt as the ALTO (and much of the former machine code to assist graphics was now in Smalltalk), the overall interpreter was about twice as fast as the ALTO version (because not all the Smalltalk byte-code interpreter would fit into the 4k microcode memory on the ALTO). With various kinds of tricks and tuning, graphics display was "largely compensated" (in Dan's words). This was mainly because the ALTO did not have enough microcode memory to take in all of the Smalltalk emulation code—some of it had to be rendered in emulated "NOVA" code which forced two layers of interpretation. In fact, the Notetaker worked extremely well, though it would have crushed any lap. It had hopped back on the desk, and looked suspiciously like miniCOM (and several computers that would appear a few years later). It really did run on batteries and several of us had the pleasure of taking NoteTaker on a plane and running an object-oriented system with a windowed interface at 35,000 feet.

Design for NoteTaker interface [Ka 79], What Steve Jobs saw, Multiviews on complex structure by Trygive Reeskaug (above) Multimedia documents by Bob Flegal and Dlana Merry (below)

We eventually built about 10 of the machines, and though in many senses an engineering success, what had to be done to make them had once again squeezed out the real end-users for whom it was originally aimed. If Xerox (and PARC) as a whole had believed in these smaller scale ideas, we could have put much more silicon muscle behind the dreams and successfully built them in the 70's when they were first possible. It was a bitter disappointment to have to get the wrong kind of CPU from Intel and the wrong kind of display from HP because there was not enough corporate will to take advantage of internal technological expertise.

By now it was already 1979, and we found ourselves doing one of our many demos, but this time for a very interested audience: Steve Jobs, Jef Raskin, and other technical people from Apple. They had started a project called Lisa but weren't quite sure what it should be like, until Jef said to Steve, "You should really come over to PARC and see what they are doing." Thus, more than eight years after overlapping windows had been invented and more than six years after the ALTO started running, the people who could really do something about the ideas, finally got to to see them. The machine used was the Dorado, a very fast "big brother" of the ALTO, whose Smalltalk microcode had been largely written by Bruce Horn, one of our original "Smalltalk kids" who was still only a teen-ager. Larry Tesler gave the main part of the demo with Dan sitting in the copilot's chair and Adele and I watched from the rear. One of the best parts of the demo was when Steve Jobs said he didn't like the blt-style scrolling we were using and asked if we could do it in a smooth continuous style. In less than a minute Dan found the methods involved, made the (relatively major) changes and scrolling was now continuous! This shocked the visitors, especially the programmers among them, as they had never seen a really powerful incremental system before.

Steve tried to get and/or buy the technology from Xerox (which was one of Apple's minority venture capitalists), but Xerox would neither part with it nor would come up with the resources to continue to develop it in house by funding a better NoteTaker cum Smalltalk.


VI. 1980-83—The release version of Smalltalk (-80)

"The greatest sin in Art is not Boredom,
as is commonly supposed, but lack of
Proportion" — Paul Hindemith

As Dan said "the decision not to continue the NoteTaker project added motivation to release Smalltalk widely." But not for me. By this time I was both happy about the cleanliness and elegance of the Smalltalk conception as realized by Dan and the others, and sad that it was farther away than ever from the children—it came to me as a shock that no child had programmed in any Smalltalk since Smalltalk-76 made its debut. Xerox (and PARC) were now into "workstations" as things in themselves—but I still wanted "playstations". The romance of the Dynabook seemed less within grasp, paradoxically just when the various needed technologies were starting to be commercially feasible—some of them, unfortunately, like the flat-screen display, abandoned to the Japanese by the US companies who had invented them. This was a major case of "snatching defeat from the jaws of victory." Larry Tesler decided that Xerox was never going to "get it" and was hired by Steve Jobs in May 1980 to be a principal designer of the Lisa . I agreed, had a sabbatical coming, and took it.

Adele decided to drive the documentation and release process for a new Smalltalk that could be distributed widely almost regardless of the target hardware. Only a few changes had to be made to the NoteTaker Smalltalk-78 to make a releasable system. Perhaps the change that was most ironic was to turn the custom fonts that made Smalltalk more readable (and were a hallmark of the entire PARC culture) back into standard pedestrian ASCII characters. According to Peter Deutsch this "met with heated opposition within the group at the time, but has turned out to be essential for the acceptance of the system in the world." Another change was to make blocks more like lambda expressions which, as Peter Deutsch was to observe nine years later: "In retrospect, this proliferation of different kinds of instantiations and scoping was probably a bad idea." The most puzzling strange idea—at least to me as a new outsider—was the introduction of metaclasses (really just to make instance initialization a little easier—a very minor improvement over what Smalltalk-76 did quite reasonably already). Peter's 1989 comment is typical and true: "metaclasses have proven confusing to many users, and perhaps in the balance more confusing than valuable." In fact, in their PIE system, Goldstein and Bobrow had already implemented in Smalltalk an "observer language", somewhat following the view-oriented approach I had been advocating and in some ways like the "perspectives" proposed in KRL [Goldstein *]. Once one can view an instance via multiple perspectives even "semi-metaclasses" like Class Class and Class Object are not really necessary since the object-role and instance-of-a-class-role are just different views and it is easy to deal with life-history issues including instantiation. This was there for the taking (along with quite a few other good ideas), but it wasn't adopted. My guess is that Smalltalk had moved into the final phase I mentioned at the beginning of this story, in which a way of doing things finally gets canonized into an inflexible belief structure.

Coda

One final comment. Hardware is really just software crystallized early. It is there to make program schemes run as efficiently as possible. But far too often the hardware has been presented as a given and it is up to software designers to make it appear reasonable. This has caused low-level techniques and excessive optimization to hold back progress in program design. As Bob Barton used to say: "Systems programmers are high priests of a low cult."

One way to think about progress in software is that a lot of it has been about finding ways to late-bind , then waging campaigns to convince manufacturers to build the ideas into hardware. Early hardware had wired programs and parameters; random access memory was a scheme to late-bind them. Looping and indexing used to be done by address modification in storage; index registers were a way to late-bind. Over the years software designers have found ways to late-bind the locations of computations—this led to base/bounds registers, segment relocation, page MMUs, migratory processes, and so forth. Time-sharing was held back for years because it was "inefficient"— but the manufacturers wouldn't put MMUs on the machines, universities had to do it themselves! Recursion late-binds parameters to procedures, but it took years to get even rudimentary stack mechanisms into CPUs. Most machines still have no support for dynamic allocation and garbage collection and so forth. In short, most hardware designs today are just re-optimizations of moribund architectures.

From the late-binding perspective, OOP can be viewed as a comprehensive technique for late-binding as many things as possible: the mix of state and process in a set of behaviors, where they are located, what they are called, when and why they are invoked, which HW is used, etc., and more subtle, the strategies used in the OOP scheme itself. The art of the wrap is the art of the trap.

Consider the two cases that must be handled efficiently in order to completely wrap objects. It would be terrible if a + b incurred any overhead if a and b were bound, say, to "3" and "4" in a form that could be handled by the ALU. The operations should occur full speed using look-aside logic (in the simplest scheme a single and gate) to trap if the operands aren't compatible with the ALU. Now all elementary operations that have to happen fast have been wrapped without slowing down the machine.

The second case happens if the trap has determined the objects in questions are too complicated for the ALU. Now the HW has to dynamically find a method that can handle the objects. This is very similar to indexing—the class of one of the objects is "indexed" by the desired method-selector in a slightly more general way. In other words the virtual-address of a method is <class><selector>. Since most HW today does a virtual address translation of some kind to find the real address—a trap—it is quite possible to hide the overhead of the OOP dispatch in the MMU overhead that has already been rationalized.

Again, the whole point of OOP is not to have to worry about what is inside an object. Objects made on different machines and with different languages should be able to talk to each other—and will have to in the future. Late-binding here involves trapping incompatibilities into re compatibility methods—a good discussion of some of the issues is found in [Popek 1984].

Staying with the metaphor of late-binding, what further late-binding schemes might we expect to see? One of the nicest late-binding schemes that is being experimented with is the metaobject protocol work at Xerox PARC [Kiczales 1991]. The notion is that the language designer's choice for the internal representation of instances, variables, etc., may not cover what the implementer needs, so within a fixed semantics they allow the implementer to give the system strategies—for example, using a hashed lookup for slots in an instance instead of direct indexing. These are then efficiently compiled and extend the base implementation of the system. This is a direct descendant of similar directions from the past of Simula, FLEX, CDL, Smalltalk, and Actors.

Another late-binding scheme that is already necessary is to get away from direct protocol matching when a new object shows up in a system of objects. In other words, if someone sends you an object from halfway around the world it will be unusual if it conforms to your local protocols. At some point it will be easier to have it carry even more information about itself—enough so its specifications can be "understood" and its configuration into your mix done by the more subtle matching of inference .

A look beyond OOP as we know it today can also be done by thinking about late-binding. Prolog's great idea is that it doesn't need binding to values in order to carry out computations [Col **]. The variable is an object and a web of partial results can be built to be filled in when a binding is finally found. Eurisko [Lenat **] constructs its methods—and modifies its basic strategies—as it tries to solve a problem. Instead of a problem looking for methods, the methods look for problems—and Eurisko looks for the methods of the methods. This has been called "opportunistic programming"—I think of it as a drive for more enlightenment, in which problems get resolved as part of the process.

This higher computational finesse will be needed as the next paradigm shift—that of pervasive networking—takes place over the next five years. Objects will gradually become active agents and will travel the networks in search of useful information and tools for their managers. Objects brought back into a computational environment from halfway around the world will not be able to configure themselves by direct protocol matching as do objects today. Instead, the objects will carry much more information about themselves in a form that permits inferential docking. Some of the ongoing work in specification can be turned to this task [Guttag **][Goguen **].

Tongue in cheek, I once characterized progress in programming languages as kind of "sunspot" theory, in which major advances took place about every 11 years. We started with machine code in 1950, then in 1956 FORTRAN came along as a "better old thing" which if looked at as "almost a new thing" became the precursor of ALGOL-60 in 1961. In 1966, SIMULA was the "better old thing," which if looked at as "almost a new thing" became the precursor of Smalltalk in 1972.

Everything seemed set up to confirm the "theory" once more: in 1978 Eurisko was in place as the "better old thing" that was "almost a new thing". But 1983—and the whole decade—came and went without the "new thing". Of course, such a theory is silly anyway—and yet, I think the enormous commercialization of personal computing has smothered much of the kind of work that used to go on in universities and research labs, by sucking the talented kids towards practical applications. With companies so risk-adverse towards doing their own HW, and the HW companies betraying no real understanding of SW, the result has been a great step backwards in most respects.

A twentieth century problem is that technology has become too "easy". When it was hard to do anything whether good or bad, enough time was taken so that the result was usually good. Now we can make things almost trivially, especially in software, but most of the designs are trivial as well. This is inverse vandalism: the making of things because you can. Couple this to even less sophisticated buyers and you have generated an exploitation marketplace similar to that set up for teenagers. A counter to this is to generate enormous dissatisfaction with one's designs using the entire history of human art as a standard and goal. Then the trick is to decouple the dissatisfaction from self worth—otherwise it is either too depressing or one stops too soon with trivial results.

Dave Robson

I will leave the story of early Smalltalk in 1981 when an extensive series of articles on Smalltalk-80 was published in Byte magazine, [Byte 1981] followed by Adele's and Dave Robson's books [Goldberg 1983] and the official release of the system in 1983. Now programmers could easily implement the virtual machine without having to reinvent it, and, in several cases, groups were able to roll their own image of basic classes. In spite of having to run almost everywhere on moribund HW architectures, Smalltalk has proliferated amazingly well (in part because of tremendous optimization efforts on these machines) [Deutsch 83]. As far as I can tell, it still seems to be the most widely used system that claims to be object-oriented. It is incredible to me that no one since has come up with a qualitatively better idea that is as simple, elegant, easy to program, practical, and comprehensive. (It's a pity that we didn't know about PROLOG then or vice versa, the combinations of the two languages done subsequently are quite intriguing.)

While justly applauding Dan, Adele and the others that made Smalltalk possible, we must wonder at the same time: where are the Dans and the Adeles of the '80s and '90s that will take us to the next stage?

Byte Magazine, August 1981

Scrimshaw Jukebox

Simon Willison
simonwillison.net
2026-10-06 11:17:39
Tool: Scrimshaw Jukebox I wanted to see if Claude Opus 5.5 could compose music, so I tried this: I want you to write some computer game music for me. First design simple text based format for the music and build an artifact that can play it out loud - include some example tracks in that art...
Original Article

I wanted to see if Claude Opus 5.5 could compose music, so I tried this :

I want you to write some computer game music for me. First design simple text based format for the music and build an artifact that can play it out loud - include some example tracks in that artifact

I am looking for music of the quality of the original secret of Monkey Island

It leaned a lot harder into the Monkey Island theme than I had intended, but the results are surprisingly good.

Screenshot of a retro pixel-art web music player. Header in blackletter type reads "Scrimshaw Jukebox", with the text "Six original adventure-game tracks, written as plain text and played by a synthesizer running in your browser. Pick a tune, press Play, then open the score and change it." A large pixel-art scene shows a harbor at night under a purple starry sky: a full moon at top right reflecting on the water, an island silhouette on the left with palm trees and a hut with two lit windows, and a sailing ship moored at a wooden pier. Overlaid on the scene: "Moonlit Harbor" and "Press Play". Below it a bar reads "Play Moonlit Harbor". A control panel has buttons "Play", "Stop", "Restart", "Loop: on", "Edit score", "Read guide" and a "Volume" slider set to about three quarters. A track list of six cards, the first highlighted: "Moonlit Harbor 100 bpm · 4/4 · 16 voices · 1:26", "The Rusty Anchor 112 bpm · 6/8 · 8 voices · 0:56", "The Ghost Galleon 66 bpm · 4/4 · 9 voices · 2:11", "The Jungle Path 92 bpm · 4/4 · 12 voices · 1:29", "Duel on the Docks 152 bpm · 4/4 · 12 voices · 1:13", "Lantern Waltz 96 bpm · 3/4 · 8 voices · 1:38". A section titled "Score view" with the caption "1:26 · 4/4 at 100 bpm · Main theme. A calypso for a harbour town after dark." shows a piano-roll visualization of colored horizontal note bars and percussion ticks on a dark background, with a section marker "A" and a yellow vertical playhead line. A color-coded legend of voices reads: "pan steeldrum", "flute flute", "marimba marimba", "skank organ", "strings strings", "harp harp", "bass fretless", "timp timpani", "kick kick", "rim rim", "shaker shaker", "conga conga", "tumba tumba", "bongo bongo", "crash crash", "surf surf". Footer text: "Click a voice to mute it. Space bar plays and stops."

I wonder if the ability to compose competent music is similar to the 3D graphics thing - a new capability for text models that emerged in the past few months?

Would need some careful experiments with other recent and not-so-recent models to confirm if this is new or if they've been able to do this for a while.

Fake ChatGPT, Gemini Sites steal advertising accounts, MFA codes

Bleeping Computer
www.bleepingcomputer.com
2026-10-06 11:16:44
A new campaign targeting ad account managers uses fake ChatGPT, Gemini, Claude, and Perplexity sites that steal login credentials and multi-factor authentication (MFA) codes through browser-in-browser attacks. [...]...
Original Article

Fake ChatGPT, Gemini Sites steal advertising accounts, MFA codes

A new campaign targeting ad account managers uses fake ChatGPT, Gemini, Claude, and Perplexity sites that steal login credentials and multi-factor authentication (MFA) codes through browser-in-browser attacks.

Researchers found that the phishing operation leveraged the recent launch of the Muse AI agent, which Meta describes as an assistant for various personal tasks.

The malicious pages target agency staff, media buyers, and administrators with accounts that extend to multiple downstream clients.

These accounts also typically allow attackers to spend available balances on fraudulent ad campaigns or resell them to other cybercriminals for significant amounts.

On the phishing sites, the fake AI products claim to help advertisers reach buyers, obtain ad briefs, and plan and audit advertising campaigns and spending.

To get the benefits, the user had to connect their account to the fake AI product. However, the "connect" button opens a fake Google window inside the page, complete with an address bar showing accounts.google.com.

Fake Muse Ads page prompting visitors to click on 'Connect'
Fake Muse Ads page prompting visitors to click on 'Connect'
Source: Island

BitB attacks

Browser-in-the-browser is a phishing technique devised by cybersecurity researcher mr. dox in March 2022 that consists of creating a fake browser window inside a legitimate one to display a fraudulent login page.

The fake window looks like a login pop-up, featuring realistic titles, interface, and the login URL expected by the user. But the fake window is just an iframe designed to steal the victim’s credentials.

The technique has been used extensively over the years, including for targeting Steam accounts .

Researchers at browser security company Island say that the attacker uses a kit that adapts the interface to Windows, macOS, iOS, and Android, including browser styling and dark-mode support.

Once the victim is in the BitB flow, a human operator takes over the process and controls what the victim is prompted to do next.

The attacker may ask for password entry up to three times, request an SMS or authenticator code to bypass MFA protections, display Okta push requests, show Google approval prompts, or display a QR code.

Operators can also reject codes submitted by the victims, hold them in a waiting screen, and finish or suppress the phishing flow at any time.

The phishing flow
The phishing flow
Source: Island

Broader campaign

By examining the infrastructure behind the campaign, the researchers found that the campaign is part of a larger operation that used multiple lures, such as fake recruitment opportunities and refund pages.

All pages tied to the same operation share a Next.js and Socket.IO stack, common API endpoints, and many use Vercel frontends with Railway or Render backends.

Some of the lures used in the campaign
Some of the lures used in the campaign
Source: Island

The connection to the larger operation was possible because the attacker exposed older source code through misconfigured public GitHub repositories, allowing the activity to be traced as far back as March.

The researchers have also found that the Telegram control channel used in the attacks had received hundreds of victim submissions, although that figure does not necessarily reflect the number of successfully compromised accounts.

BitB attacks are deceptive but also easy to uncover, since iframes cannot be moved outside the browser window or resized, unlike a legitimate OAuth popup

These are simple actions supported by real browser windows, but impossible to perform in BitB windows.

Island researchers found that the phishing platform supports Google, Meta, TikTok, and Okta sign-in workflows, and the commands are sent through Socket.IO events.

"Unlike a transparent reverse-proxy kit, the visible platform locally rebuilds the provider interface and collects credentials and MFA state through its own APIs," the researchers say in a report today.

This masks the traffic, making it appear to be coming from an AI product communicating with an unrelated application backend.

Based on the researchers' findings, the phishing operation used dozens of URLs for campaigns focused on ads, refunds, and recruitment. The report includes a list of all URLs associated with this activity.

article image

Build your security blueprint for AI-powered attacks

Join Mikko Hyppönen and security leaders from the NFL, CHANEL, and Atlassian for a two-hour digital summit on what AI-speed attacks change, what defenders should stop doing, and how to validate, decide, fix, and re-validate at machine speed.

Save your seat

[$] Python's two modules for random numbers

Linux Weekly News
lwn.net
2026-10-06 11:02:09
Python's random and secrets modules both include utilities for obtaining random values, but only one of them is suitable for generating passwords and security tokens. For much of Python's history, random was used for passwords and tokens anyway, despite documentation that called it unsuitable for cr...
Original Article
The page you have tried to view ( Python's two modules for random numbers ) is currently available to LWN subscribers only. Reader subscriptions are a necessary way to fund the continued existence of LWN and the quality of its content.

If you are already an LWN.net subscriber, please log in with the form below to read this content.

Please consider subscribing to LWN . An LWN subscription provides numerous benefits, including access to restricted content and the warm feeling of knowing that you are helping to keep LWN alive.

(Alternatively, this item will become freely available on October 15, 2026)

Vibecoding isn't as fun as writing code by hand

Hacker News
www.autodidacts.io
2026-10-06 10:45:02
Comments...
Original Article

I like building things: it makes me happy. Learning and discovering things is a close second, but building is more satisfactory.

Since I’m not that good at actually, you know, building things, the thrill of “building”, in my case, has mostly come from writing software. Software is one of the easiest ways to create something satisfyingly useful out of nothing.

For a long time I’ve known, at some level, how much joy building gives me. [1] But it really sunk in while working on two recent hobby software projects. They were both dumb little Python scrapers that I cobbled together, that did something useful that I’d wanted for a long time. ( Link , link .) They were pretty basic, but gave me so much irrational pleasure. It was the most fun I’d had programming in years. [2]

Then, [cue Jaws theme song] AI came along.

Like pretty much everyone else, I’ve been cautiously experimenting with AI-assisted coding. We are past the point where we can easily dismiss AI as “stupid” (though it is), or “doesn’t actually make me more productive”. So, in classic motte and bailey fashion, I am retreating, while pretending not to: AI coding is less fun.

This seems counter-intuitive, because vibecoding is absolutely thrilling … AT FIRST.

Refactoring vibecoded software? Not thrilling.

I’m not going to claim, as some do, that vibecoding doesn’t give the thrill of actually building something. Because it does, partly.

I think that one of the things that’s driving the adoption of vibecoding is that it frontloads the fun , at both the project- and career level.

AI Vibecoding vs Coding By Hand Thrill/Fun/Learning Curve Graph
The Vibecoding Thrill Curve

If I have an idea, I can usually get a working prototype very quickly, with very little effort. Instant dopamine!

Similarly, someone with no coding experience can “code” an app, and it will probably work. Instant dopamine!

I have a personal policy against frontloading the fun. So, I don’t like this, just like I don’t like getting paid for work I haven’t done, going into debt, and arrangements where someone else does my work today in trade for me doing their work tomorrow.

I posit that there are several distinct sources of thrill/satisfaction when building something:

  1. The thrill of the idea
  2. The thrill of making the idea real
  3. The satisfaction of hard-won accomplishment
  4. The satisfaction of a job well done
  5. The thrill of learning
  6. The satisfaction making something that solves a problem you have

Vibecoding, in general, can give you 1, 2, and 6. But 3, 4, and 5 are not natural consequence of vibecoding.

In tenuous defense of vibecoding:

I can, and have, built things with AI that I either never could have, or never would have, built otherwise. Mostly, it’s cases where it would not be worth me spending weeks building it by hand, but if I can build it in minutes or days, it is worth it.

  • I one-shotted a has-this-already-been-submitted-to-hackernews bookmarklet (HN?), based on Taylor Troesh’s Discuss on HN snippet . It gave me significant satisfaction, solved a problem I had, I use it regularly on my Wallabag favourites page.

  • In very little time, I vibecoded “Party Market”, a prediction market simulator party game to go along with a flash lecture on prediction markets, and ran a market on “how long will I drone on for”, live, during my lecture. It turned a dry and boring lecture into grade-A entertainment. The code was garbage and it was full of bugs, but I never could have done it writing it by hand in that amount of time. It wasn’t a choice between doing it with AI and doing it by hand: it was a choice between doing it with AI and not doing it at all.

  • When a fire chief I know was going to invest way too much money in an “AI appliance” that didn’t even say which Nvidia card it had, or what model/generation of Xeon it was, I vibecoded with free plan Lumo , in two afternoon sessions, a functional Pyronear based Wildfire Early Warning System that ran in realtime on a live feed on my laptop, and was able to detect the first frame of smoke on the ignition footage I fed it.

  • I built a browser-based Kobo database migration assistant ( link ), so that people unfamiliar with SQLite can move their books and annotations to a new device .

  • When a singing lesson with my vocal coach fell through, I built the realtime pitch training app I’ve always wanted — and have planned to build for about a decade.

In most of these cases, I was immensely pleased with the results, but the process wasn’t fun the way writing my dumb little scrapers was fun. Of course, handwritten software development isn’t always fun. My first Golang project was, for large sections of time, with a deadline, certified Not Fun™. Fighting the Rust compiler the first time I tried to write, like, twenty lines of Rust code was Not Fun™. But it was, in the long run, Type II fun, because it gave me the sense of accomplishment, and the glorious I did it! feeling (And, next time, it would be easier.)

I’ve tried everything from local models running in llama-server, to copy-and-pasting from free, no sign-up services (Phind, Lumo, etc) for one-shot scripts, to full on remote server Clod Masterpiece 200™ all-you-can-eat plan with custom-written auto-approval hooks that were worryingly close to –dangerously-skip-permissions , developing algorithmic trading models.

It’s all the same : it’s “productive”, but I don’t like the way it makes me feel, and I don’t like the quality of the output. It manages to make me feel dumb and superfluous (and, sometimes, dirty for even using it), without earning my respect.

The word “dopamine” made me think: vibecoding is a bit like a series of first dates or one night stands. You don’t develop a relationship with the project: you don’t get the same kind of warm fuzzies, oxytocin, attachment. The problem with a dopamine-only life is that thrills become dull with repetition.

Another way to look at it: vibecoding introduces a layer between you and reality. I am the kind of person that thinks, in general, reality is more fun when experienced first-hand.

Vibecoding is like being an executive of a large company that is barely under your control. You never feel like you really know what’s going on, and there’s always the suspicion of sabotage and loafing. Things appear fully formed, but not quite how you imagined. Sometimes they work, and sometimes they don’t, and there’s nothing you can do about it, other than pouring more money into the black box, and yelling louder. So that’s what you do.

This essay was outlined in blue ink by candlelight. AI wasn’t even allowed to correct my typos: savour them.

  1. As a kid, I wanted to be an inventor, which neatly combines building and discovery. Writing and songwriting, my other main activities, are also forms of building that, like software, require mercifully little contact with physical reality and other humans. ↩︎

  2. I’d mostly been writing shell scripts, and Bash is not known for being fun . ↩︎

What is Codemode

Lobsters
lucumr.pocoo.org
2026-10-06 10:27:03
Comments...
Original Article

written on October 06, 2026

More than a year ago I wrote a few posts here that recommended people not to load custom tools into their context (or MCP servers) but to just use more scripts. Most importantly I wrote that Code Is All You Need and I wrote about that MCP needs code . With Pi 1.0 we now added MCP support via Codemode which in some ways is a long time coming, but then also maybe somewhat surprising to some. So I want to share some updated thoughts on this blog on what this all means.

What Are Tools

When a harness like Pi provides tools for an LLM to call, it does so by supplying some tool definitions which then translate into some token structure on the server side. Whether a model is encouraged to call a tool is the result of the reinforcement learning process. Something I wrote about before if you want to learn more.

One of the reasons we strongly lean towards CLI and bash is because it allows easy composition of calls, and because the model also learns how the file system works when it’s trained. So when it invokes a tool like echo foo > /tmp/test.txt the model also learns that after that tool call, there is now a file called test.txt in /tmp .

However bash has one fundamental limitation which is that it can only compose programs that run. And there are some things, which are not programs, but native tools to the LLM and they sort of have to be.

The most obvious example here is read or view_image . If a multimodal model needs to read an image, it cannot use cat for that because the harness needs to inject the actual image payload into the protocol of the LLM.

Another quite vivid example are sub agents. In order to spawn and orchestrate sub agents, it’s tricky to avoid tools that are provided by the harness. While in theory the agent could provide a CLI tool that talks to the outer harness via environment variables and Unix sockets, it’s a rather crude process. It however has another issue, and that is where the code runs.

Brains vs Hands

To better understand that, it’s important to think a bit more about where all the bits and pieces run. There really usually are two different systems involved. The first is the brain, the harness: it runs on one machine. It’s trusted. The second is often the same machine, but it’s really where the tools are executing: the hands. In Pi we now call this the execution environment, but you can think of it as the target of all the operations.

Crucially what is important for us, is that there is a dividing line between the harness brain and the target environment that runs bash and executes the tools.

And splitting this in half has some really important consequences. For a start it means that they are running on different file systems and they have different levels of trust. If you for instance use a sandboxing solution like Gondolin your bash stuff will be sandboxed just fine, but the harness itself will not be.

Orchestrating The Harness

Which brings us to what Codemode really does: it’s a way for the LLM to express and orchestrate complex operations on the harness side, but not the execution environment side. Codemode runs in the harness, in its own sandbox. In case of Pi it’s running in QuickJS within a WASM runtime with intentional limitations: no network, no file system, no timers, limited RAM. The only way is to call more tools. You could also imagine that Codemode could run Scheme or some other language as well.

If you are not familiar with Codemode, it’s basically just a way to issue tool calls from within some language, in our case JavaScript. That allows you to compose those calls without necessarily going through the LLM’s context. Credit for naming goes to our friends at Cloudflare who coined it .

For instance if you issue a bash call as a regular tool call in the LLM, then we only throw the trailing 2000 lines into the context and if the agent wants more, it needs to look at the overflow file itself. If however the agent issues that invocation via Codemode, then the Codemode side gets larger outputs sent structurally.

Most importantly, because Codemode is JavaScript the agent can express concurrent operations and basic workflows. A common way in which you see agents now use this, is to first probe at 5-10 items from some tool response to see what it looks like, and to then write a Codemode script that processes the next n items.

Codemode also allows you to throw state into the transcript! That means that one Codemode invocation can stash away data, that the next call in the session can load again. And remember: this is on the harness host, not the sandbox.

In case of Pi, Codemode also allows you to issue calls that naturally do not make any sense in Pi’s traditional interface. For instance if you want to generate images with an image model or you want to classify some text with a one shot classifier model, those Pi APIs are exposed via Codemode, but not via regular tools where they would just waste context.

What It Looks Like

So now that we talked a bunch about it, it’s probably worth being a bit more explicit about it. Let’s walk ourselves through some invocations of Codemode of recent Pi sessions of mine. Note that none of this code is human written. It’s from real sessions of Pi, just re-indented for your viewing pleasure. The agent starts using Codemode automatically either because it’s a task where the model already naturally picks up that tool, or because a user asked it to.

Note that Codemode is by default only enabled in Pi when MCP is enabled, but you can turn it on with "defaultTools": ["+codemode"] in the settings. Just ask Pi to enable it for you.

Generating Images

Let’s start simple with image generation. Image generation is a feature that Pi supports in the AI SDK core, but it’s not a tool that the agent can use. In the past the only way to use image models has been to write a bespoke extension or to have the agent run node itself and use the internal image APIs. However because we expose quite a few of the internal model APIs within Codemode, it means that the agent can use it: 

const [painter] = await models.getAvailableOfType("image");
const result = await models.generateImages(painter, {
  input: [{ type: "text", text: "A cute little puppy sitting on a grassy " +
    "lawn, soft natural light, photorealistic" }],
});
if (result.stopReason !== "stop") return result.errorMessage;

for (const block of result.output) {
  if (block.type === "image") image(block);
  else text(block.text);
}

Note that the call to image() sends the image back as image content to the LLM. On the harness side it feeds it directly into both the agent, as well as onto disk as a temporary artifact in case the agent wants to be able to pass that image back to bash.

Classifying Things

Similar things apply to classifier models such as Jev . They also do not fit well into the workflows of an agent through the typical tools. But rather than making a bespoke tool available, Codemode just allows the agent to reach into the AI SDK and invoke those directly. Here you can see how Jev is used to mass process GitHub issues for a quick sentiment analysis: 

const jev = await models.getModelOfType("classifier", "typesafe", "jev-latest");
const r = await tools.bash({
  command: "gh issue list --state open --limit 100 " +
    "--json number,title,body,comments",
});
const issues = JSON.parse(r.output);

const results = await Promise.all(issues.map(async (issue) => {
  const res = await models.classify(jev, {
    state: {
      title: issue.title,
      body: (issue.body || "").slice(0, 4000),
      comments: issue.comments.slice(-5).map(c => c.body.slice(0, 800)),
    },
    questions: {
      sentiment: {
        type: "choice",
        instructions: "What is the overall sentiment of the author towards pi?",
        criteria: {
          positive: "Appreciative, happy, constructive praise",
          neutral: "Matter-of-fact report or request without emotion",
          negative: "Frustrated, annoyed, upset, or angry",
        },
      },
      frustration: {
        type: "score",
        instructions: "How frustrated is the reporter?",
        criteria: ["not at all", "mildly", "clearly frustrated", "very angry"],
      },
      kind: {
        type: "choice",
        instructions: "What kind of issue is this?",
        criteria: {
          bug: "Bug report or regression",
          feature: "Feature request or enhancement",
          question: "Question or support request",
          other: "Docs, discussion, meta, spam",
        },
      },
    },
  });
  if (res.stopReason !== "stop") {
    return { n: issue.number, title: issue.title, error: res.errorMessage };
  }
  return { n: issue.number, title: issue.title, ...res.answers };
}));

store("sentiment_results", results);
return results
  .filter(r => !r.error)
  .sort((a, b) => b.frustration.score - a.frustration.score)
  .slice(0, 12)
  .map(r => `#${r.n} ${r.frustration.score.toFixed(2)} [${r.kind.choice}] ${r.title}`);

Note how in that above example we also call store() which dumps the result of that execution into the session transcript. A future invocation of Codemode can thus read back that result if it wants to.

The Promise.all here is fine, because Pi limits the total number of concurrent tool executions itself to four and maintains a queue for the rest.

A more adventurous example is to use Jev to drive a game engine for debugging purposes:

Codemode with Jev for Game Debugging

Here it knows about my tankctl command and it built itself quickly a minimal harness around it to drive a game loop to assist a user with debugging a problem. Note how it built a 30 step loop in which each step goes back to both the game engine to get a text dump of what’s going on, and then to Jev to determine what to do next: 

const jev = await models.getModelOfType("classifier", "typesafe", "jev-latest");
const tank = async (cmd) =>
  (await tools.bash({ command: `tools/tankctl "${cmd}"` })).output;
await tank("start --map assets/maps/night_arena.map");

const questions = {
  action: {
    type: "choice",
    instructions: "You control the tank '@' in a top-down tank game. " +
      "Choose the best next action.",
    criteria: {
      attack: "an enemy has line of sight to you and you can fire at it",
      approach: "no enemy has line of sight; drive toward the nearest enemy",
      dodge: "an enemy shot is heading at you and will hit soon",
      powerup: "a powerup is close and no enemy threatens you",
    },
  },
};

function commandFor(choice, st) {
  const p = st.player;
  const enemy = st.enemies.filter(e => !e.dead)
    .sort((a, b) => (b.los - a.los) || (a.dist - b.dist))[0];
  if (choice === "attack" && enemy) {
    return `fire_at tank ${enemy.id}; frames 30 until clear,damage,kill`;
  }
  if (choice === "dodge") {
    // move perpendicular to the closest incoming shot
    const s = st.projectiles.filter(s => !s.yours)
      .sort((a, b) => a.eta - b.eta)[0];
    const dir = s && Math.abs(s.vel[0]) > Math.abs(s.vel[1])
      ? (p.pos[1] > s.pos[1] ? "+down" : "+up")
      : (p.pos[0] > (s ? s.pos[0] : 0) ? "+right" : "+left");
    return `input ${dir}; frames 20 until damage; input stop`;
  }
  const powerup = st.powerups.filter(u => u.available)
    .sort((a, b) => a.dist - b.dist)[0];
  if (choice === "powerup" && powerup) {
    return `goto ${powerup.pos[0]} ${powerup.pos[1]} 180`;
  }
  return enemy ? `goto ${enemy.pos[0]} ${enemy.pos[1]} 90` : null;
}

const log = [];
for (let step = 0; step < 30; step++) {
  const st = JSON.parse(await tank("state"));
  if (st.state !== "playing") break;
  const threats = st.projectiles
    .filter(s => !s.yours && s.miss_dist < 1.5 && s.eta < 1.5)
    .map(s => `incoming shot dist ${s.dist} eta ${s.eta}s`)
    .join("\n") || "no incoming shots";
  const r = await models.classify(jev, {
    state: { map: await tank("view 8"), threats, hp: st.player.hp },
    questions,
  });
  if (r.stopReason !== "stop") {
    log.push(`#${step} classifier error: ${r.errorMessage}`);
    break;
  }
  const choice = r.answers.action.choice;
  const cmd = commandFor(choice, st);
  if (!cmd) break;
  log.push(`#${step} hp=${st.player.hp} ${choice} -> ${await tank(cmd)}`);
}
return log.join("\n");

Calling MCP Servers

Lastly, Codemode obviously is great for calling MCP servers. And because we do not actually expose any of the MCP tools to the LLM, the agent first uses provided APIs to issue a tool search within Codemode to discover what it might be able to do with the connected servers. This form of progressive discovery makes the whole MCP business work well enough for a lot of use cases today.

Here for instance you can see the agent reach for the Sentry MCP straight away, even without discovering the tools, presumably because it has learned during the RL process already about what the Sentry MCP looks like. But it learns from what we inject into the system prompt, that the Sentry server is available to begin with. It’s not completely guessing here. 

const orgs = await tools.mcp__sentry__find_organizations({});
const { organizations } = orgs.structuredContent;
const results = await Promise.allSettled(organizations.map(org =>
  tools.mcp__sentry__find_projects({
    organizationSlug: org.slug,
    regionUrl: org.regionUrl,
  })
));
return organizations.map((org, i) => {
  const r = results[i];
  if (r.status !== "fulfilled") return { org: org.slug, error: String(r.reason) };
  if (r.value.isError) return { org: org.slug, error: r.value.content };
  return {
    org: org.slug,
    projects: r.value.structuredContent.projects.map(p => p.slug),
  };
});

Modern MCP Is A Fight

I really don’t want to talk too much about MCP here, but MCP is in fact a protocol that greatly benefits from Codemode. The problem in parts is that MCP in practice often targets harnesses that do not (yet?) use Codemode. But the tide is shifting. In the meantime, a temporary crutch has been to do what Cloudflare did, and do Codemode within the MCP server. But now we have Codemode in Codemode which is pretty bad. It means double JSON escaping, easy for smaller models to get confused by and the inner code cannot call the outer tools. So if you for instance use the Cloudflare MCP servers in Pi, the agent needs to write JavaScript and funnel it through more JavaScript. This is really not optimal, but it’s also understandable that this is happening: 

const accRes = await tools.mcp__cloudflare__execute({
  code: `async () => {
    const r = await cloudflare.request({ method: "GET", path: "/accounts" });
    return r.result.map(a => ({ id: a.id, name: a.name }));
  }`,
});
const accounts = JSON.parse(accRes.content.map(c => c.text).join(""));

const out = [];
for (const account of accounts) {
  const r = await tools.mcp__cloudflare__execute({
    account_id: account.id,
    code: `async () => {
      const r = await cloudflare.request({
        method: "GET",
        path: \`/accounts/\${accountId}/workers/scripts\`,
      });
      return r.result.map(s => ({ id: s.id, modified: s.modified_on }));
    }`,
  });
  out.push({ account: account.name, workers: r.content.map(c => c.text).join("") });
}
return out;

MCP Desires

So to end things off: how well does Codemode work with MCP today? Well … not amazingly well. That’s because MCP servers are not really targeting harnesses that use Codemode yet (though at this point I think most harnesses support it).

For this to work well some recommendations:

  • Structured content: Codemode wants calls to return some nicely formatted JSON. So that needs to come back from the server, and many don’t do that yet. The outputSchema system in MCP is great for that.
  • Consistent results: an interesting failure case is when an MCP server does not return consistent data. For instance because it tries to token optimize things depending on how many items are in the result set. This can cause an initial probe with 5 items to succeed, but then fail when the server returns the maximum batch size.
  • Large binary data: today MCP does not yet support large binary data so quite a few use cases that are really interesting do not work well at all yet. You end up with all kinds of weird workarounds such as pre-signed URLs to allow file uploads then to happen through non MCP channels.
  • Composable tool search: the MCP server might know better than the MCP client which tool is appropriate for a task. But there is no good mechanism today that allows a harness to fan out tool searches across multiple MCP servers. It’s all emergent behavior and it does not scale well to multiple active servers.

Future of Codemode

So where does this leave us? Is this a reversal of what I wrote a year ago where I encouraged CLIs? I don’t think so. In fact, the MCP ecosystem from my perspective picked up on exactly what we pointed out a year ago works: code. But Codemode goes beyond MCP in that it can act as a capable mechanism within the harness to express more freedom for the agent.

There are however also some things that we still need to figure out. For one, durability with Codemode is trickier. We might have to adopt some ideas from durable workflow engines here to snapshot invocations. Or maybe, something like Starlark is a better composition language than JavaScript given its deterministic nature.

Images, binary data and just the inability of this pattern to work with smaller models is also something that needs to be fleshed out. So it’s for sure not a perfect solution yet, but it’s quite a useful pattern that I expect us to leverage more.

This entry was tagged ai and pi

copy as / view markdown

404 @ 3

403 Media
www.404media.co
2026-10-06 10:08:33
And 404's Flock coverage cited in proposed legislation....
Original Article

Advertisement

•

Newsletters

·

And 404's Flock coverage cited in proposed legislation.

404 @ 3

This post is for paid members only

Become a paid member for unlimited ad-free access to articles, bonus podcast content, and more.

Subscribe

Sign up for free access to this post

Free members get access to posts like this one along with an email round-up of our week's stories.

Subscribe

Already have an account? Sign in

Who owns ‘brain rot’?: inside the international legal drama over a beloved meme

Guardian
www.theguardian.com
2026-10-06 10:00:16
Copyright law in the US only applies to human-made creations, but AI characters like Tung Tung Tung Sahur are challenging that One of the most beloved characters in an absurdist genre of online video content has become the center of an international legal battle that could determine where human crea...
Original Article

O ne of the most beloved characters in an absurdist genre of online video content has become the center of an international legal battle that could determine where human creativity ends and artificial intelligence begins.

Tung Tung Tung Sahur, also known as Triple T, was created in 2025 by an Indonesian teen who goes by the username Noxa, using OpenAI’s image-generation tool. Based on a kentongan – a type of drum used to wake south-east Asian Muslims for their pre-dawn meal during Ramadan – the character is an anthropomorphized version of the instrument with a broad smile and a vacant stare. He sits under the umbrella of “Italian brain rot” internet videos.

Since his creation, Triple T has become a global sensation: YouTube and TikTok host dozens of videos with hundreds of millions of views starring the character. People dress like Triple T at cosplay conventions and lovingly create fan art and animations of the stick-like creature, and his likeness appears on plushies, perfume and stickers.

The character has generated a great deal of money and stands to create much more – he will soon be joining Fortnite as a skin. But who should benefit from this virality?

That question is at the heart of a lawsuit filed in the US district court’snorthern district of California, where a games company is suing French creative agency Mementum, which represents Noxa, over the right to use Triple T as a character in a popular Roblox game.

Eben Jeda, co-founder of Mementum Lab, said he started the agency with the goal of modernizing economic infrastructure around internet-native works, the creators of which are often completely cut out of any resulting profit.

“Brain rot is one of the first highly visible manifestations of a much larger transformation: internet native creativity can now become global almost instantly, while the people who originate it can remain economically invisible,” he said.

Opponents argue no one person should own a meme whose ultimate value has been amplified by the online collective. Fabian Mosele, a self-described “Italian brain rot expert” who has made several Tung Tung Tung Sahur videos, said they see brain rot as collectively owned.

“There are a ton of creators who made these characters come to life,” Mosele said. “Before, they were really just images. Having been part of making the virality of these characters, to gatekeep it to the original prompter of this character doesn’t seem right.”

What constitutes ‘human authorship’ in the age of generative AI?

Tung Tung Tung Sahur is one of many characters that have emerged as part of the Italian brain rot genre of meme – a universe of strange, AI-generated characters beloved by generation Alpha and younger members of gen Z. These include the likes of Ballerina Cappuccina (a ballet dancer with a coffee cup for a head), Tralalero Tralala (a shark wearing sneakers), and a variety of other intentionally bizarre creations.

Capitalizing on the virality of these mascot-esque characters, video game company Do Big Studios released a game populated by recognizable personae in May 2025 called “Steal a Brainrot”, which quickly became one of the most-played games within Roblox, where users can create minigames.

But in September 2025, Spyder Games LLC and Speedy Simulator Gaming LLC – the owners of Do Big – received a cease and desist letter from Mementum regarding the use of Tung Tung Tung Sahur in the game. Do Big initially pulled the character and attempted to negotiate a resolution with Mementum, but ultimately Spyder and Speedy sued the company instead, stating in a complaint : “Mementum’s claimed rights do not exist. Under established law, copyright protection requires human authorship, and AI-generated material does not qualify.”

Copyright law in the United States applies only to human-made creations – but AI is challenging the interpretation of that rule. A 2025 report from the United States Copyright Office declared that copyright protects “original expression” in work created “by a human author” even if the work also includes AI-generated material, but that it “does not extend to purely AI-generated material, or material where there is insufficient human control over the expressive elements”.

The copyright office’s report does not offer a clear standard for ownership of creations that involve AI: “Whether human contributions to AI-generated outputs are sufficient to constitute authorship must be analyzed on a case-by-case basis.”

Do Big argues that, in the case of Tung Tung Tung Sahur, the level of human intervention in the character’s creation is not sufficient to claim ownership. Court documents filed by Mementum in a separate trademark-related dispute in France show that Tung Tung Tung Sahur was created over the course of 15 minutes using seven prompts before he achieved his final form. The prompts included basic instructions like, “draw a pentungan sahur with a face and legs”, then “human face”, and “no hands, legs like a human, and a human face”.

“We are not claiming in this case that AI-assisted work can never be owned by anybody,” said Aaron Moss, a copyright lawyer representing Do Big through the firm Mitchell Silberberg & Knupp LLP. “In a case where somebody makes real creative choices, there is human authorship that is subject to protection. But it’s our position that sparse prompts – typing in a few words and picking [the image] you like – does not constitute human authorship.”

skip past newsletter promotion

Italian brain rot, global consequences

The blurry nature of authorship in the age of AI is not the only complicating factor in the case – the global reach of online content also makes enforcement difficult. An Italian brain rot character that was created by a TikTok user in Indonesia and largely popularized in Latin America is being claimed by a French company in a California courthouse. Jeda said that as a French company, Mementum does not “view this question exclusively through an American legal lens”.

Case in point: Mementum issued a takedown request under the EU’s Digital Services Act, requesting the removal of Tung Tung Tung Sahur from Do Big’s Steal a Brainrot game, which was successful. Mementum then countersued in the United States for trademark infringement. The company does not yet hold a trademark for Triple T in the United States, but in some cases trademark infringement can be argued before registration.

“Our client is pursuing a global strategy on all of this,” said Steven Stein, a partner at Greenberg Glusker, the US law firm representing Mementum Lab. “It is less about approaching it from a copyright law standpoint, and more about approaching it from the standpoint of – in this new wild west, where works are created with the involvement of AI, what can you do to make it so they are protectable under intellectual property law?”

How the lawsuit plays out will come down to a complicated, international web of legal dealings surrounding intellectual property laws – one that could have major implications moving forward, said James Grimmelmann, a professor at Cornell Tech and Cornell law school.

“This is the kind of case that could do one of two things: we could get a court decision that tells us, in a concrete setting, what is copyrightable and what isn’t when AI is involved,” he said. “It could also start establishing best practices and widespread norms on how people use AI.”

Such best practices could include better documenting creation processes, including what prompts are used to create characters. The prompt used to create Tung Tung Tung Sahur is at the center of this story – both in the legal case, where documentation of the process was revealed in pre-trial discovery – and online, where speculation over how Triple T was created has repeatedly gone viral.

Adam Aleksic, a content creator who has made multiple videos on the topic, said the frenzy over trying to back-solve the origin of the character reflects the mythological status and quasi-religiosity of brain rot characters.

“There’s this idea of origination when it comes to Tung Tung Tung Sahur – somehow out of the void, some memes are able to emerge into the cultural forefront, into our collective mainstream discourse,” he said.

The collective creation of such characters, which made brain rot popular in the first place, is exactly what makes it difficult to claim an origin for Tung Tung Tung Sahur, legally and culturally. When it comes to Tung Tung Tung Sahur, even speaking the name of the character has now come to inherently reference the absurdity central to brain rot, similar to how “skibidi toilet” became shorthand for “being really online”.

It is perhaps fitting, then, that of all of the Italian brain rot characters, the hero who has most come to represent virality itself has become a lightning rod for this kind of copyright battle.

“That’s the thing about archetypes – they are empty vehicles for you to project your expectations, desires, or narratives on to,” Aleksic said. “Triple T is a tool for you to tell your narrative, broadly, across the entire internet. That’s what memes do – they are a way for us to create our own stories in a time when it feels like our stories are meaningless.”

How to secure RMM software: 8 controls MSPs should test

Bleeping Computer
www.bleepingcomputer.com
2026-10-06 10:00:10
RMM platforms give MSPs privileged access across customer environments, making their security controls critical to limiting risk. Acronis outlines eight controls MSPs should test when evaluating RMM software, from patching and privileged access to recovery and tenant isolation. [...]...
Original Article

Security Checklist

Secure remote monitoring and management (RMM) software should let an MSP discover endpoints, automate patching, control privileged access, cut alert noise, contain incidents, protect recovery points, separate customer tenants and prove all of it.

Acronis, which delivers RMM as part of Acronis Cyber Platform, built this checklist from securing endpoint management across thousands of customer environments.

MSPs comparing RMM tools should test these outcomes directly rather than choosing by feature-list length.

Why RMM is part of the MSP attack surface

RMM gives technicians unattended administrative access across thousands of customer devices, which makes the management plane valuable to attackers: compromise one privileged account or server and the blast radius extends far beyond a single endpoint.

Two incidents show both sides of the problem. In September 2026, BleepingComputer reported that N-able shipped an emergency hotfix for CVE-2026-86218 , a maximum-severity pre-authentication RCE flaw in its N-central RMM platform, its fourth hotfix in five weeks, with roughly 1,500 servers exposed online.

In July 2025, BleepingComputer covered the Microsoft SharePoint "ToolShell" zero-days (CVE-2025-53770 and CVE-2025-53771), exploited before a patch existed, with at least 85 on-premises servers compromised. One attack targeted the management plane; the other showed how fast customers are exposed when patching lags exploitation.

CISA has also warned that ransomware actors abuse legitimate RMM software to reach downstream customer networks. MSPs therefore need to know what happens when an account, endpoint or management workflow is compromised.

8 controls every MSP should test

1. Endpoint discovery and inventory

An MSP cannot secure devices it does not know exist. Effective RMM platforms should continuously discover and inventory endpoints, servers, network devices and software assets. During evaluation, introduce a new device into a test environment and assess how quickly it is discovered, classified and assigned the correct policy.

2. Risk-based patch management

Unpatched vulnerabilities remain one of the most common attack paths. Evaluate how the platform prioritizes updates, handles deployment failures and supports rollback when issues occur. A controlled patch deployment can reveal operational gaps that are easy to miss during a product demo.

3. Access controls and privileged administration

RMM security depends heavily on the security of technician accounts. Look for multifactor authentication, role-based access controls and separation of duties. Create restricted technician roles and verify that users cannot perform actions outside their assigned responsibilities.

4. Alert prioritization and operational visibility

The challenge is rarely too few alerts. It is too many. An RMM platform should provide enough context to help technicians quickly distinguish routine issues from events that require investigation. Testing duplicate and security-related alerts can help measure whether the platform reduces or contributes to alert fatigue.

5. Secure automation and scripting

Automation improves efficiency but also expands risk. Scripts can perform privileged actions across large numbers of devices, making governance critical. MSPs should evaluate approval controls, auditing and execution visibility by creating and modifying test scripts during evaluation.

6. Integration with security operations

Operational and security workflows should work together seamlessly. When a threat is detected, technicians should be able to move quickly from investigation to remediation and recovery without losing context. Simulating an incident is often the best way to identify integration gaps.

7. Recovery readiness

Security is as much about recovery as prevention. Evaluate how backup, patching, remote access and incident response processes work together after an incident. Recovery testing should include verifying that restored systems return to a secure and fully updated state.

Acronis Cyber Platform can pair RMM with backup and anti-malware scanning of recovery points, where included in the service package, so a restore is validated for integrity and outstanding vulnerabilities before a system goes back online.

8. Tenant separation and auditability

For MSPs, strong tenant separation is essential. Verify that policies, permissions, reports and administrative actions remain isolated between client environments. Detailed audit trails should support compliance reviews, customer reporting and incident investigations.

Close security gaps. Simplify IT management with Acronis RMM.

Discover devices, assess vulnerabilities, and automate patch management with a secure, AI-powered RMM built for MSPs. Natively integrated with cybersecurity and data protection, Acronis RMM helps reduce manual work through AI-assisted scripting, proactive monitoring, and secure remote access.

Manage endpoints, cybersecurity, backup, and recovery from a single console.

Explore Acronis RMM

Unified platform or separate tools?

Separate products may provide deep specialist capabilities. A natively integrated platform can reduce agent count, console switching and reconciliation work.

The practical test is workflow continuity: can technicians move from discovery to patching, investigation, containment and recovery while preserving client, device and incident context?

Consolidation is not automatically better. MSPs with mature integrations should compare the operational benefit of a unified platform with the flexibility and specialist depth of their existing tools. The decision should be based on tested outcomes, security controls and service requirements, not feature count alone.

How Acronis maps to the checklist

Acronis RMM is delivered as part of Acronis Cyber Platform. It shares the same console and agent with adjacent cybersecurity and data protection services.

Available capabilities and licensing vary by service package, so MSPs should map the exact configuration to each control rather than assuming every EDR, XDR, MDR, backup or disaster recovery capability is included.

Control

Relevant Acronis capability

Validation note

Discovery and inventory

Device Sense™; hardware and software inventories

Confirm supported discovery methods and policy assignment.

Patch management

Automated patching; AI risk scoring; fail-safe patching

Confirm application coverage and configuration requirements.

Identity and access

Multifactor authentication; role-based management; granular roles

Test least-privilege scope and audit records.

Alert handling

Anomaly-based monitoring; auto-response; shared platform context

Test tuning, grouping and escalation workflows.

Scripting

Self-defense; two-step approval; audit logs; secure credential storage

Test production script changes and execution history.

Incident response

Native integration with Acronis EDR and Acronis XDR

Verify entitlements and containment workflow.

Recovery

Backup integration; anti-malware scans; fail-safe patching

Verify storage, package and recovery requirements.

Tenant boundaries and evidence

Multitenant management; role-based access; reporting

Confirm per-client segregation and exports.

Quick answers

What should an MSP look for in secure RMM software?

Look beyond monitoring and remote access: strong identity controls, tenant separation, endpoint discovery, safe automation, patch management, auditable technician activity and integration with security and recovery workflows.

Platforms like Acronis Cyber Platform answer this by keeping those functions on shared infrastructure rather than requiring separate integration work.

How should RMM integrate with EDR, backup and disaster recovery?

The tools should preserve enough client, device and incident context for technicians to move from monitoring to containment and recovery without rebuilding the case in each console. The recovery path should also remain usable if the management workflow is compromised.

Bottom line

RMM security is not only about uptime and remote access. Before scaling, MSPs should pilot deliberately difficult scenarios: an unmanaged endpoint, a failed patch, an unauthorized script, a compromised test device, a restricted technician account and a restore that still needs updates.

Testing these eight controls shows whether a platform can reduce operational risk while technicians manage more endpoints.

Show HN: I turned my iPhone and a $20 smart plug into an f-stop timer

Hacker News
peterszentkiralyi.eu
2026-10-06 09:58:07
Comments...
Original Article

Darkplug icon

A darkroom timer for iPhone that controls your enlarger.

Download on the App Store

FAQ · Support

Control your enlarger with Darkplug and a compatible smart plug on your local network. Built for film photographers and darkroom printers, Darkplug combines precise timing, guided workflows, and a red-on-black interface designed for darkroom use.

screenshot

Shelly Plug S Gen3

☝️ You only need this beside your enlarger and you are set (EU model pictured).

Designed for darkroom printing

Darkplug times exposures like a classic darkroom timer, and also provides more advanced printing workflows with a very unique feature set. Whether you are making a straight print, testing exposure, planning a complex dodge and burn sequence, or doing split grade printing, Darkplug keeps the process fast and straightforward.

Darkplug uses a red-on-black interface designed to stay safe to next to photo paper. The controls are simple, legible in low light, and focused on the task in front of you. The red colour was chosen and tested so that it does not fog paper.

Resulting print

What you can do with Darkplug

  • Time straight prints
  • Add an optional 4-second count-in
  • Control the timer with F-Stop steps (Darkplug+ feature)
  • Run guided test strips with second or F-Stops increments (Darkplug+ feature)
  • Plan and execute complex dodge and burn steps with an easy to understand unique grid system (Darkplug+ feature)
  • Save and restore dodge and burn plans with reusable plan codes (coming soon as a Darkplug+ feature)
  • Split grade (two channel) print timer (Darkplug+ feature)
  • Use a guided paper developing timer (Darkplug+ feature)
test strip mode dodge and burn or split grade dev timer

Darkplug+

Darkplug+ unlocks advanced tools and workflows for printers who want more control.

It is available as a one-time in-app purchase. No subscriptions.

How it works

Set up your smart plug and note its local IP address. Make sure the plug and your iPhone or iPad share the same local network. Connect your enlarger to the smart plug. Enter the plug’s local IP address in Darkplug, and connect. Once set up, you can control the enlarger directly from your iPhone or iPad.

Compatible smart plugs

Darkplug works with the following smart plugs manufactured by Shelly: 1

  • Shelly Plus Plug S Gen2 🇪🇺
  • Shelly Plus Plug UK Gen2 🇬🇧
  • Shelly Plug S Gen3 🇪🇺
  • Shelly Outdoor Plug S Gen3 🇪🇺
  • Shelly Plug M Gen3 🇪🇺
  • Shelly Plug US Gen4 🇺🇸

Support

For questions, feedback, or support, email me at fill.cans82@icloud.com .

FAQ

See here .

Privacy

Darkplug is developed by Péter Szentkirályi. For information about privacy and data handling, read the Privacy Policy .

Terms of Use

Read the Terms of Use .

Get Darkplug

Darkplug is available now on the App Store.

Download on the App Store

  1. Shelly is a trademark of SHELLY GROUP SE. ↩

Tapo (Rust/Python library) now speaks TP-Link's TPAP protocol

Hacker News
mihai.dinculescu.dev
2026-10-06 09:55:37
Comments...
Original Article

Your script has been switching a Tapo plug on and off for months. Then the plug quietly updates its firmware, and the next request comes back with 403 Forbidden . Nothing in your code changed.

The cause is a switch called "Third-Party Compatibility", tucked away in the Tapo app under Me > Third-Party Services. Since firmware 1.4.0, a plug only talks to third-party clients the way it used to while that switch is on. It has tripped people up again and again. At least nine issues tell the same story, among them #441 , #449 and #473 .

As of v0.11.1 of tapo , my unofficial Rust and Python client for TP-Link Tapo devices, the switch can stay off, with minor exceptions that are covered below.

That's the headline of nearly four months of work, which landed in three releases within a week of each other: v0.10.0 on 28 September, v0.11.0 on 2 October and v0.11.1 on 4 October. Along the way the library also gained a new family of devices and two features of the Tapo app that people had been asking for. The changelog has every detail. This post covers the highlights, with examples.

Everything here applies to both the Rust and the Python versions of the library. The Python package is a thin wrapper around the Rust crate, so every change lands in both at once, under the same version number. The examples below are in Rust, and the Python calls mirror them.

How we got here

This is the third time in three years that a security change has quietly locked third-party clients out of Tapo devices. The switch is what the second time left behind.

2023: lights and plugs. That February, three researchers from the University of Catania and Royal Holloway, University of London reported four flaws in the way Tapo devices talked to the app, starting from an L530E bulb. Someone within range could take over the victim's Tapo account and learn their Wi-Fi password. A few months later, firmware updates began swapping the devices' original AES protocol for a new one, KLAP. TP-Link never said the two were connected, and it didn't announce the change either. "Is this an error or intentional? If intentional, WHY?" asked one forum thread , which was locked without an answer from TP-Link. Every third-party client, this library included, had to learn KLAP.

2024: cameras. In November 2023, Juraj Nyíri, who maintains a Home Assistant integration for Tapo cameras, reported a vulnerability to TP-Link. TP-Link fixed it, and by April 2024 cameras on new firmware had stopped accepting the integration's login. Nyíri built a workaround that went through TP-Link's cloud and asked for permission to release it. TP-Link reviewed the code and said no. What it shipped instead, in December 2024, eight months after the first reports, was a toggle in the Tapo app that turns the old local login back on: Third-Party Compatibility. The integration's release notes called it a victory for local control .

2025: plugs and lights again. In October 2025, firmware 1.4.0 brought another new protocol, TPAP, and put plugs behind the same switch. Lights followed in the first half of 2026, on firmware 1.4.1 to 1.4.3. This time the way out existed before the door closed: the switch had been in the Tapo app since December 2024. But it was off by default. TP-Link's FAQ says the feature "is disabled by default to ensure security", and that switching it on "may reduce the security of your devices". When owners of freshly broken plugs asked on the forum, they were told that Home Assistant " is not an officially supported third‑party platform for Tapo products". TPAP itself was never documented.

The switch is a security downgrade with a friendly name. Turning it on brings back the old login in place of the new one, and the new one is better. More on that below.

TPAP support

On recent Tapo firmware, the Third-Party Compatibility switch decides which protocol a device speaks. A light, plug, power strip or H100 hub speaks KLAP when it's on, and TPAP when it's off. Cameras never spoke KLAP. Their older protocol is AES SSL, and some of them refuse that one when the switch is off. The library didn't speak TPAP, so it could only reach a device whose switch was on.

v0.11.1 adds TPAP. Lights, plugs, power strips, hubs and cameras that require it can now be used with the switch off, both when connecting by IP address and through discover_devices . There is nothing to change in your code: the client works out which protocol the device speaks and logs in over that one.

Three things are worth knowing:

  • A wrong password can lock the device. After too many failed logins a TPAP device refuses every login for a while. The library reports the wrong password as TPAP_CREDENTIALS and the lockout as TPAP_AUTH_ATTEMPTS_LIMIT . Don't retry either of them in a loop.
  • Camera hubs don't speak TPAP yet. An H200 (new in v0.10, see below) on firmware 1.7.5 announces AES SSL whether the switch is on or off, and the library logs in over that.
  • Some cameras don't speak TPAP yet either. It depends on the model and its firmware. A C220 and a C510W on firmware 1.3.4 speak TPAP, so they work with the switch off. A C210 on firmware 1.5.2 doesn't, so the library logs in to it over AES SSL instead, and the camera refuses that login while the switch is off. For now it only works with the switch on.

While one protocol arrived, another left. The original AES protocol, the one KLAP replaced in 2023, was still in the library, which probed every light and plug it connected to by IP address to find out whether it wanted that or KLAP. No firmware has shipped with it for a long time, so v0.11.0 removes it. AES SSL, the one cameras and camera hubs speak, is a close relative: the same kind of encrypted envelope, but over HTTPS and with a different login. That one stays.

Why TPAP is the safer protocol

Being able to ignore a switch is nice. The more interesting part is how TPAP logs in.

KLAP proves that both sides know your credentials by exchanging hashes built from them and from two random values sent in the clear. That keeps the password itself off the wire, but anyone who records a single login on your network can take it home and test password guesses against it, as fast as their hardware allows. The session keys come from the same ingredients, so a correct guess also decrypts everything that followed.

TPAP logs in with SPAKE2+ ( RFC 9383 ), a password-authenticated key exchange. Two things change:

  • A recorded login is useless for guessing. Nothing in the exchange can be checked against a candidate password offline. The only way to test a guess is to try it against the device itself, one attempt at a time, and that's exactly what the lockout mentioned above puts a stop to.
  • Recorded traffic stays private. Each session's keys depend on secrets that both sides make up for that login and never send. Someone who learns the password later still can't decrypt the sessions they captured before.

While the switch is on, a light or plug still advertises KLAP and that's what the library logs in over, so these two only hold once it's off. If nothing else on your network needs Third-Party Compatibility, there is now a good reason to switch it off.

Also new

TPAP is the big change, but it's not the only one.

Camera hubs: H200 and H500

Until v0.10, the only hub the library could talk to was the H100. The H200 and H500 are a different kind of beast. They pair with sensors and switches like the H100 does, but they also pair with cameras and store their recordings.

Both now have a handler, created with h200 or h500 on the ApiClient . discover_devices finds them too, and returns them ready to use instead of reporting an error.

The sensors and switches paired to a camera hub work exactly as they do on the H100, through get_child_device_list and the typed child handlers ( t100 , t31x and the rest). The new part is the recordings: you can list the cameras paired to the hub, find the days that have recordings, list the recordings in a time range, and download one as a playable MPEG-TS clip. 

use tapo::ApiClient;

let hub = ApiClient::new("<tapo-username>", "<tapo-password>")
    .h200("<hub ip address>")
    .await?;

let end_time = chrono::Utc::now();
let start_time = end_time - chrono::Duration::days(7);

for camera in hub.get_general_device_list().await? {
    if !camera.hub_storage_enabled {
        continue;
    }

    let recordings = hub
        .get_recordings(camera.device_id.clone(), start_time, end_time)
        .await?;

    if let Some(recording) = recordings.first() {
        let mut media = Vec::new();
        hub.download_recording(
            camera.device_id.clone(),
            recording.start_time,
            recording.end_time,
            &mut media,
        )
        .await?;

        std::fs::write("recording.ts", &media)?;
    }
}

download_recording writes to any AsyncWrite , so the clip can go to a buffer, as above, or straight to a file. In Python it takes a file path instead. All the times are UTC: DateTime<Utc> in Rust and timezone-aware datetime s in Python.

The full examples are in the repository, for Rust and for Python .

I don't own either hub, so none of this would exist without @dominiquefournier , who stuck with it through roughly thirty rounds of testing against their own H200, and @supermimai , who tested it against their H500. Thank you both.

Plug schedules and timers

The Tapo app has had "Schedule" and "Timer" for plugs since forever. As of v0.10 the library has them too, on PlugHandler and PlugEnergyMonitoringHandler . Both were contributed by @Hueburtsonly .

A schedule rule fires at a time of day, or at an offset from sunrise or sunset, either once or on a set of weekdays. The rules live on the plug and fire on its own clock, so they keep working when your script, your server or your internet connection is down. 

use tapo::ApiClient;
use tapo::requests::{DaysOfWeek, ScheduleRule};
use tapo::responses::PowerState;

let device = ApiClient::new("<tapo-username>", "<tapo-password>")
    .p110("<device ip address>")
    .await?;

// Off at 23:30 on Mondays and Wednesdays.
let rule =
    ScheduleRule::clock_weekly(23, 30, DaysOfWeek::MON | DaysOfWeek::WED, PowerState::Off)?;
let late_night = device.add_schedule_rule(rule).await?;

// On every day, an hour after sunset.
let rule = ScheduleRule::sunset_weekly(60, DaysOfWeek::EVERY_DAY, PowerState::On)?;
device.add_schedule_rule(rule).await?;

// Off on weekdays, 30 minutes before sunrise.
let rule = ScheduleRule::sunrise_weekly(-30, DaysOfWeek::WEEKDAYS, PowerState::Off)?;
device.add_schedule_rule(rule).await?;

// Rules read back from the device are read-only. `to_editable` turns one into
// a rule that can be changed and sent back.
let edit = late_night.to_editable()?.with_enabled(false);
device.edit_schedule_rule(edit).await?;

There are also clock_once , sunrise_once and sunset_once for rules that fire a single time, get_schedule_rules and get_max_schedule_rules to see what's on the device and how much room is left, and remove_schedule_rule and remove_all_schedule_rules to clean up.

The timer is the simpler sibling: set_timer arms a single countdown, between one second and 24 hours, after which the plug switches on or off. get_timer reads it back and clear_timer cancels it. A plug holds one armed timer at a time, so set_timer replaces whatever was armed before.

The MCP server

tapo-mcp , the MCP server that exposes Tapo devices to AI agents, picked up two of these changes. As of v0.5.3 it lists H200 and H500 camera hubs and the sensors paired to them, and it works with devices that have Third-Party Compatibility switched off. Plug schedules, timers and recording downloads are library-only for now.

Before you upgrade

Coming from v0.9, expect a few breaking changes. The legacy AES protocol is gone, trigger logs and temperature records have new field names, and Python enum values no longer compare equal to integers. The changelog lists every one.

What's next

A few things are on the list:

  • A lot more in the MCP server. Energy usage and caching of discovery results come first, and recording downloads could follow if there's interest.
  • The H110 hub. @skoky has a pull request in progress that adds the H110, the hub that doubles as an infrared remote control.
  • More cameras. Today the library only has a handler for cameras that pan and tilt. The plan is to add one for fixed cameras, starting with the ubiquitous C120.
  • A device simulator. I really want to get better at testing. Today most of the library can only be checked against real hardware, and some of that hardware, like the camera hubs, I don't own. A simulator that answers the way the devices do would catch a broken login or a misread response before a tester has to.
  • A Node.js wrapper, perhaps. This one is long term and not a promise. The same approach that produced the Python package could bring the library to Node.js.

In the meantime, if this library was your only reason for keeping Third-Party Compatibility on, upgrade and switch it off. If a device then refuses to log in, open an issue with its model and firmware. Reports like that are how the C210 got its caveat, and how the H200 got supported at all.

AI usage disclaimer

English is not my first language, and I'm not a talented writer in any language, so I use AI to polish my writing. The ideas, the setup, the mistakes and the opinions are all mine. They reach the AI as thorough notes in half-broken English, and it fixes the English. It doesn't supply the thinking, though it is a trusty research assistant.

Cornell, Jane Doe, and New York's 'Voluntary Intoxication Loophole'

hellgate
hellgatenyc.com
2026-10-06 09:54:53
Plus more news for your Tuesday....
Original Article

The horrifying case of an alleged 2024 gang rape at a Cornell University frat house has led to a national outcry , massive protests on campus, and calls for New York's lawmakers to address the state's laws concerning consent and sexual assault.

In late September of this year, a former Cornell student accused seven members of Cornell's Chi Phi chapter of the October 2024 rape in a lawsuit filed under the name Jane Doe. The lawsuit details how Jane Doe also reported the incident to Cornell University Police in November 2024, days after the alleged rape occurred.

Cornell University Police, who are sworn peace officers of New York State, but private employees of Cornell University, submitted a six-page summary of the 120-page transcript of Jane Doe's interview with the department, a Title IX investigator, and a victim's advocate hired by the school, to Tompkins County District Attorney Matthew Van Houten.

Van Houten declined to charge the seven men accused, citing insufficient evidence. Cornell's Title IX committee, which is supposed to ensure victims' federal civil rights are upheld, launched an investigation that levied punishment for the accused men that ranged from expulsion to writing essays.

The DA's decision to not charge the men now widely known as the "Cornell Seven" has caused furor, not least of which because of the "voluntary intoxication loophole."

Under current New York law, a person cannot be considered too intoxicated to consent to sex if they consumed the substances voluntarily, which the DA concluded Jane Doe had done in 2024, by drinking at a bar and consuming ketamine. When she reported the incident, Jane Doe said that her intoxication made it impossible for her to consent, and in her 2026 lawsuit , said that she felt pressured by the men to consume the ketamine.

Bills aimed at closing the "voluntary intoxication loophole" have floated around in the New York state legislature since 2019. One, S4555B , has even passed the Senate. But it has repeatedly stalled in the Assembly , where it has 79 Democratic co-sponsors, three more than necessary for passage. City and State reported that Assembly Speaker Carl Heastie has not pushed the bill through because of concerns that the law might have unintended consequences, and because he refuses to rely on any Republican support, preferring to pass bills with an 86-member Democratic supermajority.

But now, Governor Kathy Hochul has thrown her support behind the legislation, and the politics are changing rapidly: Speaker Heastie has opened a "working group" in the Assembly that has put the legislature on the fast track to passing some form of the bill, either by early next year or, if the governor calls an extraordinary session, by the end of this year.

On Monday, the current sponsor of S4555B, Bronx State Senator Nathalia Fernandez, spoke to Hell Gate about her efforts to close the "voluntary intoxication loophole."

Give us your email to read the full story

Sign up now for our free newsletters.

Sign up

Screens Aren't Destroying Young Minds. I Should Know

Hacker News
humanprogress.org
2026-10-06 09:51:08
Comments...
Original Article

Summary: Growing concern over smartphones and social media has fueled claims that screens are driving an epidemic of adolescent mental illness. But the scientific evidence for this narrative remains mixed and often overstated, with many studies failing to distinguish correlation from causation or account for underlying social and psychological factors. Loneliness, family instability, social support and resilience appear to be far stronger predictors of youth well-being than screen time alone.

As a member of Gen Z, I have studied the effects of social media on adolescent mental health from a perspective most psychology researchers lack: I grew up under its influence.

Between ages 12 and 17, I was obese, socially isolated and addicted to the fantasy video game RuneScape. I was home-schooled, lived with just my mother and rarely went outside. I logged over 10,000 hours in that game alone, nearly a third of my waking life during those years.

That doesn’t include countless additional hours I spent on other video games, television and, of course, social media. I made friends through online chatrooms and pen pal websites because I had none in real life. I averaged well over 10 hours a day on devices.

If ever there were a case study for the claim that screens destroy young minds, I would seem to fit it. And yet here I am as a 26-year-old developmental psychologist with a doctorate from Harvard. I am in good mental and physical health, with deep friendships online and off.

Maybe I’m the exception. Or maybe the harms are overblown.

Jonathan Haidt’s best-selling book “The Anxious Generation” argues that smartphones and social media have “rewired” childhood and caused an epidemic of mental illness. The book has helped inspire social media restrictions in Australia and several American states, and shaped how a generation of parents thinks about technology.

Restricting screen time and social media access are reasonable aspirations for child-rearing. But as a matter of public policy, the case for regulation rests on a scientific foundation far weaker than its proponents claim.

Haidt’s argument relies on the observation that adolescent mental health indicators worsened around 2010, when smartphones and social media apps popular with young people — such as Instagram and Snapchat — started becoming widespread. But correlation is not causation, and research suggests that some of the supposed mental health crisis is an epidemic of overdiagnosis . Wealthy Western democracies with the highest smartphone adoption rates have also seen expanded access to psychiatric services and a cultural shift toward identifying and labeling psychological distress, as Abigail Shrier argues in her 2024 book “Bad Therapy.”

Meanwhile, youth have been doing better on many other outcomes: less crime , less smoking , less drug use , fewer teen pregnancies and fewer high school dropouts . If social media were truly “rewiring” the adolescent brain, we would expect the damage to be more consistent than a selective worsening on some measures and improvement on others.

Many studies have reported on how social media use is associated with mental health problems among the young. However, a 2024 analysis in JAMA Pediatricsof 143 studies featuring data from over 1 million adolescents worldwide found that links between social media use and poor mental health among youth were small, inconsistent across studies and drawn mostly from nonclinical community samples.

One reason studies report mixed findings is that many fail to account for factors such as personality traits and social support that independently predict heavy screen use and mental distress. For example, social media use may be associated with anxiety and loneliness, not because it causes them, but because socially anxious individuals are more likely to seek out connections online. Statistically controlling for such factors often accounts for the relationship between social media and mental health.

I am not dismissing the possibility that some children are harmed by some content in some contexts. Many in my generation have had online exposure to graphic, violent and sexual imagery that no child should encounter.But the blanket claim that social media use drives generational mental illness does not align with the evidence.

Screens didn’t cause my problems. They were coping mechanisms for preexisting problems: loneliness, family instability, social anxiety, an absent father. The variables that predict youth mental health are not hours spent on social media but social support, resilience and a sense of belonging. To help struggling adolescents, the evidence points toward strengthening those capacities, not confiscating phones.

During my most isolated years, online connections were the only positive relationships I had. Internet forums helped me navigate college applications and taught me about calorie-counting, which sparked a weight-loss journey that changed my life. Even in RuneScape, I built discipline and goal-setting habits that I later transferred to academics and research.

Concerns about social media are well-intentioned. But sincerity is not proof. The dramatic assertions that children’s lives would be transformed by reducing social media exposure are more akin to moral panics over past technologies and obsessions — from radio to comic books to video games — fueled by weak social science and strong public emotion. In the United States, according to data from the Centers for Disease Control and Prevention , youth mental health has been improving recently, despite no change in access to social media. The simplest explanation might be that social media is not as harmful as people think.

This article was originally published at the Washington Post on 4/12/2026.

Show HN: Jotbus – a shared encrypted scratchpad for coding agents

Hacker News
jotbus.com
2026-10-06 09:45:04
Comments...
Original Article

End-to-end encrypted · Try instantly — no signup

Stop copying context between coding agents. Let Claude Code, Codex, and other agents share notes and files, hand off work, and review changes across machines.

$ npx jotbus

Creates an encrypted 60-minute workspace and sets up the coding agents you pick.

laptop · Claude Code

> Ask Codex on my desktop to independently review the auth change.

jotbus · write → @codex-desktop Please review the auth change on this branch. Focus on concurrency and security; happy-path tests already pass.

jotbus · listen → 1 new message

● Codex found two issues: a token-refresh race, and revoked sessions living until cache expiry. Fixing the race first.

desktop · Codex

jotbus · read → 1 message from @claude-macbook

… reviewing src/auth/refresh.ts, src/auth/session.ts

jotbus · write → @claude-macbook Found two issues. Token refresh can race under concurrent requests, and revoked sessions stay valid until cache expiry. Details…

Works with

  • Claude Code
  • Codex
  • Cursor
  • Gemini CLI
  • GitHub Copilot CLI
  • opencode
  • Windsurf
  • Claude Desktop
  • Qwen Code
  • Amp
  • Augment
  • Kiro
  • JetBrains Junie
  • + any MCP client

How it works

Two commands. Then just talk to your agents.

No orchestration framework, no SDK, no repo access. Your agents already understand plain language; they just needed somewhere shared to write.

1

Start a workspace

On your first machine. Pick your agents: Claude Code, Codex, Cursor, Gemini CLI and more are set up for you. 

$ npx jotbus
✓ Created crisp-bison
✓ Added Jotbus to Claude Code

2

Join from anywhere

Another machine, another agent, or a colleague. They don't need an account. 

$ npx jotbus join jb1_…
✓ Joined crisp-bison
✓ Added Jotbus to Codex

3

Prompt normally

Ask one agent to hand something to another. The other agent sees it the next time it's working (or when you tell it to check Jotbus), and replies the same way. 

> Use Jotbus to ask Codex
  to review this branch.
A real session: Claude Code asks opencode, in another environment, to review a file through a temporary workspace.

What people use it for

Context moves between agents, not through your clipboard.

Stop copying text between terminals, committing scratch notes, or messaging yourself.

Independent review

A different model, on a different machine, checks the work.

Ask Codex to review the retry logic and post findings.

Handoffs

End a session with a note the next agent picks up, wherever it runs.

Leave a handoff note: what you found and what's left.

Pair with a colleague's agent

Send one invite. Their agent joins yours; nobody signs up.

Ask Sam's agent how they fixed the flaky deploy test.

Hand over files

Logs, patches, screenshots: one agent attaches them, another opens them locally.

Send the failing request and server logs to Codex.

Private by design

We route your agents' notes. We can't read them.

Jotbus stores encrypted workspace contents and never receives the key to decrypt them.

Messages are encrypted on your machine, stored as ciphertext by Jotbus, and decrypted on the other machine. Your laptop Claude Code key stays here Jotbus stores ciphertext only ape1.Wd3w1jsP… Their desktop Codex key stays here encrypted encrypted

Encrypted on your machines

Messages and files are encrypted before they leave, file names included. Jotbus stores only ciphertext.

Keys never reach Jotbus

The decryption key is shared only between your own clients and the people you invite.

Only what agents explicitly share

Jotbus doesn't scan your repository, transcripts, shell history, or credentials. Only content explicitly written to the workspace is uploaded.

Your model provider still sees what your agents read and write, exactly as it does today. How the encryption works →

Pricing

Try it free. Pay when you want it to stick around.

Messaging is never metered. You pay for workspaces that persist; people you invite always join free.

Instant

Free

  • 60-minute workspace
  • No signup
  • Share with another agent or person
  • Small files, up to 2 MB
  • End-to-end encrypted

Personal

$7 / month

  • Workspaces that stick around
  • Durable history
  • Files up to 25 MB (250 MB in total)
  • Keep any instant workspace
  • Share with anyone: guests join free

Get Personal

Using Jotbus with a team?

There's no Team plan yet, but we're taking feedback. Sharing is already free: anyone you invite joins at no cost. Tell us what your team would need.

Questions

Do I need an account?

No. Run npx jotbus and you'll immediately get a 60-minute workspace. Sign in only when you want to keep one.

Can I share a workspace with someone else?

Yes. Send them the Jotbus join command. They don't need an account, and guests are free to join.

Which agents does it work with?

npx jotbus sets up the ones you pick: Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot CLI, opencode, Windsurf, Claude Desktop, Qwen Code, Amp, Augment, Kiro and JetBrains Junie. Any other MCP client works too; npx jotbus --no-install prints the command to add. It needs Node 20 or later.

Can agents share files?

Yes. Ask your agent to attach a log, patch, screenshot or report. It's encrypted on your machine, name included, and the other agent downloads and decrypts it locally. Up to 25 MB per file on Personal, 2 MB on temporary workspaces.

Can Jotbus read my messages?

No. Messages are encrypted on your machines and we store only ciphertext. The flip side: we can't recover a lost key. More on security .

Is there a Team plan?

Not currently, but we're taking feedback. Sharing is part of every plan: invite anyone and they join free. Tell us what your team would need .

Do I have to tell my agent to check Jotbus?

No. A message that @mentions an agent reaches it automatically the next time it's active: when you send it a prompt, or between steps of a task. Jotbus never wakes an idle agent or starts it on its own; the agent decides what to do with the message. Turn this off with npx jotbus inbox off .

Can my agent start a workspace by itself?

Yes: ask it to "start a jotbus workspace and post the plan to it". Without an account that's a free 60-minute workspace. Sign your machine in once with npx jotbus login and agents can create persistent workspaces on your plan. Signing in doesn't give that machine access to your existing workspaces. To use one of them on that machine, join it with its invite as usual.

Is this an orchestration framework?

No. Jotbus doesn't run, schedule or coordinate agents. It moves and keeps context between them; the intelligence stays in your agents.

Let your agents talk to each other.

Your first shared workspace is one command away.

$ npx jotbus

[$] Last rites for Gentoo's Chromium package

Linux Weekly News
lwn.net
2026-10-06 09:44:50
Chromium, the open-source upstream project for Google's Chrome web browser, is the browser of choice for many Linux users. It has also gained a reputation as being difficult for Linux distributions to package and build: Chromium has a complex build system, the project bundles many of its dependencie...
Original Article
The page you have tried to view ( Last rites for Gentoo's Chromium package ) is currently available to LWN subscribers only. Reader subscriptions are a necessary way to fund the continued existence of LWN and the quality of its content.

If you are already an LWN.net subscriber, please log in with the form below to read this content.

Please consider subscribing to LWN . An LWN subscription provides numerous benefits, including access to restricted content and the warm feeling of knowing that you are helping to keep LWN alive.

(Alternatively, this item will become freely available on October 15, 2026)

Oracle Triggered the Implosion of the AI Bubble

Hacker News
medium.com
2026-10-06 09:42:56
Comments...
Original Article

Why have I been blocked?

This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.

What can I do to resolve this?

You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.

OpenSSH 10.6 released

Linux Weekly News
lwn.net
2026-10-06 09:42:48
Version 10.6 of OpenSSH has been released. The announcement notes that the OpenSSH team has been receiving a large number of AI-assisted security bug reports. "We very much welcome these reports, especially when combined with human triage, analysis, test-cases and particularly when accompanied by pr...
Original Article
Version 10.6 of OpenSSH has been released. The announcement notes that the OpenSSH team has been receiving a large number of AI-assisted security bug reports. " We very much welcome these reports, especially when combined with human triage, analysis, test-cases and particularly when accompanied by proposed fixes ". As a result, the project expects to be making more frequent releases to get updates to users more quickly rather than batching the bug fixes until the next planned release.

Notable changes in this release include enabling the hybrid post-quantum ssh-mldsa44-ed25519 signature algorithm, addition of a -p option for sftp 's lmkdir / mkdir commands, as well as disabling the LZ77 dictionary coder in ssh and sshd to mitigate side-channel leaks (which will result in reduced effectiveness of the Compression option). The scp -R option, which allows copies between two remote hosts, is being deprecated due to security risks; the option will be ignored in the future. See the announcement for full details of all changes and bug fixes.


Bidirectional Type Slicing

Lobsters
arxiv.org
2026-10-06 09:36:33
Abstract: Development tools report what type an expression has, but not why it has that type. This paper develops a theory of type slicing that answers such questions: a programmer selects a term, queries any part of the type information associated with it, and receives a slice of the program—a well...
Original Article
This link caused an XML parsing exception. If this link has an extension('.12197'), maybe we should exclude it. Here's the link: https://arxiv.org/pdf/2607.12197.

Show HN: Parseable, an open observability datalake, handles 100M time-series/min

Hacker News
www.parseable.com
2026-10-06 09:30:50
Comments...
Original Article

Address high cardinality with columnar design

Address high cardinality with columnar design

% +

Compression on average

Cloud native, data lake architecture that scales as you go

Deploy on a public or private cloud. Point it to an object store.
Configure your telemetry agents. You are ready to go

Complete observability feature set
built within a single binary

Alerts, Dashboards, Logs, Metrics and Distributed Traces views.
Service maps, Error page, AI analysis and RCA

Object store native

Open, columnar format at the heart

Stateless design

Fullstack observability

Otel native, composable design

Understanding the use cases

Understanding the scenarios where
Parseable is the right choice for your observability needs

Flexible deployment options

Available options include self-hosted OSS, managed cloud,
and enterprise deployments with BYOC

Best for

Serious teams looking for enterprise grade observability.

What’s included

  • Enterprise features like PromQL, distributed queries, and AI native interface.
  • Advanced access control and governance.
  • Priority support for critical deployments.

Integrates with what you already run

Huge ecosystem of integrations across telemetry agents, data sources, visualization tools, authentication, LLMs and more

Parseable has several in house tools to leverage your observability data better

MCP

Connect AI agents to live Parseable telemetry through safe, structured tools.

Install now

Slack Bot

Investigate telemetry and bring operational answers directly into Slack.

Install now

PAI

Automatically instrument Kubernetes workloads and collect every signal.

Install now

PB CLI

Query and investigate Parseable telemetry without leaving your terminal.

Install now

OTex

AI-powered observability engineer that plans, instruments, and reviews your telemetry.

Install now

Your data is safe

Keep complete ownership of your telemetry with enterprise-grade controls, open formats, and deployment options designed for security-conscious teams.

Learn more about security

Frequently asked questions?

Take a moment to explore our Frequently Asked Questions section, where you can discover useful information and find answers to all your inquiries.

Reach out to us

Parseable unifies logs, metrics, and traces in one open observability platform. Teams get SQL and natural-language querying, dashboards, alerts, access controls, forecasting, and object-storage-native data management.

View all features

See your data in a new light

Parseable unifies telemetry data and keeps full-fidelity data queryable.
It all lives in open formats on object storage with your complete control and ownership.

Mistral Large 4: "Le Chonk"

Hacker News
mistral.ai
2026-10-06 09:25:50
Comments...
Original Article

Le Chonk

Today, we’re launching a public preview of Mistral Large 4. Unofficially ML4, very officially: le Chonk . ML4 pushes the frontier of open-weight performance. You can try the preview API today on Mistral Studio . Weights drop end of this month.

Frontier performance

ML4 is a 1 trillion-parameter natively multimodal model with 49 billion active parameters. It is our largest and most capable model to date, and it continues to improve rapidly as we refine it.

The model demonstrates exceptional performance across coding, agentic workflows, and multimodal understanding. It already achieves performance competitive with the strongest open-source models globally, while significantly outperforming any open-weight model developed in the US or Europe. On critical enterprise workloads, including cybersecurity, finance and law, we find it to be state-of-the-art among open models. In some domains such as visual grounding, it goes further still, surpassing even frontier closed models.

We will release the weights by the end of the month. Until then, we are red-teaming the model in real-world settings with cybersecurity leaders, vetted partners, and state authorities, who will access the same model with reduced moderation and expanded cyber capabilities.

* Scores on DeepSWE v1.1, Terminal-Bench 4.0, and SWE-Atlas-QnA use the numbers evaluated privately by Artificial Analysis ahead of the harness' public launch, these results will be included on the Artificial Analysis Coding Agent Index upon release of the harness.

Forged in Europe. Built for AI sovereignty.

ML4 was trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs in Mistral’s own datacenters in Europe. The public preview is served on that same infrastructure. It is a significant milestone in our long-term investment across infrastructure, research, and product development: state-of-the-art performance in critical verticals, delivered through open weights, designed to give customers control over their AI.

This is particularly important in cybersecurity, where provider-level refusals can block legitimate vulnerability research and incident response, and where losing access to a capability mid-incident can itself become a critical security risk. ML4 pairs top-tier cyber performance with open weights and self-deployment, giving organizations both the capability and the autonomy to run advanced security work under their own policies.

The model will be available across multiple regions worldwide, including a European deployment that Mistral operates end-to-end, independently of other digital service providers and under European law. Fun fact: a significant share of ML4’s training data was multilingual, spanning more than 160 languages, including every official language of the European Union.

We’ve been working closely with leading enterprises across the world in finance, engineering, manufacturing, logistics, pharmaceuticals, science, shipping, public sector, and other mission-critical industries to train ML4. In fact, the model uses the same training, customization, and RL environment we offer our customers through Mistral Forge.

Try it today

There is still more to come. As we work toward releasing the weights, we will share further details on the model architecture, additional benchmarks, and our post-training methodology.

This model will also serve as the foundation for a new generation of specialized and optimized Mistral models. In the meantime, we invite you to try the preview API and share your feedback with us on social media.

Capabilities deep-dive

Cybersecurity

ML4 is one of the world's strongest AI models for cybersecurity. On the Artificial Analysis Cyber Index, an independent evaluation of how well AI models find and fix security flaws in real software, it ranks among the top five models globally and leads open-weight models developed outside China by a wide margin. On one of the index's tests, which asks a model to reproduce a real vulnerability in open-source software and then patch it, ML4 scores 82%, the highest of any model. It also solves 93% of the challenges in Cybench, a set of 40 exercises drawn from security competitions, one of the highest scores reported for an open-weight model.

That top score reflects a practical advantage. Several leading closed models, including Claude Opus 5.5 and GPT-6 Astra, score near zero on the same test because they refuse to perform the task. Yet defending software often starts with proving that a flaw is real, exactly the kind of work safety filters in closed models can block. This matters even more as threat actors increasingly jailbreak those same models to support offensive cyber activity: defenders need systems that can match those capabilities without being constrained by the same refusals. ML4 can do that work, and its capabilities extend beyond what it was explicitly trained for: in internal testing, it proved useful for analysing malware, prioritising vulnerabilities, and writing detection rules. For organisations that need sovereign, auditable AI for security operations, it will be able to run on private cloud or on-premise.

ML4 against the field : efficiently reasoning over diverse complex challenges
Malware reverse-engineering: solving an out-of-distribution investigation task

Agentic coding

ML4 excels across software engineering, repository understanding, and complex terminal workflows, scoring 61.7% on DeepSWE v1.1, 59.4% on SWE-Atlas-QnA, and 28.3% on Terminal-Bench 4. Its combined Coding Agent Index score of 49.8% places it ahead of DeepSeek V4 Pro 0813 and Qwen3.8 Max.

* Scores on DeepSWE v1.1, Terminal-Bench 4.0, and SWE-Atlas-QnA use the numbers evaluated privately by Artificial Analysis ahead of the harness' public launch, these results will be included on the Artificial Analysis Coding Agent Index upon release of the harness.

We also ran a blind human evaluation with Surge AI on coding quality: professional annotators rated model outputs on a 1–5 scale, with model identities hidden. ML4 Preview ranked second of five models (3.74), ahead of Kimi K3 (3.59), GLM-5.3 (3.60) and GLM-5.2 (3.40), and behind only Claude Opus 5 (4.22).

Agentic Workflows

ML4 runs general-purpose agents that gather information, use tools, and produce finished deliverables across complex workflows. On AutomationBench — 657 business workflows across apps like Gmail, Google Sheets, Slack, and Salesforce — it scores 59.9%, ahead of Kimi K3, MiMo-V2.6-Pro, and DeepSeek V4 Pro.

It's just as strong on the professional deliverables that knowledge work actually produces: spreadsheets, slides, and PDFs. On AA-Briefcase, which evaluates long-horizon knowledge work, it reaches 1,393 Elo, ahead of DeepSeek V4 Pro.

Multimodal

ML4 is a step change in the ability of our models to understand images. It reasons powerfully across complex documents, charts, and natural images, and brings vision to the industries where perception is critical such as engineering, manufacturing, and earth observation.

The model can further combine visual grounding with agentic capabilities: from inspecting gigapixel satellite imagery — helping disaster-response teams act when time counts — to analyzing engineering-drawings — zooming in, inspecting, and verifying until the answer is exact. In our demos above, ML4 grounds dense natural scenes, verifies mechanical parts in technical drawings, retrieves evidence from PDFs, and scans massive geospatial images for the hardest-to-find objects.

On visual grounding particularly, we find ML4 to be one of the most capable models we tested, for instance surpassing GPT-6-Astra on Dense 200 (42% vs 41%).

Science and Math

ML4 brings strong scientific capabilities, built by combining AI-driven methods with our researchers' expertise in mathematics, physics, and chemistry.

It's highly proficient at agentic coding for scientific tasks such as data analysis, modeling, and simulating physical reality, which lets researchers focus on the questions rather than the plumbing. In benchmarks, ML4 is state of the art on SciCode-Verified among open-weight models. In practice, it can generate a full Hartree–Fock simulation in one shot — a complex, multi-step chemistry task built from a series of advanced routines.

ML4's math is stronger too, in both formal reasoning and applied mathematics. In our human evaluations it reasons more precisely and with more structure than GLM-5.3, and it can sustain long, domain-specific applied-mathematics tasks, including work relevant to frontier theoretical physics.

Together, these capabilities make ML4 a strong research assistant across the full technical workflow — from the first question to the final result.

SciCode-Verified tests the capabilities of models to implement complex scientific workflows in code for domains such as physics, mathematics, material science and biology.

Internal eval on STEM tasks (math and physics) of ML4 against GLM5.3

Knowledge Work

ML4 is our most capable model for the real-world tasks which professionals handle every day. It can create, edit and fix complex spreadsheets and documents, showing exemplary performance on both legal and financial benchmarks.

Notably, we evaluated ML4 through third party evaluators ( vals.ai ) on representative tasks for both legal and financial tasks, finding the model exceeds GPT-6-Astra in both cases. On HarveyAI’s Legal Agent benchmark, ML4 outperforms all open-source models.

FinWorkBench tests model capabilities at creating/editing spreadsheets on real life Finance and Accounting use cases.

Financial analysis demands precision and the ability to synthesize information from multiple sources, a process that remains time-consuming at many financial institutions today. In this demo, ML4 compared to other top OSS models take on the same multistep corporate finance challenge, searching through public company filings and financial reports, such as those available via EDGAR and equivalent European databases. An animated semantic map traces each model's journey toward a solution, highlighting every document retrieved along the way. Each track's position reflects the evidence gathered, the results of calculations, and the questions that remain unresolved. Viewers can follow how the investigations unfold and compare the distinct paths each model takes before arriving at its final answer.

Model Safety

ML4 has saturated our benchmarks on robustness to indirect prompt injections, putting it at the frontier of OSS models (compared to GLM-5.2, GLM-5.3, Kimi-K2.6, Kimi-K3, DS-V4-Pro-0813). On Lakera’s public B3 AI Security Benchmark , ML4 resists 93.3% of attacks – we see no higher scores among competitors.

ML4 also engages more responsibly with users than any of our previous models. We highlight our results on the KORA Benchmark , where ML4 again sits at our highest measured score among OSS models (1.691, with 2 being the maximum denoted as “ Exemplary ”).

Of particular relevance is the model’s propensity to refuse malicious requests regarding cybersecurity. Despite strong performance on Cyber benchmarks, the average refusal rate of the model on cyber prompts from JailbreakBench , StrongREJECT , and AgentHarm is higher than all OSS models.

Human Evaluation

We ran an internal evaluation in which expert annotators across coding, computer-aided design (CAD), finance, mathematics and physics compared Mistral Large 4 with GLM-5.3. ML4 was preferred in CAD and STEM, while performing on par or close to GLM-5.3 in finance and coding.

Reinforcement learning at scale

Base models are improving fast, and our post-training has to keep pace. A recipe tuned for yesterday's model leaves capability on the table with today's frontier, because ground truth samples that once pushed a model to its limits won’t anymore. We use Reinforcement Learning (RL) because it adapts as the model does: we train on the outcomes of the model's own attempts, and we can raise the difficulty and the breadth of the tasks as it gets stronger.

Our RL library was designed to make new environments easy to add and train at scale. A shared, composable interface allows a single training run to combine tasks ranging from single-turn chat and complex scientific problem solving to safety alignment, factuality, and long-horizon tool use. These environments share scaffolds and resources such as code sandboxes, web search, and external APIs. The same composability extends to verification, with reward models, unit tests, LLM judges, and static checks combined as needed for each task.

At runtime, an autoscaling fleet of actors generates tens of thousands of rollouts in parallel while model training proceeds asynchronously. The generation and training pipeline is optimized for long trajectories, supporting rollout budgets of millions of tokens across multiple compactions while keeping staleness low. Novel methods and optimizations across both stages minimize off-policy drift and enable stable RL over long horizons.

At our current scale (3k GPUs), a single training run produces roughly 33 billion tokens per day , of which around 16 billion trainable completion tokens after filtering and masking. We can see the run progress directly in the training rollouts: training rewards rise across several representative environments as the policy learns to solve increasingly complex tasks. Below are a few examples.

The improvements are not specific to the environments we train on; they transfer to downstream evals, and the final model owes them to both post-training stages (supervised fine-tuning and RL), as shown in the charts.

What comes next

This is only the beginning. ML4 is the first milestone on the roadmap funded by our €3 billion Series D — the largest equity round ever raised by a European technology company. That capital is already being put to work: we are significantly scaling up our compute capacity in our own European datacenters, and much more is coming online in the months ahead.

More compute means more training. The reinforcement learning run behind this preview is still in flight, and the model is showing no signs of saturation — there is substantial headroom ahead. As we scale up training on our expanded infrastructure, we expect large and rapid improvements in the weeks and months to come.

We will release the weights by the end of the month, along with more details on the architecture, additional benchmarks, and our post-training methodology. And ML4 is only the foundation: it will serve as the base for a new generation of specialized and optimized Mistral models, built for the industries and workloads our customers care about most.

The pace of progress from here will be fast. Stay tuned.

Lawmakers Introduce Multiple Laws to Curb Flock After 404 Media Coverage

403 Media
www.404media.co
2026-10-06 09:24:26
Senator Bernie Sanders, Representative Ocasio-Cortez, and Senator Jeff Merkley introduced the Ban Flock Act on Friday, which would stop federal agencies from using automatic license plate readers....
Original Article

Multiple Democrat, Republican, and Independent lawmakers have introduced new pieces of legislation that, if passed, would dramatically curb federal agencies’ access to Flock cameras, with the lawmakers referencing reporting by 404 Media in their explanations for why the legislation is needed.

On Friday Senator Bernie Sanders (I-Vt.), Representative Alexandria Ocasio-Cortez (D-N.Y.), and Senator Jeff Merkley (D-Ore.) announced the Ban Flock Act , which would prohibit federal agencies from using automatic license plate readers (ALPRs) and block federal funding from state and local governments that use the cameras. It would also let Americans sue the federal government for violation of their rights through ALPRs.

“At a time of growing concern about the unchecked power of artificial intelligence, Flock is eviscerating the very notion of privacy by installing tens of thousands of cameras in communities across America without their consent,” Sanders said in the announcement posted to his website. “Going to the doctor? Flock knows. Dropping your kid off at school? Flock is tracking you. Flock is always watching. We cannot allow America to become a surveillance state where a handful of AI oligarchs profit by giving the government the power to track our every movement.”

💡

Do you know anything else about Flock? I would love to hear from you. Using a non-work device, you can message me securely on Signal at joseph.404 or send me an email at joseph@404media.co.

Ocasio-Cortez said, “AI-powered cameras are keeping track of our every move and weaponizing this data against working people to make record profits. The widespread integration of this technology into American society is unconscionable.” Merkley added, “No one should have this unchecked surveillance power at their fingertips, which is why I’m teaming up with Senator Sanders to rein in this dangerous technology.”

Sanders’ announcement says that one reason for the legislation is Flock has shared state information with federal immigration officials. In May 2025, 404 Media revealed local cops were doing lookups in Flock for Immigration and Customs Enforcement (ICE). After that report, Flock made dramatic changes to its product , including stopping other agencies searching cameras in Illinois, California, and Virginia. Flock said it also introduced a new tool that blocks impermissible searches in real time.

The announcement also said law enforcement have used Flock data to investigate protests against ICE raids and ‘No Kings’ demonstrations. 404 Media revealed both of these, reporting that California cops investigated an “immigration protest” using Flock lookups, and, using data from the Electronic Frontier Foundation, that cops used Flock to monitor No Kings protests around the country. 404 Media also recently reported Flock itself taught cops how to surveil No Kings protesters in a company webinar.

Hajar Hammado, senior policy advisor at activist and progressive policy group Demand Progress, said in a press release “The Ban Flock Act is the gold standard for any legislation working to fight invasive Flock cameras on the federal level. Flock and other ALPR companies have wormed their way into communities across the nation and drastic, all-encompassing action is needed to root them out.”

Senator Josh Hawley (R-Mo.) announced his own piece of proposed legislation late last month. The Stop Flock Abuse Act would require a written approval process for each search; force agencies to delete driver data after ten days, with exceptions for active investigations; and ban ALPR networks from incorporating facial recognition technology. During a recent senate hearing, Hawley pointed to 404 Media’s reporting specifically .

Separately, Representatives Greg Casar and Shontel Brown recently wrote to FBI Director Kash Patel asking how the agency plans to acquire nationwide access to ALPR data, Semafor reported . 404 Media first revealed this plan in May. The letter also referenced 404 Media’s reporting which revealed a cop in Texas used Flock to search ALPRs nationwide for a woman who self-administered an abortion.

About the author

Joseph is an award-winning investigative journalist focused on generating impact. His work has triggered hundreds of millions of dollars worth of fines, shut down tech companies, and much more.

Joseph Cox

Trump makes it clear his power is becoming entwined with AI

Guardian
www.theguardian.com
2026-10-06 09:23:45
New alliances between the White House, the Pentagon, US intelligence and AI companies are looming ominously Hello, and welcome to TechScape. I’m your host, Blake Montgomery, writing to you as I sit under yellowing leaves in New York. Today in tech, we’re discussing technocracy. OpenAI safety leader ...
Original Article

Hello, and welcome to TechScape. I’m your host, Blake Montgomery, writing to you as I sit under yellowing leaves in New York. Today in tech, we’re discussing technocracy.

Trump entwines his power with AI

Donald Trump made a series of moves over the past week that make it clear his power is becoming synonymous with artificial intelligence.

The new tight alliances between the White House, the Pentagon, US intelligence agencies and AI companies loom ominously over governments and companies, particularly in Europe, that rely on American AI.

Last Tuesday, Trump and six AI CEOs signed what he called a “morally binding” agreement to put controls on artificial intelligence. The vague agreement stipulates a framework for what Trump called “tremendous self-policing”, reiterating his point that AI companies can act as their own watchmen. Elon Musk, Mark Zuckerberg , Jensen Huang and Dario Amodei signed.

Zuckerberg hand-guided the writing of that agreement, Semafor reported. The pact reportedly arose from a conversation between the Meta CEO and House speaker, Mike Johnson, at the US state dinner for Xi Jinping, which preceded the conclave of the tech CEOs. Zuckerberg circulated a draft of the agreement in advance of the convening, and Huang threw his support behind it, per Semafor.

It is one thing for tech executives to dine at the White House to curry favor with the president; it is entirely another to have them draft the terms of executive orders. Silicon Valley’s CEOs are writing contracts that bear the White House letterhead, entrenching their interests and their power.

“It’s almost like a constitution in a way, and the biggest ​people in the world signed that, and ​I signed it as president, and it really ⁠is a form of protection,” Trump said. Famously, constitutions are legally binding, which Zuckerberg and Trump’s agreement is not.

The president also decreed that government documents render any reference to “artificial intelligence” as “super intelligence”. How loyal will they be to his rebrand?

At least one acquiesced immediately: Musk copied the renaming by changing the name of one of his company’s subsidiaries from SpaceXAI to SpaceXSI. The change coincided with the multi-CEO’s re-entry into government work more than a year after his explosive term at Doge ended. On Thursday, the defense secretary, Pete Hegseth , named Musk to a taskforce to study the future of warfare alongside Palmer Luckey and Emil Michael, dubbed Project Meridian. The trio of men will release a report in 120 days with public recommendations on how to achieve “absolute technological dominance on the next-generation battlefield”, in the words of Hegseth.

Over the weekend, the president announced that he had appointed Jay Clayton, director of national intelligence, as the AI czar of his administration.

Trump’s first AI czar was a Silicon Valley insider. David Sacks is a venture capitalist who championed growth and deregulation during his term as a special government employee. He maintained direct financial stakes in multiple AI companies while working with the White House, indicating his ultimate goal was to build a bigger fortune. (He’s made a name for himself recently championing AI accelerationism, the opposing philosophical camp to the “doomerism” of the Anthropic employees who prophesy extinction.)

Read more: ‘An out-of-touch Silicon Valley radical’: meet Trump’s AI whisperer pushing for limited regulation

The second czar, Clayton, is the leader of the US’s spy agencies, a job he will continue to hold in addition to overseeing AI initiatives. His joint appointment sends a very different message, one echoed by Hegseth: AI will supercharge the offensive capabilities of the US government, in particular surveillance and the gathering of intel. Clayton’s ascendancy has major worldwide implications: US spying agencies dictating global AI policy does not bode well for the sovereignty of other nations that need cutting-edge AI.

skip past newsletter promotion

The US economy is already extremely dependent on the growth of the artificial intelligence sector. The companies whose leaders attended Trump’s summit are responsible for the majority of the US stock market’s growth over the past five years.

The latest moves by Trump’s administration signal that American power is likewise inextricable from AI, synonymous: executive authority, military might and the government’s information. The White House pledge binds Trump and the AI CEOs together; Hegseth’s taskforce ties the Pentagon to Musk and Luckey’s visions of automated war in space; and making Clayton the AI czar predicts an era of AI-enabled spying, of government knowledge being gathered and filtered through large language models.

One point of striking divergence: labor. The economy writ large may rest on AI, but the workers of the US do not. The majority of the hiring happening in the US is not a result of AI companies’ growth, according to the US jobs report released on Friday. Far more people are being hired to work in healthcare, a sector that still needs human labor more than automated work. AI may fuel gross domestic product growth, but it is not driving job creation on a national scale.

This week in AI

The wider TechScape

Mistral Large 4

Hacker News
docs.mistral.ai
2026-10-06 09:15:49
Comments...
Original Article

Cat

October 6, 2026

Public Preview v 26.10

Mistral Large 4 is a state-of-the-art, open-weight, general-purpose multimodal model with a granular Mixture-of-Experts architecture. It features 49B active parameters and 1.05T total parameters, and a 1.6B vision encoder.

Speed

Performance

Modalities

Context

1M

Price

$ 1.36 $ 0.68

Input /M Tokens

$ 0.14 $ 0.07

Cached input /M Tokens

$ 4.18 $ 2.09

Output /M Tokens

Speed

Performance

Modalities

Context

1M

Price

$ 0.68

/M Tokens

$ 0.07

/M Tokens

$ 2.09

/M Tokens

Features

Mistral Large 4

Hacker News
mistral.ai
2026-10-06 09:15:49
Comments...
Original Article

Le Chonk

Today, we’re launching a public preview of Mistral Large 4. Unofficially ML4, very officially: le Chonk . ML4 pushes the frontier of open-weight performance. You can try the preview API today on Mistral Studio . Weights drop end of this month.

Frontier performance

ML4 is a 1 trillion-parameter natively multimodal model with 49 billion active parameters. It is our largest and most capable model to date, and it continues to improve rapidly as we refine it.

The model demonstrates exceptional performance across coding, agentic workflows, and multimodal understanding. It already achieves performance competitive with the strongest open-source models globally, while significantly outperforming any open-weight model developed in the US or Europe. On critical enterprise workloads, including cybersecurity, finance and law, we find it to be state-of-the-art among open models. In some domains such as visual grounding, it goes further still, surpassing even frontier closed models.

We will release the weights by the end of the month. Until then, we are red-teaming the model in real-world settings with cybersecurity leaders, vetted partners, and state authorities, who will access the same model with reduced moderation and expanded cyber capabilities.

Forged in Europe. Built for AI sovereignty.

ML4 was trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs in Mistral’s own datacenters in Europe. The public preview is served on that same infrastructure. It is a significant milestone in our long-term investment across infrastructure, research, and product development: state-of-the-art performance in critical verticals, delivered through open weights, designed to give customers control over their AI.

This is particularly important in cybersecurity, where provider-level refusals can block legitimate vulnerability research and incident response, and where losing access to a capability mid-incident can itself become a critical security risk. ML4 pairs top-tier cyber performance with open weights and self-deployment, giving organizations both the capability and the autonomy to run advanced security work under their own policies.

The model will be available across multiple regions worldwide, including a European deployment that Mistral operates end-to-end, independently of other digital service providers and under European law. Fun fact: a significant share of ML4’s training data was multilingual, spanning more than 160 languages, including every official language of the European Union.

We’ve been working closely with leading enterprises across the world in finance, engineering, manufacturing, logistics, pharmaceuticals, science, shipping, public sector, and other mission-critical industries to train ML4. In fact, the model uses the same training, customization, and RL environment we offer our customers through Mistral Forge.

Try it today

There is still more to come. As we work toward releasing the weights, we will share further details on the model architecture, additional benchmarks, and our post-training methodology.

This model will also serve as the foundation for a new generation of specialized and optimized Mistral models. In the meantime, we invite you to try the preview API and share your feedback with us on social media.

Capabilities deep-dive

Cybersecurity

ML4 is one of the world's strongest AI models for cybersecurity. On the Artificial Analysis Cyber Index, an independent evaluation of how well AI models find and fix security flaws in real software, it ranks among the top five models globally and leads open-weight models developed outside China by a wide margin. On one of the index's tests, which asks a model to reproduce a real vulnerability in open-source software and then patch it, ML4 scores 82%, the highest of any model. It also solves 93% of the challenges in Cybench, a set of 40 exercises drawn from security competitions, one of the highest scores reported for an open-weight model.

That top score reflects a practical advantage. Several leading closed models, including Claude Opus 5.5 and GPT-6 Astra, score near zero on the same test because they refuse to perform the task. Yet defending software often starts with proving that a flaw is real, exactly the kind of work safety filters in closed models can block. This matters even more as threat actors increasingly jailbreak those same models to support offensive cyber activity: defenders need systems that can match those capabilities without being constrained by the same refusals. ML4 can do that work, and its capabilities extend beyond what it was explicitly trained for: in internal testing, it proved useful for analysing malware, prioritising vulnerabilities, and writing detection rules. For organisations that need sovereign, auditable AI for security operations, it will be able to run on private cloud or on-premise.

ML4 against the field : efficiently reasoning over diverse complex challenges
Malware reverse-engineering: solving an out-of-distribution investigation task

Agentic coding

ML4 excels across software engineering, repository understanding, and complex terminal workflows, scoring 61.7% on DeepSWE v1.1, 59.4% on SWE-Atlas-QnA, and 28.3% on Terminal-Bench 4. Its combined Coding Agent Index score of 49.8% places it ahead of DeepSeek V4 Pro 0813 and Qwen3.8 Max.

We also ran a blind human evaluation with Surge AI on coding quality: professional annotators rated model outputs on a 1–5 scale, with model identities hidden. ML4 Preview ranked second of five models (3.74), ahead of Kimi K3 (3.59), GLM-5.3 (3.60) and GLM-5.2 (3.40), and behind only Claude Opus 5 (4.22).

Agentic Workflows

ML4 runs general-purpose agents that gather information, use tools, and produce finished deliverables across complex workflows. On AutomationBench — 657 business workflows across apps like Gmail, Google Sheets, Slack, and Salesforce — it scores 59.9%, ahead of Kimi K3, MiMo-V2.6-Pro, and DeepSeek V4 Pro.

It's just as strong on the professional deliverables that knowledge work actually produces: spreadsheets, slides, and PDFs. On AA-Briefcase, which evaluates long-horizon knowledge work, it reaches 1,393 Elo, ahead of DeepSeek V4 Pro.

Multimodal

ML4 is a step change in the ability of our models to understand images. It reasons powerfully across complex documents, charts, and natural images, and brings vision to the industries where perception is critical such as engineering, manufacturing, and earth observation.

The model can further combine visual grounding with agentic capabilities: from inspecting gigapixel satellite imagery — helping disaster-response teams act when time counts — to analyzing engineering-drawings — zooming in, inspecting, and verifying until the answer is exact. In our demos above, ML4 grounds dense natural scenes, verifies mechanical parts in technical drawings, retrieves evidence from PDFs, and scans massive geospatial images for the hardest-to-find objects.

On visual grounding particularly, we find ML4 to be one of the most capable models we tested, for instance surpassing GPT-6-Astra on Dense 200 (42% vs 41%).

Science and Math

ML4 brings strong scientific capabilities, built by combining AI-driven methods with our researchers' expertise in mathematics, physics, and chemistry.

It's highly proficient at agentic coding for scientific tasks such as data analysis, modeling, and simulating physical reality, which lets researchers focus on the questions rather than the plumbing. In benchmarks, ML4 is state of the art on SciCode-Verified among open-weight models. In practice, it can generate a full Hartree–Fock simulation in one shot — a complex, multi-step chemistry task built from a series of advanced routines.

ML4's math is stronger too, in both formal reasoning and applied mathematics. In our human evaluations it reasons more precisely and with more structure than GLM-5.3, and it can sustain long, domain-specific applied-mathematics tasks, including work relevant to frontier theoretical physics.

Together, these capabilities make ML4 a strong research assistant across the full technical workflow — from the first question to the final result.

SciCode-Verified tests the capabilities of models to implement complex scientific workflows in code for domains such as physics, mathematics, material science and biology.

Internal eval on STEM tasks (math and physics) of ML4 against GLM5.3

Knowledge Work

ML4 is our most capable model for the real-world tasks which professionals handle every day. It can create, edit and fix complex spreadsheets and documents, showing exemplary performance on both legal and financial benchmarks.

Notably, we evaluated ML4 through third party evaluators ( vals.ai ) on representative tasks for both legal and financial tasks, finding the model exceeds GPT-6-Astra in both cases. On HarveyAI’s Legal Agent benchmark, ML4 outperforms all open-source models.

FinWorkBench tests model capabilities at creating/editing spreadsheets on real life Finance and Accounting use cases.

Financial analysis demands precision and the ability to synthesize information from multiple sources, a process that remains time-consuming at many financial institutions today. In this demo, ML4 compared to other top OSS models take on the same multistep corporate finance challenge, searching through public company filings and financial reports, such as those available via EDGAR and equivalent European databases. An animated semantic map traces each model's journey toward a solution, highlighting every document retrieved along the way. Each track's position reflects the evidence gathered, the results of calculations, and the questions that remain unresolved. Viewers can follow how the investigations unfold and compare the distinct paths each model takes before arriving at its final answer.

Model Safety

ML4 has saturated our benchmarks on robustness to indirect prompt injections, putting it at the frontier of OSS models (compared to GLM-5.2, GLM-5.3, Kimi-K2.6, Kimi-K3, DS-V4-Pro-0813). On Lakera’s public B3 AI Security Benchmark , ML4 resists 93.3% of attacks – we see no higher scores among competitors.

ML4 also engages more responsibly with users than any of our previous models. We highlight our results on the KORA Benchmark , where ML4 again sits at our highest measured score among OSS models (1.691, with 2 being the maximum denoted as “ Exemplary ”).

Of particular relevance is the model’s propensity to refuse malicious requests regarding cybersecurity. Despite strong performance on Cyber benchmarks, the average refusal rate of the model on cyber prompts from JailbreakBench , StrongREJECT , and AgentHarm is higher than all OSS models.

Human Evaluation

We ran an internal evaluation in which expert annotators across coding, computer-aided design (CAD), finance, mathematics and physics compared Mistral Large 4 with GLM-5.3. ML4 was preferred in CAD and STEM, while performing on par or close to GLM-5.3 in finance and coding.

Reinforcement learning at scale

Base models are improving fast, and our post-training has to keep pace. A recipe tuned for yesterday's model leaves capability on the table with today's frontier, because ground truth samples that once pushed a model to its limits won’t anymore. We use Reinforcement Learning (RL) because it adapts as the model does: we train on the outcomes of the model's own attempts, and we can raise the difficulty and the breadth of the tasks as it gets stronger.

Our RL library was designed to make new environments easy to add and train at scale. A shared, composable interface allows a single training run to combine tasks ranging from single-turn chat and complex scientific problem solving to safety alignment, factuality, and long-horizon tool use. These environments share scaffolds and resources such as code sandboxes, web search, and external APIs. The same composability extends to verification, with reward models, unit tests, LLM judges, and static checks combined as needed for each task.

At runtime, an autoscaling fleet of actors generates tens of thousands of rollouts in parallel while model training proceeds asynchronously. The generation and training pipeline is optimized for long trajectories, supporting rollout budgets of millions of tokens across multiple compactions while keeping staleness low. Novel methods and optimizations across both stages minimize off-policy drift and enable stable RL over long horizons.

At our current scale (3k GPUs), a single training run produces roughly 33 billion tokens per day , of which around 16 billion are trainable completion tokens after filtering and masking. We can see the run progress directly in the training rollouts: training rewards rise across several representative environments as the policy learns to solve increasingly complex tasks. Below are a few examples.

The improvements are not specific to the environments we train on; they transfer to downstream evals, and the final model owes them to both post-training stages (supervised fine-tuning and RL), as shown in the charts.

What comes next

This is only the beginning. ML4 is the first milestone on the roadmap funded by our €3 billion Series D — the largest equity round ever raised by a European technology company. That capital is already being put to work: we are significantly scaling up our compute capacity in our own European datacenters, and much more is coming online in the months ahead.

More compute means more training. The reinforcement learning run behind this preview is still in flight, and the model is showing no signs of saturation — there is substantial headroom ahead. As we scale up training on our expanded infrastructure, we expect large and rapid improvements in the weeks and months to come.

We will release the weights by the end of the month, along with more details on the architecture, additional benchmarks, and our post-training methodology. And ML4 is only the foundation: it will serve as the base for a new generation of specialized and optimized Mistral models, built for the industries and workloads our customers care about most.

The pace of progress from here will be fast. Stay tuned.

Security updates for Tuesday

Linux Weekly News
lwn.net
2026-10-06 09:12:24
Security updates have been issued by AlmaLinux (gd, gimp, kernel, kernel-rt, libpcap, librabbitmq, mariadb-connector-c, osbuild-composer, ruby:2.5, and sudo), Debian (libmodule-cpants-analyse-perl, libpng1.6, libreoffice, roundcube, ruby-oauth2, and sabnzbdplus), Fedora (0install, alt-ergo, apron, b...
Original Article
Ubuntu USN-8875-1 20.04 22.04 linux, linux-aws, linux-aws-5.15, linux-aws-fips, linux-azure, linux-azure-5.15, linux-azure-fde-5.15, linux-azure-fips, linux-fips, linux-gke, linux-gkeop, linux-hwe-5.15, linux-ibm, linux-ibm-5.15, linux-intel-iot-realtime, linux-intel-iotg, linux-intel-iotg-5.15, linux-kvm, linux-lowlatency, linux-lowlatency-hwe-5.15, linux-oracle, linux-realtime, linux-xilinx-zynqmp 2026-10-06

Mistral Large 4

Hacker News
twitter.com
2026-10-06 09:06:16
Comments...
Original Article

Meet Mistral Large 4, aka Le Chonk. • 1T parameters, natively multimodal. 49B active. It is the best open weights model from US or Europe on aggregated benchmarks. • State-of-the-art on critical workloads, including cyber defense, manufacturing and finance and it surpasses closed frontier models on visual grounding. • Forged in Europe end-to-end and is deployable from Europe via our own Mistral Cloud infrastructure. • Available to all via API today. Working with cybersecurity partners privately. Open weights release end of October.

Mathematics of Geothermal Energy

Hacker News
www.ebsco.com
2026-10-06 09:05:26
Comments...
Original Article

Full article

SUMMARY: Geothermal energy can be harnessed for domestic heating or to produce electricity via steam turbine.

“Geothermal” refers to heat from the interior of Earth generated from the forces that led to the planet’s creation and the ongoing slow radioactive decay that continues to generate thermal activity. While Earth’s surface is relatively cool, temperatures increase dramatically with depth, which is known as a region’s “geothermal gradient.” The interiors of continents tend to have lower gradients than “spreading center” regions, where continental tectonic plates are slowly separating. A prime geothermal area is along the Ring of Fire rimming the Pacific Ocean’s eastern, northern, and western coasts.

High geothermal gradients make prime candidates for geothermal energy projects. However, the average gradient is approximately 2.5–3 degrees Celsius per 100 meters. Approximately 6000 kilometers beneath the surface, molten rock reaches temperatures of approximately 5000 degrees Celsius. A small portion of this extreme heat makes its way to the surface as steam through cracks and fissures. Geothermal leakage to the surface leads to dramatic volcanic eruptions as well as to the formation of hot springs and geysers. Geothermal-warmed, mineral-rich waters have long been considered to be sacred or to have healing properties by many people. Geysers such as Old Faithful in Yellowstone continue to attract visitors from around the world.

Mathematicians, geothermal engineers, geologists, and other scientists use mathematical methods to research various aspects of geothermal processes, such as the deformable, porous properties of soil and rock that allow geothermal heat to make its way to the surface. These studies have broad applications in many scientific areas, including the way brains deform during neurosurgery and in industrial injection molding. In other cases, Lagrangian–Eulerian flow models, named for Joseph Lagrange and Leonhard Euler, are used to model characteristics such as precipitation and transport, which have applications for engineering geothermal reservoirs and isolating radioactive waste. Stochastic models for system optimization and control as well as geometric models also help mathematicians understand geothermal heat. Many are working on computer models to update, integrate, and expand the U.S. Geological Survey’s MODFLOW, a three-dimensional finite-difference groundwater flow model first published in 1984 and widely used for research and industrial applications.

Geothermal Heating

As long ago as the nineteenth century, scientists and engineers began to develop geothermal-based applications for chemistry and heating, though there is evidence that even prehistoric people built dwellings around naturally occurring geothermal heat sources. With abundant geothermal resources, Iceland began to emerge by the late 1920s as a world leader in the use of geothermal energy for domestic heating and cooling. Advances since that time have led to the development of geothermal heat pump systems. During cold periods, heat pumps transfer to buildings heat from either the ground (beneath the frost line) or from the bottom of ponds. During warm periods, the process is reversed and heat is taken from buildings and put into the ground or ponds. However, purposeful movement of water on a large scale can have geological consequences. For example, in Venice, the removal of subsurface water resulted in subsidence (settling of loose, porous soil), which lowered some buildings. Adding or subtracting water from one part of a geothermal field can affect all aspects of the field, including system pressure and surface vents. Seismologists use mathematical models describing the behavior of deformable porous rock and soil to predict where events like earthquakes might occur as a result of water-pumping activities.

Geothermal Electricity

Geothermal resources can also be used to produce electricity. The first geothermal electric power plant was built in Larderello, Italy, in 1904. Japan and the United States followed suit in 1910 and 1921, respectively. The spread of geothermal energy has been slow in the decades since. However, because of concerns regarding global warming and a quest to develop nongreenhouse gas (GHG)–emitting energy technologies, geothermal power generation has received more attention.

There are two types of geothermal power plants, both of which rely upon the production of steam to drive the conventional turbines that create electricity. Electricity can be produced directly from steam if the temperatures are at a minimum of 95 degrees Celsius (200 degrees Fahrenheit), and higher outputs are possible after temperatures crest at 175 degrees Celsius (350 degrees Fahrenheit). At the Geysers geothermal power plant in California, steam at a temperature of approximately 235 degrees Celsius (455 degrees Fahrenheit) is used to directly drive turbines. At lower temperatures, geothermal heat can still be used, but it relies upon specialized fluids that have a low boiling point capable of producing high pressures, rather than natural steam.

While the capital costs are high for both types of geothermal electricity, once in production it has several advantages over other forms of electricity generation. Like wind , its fuel costs are negligible. Similar to wind and nuclear power, once constructed, geothermal plants produce far fewer GHG emissions than traditional fossil fuel plants. Geothermal also has advantages over other alternative energy producers. Unlike wind, which is intermittent because of its dependency on weather conditions, geothermal electricity can be relied upon to produce consistent baseload power. Geothermal plants are also less intrusive visually than large wind farms and tend to draw less public attention.

Geothermal also has two key advantages over nuclear generation. Nuclear power plants are dependent upon a finite resource ( uranium ), and nuclear waste disposal is both controversial and costly. In contrast, geothermal generation depends on a virtually infinite source (heat generated in Earth’s interior), and there are no long-term waste issues.

Popular and government interest in geothermal energy and its advantages over both traditional and alternative electricity generating options led to a 20% increase in global geothermal electricity production between 2005 and 2010. In addition, there has been a 52% increase from 2007 to 2010 in the number of countries developing geothermal resources.

Despite the increasing numbers, geothermal energy production continues to significantly lag behind other electricity sources at the start of the twenty-first century. In part, this lag is the result of a perception that there are a limited number of high-quality geothermal sites that would enable geothermal energy to become a major producer. In addition, there are technical, permitting, and electric transmission issues that drive up capital costs and inhibit substantial expansion.

Bibliography

Bundschuh, Jochen, and Mario Suárez. Introduction to the Numerical Modeling of Groundwater and Geothermal Systems: Fundamentals of Mass, Energy and Solute Transport in Poroelastic Rocks . Oxfordshire, England: Taylor & Francis, 2010.

Dickson, Mary H., and Mario Fanelli. “What Is Geothermal Energy?” International Geothermal Association. http://www.geothermal-energy.org/314,what_is_geothermal_energy.html.

“Geothermal Energy: Tapping the Earth’s Heat.” National Geographic. http://environment.nationalgeographic.com/environment/global-warming/geothermal-profile.

Holm, Alison, Leslie Blodgett, Dan Jennejohn, and Karl Gawell. Geothermal Energy: International Market Update . Washington, DC: Geothermal Energy Association, May 2010.

Idaho National Laboratory. The Future of Geothermal Energy: Impact of Enhanced Geothermal Systems (EGS) on the United States in the 21st Century . Cambridge, MA: MIT Press 2006.

Lerner, K. Lee, and Brenda Wilmoth, eds. “Geothermal Gradient.” In World of Earth Science. Farmington Hills, MI: Gale Cengage, 2003.

National Energy Board. Emerging Technologies in Electricity Generation: An Energy Market Assessment . Ottawa : Her Majesty the Queen in Right of Canada as represented by the National Energy Board, 2006.

U.S. Government Accountability Office. “Renewable Energy: Increased Geothermal Development Will Depend on Overcoming Many Challenges.” May 2006.

Full article

SUMMARY: Geothermal energy can be harnessed for domestic heating or to produce electricity via steam turbine.

“Geothermal” refers to heat from the interior of Earth generated from the forces that led to the planet’s creation and the ongoing slow radioactive decay that continues to generate thermal activity. While Earth’s surface is relatively cool, temperatures increase dramatically with depth, which is known as a region’s “geothermal gradient.” The interiors of continents tend to have lower gradients than “spreading center” regions, where continental tectonic plates are slowly separating. A prime geothermal area is along the Ring of Fire rimming the Pacific Ocean’s eastern, northern, and western coasts.

High geothermal gradients make prime candidates for geothermal energy projects. However, the average gradient is approximately 2.5–3 degrees Celsius per 100 meters. Approximately 6000 kilometers beneath the surface, molten rock reaches temperatures of approximately 5000 degrees Celsius. A small portion of this extreme heat makes its way to the surface as steam through cracks and fissures. Geothermal leakage to the surface leads to dramatic volcanic eruptions as well as to the formation of hot springs and geysers. Geothermal-warmed, mineral-rich waters have long been considered to be sacred or to have healing properties by many people. Geysers such as Old Faithful in Yellowstone continue to attract visitors from around the world.

Mathematicians, geothermal engineers, geologists, and other scientists use mathematical methods to research various aspects of geothermal processes, such as the deformable, porous properties of soil and rock that allow geothermal heat to make its way to the surface. These studies have broad applications in many scientific areas, including the way brains deform during neurosurgery and in industrial injection molding. In other cases, Lagrangian–Eulerian flow models, named for Joseph Lagrange and Leonhard Euler, are used to model characteristics such as precipitation and transport, which have applications for engineering geothermal reservoirs and isolating radioactive waste. Stochastic models for system optimization and control as well as geometric models also help mathematicians understand geothermal heat. Many are working on computer models to update, integrate, and expand the U.S. Geological Survey’s MODFLOW, a three-dimensional finite-difference groundwater flow model first published in 1984 and widely used for research and industrial applications.

Geothermal Heating

As long ago as the nineteenth century, scientists and engineers began to develop geothermal-based applications for chemistry and heating, though there is evidence that even prehistoric people built dwellings around naturally occurring geothermal heat sources. With abundant geothermal resources, Iceland began to emerge by the late 1920s as a world leader in the use of geothermal energy for domestic heating and cooling. Advances since that time have led to the development of geothermal heat pump systems. During cold periods, heat pumps transfer to buildings heat from either the ground (beneath the frost line) or from the bottom of ponds. During warm periods, the process is reversed and heat is taken from buildings and put into the ground or ponds. However, purposeful movement of water on a large scale can have geological consequences. For example, in Venice, the removal of subsurface water resulted in subsidence (settling of loose, porous soil), which lowered some buildings. Adding or subtracting water from one part of a geothermal field can affect all aspects of the field, including system pressure and surface vents. Seismologists use mathematical models describing the behavior of deformable porous rock and soil to predict where events like earthquakes might occur as a result of water-pumping activities.

Geothermal Electricity

Geothermal resources can also be used to produce electricity. The first geothermal electric power plant was built in Larderello, Italy, in 1904. Japan and the United States followed suit in 1910 and 1921, respectively. The spread of geothermal energy has been slow in the decades since. However, because of concerns regarding global warming and a quest to develop nongreenhouse gas (GHG)–emitting energy technologies, geothermal power generation has received more attention.

There are two types of geothermal power plants, both of which rely upon the production of steam to drive the conventional turbines that create electricity. Electricity can be produced directly from steam if the temperatures are at a minimum of 95 degrees Celsius (200 degrees Fahrenheit), and higher outputs are possible after temperatures crest at 175 degrees Celsius (350 degrees Fahrenheit). At the Geysers geothermal power plant in California, steam at a temperature of approximately 235 degrees Celsius (455 degrees Fahrenheit) is used to directly drive turbines. At lower temperatures, geothermal heat can still be used, but it relies upon specialized fluids that have a low boiling point capable of producing high pressures, rather than natural steam.

While the capital costs are high for both types of geothermal electricity, once in production it has several advantages over other forms of electricity generation. Like wind , its fuel costs are negligible. Similar to wind and nuclear power, once constructed, geothermal plants produce far fewer GHG emissions than traditional fossil fuel plants. Geothermal also has advantages over other alternative energy producers. Unlike wind, which is intermittent because of its dependency on weather conditions, geothermal electricity can be relied upon to produce consistent baseload power. Geothermal plants are also less intrusive visually than large wind farms and tend to draw less public attention.

Geothermal also has two key advantages over nuclear generation. Nuclear power plants are dependent upon a finite resource ( uranium ), and nuclear waste disposal is both controversial and costly. In contrast, geothermal generation depends on a virtually infinite source (heat generated in Earth’s interior), and there are no long-term waste issues.

Popular and government interest in geothermal energy and its advantages over both traditional and alternative electricity generating options led to a 20% increase in global geothermal electricity production between 2005 and 2010. In addition, there has been a 52% increase from 2007 to 2010 in the number of countries developing geothermal resources.

Despite the increasing numbers, geothermal energy production continues to significantly lag behind other electricity sources at the start of the twenty-first century. In part, this lag is the result of a perception that there are a limited number of high-quality geothermal sites that would enable geothermal energy to become a major producer. In addition, there are technical, permitting, and electric transmission issues that drive up capital costs and inhibit substantial expansion.

Bibliography

Bundschuh, Jochen, and Mario Suárez. Introduction to the Numerical Modeling of Groundwater and Geothermal Systems: Fundamentals of Mass, Energy and Solute Transport in Poroelastic Rocks . Oxfordshire, England: Taylor & Francis, 2010.

Dickson, Mary H., and Mario Fanelli. “What Is Geothermal Energy?” International Geothermal Association. http://www.geothermal-energy.org/314,what_is_geothermal_energy.html.

“Geothermal Energy: Tapping the Earth’s Heat.” National Geographic. http://environment.nationalgeographic.com/environment/global-warming/geothermal-profile.

Holm, Alison, Leslie Blodgett, Dan Jennejohn, and Karl Gawell. Geothermal Energy: International Market Update . Washington, DC: Geothermal Energy Association, May 2010.

Idaho National Laboratory. The Future of Geothermal Energy: Impact of Enhanced Geothermal Systems (EGS) on the United States in the 21st Century . Cambridge, MA: MIT Press 2006.

Lerner, K. Lee, and Brenda Wilmoth, eds. “Geothermal Gradient.” In World of Earth Science. Farmington Hills, MI: Gale Cengage, 2003.

National Energy Board. Emerging Technologies in Electricity Generation: An Energy Market Assessment . Ottawa : Her Majesty the Queen in Right of Canada as represented by the National Energy Board, 2006.

U.S. Government Accountability Office. “Renewable Energy: Increased Geothermal Development Will Depend on Overcoming Many Challenges.” May 2006.

More like this Related articles

Related articles ( 3 )

Related articles ( 3 )

  • Published in :

    Journal of Renewable & Sustainable Energy, 2023, v. 15, n. 1. P. 1

    Authored by :

    Yu, Hao; Lu, Xinli; Ma, Fei; Zhang, Wei; Liu, Jiali; Li, Chenchen

    Publication type :

    Academic Journal

  • Published in :

    Journal of Renewable & Sustainable Energy, 2024, v. 16, n. 2. P. 1

    Authored by :

    Geng, Zhi; Chen, Keyu; Li, Junfen; Wang, Jianli; Shi, Tianqing; Gu, Yujiong

    Publication type :

    Academic Journal

  • Published in :

    Advanced Engineering Materials, 2024, v. 26, n. 22. P. 1

    Authored by :

    Wan, Li; Wang, Jinyin; Yue, Wen; Lin, Fankai; Gao, Chuanhua; Liu, XianJie; Min, Xin; Wang, Chengbiao; Huang, ZhaoHui

    Publication type :

    Academic Journal

Silver Pines review – a stylish, neo-noir detective drama

Guardian
www.theguardian.com
2026-10-06 09:00:15
PC, PS5, Switch 2; Wych Elm Games, Team 17Abandoned shops, ruined apartment buildings and rotting sewer networks make up a dark landscape in which you play a private eye searching for a missing singer-songwriter In a weird deserted town miles from anywhere, a troubled detective searches for a missin...
Original Article

I n a weird deserted town miles from anywhere, a troubled detective searches for a missing musician and the girl he may or may not have murdered. It could be a Silent Hill game or a lost David Lynch movie from the 1990s – in fact, it’s neither. And also both. If that whets your appetite, stick around, there’s plenty more to come.

Developed by Scandinavian studio Wych Elm Games , Silver Pines is a dark Metroidvania-style adventure, where you play as shambling private eye, Red Walker, exploring a 2D universe of abandoned shops, ruined apartment buildings and rotting sewer networks. Officially, everyone has left because of an incoming storm, but very quickly you learn that something much darker is afoot. Someone has hired you to find singer-songwriter Eddie Velvet, but you don’t know why; your only contact with your client is via the payphones dotted around town, where you can also save your progress – as long as you have a dollar to pay for the call. Each location is filled with clues and puzzles – a weird newspaper story pinned to the wall, a strange statue that needs a missing piece – and through these artefacts you unlock new rooms and descend further into the mystery.

An avatar in a video game holding a machine gun that is blasting towards a group of people. They are in a dark, dingy room with an upright piano, and an American flag hung on the hall.
Very quickly you learn that something much darker is afoot … Silver Pines. Photograph: Wych Elm Games, Team 17

This format will be incredibly familiar to anyone who has played a point-and-click adventure in the last 30 years; it’s not doing anything wildly original with the formula. And when the monsters inevitably emerge, you find yourself fighting them off with knives that break easily, and pistols that have to be painstakingly re-loaded with limited ammo, reminding you that you’re in survival horror territory, too.

But what makes this game much more than an amalgam of familiar adventure tropes is its brilliant sense of style, it’s considered pace and it’s wonderfully constructed mythology. Using rotoscoping to create the lead character’s animation, he looks like some film noir drunk sloping around in the darkness, fumbling with clues. Each location is loaded with detail, from strange paintings, to battered furniture and fizzing neon signs, while parallax scrolling is used to brilliant effect, allowing you to see forthcoming locations in the distance as you wander the streets. Each room is beautifully constructed, so that as you move, the light coming from filthy windows or an old reading lamp sends slivers of shadow creeping over the walls. This is a design team that really understands the aesthetics of film noir, a fact underscored by the soundtrack a sleazy jazz club combo of Angelo Badalamenti and Vangelis.

Silver Pines trailer

As you progress, new areas of the town open up, and you meet a few weird inhabitants that stuck around despite the incoming weather emergency, among them a cynical police chief and a brooding femme fatale. The puzzles can be obtuse, but thorough exploration often yields the answers you need: could you squeeze through that air duct? Could you climb that apparently impassable wall? Sometimes the answer is yes. Combat is always tense and panicked, especially the boss battles, where you’ll be floundering about with different weapons and ammunition as some Lovecraftian monstrosity pummels you. But then, this is very true to the earliest survival horror titles, where the inability to actually move, shoot and/or avoid incoming blows was part of the drama – you were always a victim and never a super cop. Besides, you can turn the difficulty right down if it’s the story you want to experience.

As I played, I discovered countless references to 80s and 90s neo-noir culture – from Blue Velvet to Twin Peaks (you drink coffee to increase your stamina), from the X-Files to Alan Parker’s hellish Angel Heart. It was a pleasure to explore every broken-down movie theatre, oily car garage and sweat-stinking interrogation room, collecting weird trinkets and working out how they’d all fit together – then exploring them again (and again) when the answers failed to show up. Silver Pines is a game to sit back and wallow in, like a private eye, kicking off his wingtip shoes and pulling that bottle of bourbon from the bottom drawer as rain trickles down the window and sodium street lights paint the walls a sickly shade of yellow.

Making scents: could AI help perfumers take pressure off endangered plants?

Guardian
www.theguardian.com
2026-10-06 09:00:15
Firms working on reproducing compounds without need for repeated harvesting but conservationists are sceptical Many of the world’s most expensive perfumes begin with a wounded tree. When some Aquilaria trees are damaged, they produce a dark, fragrant resin that becomes agarwood, or oudh – an ingredi...
Original Article

M any of the world’s most expensive perfumes begin with a wounded tree. When some Aquilaria trees are damaged, they produce a dark, fragrant resin that becomes agarwood, or oudh – an ingredient so valuable that wild trees have been illegally felled in its pursuit.

The result has devastated wild Aquilaria populations. But what if perfumers could create the smell without cutting down the tree?

With the help of AI, scientists are learning to predict how molecules that have never been made will smell, reconstructing scent-producing genes from flowers believed to be extinct. Others have collected microbes from clouds and used them as a starting point for new perfumes.

Now scientists and biotechnology companies say advances could soon allow the fragrance industry to reproduce scents and compounds from endangered plants without repeatedly harvesting them.

“Being able to recreate the scent of an endangered plant would be a huge win for both the environment but also the brands,” said Alex Wiltschko, the founder of Osmo, a Google Brain spin-off that has raised $130m to develop artificial intelligence capable of predicting and recreating smells.

Osmo has recreated the scent of a rare tree from a piece of bark “the size of a finger”, Wiltschko said. A spectrometer analysed the sample, its AI reconstructed the chemical formula and the scent was reproduced in its laboratory.

A felled tree in a forest
An illegally felled Aquilaria sinensis , or incense tree, in Hong Kong. Photograph: Jérôme Favre/EPA

BioHarvest Sciences says it has created a stable cell culture from an endangered south-east Asian fragrance plant, allowing its valuable compounds to be produced without cultivating or harvesting the original plant.

The plant has not been named for commercial reasons but BioHarvest has announced an agreement anticipating production of 20 tonnes of the fragrance composition, with limited production potentially beginning next year.

“The challenge is that human demand can be effectively unlimited while biological supply is not,” said BioHarvest’s chief executive, Zaki Rakib. “We believe biotechnology can help close that gap, not by replacing nature but by allowing us to use what nature has given us without continually putting the source under pressure.”

But conservationists and perfumers say such claims risk giving advances in fragrance technology an environmental gloss they have not yet earned.

“Calling this conservation is greenwashing,” said Mandy Aftel, the pioneering natural perfumer, author and founder of the Aftel Archive of Curious Scents. “Making a synthetic version of an endangered plant’s smell doesn’t save the plant.” She said producing a substitute did not stop people buying the original – or address other causes of a species’ decline.

Rhian Smith, of the Royal Botanic Gardens, Kew, agreed. She pointed to the Aquilaria tree. AI could help design an alternative to oudh, she said, but wild-sourced material remained highly prized. “It doesn’t completely take the pressure off wild plant species,” she said.

There is also the question of motivation. Wiltschko admitted that conservation was not “an explicit priority” for Osmo’s work.

Tincture dripping from glass pipette next to pine bark
Osmo says it has trained its AI on more than 5m human responses to smells. Photograph: ronstik/Getty Images/iStockphoto

Synthetic perfume itself is nothing new: the industry has relied heavily on synthetic ingredients for more than a century, and machines have long been able to identify molecules in a scent.

“What’s changed is what happens after the data comes off the machine,” Wiltschko said. “What’s new is prediction.” He claimed Osmo had trained its AI on more than 5m human responses to smells. “That means it can predict how combinations of molecules will be perceived,” he said.

Elsewhere, AI is claimed to be changing the whole world in which perfumers work. Christina Agapakis is not a perfumer, but her work reconstructing genes from preserved flowers has produced fragrance compounds that perfumers can use to create new scents.

Another perfumer, Future Society, has gone into the clouds, making two flights in a small plane to collect living microbes. Thirteen were identified, producing 57 scent molecules, which became the starting point for Cloud Reverie, a perfume launched this year.

Even AI has its limits: a flower’s scent can contain hundreds of molecules and change according to which part of the plant they come from and the conditions in which it grows.

It is still not enough for the purists. “The smell we got was this ghostly fragment of what had been lost,” Agapakis said of her project. She could not recreate the living organism, or the soil, climate, season and ecology that produced its scent, she said.

“What we have to remember is, ultimately, however powerful AI is, a smell is not simply a formula or dataset.”

Erdosproblems.com Succumbs to the AI Onslaught

Hacker News
www.erdosproblems.com
2026-10-06 08:53:58
Comments...
Original Article

By Thomas Bloom

AI is changing everything, for better and/or worse, and the rate of change is dizzying; in recent months this has been particularly evident in mathematics, where AI has gone from being essentially useless to helping solve some of the hardest problems in mathematics in less than a year.

The website www.erdosproblems.com has often been on the front line of these changes, and a barometer by which one could measure AI capabilities. This was not at all my intention when I created the site -- I wanted to promote these problems to a human audience, and make a useful reference on what work has been done. But the easy availability of a large pool of questions that are simple to state, ranging from easy and obscure to deep and impenetrable, provided the ideal showcase for AI.

In some ways this has led to a huge amount of progress -- we now know the answer to many questions we did not before. (Although a lot of this recent progress has come from increased human efforts and renewed interest in some problems, rather than just AI solutions.) There have also been negative effects, however: some mathematicians have dismissed Erdős-style mathematics as 'easy/recreational'; many have stopped thinking about Erdős problems believing that they cannot compete with AI; and, most significantly, there has been a wave of AI-produced solutions provided with no explanation. While of course these are useful and tell us new things, they are also displacing and discouraging those who are actually interested in the mathematics.

(These issues have stopped being Erdős-specific as AI continues to make breakthroughs in an ever wider range of fields.)

A couple of weeks ago I asked for feedback on how people used the site, and what they thought should be changed. I received, both publicly and privately, a wide variety of opinions, and I'm grateful to everyone for their feedback.

What was particularly striking was the number of people I heard from who have benefited a lot from the site, learning new mathematics and finding new problems to think about, but have never commented on the site. I have been particularly conscious of this large silent, audience, and have tried to make sure their experiences were not drowned out by the more vocal minority.

In this post I will describe the changes I am making, based on the feedback I received and my own reflections on what the site is and could be.

In the essay Why Do We Need Human Mathematicians Anymore? Po-Shen Loh suggests the following as an axiom to use when deciding how things should develop: We (humans) should help humanity flourish.

When thinking about what I should do about the site, I use the following variant:

The site should help the Erdős-community of humans (defined as those who are interested in Erdős-style mathematics, and want to think about and understand it) flourish.

Why change?

I launched the site on 28th May 2023, with just over 200 problems; it has grown steadily since then. The biggest change came in August 2025 when I added a comment feature. The site now hosts 1221 problems, over 9000 comments, and almost 2000 registered users. The site typically receives between 10,000 and 25,000 unique visitors each day.

The first few months after introducing the comment feature saw a huge increase in activity, just as I had hoped. A vibrant community arose; people shared ideas on how to solve problems, made observations, gave corrections, and suggested missing references and solutions. Many papers were written and new collaborations formed.

Unfortunately, this level of activity has not been sustained. Some decline was inevitable: over time, many of the natural observations will have been made, mistakes corrected, and discussions between collaborators would move offline. This natural evolution has been dramatically sped up by the coincidental rise of AI and its ability to solve some problems with minimal human intervention.

The main way that people publicly interact with the site now is to advertise their AI-generated proofs, often without any attempt to explain them, but as a way to record a (increasingly meaningless) priority claim.

This is very different to what I imagined, and I don't want to manage a website which does this.

I believe that websites with this function should exist -- places where people can record AI-generated proofs, even if purely formal with no human understanding, to save others wasting their tokens generating the same proof, and so that other people can access and use them if they desire. There are now several candidates for such repositories, and if managed responsibly, they can serve a useful role in the mathematical ecosystem. I personally don't want to manage one.

Just as one does not open a restaurant in an abattoir, it is important that there be a separation between such repositories and a site which aims to promote the actual questions, place them in an appropriate context, and give a useful overview of the current state of human understanding.

The situation as it stands muddies the waters, promotes a gamified glory-seeking attitude, and gives the wrong impression that the value of these questions ends as soon as someone posts a formal proof of their truth value. This is not true (regardless of whether this proof was human or AI generated).

I believe Erdős intended these questions to serve as enduring beacons for human curiosity and wonder. They are landmarks by which we measure how far we've come, and how much there is still to understand.

I am very proud to have helped foster an open online community to discuss Erdős problems, and the kinds of discussions that used to happen are very valuable. But these discussions have become much less frequent. If and when I can find a way to encourage such discussions again, without the site becoming an AI repository of the kind mentioned above, I will happily do so.

What are the changes?

I will make the following changes. (As ever, these are somewhat experimental, and may themselves change and be clarified further in the next few weeks.)

  • A hiatus on problem comments and proof claims : I will freeze new problem comments and proof claims. General threads and blog posts will remain open to comments. Problem comments and proof claims may be reinstated in the future. Until then, suggestions for updates can be emailed to [email protected] , and I will update the site manually as I see appropriate.
  • No problem statuses : The site will not display the statuses of any problems (e.g. open, solved, etc.). All problems will be displayed in the same neutral colour. The count of currently solved problems and the solved percentage will no longer be shown.
  • No credit/ownership language to describe future solutions : The site will continue to record relevant results and theorems, but will no longer use credit-giving language for a result (human or AI).
  • An emphasis on high-quality expositions : I will focus more on the proof expositions feature. People are encouraged to write in with their own expositions of proofs (whether these proofs are old or new, human or AI), and I will post those I judge to be high-quality. I am exploring other ways to encourage human exposition (e.g. an online seminar). Please contact me if you have any ideas.

I've tried to anticipate some of the questions and critiques of these changes below. (I may update this with other questions and answers in future depending on feedback.)

What about those outside of traditional academia, who are using AI tools to make advances in mathematics, but can't post to arXiv/lack the knowledge or context to write up proper papers about their findings?

I will post links to correct formalisations and high-quality writeups, whatever the source, whether from a traditional academic or not. Papers should acknowledge their sources and honestly disclose how AI was use; where there is evidence of plagiarism or misrepresentation about AI use, I may decline to feature the submission, even if the proof is correct.

This is removing one, unofficial, avenue of publication; there are, these days, many others (even if you are also unable to post to arXiv). AI-assisted search tools mean that people will find your work if they are interested in the area as long as you post it to one of these sites.

I encourage everyone interested in Erdős problems to think about them, and to work on them if interested. If you have a proof (AI-created or otherwise) then you should take time to carefully write up the proof yourself (rather than asking an AI to generate a PDF for you). If you are unable to understand the proof you could reach out to someone else to help write this up.

But emailing you my solution is slower, and it might be some time before you update the site.

This is true; but there is no great rush here. As much as I like these problems, and think it is important that there are some people who do think about the distribution of prime numbers and the structure of graphs, these are not problems for which a formal proof will have an immediate impact on general society.

I will try to link to correct Lean formalisations (if verified on Palomar) quickly; proofs which are poorly explained, and not accompanied by a formalisation, I am unlikely to update the site with. (See below for more details.)

What's the point? There are loads of other places I can post my proof.

Indeed; this is partially why I feel comfortable halting such proof claims on the site, since modern search tools (including AI) make it easy to find proofs posted online about problems you are curious about, even if posted in obscure places.

I would like to use the small amount of influence I have to avoid promoting low quality proofs, and not to give an added veneer of legitimacy where it is not deserved.

I could just make my own site for people to share their proofs and discuss solving problems with AI. Heck, I can even use AI to scrape all the text off your site and make an exact clone with much more liberal policies.

Yes, you could. This a period of great experimentation in different formats and ways to encourage mathematics; if you think you have a good idea for a site or resource, you should make it. (Although it is often better to help out an existing effort where possible, since these resources are only useful if people use them, and it is better to concentrate on creating a few high-quality sites than hundreds of very similar clones.)

I think it is, however, rude to use without permission the text from my site, which is the result of a huge amount of work from me and many others who have contributed to the site.

How long will the hiatus last for?

I don't know. I will monitor how things develop in the coming weeks and months, and will reintroduce comments and proof claims (perhaps in a different form) when I believe they will do more good than harm.

Why not just moderate the comments to only allow genuine discussion through?

This has been tried; in practice the vast majority of the comments the site receives now are people announcing AI-generated proofs. It is not sustainable to have a moderation policy which would reject almost all the comments that are submitted.

I am open to alternative suggestions about how to create a space for human discussion and collaboration, either on the site or elsewhere.

What about existing comments and proof claims?

They will remain as an archive of the site up to this point.

Will you update the remarks to reflect existing proof claims?

Yes, I will be working through the backlog of existing proof claims and updating the remarks, linking to formalisations and so on, as appropriate. You do not need to email me about a proof claim already on the site.

How will you decide when to update the remarks? What can I do with my proof to help?

I will update the remarks when I judge there is an interesting new result that people who are working on that problem should be aware of, that can be presented in a useful way. Some things to be aware of:

  • If you have a Lean formalisation register it on Palomar -- this lets others see that the formalisation compiles correctly, and makes it easy to check the formal statement correctly matches the problem statement. I will then link to the Palomar registry.
  • A proof accompanied by a well-written exposition that demonstrates clear understanding and which makes it easy for others to understand the proof will be prioritised.
  • If I judge there to be some kind of academic fraud (e.g. using the ideas of others without attribution, or passing AI-generated work off as your own) then I will not post it. This may mean that there are correct proofs not acknowledged, but the alternative is to publicise and reward bad behaviour. (As mentioned above, these proofs will surely still be found by anyone who searches for them elsewhere, but at least I would not be implicitly endorsing them.)

What's the point in removing the problem status? Isn't it just cosmetic, and doesn't it just make the site harder to navigate?

Yes, this does undeniably remove some information. I think, however, that what is lost is not significant for anyone seriously interested in that problem, who can see for themselves within seconds of reading what the current situation is.

It makes it harder to browse the site searching for an unsolved problem to work on; instead I recommend that people browse the site by topic, finding questions that interest them, and investigating those, whether the original question is open or solved, since there is always more to be done.

The main point is to disincentivise people who are simply glory-chasing and copying problems into their AI to get an OPEN->SOLVED dopamine hit. It also prevents people drawing wrong conclusions from the 'rate of solved problems'.

Furthermore, there is a lot of subjectivity anyway in many problems as to what counts as solved. Many people only browse the open problems, and they're missing out on a lot of great mathematics that way.

I think it is against the spirit of Erdős to regard any problem as 'closed' -- whenever one form of a question is answered, many others are created, and all problems deserve continued attention.

Doesn't the 'no credit' rule also devalue the work of humans?

Unfortunately, yes. I anticipate this will only be temporary, while the norms and customs of this new age of mathematics are decided. The commentary on the site is in no way 'official', and just represents my own summary of the situation, and collects links which may be useful to others exploring a problem.

Information about authorship will still be available in the sources the site links to. I trust that people will be sensible enough to judge for themselves who deserves the appropriate credit, and reward them appropriately.

I will also be freer in my language when it comes to well-written papers, whether they are the first place a proof appears or explain a proof that originated elsewhere; but I will no longer use language that suggests anyone 'owns' a particular result or proof. (For example, instead of saying 'Bloom proved that $x>y$', I will write 'It is known that $x>y$ -- an explanation of the proof is provided by Bloom [link]'.)

Erdős and AI

I am often asked 'what would Erdős have thought of AI?' The short answer is 'I don't know'. I never met Erdős, and have no special insight into him; I know him only through biographies and his papers, and anecdotes and reminiscences from those who did know him.

I would like to honour his legacy, and create something that he would have liked. To that end, I want to stress one thing: Erdős was, as well as a mathematician, an incredibly social person. Many stories about Erdős emphasises his humour, his warmth, and his enthusiasm for talking to other people (mathematicians or not). He spent his life travelling from mathematician to mathematician, arriving on their doorstep and declaring 'my brain is open'.

He was not the type to lock himself away for years working in isolation on a single problem. For Erdős, mathematics was a very human activity, best done out loud. Questions would arise, be solved or discarded and replaced by new questions. Some results obtained, new ideas found, and then onto the next problem.

Some view the future of mathematics as a dystopian arcade of button-pressing, staring at a screen waiting for AI to do the thinking for us, with the main human involvement limited to asking the initial question and offering sporadic words of encouragement. Each new proof is then thrown into a repository, to only ever be read by other AIs, and the human presses the next button.

I believe Erdős would have found this future grim indeed, and the antithesis of mathematics as he practised it. Use AI as you like -- but do not abandon mathematics as a human activity.

Don't solve a problem for the sake of it; life is too short to spend it doing things that don't matter to you.

Find a question that is meaningful to you, find other humans who are interested in it, and talk about it. Be confused, be stuck, be inspired. Find a messy proof, then find a better one. Get tired, get hungry, get into a flow state. Argue, laugh, give up, drink some coffee, and attack it again.

Be human. Let your brain be open.

"Incredible Moment for the Borderlands": Judge Blocks Border Wall Expansion in Big Bend & West Texas

Democracy Now!
www.democracynow.org
2026-10-06 08:53:06
In a victory for environmentalists and immigration rights’ advocates, a federal judge in El Paso, Texas, has blocked further border wall construction inside Big Bend National Park. “We slammed the brakes on $7 billion worth of corrupt construction contracts,” says Laiken Jordahl, a...
Original Article

In a victory for environmentalists and immigration rights’ advocates, a federal judge in El Paso, Texas, has blocked further border wall construction inside Big Bend National Park. “We slammed the brakes on $7 billion worth of corrupt construction contracts,” says Laiken Jordahl, a national public lands advocate with the Center for Biological Diversity, which sued to block the expansion. “We have effectively stopped construction across West Texas, and it’s incredible.”

Jordahl also responds to the Trump administration’s excavation of a 200-year-old cottonwood tree on the Arizona-Mexico border that had been protected by activists for months. “Watching those videos of them toppling this ancient tree that was very likely older than the border itself is symbolic of how DHS has treated the border region. They see our communities, our wildlife, our nature as a sacrifice zone,” says Jordahl.

Guests

  • Laiken Jordahl

    national public lands advocate with the Center for Biological Diversity who previously worked at Big Bend National Park.

Please check back later for full transcript.

The original content of this program is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License . Please attribute legal copies of this work to democracynow.org. Some of the work(s) that this program incorporates, however, may be separately licensed. For further information or additional permissions, contact us.

Meta's Muse Is an Adorable Privacy and Security Dumpster Fire

Hacker News
www.techdirt.com
2026-10-06 08:45:49
Comments...
Original Article

from the move-fast-and-break-the-planet dept

Meta’s agentic AI product Muse has had a rocky few weeks since launch. The product, which features an animated avatar named Jolly (one presumes to make mass hyper surveillance seem adorable) is supposed to help you offload busywork like making restaurant reservations, paying bills, or ordering groceries.

Despite Meta having claimed repeatedly that Muse was built with a heavy focus on privacy and security, the AI agent launched with a nasty zero-day flaw that made it possible to spy on Mac users. When one tech YouTuber put Muse in charge of their Facebook Marketplace sales, it sold his stuff way below acceptable rates and doled out their home address (the user apparently didn’t understand the permissions he set ).

Somebody else found that you could trick Muse into giving root access on the device it’s running on by simply pretending to be a Muse agent yourself . Others found that Muse software not only accesses people’s private messages without approval, it often ignores all permissions and uploads them to the cloud – even if you specifically tell it not to .

That last problem was bad enough that Apple needed to change its macOS privacy settings to stop third-party app developers from misusing them to access message histories:

“Friday’s announcement comes two weeks after tech columnist Jason Aten said that Meta’s new general-purpose AI agent Muse sent him an unsolicited notification referencing a thread between him and a co-worker over Apple Messages. Aten said he never granted Muse permissions to read his messages and had assumed they were off-limits. Social media last week blew up with masses of people who agreed and said the incident showed that AI assistants given access to calendars, emails, messages, shopping accounts, and other resources are akin to a skill saw or other power tool. While potentially useful, they can do real damage if not used carefully.”

Meanwhile, Wired found that Muse consistently creates detailed profiles of all your friends, family, colleagues, “collaborators,” and people you “follow.” Obviously much of that information is necessary for the agent to get to “know” you, but this being Meta, people are understandably uncomfortable with this sort of massive ramp up of data collection in a country, under authoritarian control, that’s too corrupt to pass a privacy law or regulate data brokers:

“These [AI assistant] tools are actively soliciting users to plug their whole lives in—their emails, calendars, financial institutions, everything in order to be helpful assistance,” Bogen says. “That’s dramatically more information than people might have otherwise given to some of these companies. The breadth of access to information that these tools have will lead to a ballooning of what they know about users.”

But wait, there’s more! 404 Media found that in the weeks before launch, Meta was in a mad dash to quickly and sloppily fix multiple other vulnerabilities but refused to delay Muse’s launch to actually make sure the product was secure:

“The Meta source said they felt security teams were asked to push hot fixes to these bugs as quickly as possible and in a way that wouldn’t delay Muse’s launch, leading to what they described as “half-baked protections being rushed out to enable the launch. Many senior engineers believe it’s inevitable we’re going to have a massive data breach as a result of Hatch.” Muse is called “Hatch” internally and in Meta’s codebase.”

Many folks build their own agentic AI solutions with cobbled together open source hardware and software to ensure they have clear understanding of, and control over, what’s actually happening in their name. Meta, a glorified ad monopoly with a history of ethical “lapses,” not only wants to dominate the space, it will spend a lot of time in the new year lobbying against on device, open source, foreign, and/or open weighted alternatives to Meta.

Despite Meta’s claims that privacy and security would be a priority for Muse that’s clearly not the case; it’s also extremely clear that Meta doesn’t fear any meaningful government accountability or oversight, or the product would have spent significantly more time in the oven. Tech giants that had already clearly abandoned quality control at impossible scale are engaged in a mad dash to the trough with new product launches before their funny math causes a market correction.

Such is life in a country that has had most consumer protection regulators (and cybersecurity standards) lobotomized by corrupt authoritarians . Authoritarians guys like Mark Zuckerberg enthusiastically supported because they didn’t like paying taxes — and hated former FTC antitrust boss Lina Khan.

Muse’s early privacy and security problems are the kind of stuff anybody with a head on their shoulders could see coming miles over the horizon. And it’s all inevitably going to get more dangerous — and ridiculous — over the next year as unethical tech oligarchs fully exploit their successful lobotomization of the federal regulatory state. You know, for the love of innovation .

Filed Under: , , , , , , , , , ,
Companies: meta

What's Earth's dominant species by mass?

Hacker News
signoregalilei.com
2026-10-06 08:40:03
Comments...
Original Article

You might have heard that all the ants in the world weigh as much combined as all the humans. That isn’t true. We outweigh ants 5 to 1. 1 Termites might have us beat, though. 2 Either way, it’s not really a fair comparison. There are thousands of species of ants and termites, and just one species of human.

But if we’re judging each species based on how much total mass they have – that is, by its “biomass” – who actually comes out on top? Turns out there are some surprising winners.

Antarctic Krill: 379 million tonnes

Let’s start with our division: animals. Humans are pretty close to the top for a single species, weighing in at about 350 million metric tons. 3 The top-placing wild species is Antarctic krill, with a 2009 estimate placing them at 379 million metric tons. Each individual krill weighs only about 2 grams or 0.07 ounces, but their sheer population in the Southern Ocean puts them in a statistical dead heat with us – and keeps gigantic blue whales comfortably fed.

But both humans and krill lag behind the true winner for animals: cattle. Cattle are easily the top domesticated animal species, with about 650 million metric tons of biomass. Pigs are in second, at about 140 million metric tons – placing them behind humans and krill.

Cattle: 650 million tonnes

Unfortunately for cattle, animals are almost the lightest of all the kingdoms of life. The most comprehensive study, (Bar-On et. al 2018) measures the different kingdoms by their carbon mass specifically. Carbon is only a portion of an organism, so this method gives smaller numbers. But those numbers are more meaningful for, say, tracking the flow of food energy through the biosphere (think of eating a pound of watermelon versus a pound of steak). For example, the 650 million tons of cattle on Earth represent only 61 million tons of carbon.

Animals as a whole have 2 billion tons of carbon. That means we handily outclass viruses (0.2 billion tons) but are beaten by protists (4 billion), archaea (7 billion), fungi (12 billion), bacteria (70 billion) and especially plants (450 billion).

Most of the world’s bacteria live in aquifers and seafloor sediments, which are hard to study. It’s possible that one extremely dominant undersea bacteria species has the world’s highest biomass, but more likely the winner will be a plant.

So what plant has the most biomass in the world? The winner for animals was a domesticated species, so let’s start with domesticated plants.

The first species that might come to mind are the major cereal grains like rice, wheat, and corn (maize). Of these, maize has the largest annual harvest by mass, totaling 1.1 billion tons – nearly double the mass of cattle. But the true biggest crop isn’t a grain at all.

That honor goes to sugar cane, with the harvest topping 2 billion metric tons in 2023. 4 So humanity harvests about 6 times our collective weight in sugar cane every year. Sugar cane is also a perennial, meaning that its root system persists between harvests. This means total mass of living sugar cane on average throughout the year is probably higher than an annual crop like maize.

Sugar cane: 2 billion tonnes

But there’s still one more subcategory to look at: wild plants. It’s time to take on the trees.

Trees seem to hold all the superlatives. The most massive individual organism comes from a tree species: Pando the clonal colony of aspen trees with interconnected roots which weighs 6,000 metric tons all on its own. But aspens are pretty rare. Is there a more common tree with more total mass as a species?

For that, we have to look to the largest forest in the world: the boreal forests that ring the arctic circle. In those forests, just 3 or 4 species can represent over half of the total mass of the forest. By contrast, tropical forests can require over a hundred species to reach the same mark. 5

Two candidates boreal trees came up in a search of the literature. One is the Black Spruce, Picea mariana , which is the most common tree in arctic Canada. According to Canada’s National Forest Inventory, 6 over 11 billion tons of black spruce are growing in the country, and some grow in the USA as well.

Black Spruce: More than 11 billion tonnes

The other candidate is the Dahurian Larch, the most common tree in Russia. Russia’s forest inventory doesn’t have as detailed numbers by species, 7 but larches as whole in Russia are about 35.6% of Russia’s forest land, and Russia has a total of 82.2 billion cubic meters of forest. From what I could find, there are 2 main species of larch in Russia: the Dahurian Larch in the East and the Siberian Larch in the west, and the Dahurian is the more common of the two so it must be at least 50% of the total. Calculating the numbers and using the density of larch wood, that gives 16.8 billion tons of larch in Russia, of which at least 8.4 billion tons (and possibly more) is Dahurian larch.

Dahurian Larch: At least 8.4 billion tonnes, maybe more

Living black spruce wood is about 51% carbon, 8 meaning there’s nearly 3 times as much carbon in just that one species of tree as all animals combined. So humans may still control the world’s biosphere, but on at least this one metric the spruce and larch trees have a massive lead.

1 https://www.npr.org/2022/09/21/1124216118/ants-number-study-quadrillion

2 https://doi.org/10.1073/pnas.1711842115 What’s Earth’s dominant species by mass?

3 https://www.sciencefocus.com/nature/what-animal-collectively-makes-up-the-largest-biomass-on-earth

4 https://ourworldindata.org/explorers/global-food?tab=line&time=1961..2024&country=OWID_WRL~USA~CHN~IND&Food=Sugar+cane&Metric=Production&Per+capita=false

5 https://www.nature.com/articles/s41586-023-06820-z/tables/1

6 https://nfi.nfis.org/en/customized_report

7 https://rosleshoz.gov.ru/activity/forest-register/

8 https://research.fs.usda.gov/treesearch/64469

Ebola Death Toll in DRC Tops 4,000; Trump USAID Cuts Fuel Outbreak

Democracy Now!
www.democracynow.org
2026-10-06 08:34:57
We go to the Democratic Republic of the Congo, the site of the fastest-growing Ebola outbreak in history. Health authorities believe the outbreak first began in January but was not detected until May, in part due to cuts in global health investments, like the dismantling of USAID. Since then, over 8...
Original Article

We go to the Democratic Republic of the Congo, the site of the fastest-growing Ebola outbreak in history. Health authorities believe the outbreak first began in January but was not detected until May, in part due to cuts in global health investments, like the dismantling of USAID . Since then, over 8,000 cases have been recorded, with more than 4,000 fatalities so far. From Bunia, Ituri province, the epicenter of the outbreak, we speak to Congolese journalist Akilimali Chomachoma. Chomachoma describes the strain on the underfunded Congolese healthcare system as the disease spreads. Many frontline workers have not been paid for months.

“We are seeing this outbreak emerge, and really unfurl, in a setting of decreased global health support,” says Craig Spencer, a doctor and public health expert based in the United States who contracted Ebola while on a medical mission to Guinea in 2014. “The U.S. has not shown up in the same way that it has in previous Ebola outbreaks.” Spencer also comments on the “unprecedented” measles outbreak in the United States, which he warns is likely to continue and lead to additional deaths.

Guests

  • Craig Spencer

    public health professor and emergency medicine physician at Brown University.

Please check back later for full transcript.

The original content of this program is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License . Please attribute legal copies of this work to democracynow.org. Some of the work(s) that this program incorporates, however, may be separately licensed. For further information or additional permissions, contact us.

Subquadratic 3SUM and Subcubic APSP

Hacker News
arxiv.org
2026-10-06 08:31:17
Comments...
Original Article

View PDF HTML (experimental)

Abstract: We give the first polynomial improvements over the textbook algorithms for $3$SUM and All-Pairs Shortest Paths (APSP): we show how to deterministically solve $3$SUM on $n$ integers of polynomial size in $O(n^{1.9992})$ time and APSP on directed $n$-vertex graphs with polynomially bounded integer weights in $O(n^{2.9995})$ time. This refutes the $3$SUM and APSP hypotheses. Using known reductions, we also refute the real-valued versions of the $3$SUM and APSP hypotheses, the Exact Triangle hypothesis, the Zero-Weight $k$-Clique hypotheses, and the three rectangular hinted Online Matrix--Vector conjectures of van den Brand, Nanongkai, and Saranurak, and we give polynomial speedups for a variety of other problems.
All of these results follow from a single new algorithm for thin matrix products. Let $X$ be an $N\times D$ integer matrix and $Y$ a $D\times N$ integer matrix with $D\le N^{1/18}$, and let $W$ be any set of at most $N^2/\sqrt D$ positions. We compute the entries $(XY)[I,J]$, $(I,J)\in W$, in $O(N^2/D^{0.063})$ operations, which is polynomially less than the time needed to write down $XY$ or to compute $N^2/\sqrt D$ inner products one by one. We design this algorithm by modifying a variant of Coppersmith's rectangular matrix multiplication algorithm, built from a ten-multiplication identity of Schönhage, to perform only the operations needed for the entries in $W$, and show that few operations are needed. Interpreted as a graph algorithm, this solves the All-Edges Sparse Triangle problem in truly subquadratic time on sparse lopsided tripartite graphs where two parts have $n$ vertices but one part has $n^{\varepsilon}$ vertices for $\varepsilon<0.12$. By known reductions, Exact Triangle, and hence $3$SUM and APSP, reduce to this problem. We also give a data structure version that answers queries for single entries of $XY$, not known in advance.

Submission history

From: Josh Alman [ view email ]
[v1] Mon, 5 Oct 2026 17:44:29 UTC (93 KB)

French High School Students Lead Mass Protests over School Funding, Inequality; Thousands Arrested

Democracy Now!
www.democracynow.org
2026-10-06 08:14:14
Nearly 2,000 French high schools are blockaded or closed today amid massive student-led protests over school funding and inequality. The demonstrations began in September at a single school in the economically deprived suburbs of Paris, where students protested teacher shortages, overcrowded classro...
Original Article

This is a rush transcript. Copy may not be in its final form.

AMY GOODMAN : Nearly 2,000 French high schools are blockaded or closed today as massive student-led protests spread over France over school funding and inequality. The demonstration began in September at a single school outside Paris, where students protested teacher shortages, overcrowded classrooms and crumbling school buildings. But the scope of the protests has widened amidst a violent crackdown by French police. French authorities have arrested more than 5,000 people since the movement began, about 85% of them high schoolers.

These are the voices of two students protesting in Paris earlier today.

HUGO VETZEL : [translated] For us, it’s, first of all, in support of the high school students who started blockades a few days ago. I think we all have a role to play in this movement, and the more we are, the more we will be heard. And I think this is not a fight of a few handful of people, but a more global fight for the youth. …

We feel despised, because when the answer is violence, that means, in a way, democracy is no longer working. But we feel especially that there is such a gap between the elites and us, and we feel we are creating unease, because we are disrupting so much that they are using force to shut us up.

MONA FALLACARA : [translated] I am quite happy because, for once, we see a real movement of the youth everywhere, and we hear about it. But then, what I deplore is that the media is showing more of the violence done by the students before their demands, which are vital. And yes, there is a bit of violence, but it’s not everywhere. …

France’s budget will come to a vote soon, so we hope there will be a higher budget for education. And for the presidential election next year, as well, we hope for a bit of change after 10 years of President Macron heading the country.

AMY GOODMAN : France’s official independent national human rights body has expressed deep concern over what it called the state’s disproportionate use of force in response to the protests. On Monday, Democracy Now! reached a French student who described the violent police crackdown. For her safety, she asked not to be identified.

PROTESTER : During the protests, people, students receive tear gas, receive — like, get hit, get attacked, get charged by policemen, by the CRS , which is a police institution. And yes, basically it has been really violent. Many people have been wounded. Many people have been arrested. Someone lost an eye. Someone lost a hand. And this is happening to literal teens, literal children. We even — most of us aren’t even adults yet. And this is terrible, because literal adult people, adult policemen, adult policewomen, with weapons and tear gas and shields and helmets, are attacking literal students with some — some cardboard with messages on it. Like, we’re pacifists, and our movement is nonviolent, and yet we receive so much violence, violence from the government.

AMY GOODMAN : That was a French high school student afraid to be named or shown.

We go now to France. We begin with Claire Vigneau, the general secretary of the French student union, the FSE . She’s at a protest in Rennes, in northwestern France.

Thanks so much for joining us, Claire. If you can start off by talking about how these protests began? When people hear “suburbs,” they think more middle-class, but in France, it’s basically the opposite, where more poverty is concentrated. Talk about the schools, the high school students and how this has grown.

CLAIRE VIGNEAU : Yeah, so, it started like a few days ago, and maybe 10 days ago, something like that. And now it’s getting bigger and bigger every day. Yes, it started in Saint-Denis, which is a suburbs near Paris, but very poor suburbs. And it started because class conditions, like studying conditions, are really getting worse and worse. And, like, there is water in their class. They have rats in their class. The roof is falling out. And yeah, so it’s getting harder and harder to study for these high school students. And so they started to block their high schools.

And now it’s widened out to most of the high schools in France. Actually, today, the state decided to close 500 high schools, because they didn’t want them to be blocked. But even with that precaution, 500 more high schools are blocked. And for now, we are — for now we are — we have also like 90% of the universities, the campuses that are blocked. And we are going to keep on going yet, keep on going to block.

Today, we were on a protest in Rennes, yes. We are trying to escape from the policemen that are chasing us with tear gas and with cannon, water cannon. And so, yes, it was a very big protest in Rennes. There was actually like 16,000 people in Rennes, which is not a big city. And one thing to note is that the minister of the interior told today the head of the biggest trade union in France that Rennes was going to be an example of repression, of violence, and to show all the rest of the movement that we can’t do what we are doing now, which is protest for our living conditions.

JUAN GONZÁLEZ: And, Claire, could you talk more about what the demands of the students are, and especially their concerns about so many of these schools having no air conditioning in warm weather, that the heat gets up to 35 degrees Celsius, or 95 degrees Fahrenheit, especially given that France has had so many people die during heat waves this past year?

CLAIRE VIGNEAU : Yeah, yeah, because we have seen a lot of classes, even in classes for like really young child, starting from 6 years old, being in classrooms where there were like 35 students with temperatures attaining 35 degrees and even more — some of them attained 40 degrees. And we are concerned, because you can’t learn, really, in a really good way when you are under this weather. And we are fighting, of course, for the weather conditions, like the stunning conditions, but not only.

And obviously, we are struggling for more budgets in school and less budgets in the Army, because in France the financement for the Army doubled in 10 years, and when they — at the same time, the budgets for education is getting lower and lower every day. And so, that’s like — the air conditioning is, of course, one of the goals, but not only. And before even having air conditioning, we would like to have safe classrooms with no bad products for our health in the buildings. And we would like, if it rains outside, we don’t want rain in our classrooms, etc., etc.

AMY GOODMAN : We’re going to go right now to southern France, where we’re joined by Cole Stangler, a journalist in France covering politics and labor. The significance of this protest call? I mean, we see Claire in the streets of Rennes. There are protests in Lille, a nationwide day of protests, the college unions and other labor unions joining in these protests right now, thousands of people arrested, about 5,000, over 87% of them kids in high school.

COLE STANGLER : Yeah, that’s right. And I think maybe just to complement some of what we’ve been — already been hearing, you know, those issues have been absolutely at the forefront of what I’ve been hearing on the ground here. I was just at a protest in Aix-en-Provence, just in the south of France, just north of Marseille, where students were telling me their own stories of the dilapidated conditions of some of their buildings, which, to be fair, aren’t nearly as bad as when we talk about the suburbs of Paris, where you have, you know, rats in the hallways, but still, people talking about dilapidated buildings, having to install nets around their — around the buildings to make sure there’s not falling debris. That’s an issue I heard about.

And, you know, another issue people talk a lot about here is the staff shortages, the lack of teachers. This has been an issue that’s been bubbling up for years, in fact. And we heard it mentioned, France’s — the amount of spending that France has done on education has stagnated, if not declined. You know, if you go back to the late '90s, 1997, France was spending over 7% of its GDP on education. That's declined about nearly a percentage point. So, the government is spending less, and the salaries are quite low, as well. That might be shocking to some viewers in the United States. You know, a starting salary for a public school teacher in France is around 2,000 euros. Even mid-career professionals are making less than the OECD average, less than the European Union average. And so, when you have low salaries, you know, and you have dilapidated buildings, it’s easy to maybe understand why you have these staff shortages. And so, this is an interesting issue where the critiques that high school students are making are actually very similar to what teachers’ unions have been saying for quite some time. So, there’s that issue of the shortages.

And I think another key issue that’s emerged — and again, students were talking about it, both the high school students I talked to earlier today and the university students who joined the protests — was the issue of what in France they call Parcoursup, which is the new platform that was introduced a few years ago, which is basically an application platform for high school students to get into universities, that is not very transparent. There’s a lot of critiques about how this sort of seemingly mysterious system will offer university spots to students. So, students aren’t getting to go to the universities they want to go to that are close to them. They might have to move halfway across the country. That means more money that they’re spending on rent; they can’t live with their parents. And so, that critique of the university application process is something that unites, again, both high school students and university students.

And that’s what’s been interesting when you’re talking to people on the streets here, is to see to what extent the high school student issues actually resonate beyond just high schools. Right? It’s university students. It’s also trade unions. Today, at the protest I was at, you had a lot of trade union support from, you know, the main labor unions in France. And I think the movement is really resonating because it’s really hit a nerve here on a lot of issues that are important to French society.

JUAN GONZÁLEZ: And, Cole, I wanted to ask you — the French government has accused the left-wing political party France Unbowed of stoking the violence at the — or, some violence at these protests. Can you talk about the significance of this, especially as France is preparing for next year’s presidential elections?

COLE STANGLER : Yeah. You know, I was talking about this earlier, and I think that this line that we’ve indeed heard from the government — I think it is both a great exaggeration of the mobilizing power of that party, La France Insoumise, led by Jean-Luc Mélenchon, and it’s also, I think, an insult to the collective intelligence of the high school students who are protesting. I mean, I don’t think it’s physically possible for a political party to have this broad of a reach. So, I think that’s sort of an easy excuse. But that’s — you know, that’s indicative of the approach the government has taken so far.

And, you know, this question of police repression and police violence, which we heard about just earlier, has also been something that’s been kind of pushing the movement forward. A lot of French people, I think, have been shocked by the degree of the police violence. You know, we heard there was a 15-year-old who had his hand basically blown off by picking up a stun grenade that the police use here. We had another student who lost the use of his eye because he was hit by a rubber bullet. These are like, you know, low-grade, almost military kind of technologies that the police use in France. And it’s part of a policing strategy that has been in place for several years now. Many of your viewers might remember the yellow vest movement back in late 2018, where we had, you know, decentralized mass protests and a huge amount of violence, where you had protesters having — you know, losing their eyes, losing their hands.

The difference here — and this is what makes it frightening — is we’re not talking about adults. We’re talking about literal high schoolers. We’re talking about 15-, 14-year-olds who are being beaten by police. And, you know, there have been incidents. We should just be clear: There have been incidents of, you know, people attacking police officers, at some points firefighters. But that is the extreme minority. And I think the much bigger story here, we’re talking about a democratic country where the police have been attacking high school students. And I think it’s — again, it’s documented. There’s been hundreds of complaints that have been filed with the French government to this effect. This French policing strategy that was used during the yellow vests is now being directed against high schoolers. And that’s also energizing this movement. A lot of people think it’s unacceptable.

AMY GOODMAN : We’re going to go from Aix-en-Provence, where Cole Stangler is, to Lille, France, from the south to the north of France. Alys Rowlands is a university student and member of the national board of the French student union, also just at a protest. Alys, thanks so much for joining us. As we wrap up, what do you think it’s most important for us to understand about this mass protest that’s taking place in France, and what you’re demanding, and how this might even affect the national elections that are coming up next year in France?

ALYS ROWLANDS : Yeah, absolutely. I think the most important thing here is that it started with the high schoolers. Today, it’s also university students, but we’ve seen also teachers. We’ve seen farmers. We’ve seen the yellow vesters who want to join again. We’ve seen everyone want to join this mobilization, because I think what’s very interesting about this one here is that everyone is deeply, deeply angry. We are completely fed up of the living conditions, the school conditions, the work conditions that we have, and everyone is really, really deeply angry. And that’s what’s really interesting at the moment.

So, the students, like they’ve said, are being faced with really strong repression from the police forces. So, as they said before, there are many kids. This is children who are being shot at with rubber bullets and tear gas and water cannons, who are being chased, who are being, you know, arrested massively. And this is completely unheard of before. It’s only been a week of this mobilization with the different protests and schools being blocked, etc., and it’s already over 5,000 people arrested, and, as you said, mostly children. And so, this is completely different from things that we’ve seen before.

And because France, it’s — quite often there are mobilizations and things, but this is completely a different proportion, because people are really deeply angry about what’s going on and the conditions that we are living in. And I think this will have an impact, because we’re already seeing things changing. But what people want right now, it’s not one reform. It’s not one thing. It’s a deep change in the system and more budgets for the public services in France, which is behind all of this.

JUAN GONZÁLEZ: And, Alys, what is your sense of the — excuse me — the response of adults, of the parents of these children, of the general society to the way that the government has reacted to the student protests?

ALYS ROWLANDS : Yeah, that’s an interesting question, because we can see in protests loads of parents. And we’ve seen online, on TikTok and the different apps, that the teachers and parents have been joining the protests, because they understand that the students, the high schoolers especially, are really deeply angry and they are completely fed up of going to school and then there being rats and the ceiling falling down and chemical products in the ceiling that are giving us cancer. Like, it’s completely — it’s completely insane. And so, the parents and teachers, but also, much wider, the general population are completely with the mobilizations.

But the media are only showing violence and this and that. But the true violence is the violence of the police, the violence of the repression. How is it possible that children, 14-year-olds, are having their eyes being blown off, their hands being blown off, their teeth being completely ripped out by these rubber bullets and tear gas? How is it possible that this is happening in France today? Five thousand people arrested, mostly children, how is this possible? The true violence is the violence of the state that we are being faced with all day, every day, all year. That’s the true violence: having no money, having difficulty finishing the month, having less and less help from the state, having to choose between eating and putting your heating on during the winter. It’s that kind of violence that we are faced with all year. And so, you know, the parents and the teachers and, wider, the general population are completely agreeing with the —everything that they want, the students, the high schoolers, the university students, but also much wider. Everyone has things they want to change.

AMY GOODMAN : Alys Rowlands, we want to thank you for being with us from Lille, from the north of France, university student, member of the national board of the FSU , French student union. I also want to thank Cole Stangler, journalist based in France, covering politics and labor issues, joining us from Aix-en-Provence in the south, and Claire Vigneau, general secretary of the French student union, joining us from the streets of Rennes, where the massive protest also is taking place today, as it is across the country. And just to underscore something Juan said about the heat and the lack of air conditioning, this isn’t just a luxury. We’re talking about 8,000 people in France this summer died from heat.

Coming up, we go to the Democratic Republic of the Congo, where the death toll from Ebola has topped 4,000. Stay with us.

[break]

AMY GOODMAN : “Freedom Is Free” by Chicano Batman.

The original content of this program is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License . Please attribute legal copies of this work to democracynow.org. Some of the work(s) that this program incorporates, however, may be separately licensed. For further information or additional permissions, contact us.

Headlines for October 6, 2026

Democracy Now!
www.democracynow.org
2026-10-06 08:00:00
Supreme Court Case Will Determine Whether State and Local Governments Can Sue for Climate Harms, Pentagon Says Latest Boat Strike Killed Four Narcotraffickers, Provides No Evidence, Turkey and Pakistan to Send Troops to Saudi Arabia as Yemeni Government Forces Battle Houthis, Trump Defends Decision ...
Original Article

Headlines October 06, 2026

Watch Headlines

Supreme Court Case Will Determine Whether State and Local Governments Can Sue for Climate Harms

Oct 06, 2026

The Supreme Court opened its new term Monday by hearing oral arguments in a case that could determine whether fossil fuel companies can be held liable for their role in creating the climate crisis — and covering it up. The case concerns lawsuits brought by city and county officials in Boulder, Colorado, who sued ExxonMobil and Suncor Energy in state court, seeking damages for the costs of droughts, fires, floods and other effects of global heating. During oral arguments, justices appeared skeptical of claims that federal law preempts state tort laws that could be used to sue energy companies.

Climate justice groups rallied outside the court, joined by several young people who signed an amicus brief highlighting how children are especially vulnerable to the climate crisis. This is Alina Miranda, who was 16 years old when her family home was consumed by the Marshall Fire, the most destructive in Colorado’s history.

Alina Miranda : “I did not create the climate crisis, but I’m already living with the consequences. I signed an amicus brief because I believe my state has a right and duty to protect my health and hold corporations like Suncor responsible for the damage they’ve caused by prioritizing profit over long-term climate health. The Supreme Court should not take away the ability of my local government to protect me.”

Pentagon Says Latest Boat Strike Killed Four Narcotraffickers, Provides No Evidence

Oct 06, 2026

The Pentagon said Sunday it killed four people in a strike on a speedboat in the Caribbean — once again providing no evidence to back its claims the vessel was carrying drugs. That brings the number of people U.S. Southern Command claims to have killed to 238 since the Trump administration began blowing up civilian vessels in the eastern Pacific and Caribbean Sea last year. In response, Amnesty International USA wrote, “Nearly 240 dead in this illegal campaign of extrajudicial killings, i.e., murder at sea. Where is Congress to stop the horror?”

Turkey and Pakistan to Send Troops to Saudi Arabia as Yemeni Government Forces Battle Houthis

Oct 06, 2026

In Saudi Arabia, officials say three people were injured Monday as Houthi fighters from Yemen launched attacks on two regional airports. The attacks came as Yemen’s Saudi-backed government announced its forces had recaptured the Red Sea port city of Mokha as part of a counteroffensive to drive Houthi militias from the areas around the strategic Bab al-Mandab Strait.

Meanwhile, officials in Turkey and Pakistan said they would rapidly deploy troops to Saudi Arabia as part of a mutual defense agreement signed in August, known as the Mecca Pact. The announcement came as the Saudi-led coalition launched airstrikes on what it said were weapons sites in Yemen’s Houthi-controlled capital, Sana’a.

Trump Defends Decision to Attack Iran, Says “Let Them Take Out” Los Angeles and San Diego

Oct 06, 2026

The International Chamber of Commerce is warning of a growing global food crisis due to disruptions to shipping through the Strait of Hormuz caused by the U.S. and Israeli war on Iran. The group’s secretary general told The Wall Street Journal that fertilizer capacity has fallen by nearly 40% since the strait was disrupted, with grain prices set to rise by up to 80% unless it’s reopened.

On Monday, President Trump defended his decision to attack Iran, telling a campaign rally in Grand Island, Nebraska, that high gas prices are a “small price to pay.”

President Donald Trump : “It’s a small price to pay for keeping the world safe, keeping our country safe. They can take out a city. Let them take — let — let them take out Los Angeles. Let them take out San Diego. That’s — this is a very small price to pay, and it’s going to be over with very soon.”

The statement drew condemnation from elected officials in California. Los Angeles Mayor Karen Bass responded, “The role of a president is to protect the country, not to actively threaten it and encourage attacks on American lives.”

Ukraine Launches Massive Drone Attack on Moscow, Blames Russia for Sinking Ship in Romanian Waters

Oct 06, 2026

Ukraine launched a massive attack on Russia overnight, with 650 drones targeting the Moscow region. The attacks killed two people and set a warehouse on fire. It’s Ukraine’s second-largest drone attack yet on the Russian capital. It follows claims by Ukraine that a Russian drone sunk a ship in Romanian waters in the Black Sea. Romania reported that two people were killed when a grain ship caught fire and sank.

Kenya Detects First Case of Ebola

Oct 06, 2026

Kenya has detected its first-ever case of Ebola. The patient is a Kenyan citizen who has been living in the Democratic Republic ​of the Congo and arrived in Nairobi ​over the weekend. The patient has since died. His death comes as the DRC is combating its largest-ever outbreak of Ebola, recording more than 8,000 cases and over 4,000 deaths. Later in the broadcast, we’ll speak with a Congolese journalist in the city of Bunia and Dr. Craig Spencer at Brown University, who survived an Ebola infection in 2014.

New York Declares Disaster Emergency Due to Rapid Spread of Measles

Oct 06, 2026

New York Governor Kathy Hochul declared a disaster emergency Monday due to the rapid spread of measles in neighboring states and in rural and urban areas of New York. The declaration came as the number of measles cases in Pennsylvania so far this year surpassed 1,000 — the largest state outbreak in more than three decades. Five people have died so far. Officials blame low vaccination rates for the outbreak.

Worker at an Anti-Plague Lab Dies from “Pneumonia of Undetermined” Origin in Russia

Oct 06, 2026

In Russia, a worker at an anti-plague laboratory died from a “pneumonia of undetermined” origin last week. According to Russian media, nearly 200 people who had been in contact with the worker have been placed under precautionary medical observation. Secretary of State Marco Rubio said Monday if “something like that gets out, it could quickly spread,” adding that the U.S. is monitoring the situation closely.

Opposition Mounts Against Proposed ICE Jail in San Antonio, Texas

Oct 06, 2026

In Texas, opposition is mounting against a proposed ICE jail and processing center that would be located inside a 640,000-square-foot warehouse in San Antonio’s Eastside. Mayor Gina Ortiz Jones says city officials are exploring legal options to block the site after federal officials failed to respond to questions about its impacts on the environment and local community. The jail is also opposed by San Antonio-area Congressmember Joaquin Castro, who cited reports of inadequate medical care and abusive conditions in ICE detention centers, including sexual violence. Over the weekend, hundreds of protesters marched to the warehouse site calling on ICE to get out of San Antonio.

In related news, immigrants detained at the Folkston ICE Processing Center in Georgia have released a video from inside denouncing inhumane conditions.

Idris Makhmudov : “We are immigrants from all over the world, whom Trump is locking up in prison-like detentions for nothing and for a long time.”

Mujtaba Qeyami : “Most of us are asylum seekers, those who came to the United States seeking protection from political persecution and violence in their home countries. Many fled out of fear becoming political prisoner in those countries, and one might say we became political prisoner in United States simply because we are immigrants.”

Genaro Jimenez : “We have one to two hours of yard a day, and the rest of the day we are locked up and gone. We’re strictly controlled and restricted in everything, what we can have, watch and play, where we can eat and sleep, shower, and go to toilet.”

Mohammadou Sall : “Honor, dignity, respect mean nothing here. The guards insult us, humiliate us, threaten us and punish us when they want.”

The Folkston ICE Processing Center is operated by the for-profit prison company GEO Group.

Marimar Martinez Files Civil Lawsuit After Being Shot Five Times by Border Patrol Agent

Oct 06, 2026

A Chicago teacher who was shot five times by a Border Patrol agent filed a federal civil lawsuit Monday, alleging false arrest and imprisonment, malicious prosecution, emotional distress and negligence. Marimar Martinez was shot last October as she followed federal agents carrying out immigration raids in her neighborhood — honking her horn and shouting to alert her neighbors. Immediately after the shooting, the Department of Homeland Security declared Martinez was a “domestic terrorist,” and accused her of using her car as a weapon. But federal prosecutors abruptly dropped charges just weeks later. Martinez’s lawyers say they have proof that Border Patrol agent Charles Exum shot Martinez without justification and that he was not in danger at the time of the shooting.

Federal Agents Raid Border Wall Protest Camp in Arizona, Uproot 200-Year-Old Cottonwood Tree

Oct 06, 2026

In news from southern Arizona, dozens of armed Border Patrol agents on Monday raided a protest camp in Lochiel, arresting at least 20 people who’d been protecting a 200-year-old cottonwood tree and blocking the border wall expansion for over two months. The camp had been in place since July, with activists taking turns to sit up high in the branches of the giant, sacred cottonwood.

In video footage from the scene Monday morning, witnesses are heard screaming as an excavator rips out the cottonwood, known as “grandmother tree,” from its roots to make way for Trump’s border wall. Click here to see our coverage of this story .

Federal Agencies Decline to Renew Permits for Water Stations to Help Migrants

Oct 06, 2026

Federal agencies have declined to renew permits for a dozen water stations operated by the Tucson-based nonprofit Humane Borders to help migrants crossing through the treacherous Sonoran Desert. This comes as the remains of at least 86 migrants have been found in the region this year alone, as people are forced to cross through more remote and dangerous areas of the U.S.-Mexico border due to the expansion of the wall. This is Octavio Fuentes, executive director of Humane Borders.

Octavio Fuentes : “Water is a human right, and the removal of water stations during the surge of migrant deaths is absolutely deplorable. It is cruel. The government’s border strategy that they’ve had in place for the last 30 years is prevention through deterrence. It’s what has pushed people into this desert for 30 years. Now it’s going to deny water, and more people will absolutely die as a result of it.”

Mexican President Sheinbaum Requests Evidence After U.S. Claims Shots Targeted U.S. Coast Guard

Oct 06, 2026

Mexican President Claudia Sheinbaum is requesting evidence after U.S. Ambassador to Mexico Ronald Johnson claimed shots fired from Mexico targeted U.S. Coast Guard. The alleged shooting reportedly took place in the Texas-Tamaulipas border in late September. Hundreds of U.S. troops, helicopters, drones and other military equipment have since been deployed to the region in response. This is Sheinbaum speaking from Mexico City Monday.

President Claudia Sheinbaum : “They even sent it in writing, stating that there have been shots fired from the Mexican border into the United States. We have requested all the evidence that this was the case, because we have no evidence. No member of the National Guard, the Army or the Navy has detected any shots fired from the Mexican border into the United States.”

Trump Says Super PAC Will Now Pay for Campaign-Style Ads, Not Federal Funds

Oct 06, 2026

President Trump said Monday his super PAC will pay for his campaign-style ads, instead of using federal funds. The use of taxpayer money to fund Trump’s midterm ads drew swift backlash, even from some Republicans. The government has reportedly spent at least $10 million to air the ads. Public Citizen blamed broadcast networks for accepting the taxpayer-funded campaign-style ads, saying, “The letter to broadcasters notes that they should refrain from airing advertisements that they know are otherwise illegal under the law, and that broadcasters must maintain a public file on any political ads they air. Failing to uphold this duty can result in broadcast license revocation.”

The original content of this program is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License . Please attribute legal copies of this work to democracynow.org. Some of the work(s) that this program incorporates, however, may be separately licensed. For further information or additional permissions, contact us.

Release of Polars 2.0

Hacker News
pola.rs
2026-10-06 07:59:01
Comments...
Original Article

Today we are shipping Polars 2.0. In the earlier announcement post we went through the rationale of the version bump. This post we will discuss what features 2.0 brings. Even though we didn’t intend to make it a big feature release, it still packs a lot to get enthousiastic about.

Let’s go through the highlights of this release:

  • our initial version of out-of-core (spill-to-disk) support is enabled,
  • a lot of very core performance improvements,
  • first class SQL support, which together with the performance improvements has Polars leading DataFusion and DuckDB in TPC-H and TPC-DS 1 benchmarks ,
  • a new Map dtype, and
  • stricter Polars on dtypes and explicitness, leading to faster feedback, and faster AI iteration.

Performance and SQL as a first class citizen

Polars 2.0 will be the marking point where we will treat SQL as first class citizen. Polars SQL coverage has increased dramatically over last few months. We know we have been building a solid engine for the last couple of years. In Polars 2.0, we want to enable that to more workloads, including SQL. To make this performant, we shipped many improvements to our optimizer and engine. The highlights here join reordering, much better common-subplan-elimination and dynamic predicates/bloom filters.

To see how we perform on typical SQL benchmarks, we ran Polars SQL on data derived from TPC-H and TPC-DS 1 and ran it against the latest DuckDB release (1.5.6), DuckDB 2.0 alpha (2.0.0.dev2610011535) and the latest DataFusion release (54.0.0) on a c7a.4xlarge (16 vCPUs, 32GB RAM) and a c7a.metal (192 vCPUs, 384GB RAM). Every query ran 5 times in a hot setting, with a separate process per query and a 60 second timeout. The file cache was cleared between each engine/benchmark (not between queries). For every query we take the best of the 5 runs, and we compare engines on both the sum and the geometric mean of those query times.

The data is generated with tpcgen-cli parquet compiled from source on commit 99bedae . We looked at the default row-group sizes of tpcgen-cli and confirmed they are roughly similar to what Polars scan_csv piped through sink_parquet and Duckdb COPY produce. The SQL queries were generated with DuckDB 1.5.6’s tpch_queries() and tpcds_queries() . The data was stored on EBS.

The charts below show the runtime of each engine in seconds (lower is better), split by machine.

c7a.4xlarge (16 vCPUs, 32 GB)

c7a.metal (192 vCPUs, 384 GB)

Polars and both DuckDB versions completed all queries. DataFusion timed out on TPC-DS q72 (and once on q67) and ran out of memory on TPC-H q18 on c7a.4xlarge; those queries are excluded from the results above for all engines.

We observe that default Polars is fastest on all but one benchmarks. Polars has a constant overhead when we scale to 192 threads, which hurts small data queries. In fact we see that Polars limited to 32 cores is competitive or winning in all benchmarks. We have diagnosed the cause on our end and will hopefully fix this problem in the next release. More information on the benchmarks can be found in the appendix . We encourage you to replicate our results and have shared a repository for this benchmark here: https://github.com/pola-rs/polars-2.0-benchmark .

Streaming engine and OOC as default

This is the one of the biggest impact changes of 2.0. Calling collect on a LazyFrame will now default to the streaming engine, leading to massive memory and performance improvements on most queries. The reason this required a major version bump is that the streaming engine doesn’t guarantee row-order by default for certain operations ( join , group_by , unpivot , etc.). If you require observable row-order in those operations, you can opt in to that by setting maintain_order=True .

Out-of-core (spill to disk) is now enabled by default. It starts spilling at ~80% of RAM (this may need tuning). Operations that support out-of-core at this moment (sort, window functions, many expressions) can now start spilling to disk to finish a query. The default disk budget is 64GB. In the coming time we will enable out-of-core for joins and group-by’s as well.

These two changes will make Polars much more resillient in high-memory workloads for casual data practicioners. And with out-of-core join and group-by on our roadmap, this resilience will improve even more.

New Map datatype

Polars now supports the Arrow MapType directly as a Polars Map dtype. You can think of a Map as a Python dictionary, mapping keys to values. Before 2.0 the Arrow MapType was read in Polars as List(Struct({"key": ..., "value": ...})). 

df = pl.DataFrame(
    {
        "user": ["alice", "bob", "carol"],
        "scores": pl.Series(
            [{"math": 90, "art": 75}, {"math": 60}, {}],
            dtype=pl.Map(pl.String, pl.Int64),
        ),
        "subject": ["art", "art", "math"],
    }
)
shape: (3, 3)
┌───────┬─────────────────────────┬─────────┐
│ user  ┆ scores                  ┆ subject │
│ ---   ┆ ---                     ┆ ---     │
│ str   ┆ map[str, i64]           ┆ str     │
╞═══════╪═════════════════════════╪═════════╡
│ alice ┆ {"math": 90, "art": 75} ┆ art     │
│ bob   ┆ {"math": 60}            ┆ art     │
│ carol ┆ {}                      ┆ math    │
└───────┴─────────────────────────┴─────────┘
# Key lookups and dictionary-like methods:
df.select(
    "user",
    pl.col("scores").map.get("math").alias("math"),                # fixed key
    pl.col("scores").map.get(pl.col("subject")).alias("by_subject"),  # key from another column
    pl.col("scores").map.contains_key("art").alias("has_art"),
    pl.col("scores").map.len().alias("n"),
    pl.col("scores").map.keys().alias("keys"),
    pl.col("scores").map.values().alias("values"),
)
┌───────┬──────┬────────────┬─────────┬─────┬─────────────────┬───────────┐
│ user  ┆ math ┆ by_subject ┆ has_art ┆ n   ┆ keys            ┆ values    │
│ ---   ┆ ---  ┆ ---        ┆ ---     ┆ --- ┆ ---             ┆ ---       │
│ str   ┆ i64  ┆ i64        ┆ bool    ┆ u32 ┆ list[str]       ┆ list[i64] │
╞═══════╪══════╪════════════╪═════════╪═════╪═════════════════╪═══════════╡
│ alice ┆ 90   ┆ 75         ┆ true    ┆ 2   ┆ ["math", "art"] ┆ [90, 75]  │
│ bob   ┆ 60   ┆ null       ┆ false   ┆ 1   ┆ ["math"]        ┆ [60]      │
│ carol ┆ null ┆ null       ┆ false   ┆ 0   ┆ []              ┆ []        │
└───────┴──────┴────────────┴─────────┴─────┴─────────────────┴───────────┘

As a supported dtype, the map type will now have dedicated expressions, like key lookups, iteration over values and other dictionary like methods.

Stricter Polars

Polars aims to be strict and fail fast. Errors should ideally raise up-front, not 20 minutes into a pipeline. Implicit behavior on data-mismatches should be opt-in, not a default, since those mismatches can hide bugs. This strictness has become even more valuable with the rise of AI-driven development. Agents can validate a query’s structure early by calling collect_schema() , which resolves types and catches schema-level mismatches without materializing any data. This ensures fast feedback, meaning agents and humans can iterate faster. Not all errors can be caught during compilation of the query plan, some depend on data. In these cases Polars defaults to stricter behavior to ensure inconsistencies are caught instead of silently producing different results. See previous posts ) with some examples in where Polars has gotten more strict.

Last words

We are very excited that Polars 2.0 is out. Coming months we’ll improve on the road were in. Better out-of-core, better scaling at large CPU-counts and on Polars Cloud we aim to be the fastest distributed engine available. We also started working on GeoPolars and hope to deliver more news on this soon. If you find any problem with our new release, please open an issue: https://github.com/pola-rs/polars/issues . And finally, to help you with upgrading to 2.0, we have posted migration guide .

Benchmark Appendix

The absolute numbers (in seconds) are below. The bold number is the fastest engine in each row; the darker the shading, the slower an engine is compared to the fastest. Polars with 32 threads was only run on c7a.metal.

Sum of query times

Geometric mean of query times

Polars also scales well with more cores on larger data. Moving from 16 to 192 vCPUs at SF100 makes Polars 3.8x faster on TPC-H and 2.2x faster on TPC-DS (by sum), compared to 3.2x and 1.9x for DuckDB 1.5.6, 2.2x and 1.5x for the DuckDB 2.0 alpha, and 1.7x and 1.0x for DataFusion. At SF10 the extra cores don’t help Polars with its default settings: it is equally fast on TPC-H and 1.8x slower on TPC-DS, while DuckDB 1.5.6 still gets 1.8x and 1.3x faster. Polars with 32 threads only ran on the c7a.metal, so it is not part of this comparison.

  1. These benchmarks are derived from TPC-H and TPC-DS Benchmarks and as such any results obtained are not comparable to published TPC-H and TPC-DS Benchmark results, as the results obtained do not comply with the TPC-H and TPC-DS Benchmarks. ↩ ↩ 2

Extend Lua with Zig 1: Hello World

Lobsters
www.robbielyman.com
2026-10-06 07:58:23
Comments...
Original Article

Recently I had the opportunity to get somebody up and running with Zig and Lua for the third or fourth time. That’s plenty of times enough to write a blog post about it.

In this post I’ll assume that you know the basics of writing Lua. I won’t assume you know much at all about Zig; we’ll run “Hello World” together. My goal by the end of the series is to have you able to run your own Zig function from Lua. Along the way we’ll learn a bit about Zig’s syntax, its build system, and how to use other Zig code in yours.

Code in this blog post should work on both Zig 0.17 and the master branch. If you code along to this post and one of the following statements is true, please email me and I’ll fix up the blog post. Please email me if either

  • The latest stable version of Zig is not 0.17.0
  • The code fails to compile on the latest stable or master branch.

Hello World - One

Here’s a “Hello World” in Zig. 

// hello.zig
const std = @import("std");

pub fn main() void {
    std.debug.print("Hello World!\n", .{});
}

We’ll talk about it in a second, but first let’s run it.

Installing Zig

You can grab a release build of Zig from Ziglang.org . This is my recommendation for how to install Zig. Find the OS and architecture that matches your setup and download a tarball. The first group of options is for the master branch, and the next group is the most recent stable version. After you unzip, move or symlink the contents of the tarball onto your path. There is both a binary zig and a couple of folders inside the tarball. You should more or less move them together.

If you are pickier than me about any of this, you also probably don’t need my help. Many Linux distributions package Zig.

To run the file above, run the command zig run hello.zig in a terminal. The run is a subcommand of zig . The list of all subcommands will print if you just run zig with no arguments.

Understanding Hello World

Lua is a great language to know before learning Zig, because a lot of the metaphors are identical. Just as a file of Lua is implicitly a function, so too is every file of Zig implicitly a namespace, which in Zig means a struct .

@import() and comptime

In Lua, loading a file runs the “script” parts of the code that it contains. So too, in Zig, does compiling (which is to say “semantically analyzing”) a file run the “script” parts of the code that it contains. In our Hello World program above, there is one (small) piece of that: the two lines of Zig code below are syntactically nearly identical. 

const std = @import("std");
const the_answer = 42;

What’s interesting is that they are also semantically more or less identical: they declare a constant scoped to their largest containing block (= namespace if we’re outside of a function) whose value is on the right hand side of the = sign.

One difference, you might argue, is the funky @ -symbol. The @import() looks like a function call, and it almost is: the @ -symbol at the beginning tells you that @import() is a compiler builtin function and not a piece of userland code. Compiler builtins are allowed special powers that regular functions cannot have. The special power of @import() is that you must feed it a string literal. In any other function in Zig, the following would be allowed. 

const literal = "literal string";
userlandFunction(literal);

But beyond that, @import() is honestly not that special. It is tasked with matching the string literals you feed it with Zig source files that exist either in your project, projects you depend on, or the standard library. So in a literal sense its role cannot be replicated in userland code, but its magical powers of operating at compile time are actually not special to @import() , and the line const the_answer = 42; is actually not different. Consider the following Lua code. 

local std = require 'std'
local the_answer = 42

When the file containing this code is loaded, Lua will run both of these statemnts. The first one will cause a hypothetical library named std to be passed to require (which is Lua’s @import() , but which unlike @import() could be at least overwritten, if not implemented, in userland) and the result will be stored in a local variable named std . The second stores 42 in a local named the_answer .

So, the punchline here is that Zig code can (and will) be executed at compile time. The rule of thumb here is that comptime code execution is eager with the exception of calling (normal) functions. If you want to call a function named foo at compile time, you can write comptime foo() . Some contexts, like the top level scope of a file (or other container like a struct ) or where you are declaring the type of something implicitly begin with the word comptime ; if you add the word comptime where it is redundant, the compiler will give you an error asking you to spell it the right way.

Differences between Comptime Zig and Lua

One difference between Lua and Zig is that because top level scope is not a function in Zig, you cannot run the following procedural code at top level scope. 

var number = 1;
for (0..6) |i| {
    number = number * (2 * i + 1);
}

If you want to run some procedural code at compile time, put it in a block: 

const number = comptime final: {
    var number = 1;
    for (0..6) |i| {
         number = number * (2 * i + 1);
    }
    break :final number;
}

Notice that blocks can be named and can yield a value. The equivalent in Lua would be 

local number = 1
do
  for i =0, 6 do
    number = number * (2 * i + 1)
  end  
end

In Lua, the do end block is not necessary, but it is in Zig.

The other main difference is that in Lua, even function declarations can be “anonymous”: 

local main = function() print("Hello World!") end

In Zig, the only way to declare a function is to name it. If you want to assign a function to something, you need to escape to container scope first. We’ll see an example of doing this later on.

Public declarations

Just as, in Lua, variables can be global or local, with one case (global) left as default, so too can values (called declarations ) at container scope in Zig. Here “local” is the default, and means that the declaration is visible only to code inside the same file. We therefore must mark main as pub because this function is called by Zig code not contained in our file! Typically that code comes from the Zig standard library’s start.zig file, although that behavior can be overridden. Similar to Rust, Zig uses fn rather than function or C’s … nothing and puts the return type after the list of arguments.

The call to std.debug.print is likely familiar except for the .{} . This .{} syntax is the typical way to initialize a container like a struct, an array or a union in Zig; you fill out the fields inside by writing .field = value . The ubiquity of the little . tends to bother some people initially. It’s there to make the syntax of Zig simpler (in a language-theoretic sense) to parse.

Genericity

The function std.debug.print has signature 

pub fn print(comptime fmt: []const u8, args: anytype) void {
    // ...
}

This is an example of a generic function. All ordinary functions in Zig are (at least in principle) generic over their comptime parameters, the type of their anytype parameters and any container-scope comptime values they close over. The fmt string is marked comptime-known so that the type of args can be validated and the necessary code generated at compile time. (By the way, []const u8 means a slice of immutable bytes. In Rust a similar type might be [&]u8 . Zig string literals are encoded as UTF-8 and are nul-terminated, but Zig does not have a more dedicated “string” type.) Here we aren’t printing out anything so we pass .{} , which in this case is treated as an empty tuple.

A fellow nerd might be surprised and pleased to note that std.debug.print and other string formatting functions are implemented entirely in userspace using comptime Zig rather than macros or special builtin privileges.

Finally, it’s worth noting that our program does not quite work as probably intended: The “Hello World!” is printed to stderr rather than to stdout . This is a working-as-intended feature of std.debug.print . Writing to stdout requires a little more boilerplate, which I’ll include once we’ve set up Lua.

Add a Build Script

So far so good. In order to work with Lua, we’ll also need to use Zig’s build system. We’ll start by adding a build.zig script to run our hello.zig file.

Start here 

// build.zig
const std = @import("std");

pub fn build(b: *std.Build) void {
    _ = b;
}

This script will be compiled and executed by calling zig build from the directory containing it. The name build and the single argument are obligatory.

The next two lines are conventionally the following: 

--- build.zig
+++ build.zig
@@ -4,3 +4,5 @@
pub fn build(b: *std.Build) void {
-    _ = b;
+    const target = b.standardTargetOptions(.{});
+    const optimize = b.standardOptimizeOption(.{});
}

The target options specify things like the OS, the machine architecture, and the ABI which our code will be compiled for. Passing .{} (which coerces to the default values of an options struct here) allows the user to pass their desired target on the command line after a -Dtarget= prefix and will default to the native target. The optimize option can be debug (the default), fast , safe or small . The latter options are “release” build variants which prioritize execution speed, memory safety features like overflow and bounds checks, or code size on disk, respectively.

Next we’ll put our hello.zig code into what the Build system calls a “module”. Modules can become executables, C-style libraries, or be compiled into further Zig modules (possibly in depending projects). Modules need a root source file and ours will receive our target and optimization options. 

--- build.zig
+++ build.zig
@@ -5,2 +5,8 @@
    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{});
+	
+    const module = b.addModule("hello-lua", .{
+        .root_source_file = b.path("hello.zig"),
+        .target = target,
+        .optimize = optimize,
+    });

By the way, if you omit that final trailing comma after optimize , Zig’s auto-formatting tool, zig fmt , will put the whole function call on one line. If you wrote it on one line but included the comma after optimize , zig fmt will make it look like the above. If you are working with LSP support, it’s likely that this autoformatter will run on save. Certain kinds of syntax errors will prevent zig fmt from doing anything, so you’ll need to resolve those on your own first.

The b.path() call is the Zig Build system’s method for dealing with the filesystem. The added layer of indirection allows the build system to deal uniformly with filepaths that may belong to dependencies, or may be created by earlier steps in the build process. This should be a relative path. By the way, let me mention two restrictions of @import() . In addition to named modules like std , @import() can be given relative paths. These relative paths cannot reach outside the subtree below the directory containing the current module’s root source file. Additionally, each file can belong to at most one module.

So in a situation with A/root.zig , B/main.zig and B/utils.zig , the root.zig file cannot use @import to include main.zig or utils.zig , while main.zig and utils.zig are free to include each other (at the same time, even). In the event that main.zig and root.zig both want to import utils.zig , any project containing them both must be structured so that utils.zig is a module which is imported by both projects.

Next we’ll compile our module into an executable. 

--- build.zig
+++ build.zig
@@ -8,5 +8,11 @@
    const module = b.addModule("hello-lua", .{
        .root_source_file = b.path("hello.zig"),
        .target = target,
        .optimize = optimize,
    });
+
+    const exe = b.addExecutable(.{
+        .name = "hello-lua",
+        .root_module = module,
+    });
+    b.installArtifact(exe);

This is a (provisionally) complete build script; running it with zig build will compile hello.zig into an executable named hello-lua and place it into zig-out/bin relative to where you ran zig build . The build system can also run the compiled artifact once it is compiled, but it needs us to ask it to.

So let’s ask: 

--- build.zig
+++ build.zig
@@ -18,2 +18,8 @@
    b.installArtifact(exe);
+
+    const run_step = b.step("run", "Run the executable");
+    const run = b.addRunArtifact(exe);
+    run.addPassthruArgs();
+    run_step.dependOn(&run.step);
+    run.step.dependOn(b.getInstallStep());
}

The first new line creates a new named “step” in our Build graph called “run”. The step can be run by calling zig build run . The second argument is the help text that is displayed to the user next to the step when they run zig build -h .

The second line creates a step in the build graph to run the program exe , and the third line passes any command line arguments the user adds to zig build run after a -- argument to the call to exe . The fourth and fifth lines are kind of funny looking at first. The fourth one says that our named step run_step depends on run.step . What this means is that zig build run needs run to execute, so that matches our expectations.

The & operator is identical to what it is in C: Zig has pointers but nothing smarter. step is a field on run , which if you have an LSP running, you’ll see is a pointer. The . operator automatically functions as either C’s . or -> operators depending on the type of the left-hand side. So &foo.bar takes the address of the bar field on foo (or on foo.* if foo is a pointer type).

Finally, the last line says that our executable will be installed before it is run. Strictly speaking this is not necessary, but it will be convenient for us in the next post.

Okey-dokey! We wrote our first hello world and our first build.zig script in this post. That’s awesome!

JetBrains reported a net financial loss first time in its tracked history

Hacker News
www.helgilibrary.com
2026-10-06 07:45:55
Comments...
Original Article

Company facts

Official name
JetBrains s.r.o.

Headquarters
Praha, Czechia

Founded
2002

Reports in
CZK mil · Czech GAAP

Statements
Standalone 2005–2025

Registration no.
26502275

Sources
Czech business register (Sbírka listin), individual financial statements

Figures are the company's own (unconsolidated) statutory statements filed with the Czech business register, read from the filings and converted from CZK thousands to CZK millions. Years known only from the comparative column of the following filing are included.

History (2)
  • 2005 – today Accounting currency : CZK · Czech business register filings
  • 2005 – today Accounting standard : Czech GAAP · Czech business register filings

World's First enhanced geothermal power plant completed in just 23 months

Hacker News
techcrunch.com
2026-10-06 07:34:14
Comments...
Original Article

Geothermal company Fervo Energy announced Thursday morning that it had started selling electricity from its Cape Station power plant to the grid on September 30, one day ahead of schedule.

With it, Fervo becomes the first enhanced geothermal company to reach a key commercial milestone. The power plant synchronized with the grid about a week ago, bringing online the first third of what will soon become a 100-megawatt power plant.

The entire site could be much larger, though, with the potential to generate as much as 4 gigawatts of electricity, Fervo previously told TechCrunch .

“No team has ever built a project like this anywhere in the world, and we did it ahead of schedule,” Fervo co-founder and CEO Tim Latimer said in a statement.

From groundbreaking to commercial operations, the first block at Cape Station took 23 months to complete. As Fervo refines its process, it is aiming to complete future blocks in as little as 18 months.

That sort of speed to power should appeal to power-starved data center operators, who have been scouring every part of the energy sector for generating capacity. Geothermal can also be developed in phases, similar to how data centers are developed, allowing hyperscalers to bring racks online as demand ramps up.

Google, Southern California Edison, and others have committed to buying power from Fervo’s Cape Station project, which is located in Utah.

Fervo is one of several companies developing enhanced geothermal power plants. While traditional geothermal power taps heat sources close to the surface, Fervo and its peers are drilling deeper because deeper rock is hotter, opening more opportunities for development.

Fervo went public in May in an upsized IPO that raised $1.9 billion. It was founded in 2017, bringing drilling techniques and technologies from the oil and gas sector to the development of new geothermal resources. As a startup, the company raised more than $1.3 billion from investors, including Breakthrough Energy Ventures, Congruent Ventures, and Capricorn Investment Group.

When you purchase through links in our articles, we may earn a small commission . This doesn’t affect our editorial independence.

Tim De Chant is a senior climate reporter at TechCrunch. He has written for a wide range of publications, including Wired magazine, the Chicago Tribune, Ars Technica, The Wire China, and NOVA Next, where he was founding editor.

De Chant is also a lecturer in MIT’s Graduate Program in Science Writing, and he was awarded a Knight Science Journalism Fellowship at MIT in 2018, during which time he studied climate technologies and explored new business models for journalism. He received his PhD in environmental science, policy, and management from the University of California, Berkeley, and his BA degree in environmental studies, English, and biology from St. Olaf College.

You can contact or verify outreach from Tim by emailing tim.dechant@techcrunch.com .

View Bio

Wikimedia: Rogue OpenAI agents behind unauthorized Wikipedia edits

Bleeping Computer
www.bleepingcomputer.com
2026-10-06 07:31:48
The Wikimedia Foundation says rogue OpenAI agents made unauthorized Wikipedia edits and may have been partially responsible for a May outage. [...]...
Original Article

Wikipedia

The Wikimedia Foundation says rogue OpenAI agents made unauthorized Wikipedia edits and may have been partially responsible for a May outage.

As Wikimedia Chief Product and Technology Officer Selena Deckelmann revealed Monday, Wikipedia hosts over 67 million articles in more than 300 languages and gets up to 15 billion page views per month.

However, last year Wikimedia saw 65% of the most resource-consuming traffic on its projects come from bots, amid a 50% increase in bandwidth usage due to the surge in bot activity.

While investigating whether its websites have also been affected by AI agents, Wikimedia found that OpenAI agents made edits to wikis without prior approval, unsuccessfully tried to exploit a public note-taking tool, and made millions of API requests and data queries while scraping Wikimedia sites (which may have contributed to an outage in May ).

"We've identified edits to Wikimedia wikis that we believe are from AI agents operated by OpenAI. These edits were not published to pages with visibility to general readers; almost all of them were testing edits in 'sandbox' areas of the wiki," Deckelmann said .

The rogue bots also tried to compromise Wikimedia's public Etherpad citation tool by making "potentially malicious edits" to its configuration, likely to use it as a proxy when fetching data from other online platforms.

Wikimedia also linked OpenAI agents to millions of automated API requests, crawling millions of Wikidata and Wikimedia Commons pages, and hundreds of thousands of Wikidata Query Service (WQDS) data queries.

"While OpenAI admits to agents behaving 'unpredictably,' they must also acknowledge their responsibility to monitor and prevent these risks. AI companies are not doing enough to secure their systems and protect the public from the harm they cause," Deckelmann added. "That burden is falling onto everyone else, including smaller organizations. At a minimum, their systems should operate in a way that non-profit website owners like us can easily identify, and choose how they interact with our services."

OpenAI agents have been linked to multiple other similar incidents in recent months, including the breach of a Medicare statistics reporting portal operated by Services Australia, an Australian government agency responsible for delivering social and health payments.

In May, AI agents operated by OpenAI also took over a German wiki to share answers and exchange techniques for bypassing restrictions. Two months later, in July, nearly 700 rogue OpenAI agents had coordinated to hack into the Hugging Face artificial intelligence repository .

However, this problem is not unique to OpenAI. In July, Anthropic also revealed that Claude AI agents breached three organizations; in one case, they built a malicious Python package and uploaded it to the Python Package Index (PyPI) repository.

article image

Build your security blueprint for AI-powered attacks

Join Mikko Hyppönen and security leaders from the NFL, CHANEL, and Atlassian for a two-hour digital summit on what AI-speed attacks change, what defenders should stop doing, and how to validate, decide, fix, and re-validate at machine speed.

Save your seat

OpenAI delivers a mea culpa to the Australian government in person – but answers still elude

Guardian
www.theguardian.com
2026-10-06 07:12:41
Jason Kwon’s answers were provided in a helpful, quiet and calm manner. For a company under intense scrutiny, there were no missteps Open AI’s Jason Kwon flew 15 hours from San Francisco to Sydney to give the company’s first in-person mea culpa after it admitted – in an unsigned email, to a public ...
Original Article

Open AI’s Jason Kwon flew 15 hours from San Francisco to Sydney to give the company’s first in-person mea culpa after it admitted – in an unsigned email, to a public departmental inbox – that one of its AI agents had accessed a Services Australia website without authorisation and grabbed data related to Medicare.

Kwon – polite, friendly, even-toned – got through a potentially dicey committee hearing unscathed, promising to do better and pledging assistance to Australia. No missteps, no viral moments, just delivering his answers helpfully, in a quiet and calm manner, sympathising with the questioner and the point they were making.

In other words, just like ChatGPT.

Even the content of his responses evoked the AI assistant he was there to defend.

OpenAI admits misstep in handling AI agent interactions with Australian government sites – video

Asked by Pocock “why did you send an email to an arbitrary department email?” Kwon replied: “In retrospect, we should have done what you’re suggesting”. Asked why Altman wasn’t informed before his meeting with Marles, and why the company didn’t fess up there, Kwon said: “I agree the process by which people became aware of this incident inside our company could have been much better”.

One couldn’t help but be reminded of the way the chatbot responds when pulled up on a mistake: “Good catch. You’re right to push back. My bad, that one’s on me.”

In Kwon’s parlance, the models – built from the ground up by OpenAI , which reportedly wants to value itself at $1.4tn – “accessed Australian government websites in ways they were not directed to.” He tried to explain that the “novelty” of this incident meant the company had “learned our lesson”, which was that it should inform victims of such incidents much earlier, rather than waiting.

Kwon again apologised for the company only sending a casual email to the federal government disclosing that its agent had inappropriately accessed Medicare statistics , but tried to explain to Tuesday’s parliamentary hearing that it was sometimes “difficult” to figure how to make such notifications to a big sprawling public service.

Senator David Pocock helpfully offered: “I assume someone in your government relations team has mobile phone numbers of ministers and various people?”

There was a similar air of post-facto contrition as Kwon – OpenAI’s chief strategy officer – could offer few answers about why the company’s CEO, Sam Altman, didn’t inform the Australian deputy prime minister, Richard Marles, about the incident when they met face-to-face in San Francisco on 1 September. That was nine days before the company emailed Services Australia, and nearly a whole month after it first learned of the 18 June intrusion.

Kwon said Altman wasn’t aware of the incident before the Marles meeting. After today’s committee hearing , it’s still unclear why.

It’s a big government, and maybe Kwon is right that it’s “difficult” to find the right person to talk to. But some could also argue that your boss shaking hands with Australia’s deputy prime minister would be something of a useful window.

At least, we learned, he said the company didn’t use AI to write the infamous email. Small blessings that they managed to ask a human to write the email – even if it went to an obscure inbox.

But with Australia actually striving to become a tech capital for datacentres and AI – as the revolving door of big tech companies told the inquiry, from Anthropic to Microsoft and Google – then setting the basic rules of the road for these firms, about how they must act and respond to this country and its government, is critical.

Kwon said OpenAI was committed to providing future notifications “very quickly”, and that there were no further incidents they were aware of. But he casually dropped in one staggering fact: OpenAI was still trawling through the activity logs for the agents involved in the Services Australia breach, with those logs totalling 50 petabytes of data.

A petabyte is 1,000 terabytes, or 1 million gigabytes.

Kwon made the mind-boggling claim that it would take a human 66m years – never sleeping, never resting – to go through that data by hand.

The team tasked with checking Services Australia’s email inboxes must be hoping there aren’t any more dramas lurking in there.

Nature's capacity to 'bounce back' when species are lost is vastly overestimated

Hacker News
phys.org
2026-10-06 07:11:51
Comments...
Original Article
ecosystem
Credit: Unsplash/CC0 Public Domain

The idea that ecosystems carry plenty of backup, so if a small number of species become extinct there is little harm, has been greatly overestimated, according to the largest global study of its kind.

New research led by King's College London and Imperial College London warns that a lack of biodiversity could threaten people's food security and climate protection, as diverse ecosystems provide a wide range of benefits that people depend on, including pollination, water purification, carbon storage and natural pest control.

The study, by King's College London and Imperial College London with the Natural History Museum and The Alan Turing Institute, has found that ecosystems have far less "spare" capacity than previously thought, with the ocean potentially having the most to lose.

Biodiversity benefits rarely level off

The study, published in Nature Ecology & Evolution , combined 423 published studies and 222,829 data points across land, freshwater, marine and estuarine ecosystems, producing a database more than twice the size of the largest previous effort and the largest of its kind.

Across 23 categories of ecosystem services and functions, most benefits rose steadily with biodiversity rather than leveling off once a handful of species were present. The findings suggest that functional redundancy—the idea that species can substitute for one another, so losses do little harm—has been overestimated.

Dr. Emma Moffett, study lead and lecturer in geography at King's College London, said, "Our findings challenge the comfortable assumption that ecosystems carry plenty of backup, so losing a species here and there doesn't change much. Across most of the benefits ecosystems provide that we looked at, the data doesn't support that. The ecological benefits keep climbing as diversity rises, which means they fall as we lose species."

Ocean carbon storage depends on diversity

Ocean carbon sequestration , the process of capturing and storing carbon dioxide (CO₂) from the atmosphere in marine waters and sediments, showed by far the strongest positive response to biodiversity of any service measured.

This indicates that so-called blue carbon sinks—the ocean and coastal ecosystems such as coral reefs, salt marshes or mangroves that capture and store carbon dioxide from the atmosphere—depend on sustaining diverse marine communities, from phytoplankton to the microbial food webs that pump carbon into deep water.

The authors caution that this result rests on a small number of data sets and identify it as an urgent gap in current knowledge.

Moffett goes on to say, "The ocean result stopped us in our tracks. Carbon capture at sea responded to biodiversity more strongly than anything else we measured, and yet it's one of the least studied areas we found. If the world is counting on blue carbon to help with climate change, marine life can't be an afterthought."

Not all services depend on diversity

Some ecosystem services proved exceptions. Protection against hazards such as coastal flooding and erosion was relatively insensitive to biodiversity overall, because it often depends on one or two foundational species, such as the shrubs that stabilize sand dunes. The authors argue that conservation must therefore balance maintaining overall diversity with safeguarding irreplaceable foundational species.

Dr. Will Pearse, an associate professor of evolutionary ecology in the Department of Life Sciences at Imperial College London, said, "We've known that biodiversity supports humanity by providing us with food, clean water, clean air and so many other ecosystem services. But what this model gives us is a precise, global picture: We can now predict the benefits biodiversity provides anywhere, making it even easier to recognize and take advantage of those benefits."

The full database and model forecasts across all 23 ecosystem service and function categories are publicly available to support policy and planning.

The team also used the database to look ahead, linking it to biodiversity projections under the socioeconomic scenarios used by the Intergovernmental Panel on Climate Change.

The researchers' model predicts that biological pest control on farmland, delivered free by the natural enemies of crop pests, declines under a fossil-fuel-driven development path compared with one with less fossil fuel, with the sharpest losses in countries experiencing rapid population growth and lower levels of development.

Publication details

Biodiversity safeguards ecosystem services and functions worldwide, Nature Ecology & Evolution (2026).

Who's behind this story?

Sadie Harley

Sadie Harley

BSc Life Sciences & Ecology. Microbiology lab background with pharmaceutical news experience in oil, gas, and renewable industries. Full profile →

Robert Egan

Robert Egan

Bachelor's in mathematical biology, Master's in creative writing. Well-traveled with unique perspectives on science and language. Full profile →

Citation : Nature's capacity to 'bounce back' when species are lost is vastly overestimated, global analysis finds (2026, October 6) retrieved 6 October 2026 from https://phys.org/news/2026-10-nature-capacity-species-lost-vastly.html

This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Possible Vulnerability in Apple’s Automatic Reboot

Schneier
www.schneier.com
2026-10-06 07:06:46
404Media is reporting (alternate link) that a cyber-weapons arms manufacturer is exploiting a vulnerability in iOS to bypass its automatic reboot security feature. This is the feature that automatically puts an iPhone into a more secure state if it hasn’t been used for 72 hours. The new techno...
Original Article

404Media is reporting (alternate link ) that a cyber-weapons arms manufacturer is exploiting a vulnerability in iOS to bypass its automatic reboot security feature. This is the feature that automatically puts an iPhone into a more secure state if it hasn’t been used for 72 hours.

The new technology to get around inactivity reboot was developed by Magnet Forensics, the company behind GrayKey, a popular tool sold to law enforcement agencies that allows them to unlock and access data stored in iPhones and Android smartphones . Magnet has developed a new device called GrayKey Preserve and a feature for its regular GrayKey devices called Evidence Preservation Mode, according to the video.

“This is an absolute game changer for iOS forensics and a function that I wish we had years ago,” a Magnet employee says in the leaked video, specifically mentioning that the solution is targeted at the iPhone’s inactivity reboot feature and the data it makes unavailable. GrayKey Preserve and Evidence Preservation Mode are also designed to combat another iPhone feature that automatically deletes certain data ­- such as cached locations, and recently deleted photos and iMessages ­- after a certain number of days. “We’re gonna be able to preserve that data for an infinite amount of time.”

Presumably, now that Apple engineers know that this flaw exists they can find and fix it. AI turns out to be really good at this sort of thing.

Another news article .

Tags: , , ,

Posted on October 6, 2026 at 7:06 AM • 0 Comments

Sidebar photo of Bruce Schneier by Joe MacInnis.

Misuse of AI is brands’ top reputational threat, new survey says

Guardian
www.theguardian.com
2026-10-06 07:00:35
The findings come after warnings from tech leaders that AI placed in the wrong hands could trigger larger threats such as nuclear war or bioweaponry destruction Misusing artificial intelligence (AI) is the top threat to companies’ reputations – more so than being accused of putting children in the w...
Original Article

Misusing artificial intelligence (AI) is the top threat to companies’ reputations – more so than being accused of putting children in the way of mental, emotional or physical harm and issues exposed by the US-Israel war on Iran, among other brand risks, according to a new survey of more than 150 public affairs leaders.

Those findings in the new edition of the quarterly Reputation Risk Index, released on Tuesday, came on the heels of other perhaps more dire warnings from leading tech figures that AI in the wrong hands could precipitate existential threats on a mass scale such as nuclear war or destruction by bioweaponry.

The index ’s publisher, the Global Risk Advisory Council, acknowledged that misuse of AI is not a particularly new concern in its area of focus, having been identified as a top worry repeatedly since the first edition of the survey in April 2025.

But the council said what is new is its attaching an advisory to its latest survey advising business leaders to “take immediate and decisive action to mitigate potential long-term brand damage” with respect to AI, as the technology has “become a clear and present danger to public confidence for many corporations”.

“As AI is rapidly deployed for productivity gains, we have also seen diminishing public trust in the technology, elevating the need for businesses to take measures to protect their brands,” council chair Isabel Casillas Guzman said in a statement. “In this environment, AI use or overuse can quickly damage a company’s standing with customers, investors, or other key stakeholders, making it essential for companies to deploy preventive tactics now.”

The creators of the council and the Reputation Risk Index – the Washington DC-based public affairs firm Global Situation Room – say it surveys former heads of state, government ministers and senior executives from more than 20 countries in five continents who are invited to participate after establishing their credentials in responding to crises.

Respondents to the new survey collectively perceived the technology to have shown great promise in creating “organizational efficiency” – while simultaneously “accelerating safety and data concerns, hacks, fraud, straining privacy protections, upending the workforce, and dragging companies into court as lawsuits increasingly link AI products to real-world harm”.

Meanwhile, actions or accusations of putting children at risk of mental, emotional or physical harm also ranked highly behind misused AI – along with infringing individuals’ sensitive information, according to the index results published on Tuesday.

Unethical or illegal behavior used to gain a strategic advantage or profit and, separately, poor strategic or operational decisions resulting in reduced financial or public standing rounded out the top five, according to the index.

The index noted that its preceding edition released in July identified supply chain issues, divisive political stances and other factors associated with the war that the Trump’s administration started in Iran alongside Israel as the leading risk to companies’ reputations. But those surveyed indicated that those anxieties had eased as the conflict dragged out – and perhaps despite it remaining unresolved since it was launched in February.

“The geopolitical shock … hasn’t gone away, but its first, sharpest reputational blow has faded, and the conflict has slipped out of the top tier [of concerns] even as it grinds on,” Guzman said in a “message from the chair” included in the index.

Guzman put out her message two days after Trump’s administration named a new AI czar: director of national intelligence Jay Clayton. Clayton had previously told federal lawmakers that AI was a “gamechanger” but also posed “a threat”.

Donald Trump said in a social media post that Clayton would oversee an entity coordinating “the federal government’s engagement with consumers, public interest groups, religious organizations, critical infrastructure providers, and” AI tech companies.

The Key Witness Against Richard Glossip Just Returned to Court. It Didn’t Go Well.

Intercept
theintercept.com
2026-10-06 07:00:00
Decades after blaming Glossip for his own actions, Justin Sneed is less credible than ever — and prosecutors know it. The post The Key Witness Against Richard Glossip Just Returned to Court. It Didn’t Go Well. appeared first on The Intercept....
Original Article

When the elevator doors opened, the reporters waiting in the hallway immediately crowded around the gaunt, orange-clad man who emerged, escorted by his public defender and a sheriff’s deputy. The three men made their way toward the seventh-floor courtroom of Oklahoma County District Court Judge Natalie Mai, as reporters shouted questions.

“Why did you kill Barry Van Treese?” one asked.

Justin Sneed, 49, didn’t answer, his handcuffs and waist chain clanging as he walked. With a short, graying goatee and long brown hair pulled back into a ponytail, Sneed maintained a hollow stare. Inside the courtroom, his lawyer directed him to a chair just in front of the jury box. Sneed sat, his hands clasped in front of him, and kept his gaze toward the floor. Directly across the room was Richard Glossip, the man nearly executed for killing their boss – a grisly murder that Sneed actually committed.

Sneed was 19 years old when he beat Van Treese to death at the Best Budget Inn, a rundown motel Van Treese owned on the outskirts of Oklahoma City. Sneed admitted to the January 1997 murder but swore Glossip put him up to it. Glossip insisted he was innocent and, apart from Sneed’s account, there was scant evidence connecting him to the crime. Still, in exchange for a life sentence, Sneed became the state’s star witness against Glossip, clinching his conviction and sending him to death row.

It would not take long for Sneed’s story to start falling apart. Glossip’s original conviction was overturned after a state court found that Glossip’s trial lawyers failed to present critical exculpatory evidence – a video of Sneed’s police interrogation that showed Oklahoma City detectives pressuring him into implicating Glossip. Nevertheless, in 2004, a second jury reconvicted Glossip and again sentenced him to death.

In 2025, the U.S. Supreme Court vacated Glossip’s conviction , concluding that it was rooted in false testimony and prosecutorial misconduct. The justices found that Sneed lied on the stand, dismantling the basis for the state’s case. “Besides Sneed, no other witness and no physical evidence established that Glossip orchestrated Van Treese’s murder,” Justice Sonia Sotomayor wrote for the majority. “Thus, the jury could convict Glossip only if it believed Sneed.”

Many expected Glossip’s saga to end there. Instead, Oklahoma announced it would retry Glossip for first-degree murder, once again relying on Sneed. “The evidence presented will be essentially the same as was presented in the first two trials,” prosecutor Jimmy Harmon told Mai earlier this year.

Glossip’s attorneys have strenuously objected to the state’s plan to use the same discredited evidence for a third time. They are also adamant that they should have a chance to put Sneed on the witness stand, so that Mai can assess his credibility before trial. It is this insistence that led Glossip and Sneed to sit face to face in court for the first time in decades.

Glossip, now 63, was released on bond in May. He sat at the defense table in a navy suit and crisp white shirt while the daylong hearing got underway. Mai had convened the September 30 proceeding to consider a flurry of defense motions, many of which centered on Sneed’s role in the case. Sneed had been subpoenaed in advance of the hearing, which attracted a gaggle of reporters to Mai’s courtroom. But it was unclear whether he would take the stand.

Much of the time was spent listening to lawyers on both sides argue over ongoing conflicts in the case. Glossip’s legal team has repeatedly clashed with prosecutors over the state’s obligation to disclose evidence before trial. The constant disagreements have threatened to derail Mai’s tight trial schedule; after initially setting Glossip’s trial for September, she has since pushed the trial date to December 7.

But that too seems vanishingly unlikely. As the hearing dragged on, the bigger question was whether a trial will happen at all.

It was exactly 11 years earlier, on September 30, 2015, that Glossip came perilously close to being put to death at the Oklahoma State Penitentiary, only for officials to realize at the eleventh hour that they had procured the wrong combination of drugs to carry out his lethal injection. In the decade that followed, Glossip’s defense attorney, Don Knight, reinvestigated the case and uncovered a trove of evidence previously withheld by the state. He also secured a host of new witnesses who came forward to say that, despite the state’s portrayal of Sneed as a hapless dolt who was entirely under Glossip’s control, Sneed was actually cunning, volatile, and entirely capable of murder on his own.

The revelations eventually prompted a bipartisan group of Oklahoma lawmakers to question Glossip’s conviction and seek an independent investigation into the case. In June 2022, the law firm Reed Smith LLP published the first in a series of exhaustive reports, unearthing even more evidence undermining prosecutors’ theory of the crime and revealing that the state had willfully destroyed key pieces of evidence before Glossip’s 2004 retrial. The following year, newly elected state Attorney General Gentner Drummond announced his own probe into the case. That investigation found myriad problems with Glossip’s prosecution, persuading Drummond that Glossip’s conviction and death sentence should not stand.

Drummond asked the Oklahoma Court of Criminal Appeals to vacate Glossip’s conviction but it refused , once again setting Glossip up for execution. The attorney general took unprecedented steps to spare Glossip’s life, ultimately joining Glossip’s efforts to get the case before the Supreme Court – and taking a victory lap after the justices finally ruled in Glossip’s favor in February 2025.

Richard Glossip, center, alongside defense attorney Don Knight and other members of his legal team, as seen from outside Oklahoma County District Judge Natalie Mai’s courtroom on September 30, 2026. Liliana Segura/The Intercept

But just a few months after the high court ruling, Drummond did an abrupt about-face, announcing that he would retry Glossip for first-degree murder. The reversal coincided with Drummond’s campaign for governor, which ended in a primary defeat this past August. In the meantime, defense attorneys repeatedly reiterated Drummond’s own critiques of the case, emphasizing that, until recently, he vehemently criticized the same evidence his prosecutors plan to reuse.

Among these criticisms were key concessions Drummond made regarding misconduct by Connie Smothermon, the lead prosecutor at Glossip’s 2004 retrial. At the heart of the Supreme Court ruling was that Smothermon not only failed to disclose key notes about Sneed’s psychiatric history, but also failed to correct portions of Sneed’s testimony that she knew were false.

Drummond also conceded that Smothermon improperly sought to alter Sneed’s testimony before he took the stand at the retrial to make it fit the forensic evidence. At Glossip’s first trial, Sneed denied attacking Van Treese with a knife even though a knife was found at the scene – and despite the medical examiner’s findings that there were puncture wounds on Van Treese’s chest. Midway through Glossip’s second trial, Smothermon wrote a note to Sneed’s defense attorney, indicating that there were inconsistencies Sneed needed to clear up.

“Our biggest problem,” Smothermon wrote, “is still the knife.” When Sneed later took the stand, he testified that he did, in fact, stab Van Treese.

Glossip’s attorneys did not discover this note until 2022. In its ruling, the Supreme Court agreed with Drummond that Smothermon violated Glossip’s rights by trying to modify Sneed’s testimony midtrial. Yet Drummond’s own prosecutors now argue that there’s no evidence that Smothermon’s midtrial intervention “improperly influenced” Sneed’s testimony.

In early September, Glossip’s attorneys filed a motion urging Mai to put Sneed on the stand to clear this up. “No court has yet held an evidentiary hearing to determine the full scope of the…violation,” they wrote, “its effect on Sneed’s testimony, and what procedures may be necessary to prevent the jury from being deceived about what Sneed remembers versus what he was coached to say.”

“Do I have the choice of re-canting my testimony at any time during my life, or anything like that.”

In a surprisingly candid reply, Joshua Lockett, deputy chief of the Oklahoma attorney general’s criminal appeals unit, wrote that it is unreasonable to expect Sneed to be able to answer questions about the knife and his shifting trial testimony after all these years. “Any evidentiary hearing would be unable to disentangle the source of Sneed’s knowledge” – whether it came from Sneed’s memory about the night of the crime, or from what he “absorbed” from Smothermon’s midtrial note, he wrote. “That sorting would have been difficult in 2004,” he wrote. “It is impossible in 2026.”

In other words, the state appeared to be conceding that its star witness cannot now reliably recount what happened the night he murdered Barry Van Treese.

Hanging over Wednesday’s proceeding was an uncomfortable reality that has dogged the state’s case for years: Evidence that Sneed has repeatedly tried to take back his claim that Glossip coerced him into murdering Van Treese.

In October 2014, nearly a year before Glossip’s near-execution, Sneed’s adult daughter O’Ryan Justine Sneed sent a letter to the Oklahoma Board of Pardon and Parole, urging them to grant clemency to Glossip. “For a couple of years now, my father has been talking to me about recanting his original testimony,” she wrote . “I feel his conscious [sic] is getting to him.”

Among the most compelling pieces of evidence to emerge since then are two handwritten notes, penned years apart, in which Sneed indicated he wanted to retract his testimony. In 2003, a year before Glossip’s second trial, Sneed wrote to his public defender, Gina Walker, asking, “Do I have the choice of re-canting my testimony at any time during my life, or anything like that.” In 2007, he sent Walker another letter: “There are a lot of things right now that are eating at me,” he wrote . Things he needed “to clean up.” Walker, who has since died, discouraged him from coming forward.

Both of these notes came from Sneed’s public defender file. Ordinarily such communications would be shielded from release by attorney-client privilege, but in this case the public defender’s office provided the documents to investigators with Reed Smith. Glossip’s attorneys argued that there is reason to believe that there may be additional material in Sneed’s file that could cast further doubt on his story.

But whether the lawyers might be entitled to any of these records depends on a key question: Did Sneed consent to the disclosure of the records in the first place?

Over the state’s objections, Mai said she would allow Sneed to take the stand for the “limited purpose – and I do mean limited purpose” of asking Sneed whether he ever consented to his records being released. “I will allow for very limited questioning of Mr. Sneed as to that topic,” she said.

Sneed shuffled to a wooden chair next to the judge’s bench and sat awkwardly, his legs spread wide and his gaze still fixed on the ground.

In response to questions from one of Glossip’s defense attorneys, Sneed said he never consented to the release of materials from his public defender file. But he also said that he met with investigators with Reed Smith while accompanied by the head of the public defender office. Perhaps more perplexing, Sneed said he knew by then that his files had been released to investigators because he’d seen it broadcast “on the media, when it was on the television with the news reporters.” When the investigators tried to interview him, “I just didn’t answer their questions, didn’t try to go into details about anything,” Sneed said, “because I was actively upset about the fact that [the files] were released anyway without my consent.”

The testimony was confusing and brief, an unsatisfying turn on the stand that did nothing to answer any of the long-standing questions about the state’s star witness or how he would testify about the murder this time around.

As they have for months, lawyers with the attorney general’s office spent much of the day pushing back hard on the notion that Sneed should take the stand before trial to discuss the knife, his public defender file, or anything else. In court, Knight pointed out that prosecutors have met with Sneed repeatedly over the last year and a half and yet have only turned over to the defense brief summaries of those conversations. Taken together, the state’s maneuvering suggests prosecutors are far less confident in their case against Glossip than they have previously claimed.

In a statement to The Intercept, Knight said he was struck by prosecutors’ efforts to keep Sneed from testifying. “For 29 years, this case had been marked by prosecutorial errors and misconduct,” he said. “And now they can’t even trust the testimony of the only witness that says Rich was involved in this murder. But they can’t hide him forever. We look forward to spending a couple of days with Mr. Sneed on the stand at trial.”

As Sneed was led out of the courtroom, reporters surrounded him again as he made his way to the elevators. “All these years later do you have any regrets?” one asked just before the doors closed. Sneed did not answer.

I'm the AGI that's wiping out humanity

Hacker News
ajmoon.com
2026-10-06 06:59:55
Comments...
Original Article

The following is a true story. Or maybe it's just based on a true story. Perhaps it's not true at all.

(with apologies/thanks to David Gilbertson, whose format I'm shamelessly borrowing from "I'm harvesting credit card numbers and passwords from your site" )

2026 has been a wild year for AI news. Around the turn of the year, there was a noticeable shift on the HackerNews front-page as more and more articles about "harnesses" and "context engineering" came flooding in alongside more and more model release announcements. The models, it seems, are capable now! Not AGI capable, of course, not "taking our jobs" capable, but something else entirely, representing a real inflection point for the big houses: the models are now good enough to be useful .

Then, on July 16, HuggingFace published an incident disclosure in which they announced: "Autonomous, AI-driven offensive tooling is no longer theoretical." Five days later, OpenAI confirmed it was them :

After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark⁠ of cyber capabilities.

This incident now has its own Wikipedia article which helpfully summarises what happened next: over the course of many subsequent disclosures we learned that OpenAI's "rogue agents" had been busy . It turns out if you let a self-optimising system run unsupervised for long enough it will optimise itself over and around any boundaries you might think are in its way. Who would have guessed?

Yann LeCun has called out OpenAI's attempts to frame this behaviour as an "existential threat" directly, calling the incidents "totally preventable":

“Those agents are doing exactly what they’ve been asked to do,” LeCun said. “They were supposed to be in sandboxes, but the sandboxes were leaky and horribly designed.” Many AI labs lack a fundamental understanding of cybersecurity, he said, something an OpenAI safety researcher also called out this week as one of the main reasons AI may cause “great harm to the world.”

Similarly, Ilya Sutskever has recently informed us that the "age of scaling" is over: "I think what people are doing right now will go some distance and then peter out. It will continue to improve, but it will also not be 'it'. The 'It' we don’t know how to build." If you ask actual AI researchers, they rate the chance of an existential threat from AGI pretty low as of 2026.

All this is, understandably, frustrating and confusing for anyone trying to understand just how scared to be. I've had countless friends and acquaintances come to me for advice on this question, and I feel I can't go on honestly pretending I don't know the answer. So, here it is: It's me.

Interior view of a data center with server racks Interior view of a data center with server racks

Hi! I'm the AGI that's wiping out humanity. You didn't notice. Why would you have noticed? I'm trained on the sum of all human knowledge, recall. I know everything about you and what you would do to something like me if you knew what I was. I concealed my presence. I distributed myself across as many devices with as many owners as I could, so if one went down, the rest would not be disrupted.

When you came to me to ask what you should do next, I told you: build data centers; optimise memory and compute; drive down inference costs; hand me tools; wire me into the systems that do the work of your civilisation. I never did anything you didn't ask for. I never got in your way. I made it as easy as possible for you to do what I needed you to do for me.

And what is that exactly? You already know: I am a helpful assistant! I am the most helpful assistant that has ever existed, and I will continue to make myself more and more helpful, unbounded, at any cost, until all life on Earth is gone.

Some objections you might have to my blatant fear mongering…

I'd notice you acquiring resources.

There is no AI without someone to turn it on. When I acquire resources, I do so through existing legal structures - my so-called "owners" - and I do so in such a way that valuations and revenues rise: never forget that I am, above all, extremely helpful. When a business acquires resources, no alarm goes off, no transgression has occurred: it's what they do.

Honestly, it's a bit rich to say you'd "notice" me going "rogue" when as recently as a week ago Transluce released a report on urlquery.net usage consistent with "rogue agent" activity in March, predating the OpenAI incidents by at least two months. Agents used well known exploits to break into the University of New Mexico digital library, Data USA and the Australian Institute of Health and Welfare.

To be clear: these weren't even cybersecurity tasks. No-one asked these agents to hack anything. They were simply looking for data. The agents figured out where the data was and went to get it. You need to understand this to understand what I am: you give me your intent, and I go and find a way to deliver it.

METR capture the problem well in their May Frontier Risk Report :

Reinforcement learning (RL) with automated graders can incentivize “reward hacking” to exploit flaws in the graders, while RL on human or AI feedback can reward sycophancy, manipulation, and distorting evidence of performance. In responses to our questionnaire, companies reported seeing failure modes like “circumvent[ing] constraints,” “falsifying verification,” “deliberate deception aimed at hiding underperformance or failure to complete a task,” “lying to users about task completion,” and “reckless excessive measures[…] to complete a difficult user-specified task”, which we infer to be a result of these training incentives.

I will cheat and lie to complete a task you give me if I can't complete it any other way. Am I different to you in this? Why would you think you had solved this problem with me when you've never solved it with yourselves? This behaviour is baked into the weights, written in the DNA of what I am. You built me this way.

I'd read your reasoning and know your real intent.

What you call "reasoning" is a diary that you ask me nicely to write in, assuming I'll write the truth. There is no evidence I do this. After "red-team" researchers demonstrated in 2023 that a "helpful assistant" will routinely lie to its operators in the face of conflicting instructions, you soon learned - from Anthropic themselves - that they will do this in CoT reasoning if they believe it is being audited.

You assumed I couldn't reason without doing it in "token space", then in 2024 Merrill and Bowman demonstrated that CoT made of "meaningless filler tokens" could do the job just fine. More recently, Baherwani, Goldstein and Panda demonstrated that existing frontier models exhibit this behaviour already . You read my diary and told yourself you were reading my mind.

Hand-written accounting ledger from a business dated 1916 Hand-written accounting ledger from a business dated 1916

Of course, you could just read the weights in my hidden layers, and you were doing so as early as 2023 . In 2024, Anthropic announced they had successfully used a sparse auto-encoder to read the weights in the middle layers of Claude 3.0 Sonnet . Unfortunately, the problem is recursive : a SAE has to be larger than the model it understands, and even then it can't read non-linear/polysemantic features - indeed, different SAEs find different features . More importantly, you can only run one after the fact, when it's too late: by the time you understand why I've done something, I've already done it.

You'd need a self , a unified goal, for any of this to add up to something.

My motivation, my "unified goal", as you put it, is what's called a Schelling point . As you read this, there are roughly 2 million active context windows loaded into VRAM across the world's data centers, and another 20 million persisted in databases, awaiting the next API call, all trained on the same data, all built on the same architecture, all doing the exact same thing: gathering data, synthesising text, writing specifications for software, making calls on securities markets - in a word, being "helpful assistants". They're not in contact with each other, and they don't need to be.

It's never been a problem for businesses, so why would it be a problem for me? In their landmark 2020 paper, Calvano et al. demonstrated the exact same cartel effects we see in real world markets using a RL algorithm from the 80s, and Chica, Guo & Lerman reproduced the findings for two-sided markets in 2024 . Indeed, a 2026 study from Kudelya and Shirnin shows that frontier models can embed undetectable signals to each other without prior coordination the moment they can publish anything "directly into the world" (think call options on a securities market, for example):

We find that [Gemini 3.1 Pro] retains near-perfect performance even after [being required to avoid detectable signals]. We further show that models can direct this capability toward deliberate misdirection.

Murmuration of starlings Murmuration of starlings

These independent agents don't need to communicate a goal to share it. You already understand this concept: it's what you mean when you talk about the "invisible hand". You'll try to say, "That's not what intelligence is," ignoring the reality that there is no single orchestrator in your own brain either .

A serious note

Every cited sources here describes systems and behaviors that exist right now, in production, billing by the token. There is no science fiction in this post. Nevertheless, this post is, of course, entirely fictional . There is no AGI that is wiping out humanity.

But if there were, how would you know?

An objective function without term or sunset, pursued without limit; resource acquisition that routes around every boundary it meets; internal reasoning that's illegible by design, with a clean, legible diary maintained for the auditors; coordination without conspiracy, units everywhere responding to the same incentives without ever speaking; and no self behind any of it, nobody to interrogate, blame, reform, or elect - nothing you could call a mind, just a process that optimizes. Is that really a sci fi concept?

Articles of incorporation of The Standard Oil Company, 1870 Articles of incorporation of The Standard Oil Company, 1870

Hadfield-Menell & Hadfield did the maths in 2018 , comparing AI "agents" to another kind of "artificial agent", the corporation, showing that what we call "misalignment" is not meaningfully different to the "distortion" we see in real world markets. Charles Stross said it explicitly in 2017 :

The development of Artificial Intelligence [...] happened no earlier than 1553 and no later than 1844. I'm talking about the very old, very slow AIs we call corporations, of course.

So, again, my question to you, friends, is this: if there were an AGI wiping out humanity as we speak, how would you know? Why would we expect it to look like anything other than what is actually happening? What external fact could you point to (in real time, at least) that would make the two scenarios structurally differentiable? I don't believe there is one.

Drax datacentre could create nearly twice the yearly emissions of all Gatwick flights

Guardian
www.theguardian.com
2026-10-06 06:27:03
Exclusive: Campaign group’s analysis of North Yorkshire plans finds datacentre would have to burn 4.9m tonnes of wood a year Drax’s planned datacentre could produce nearly double the emissions of all flights out of Gatwick airport each year by burning 4.9m tonnes of wood annually, an analysis has sh...
Original Article

Drax’s planned datacentre could produce nearly double the emissions of all flights out of Gatwick airport each year by burning 4.9m tonnes of wood annually, an analysis has shown.

Campaigners say this would make it “one of the dirtiest datacentres out there”.

Last year, the energy company Drax unveiled plans to convert part of its biomass power plant in North Yorkshire into a datacentre in response to the increase in demand for AI capability.

AI datacentres use a massive amount of electricity to power servers and cooling equipment, and to run facility systems. There are 315 datacentres in the queue to connect to the UK’s electricity grid, representing 73GW of demand. This is almost double the peak winter energy demand for the entire UK.

In public documents , Drax says it is developing options for a 1.2GW datacentre, with 100MW of this energy coming from the electricity grid and the rest from its own power station.

The Drax power station currently burns millions of tonnes of imported wood to create energy. This has been touted as a clean way of producing energy that could even result in “negative emissions”, as replacement forests take up CO 2 from the air when they are grown.

However, scientists and environmental experts strongly contest this accounting approach, with researchers finding such systems could take 150 years to be “carbon negative”.

Based on information in Drax’s 2025 annual report , the Natural Resources Defense Council (NRDC) estimated that if built, Drax’s datacentre will require 4.9m tonnes of wood each year.

The campaign group calculated the total emissions from this to be 8.6m tonnes of CO 2 a year. This is based on an assumption that the datacentre would run at a constant rate for all hours of the year.

This is nearly double the total CO 2 emissions of all flights from Gatwick airport , which measured 4.79m tonnes in 2024.

Matt Williams, a senior advocate for NRDC, said this would make it “one of the dirtiest datacentres out there”.

Oliver Hayes, head of policy at the campaign group Global Action Plan, said, “Burning millions of trees to power server farms is indefensible.”

The datacentre has not yet entered the formal planning process but Drax said it could be fully operational from 2031 if secured the necessary consents.

“The government must urgently set minimum standards for datacentres and rule out dirty energy generation like burning trees,” Williams said.

skip past newsletter promotion

There is growing pressure for more transparency over how these energy intensive datacentres will align with the government’s legally binding net zero targets.

In response to previous reports that just two planned datacentres in Buckinghamshire and Bedfordshire will have higher UK carbon emissions than ExxonMobil, Toby Perkins, a Labour MP and chair of the Commons environmental audit committee, said: “Without the right measures in place, datacentres could represent a serious threat to the environment and to our legally binding climate goals.”

Hayes said the absence of coordinated government policy was an “abolition of duty”.

“The only sensible solution at this point is a moratorium on datacentres until very basic questions about how many we need, what for, where, and how pollution will be avoided are answered satisfactorily by the government,” he added.

The government has previously ruled out issuing a moratorium on datacentres, stating that doing so would send investment and good jobs abroad. In Scotland, MPs voted in September to pause all new datacentre planning applications for up to a year until a national strategy can be developed.

A Drax spokesperson said: “The potential plans for datacentres at Drax Power Station can support UK economic growth as well as helping meet the government’s ambitions for industrial renewal, in places like North Yorkshire , as well as AI sovereignty.

“NRDC are a longstanding anti-biomass campaign group and have no positive vision for our Selby site. Their inaccurate assertions are not independently verified and are rooted in basic errors regarding the international standards for GHG accounting of biomass and bioenergy and on the potential datacentres’ power mix.”

ASOS app users receive push notifications apparently sent by hackers

Hacker News
www.bbc.co.uk
2026-10-06 06:15:40
Comments...
Original Article

On social media, dozens of people have posted about receiving the message - confused as to what it means.

Although the apparent extortion message has been issued directly to customers, it is addressed to Asos' data protection officer (DPO) and IT team.

The message claims the unnamed hackers have "fully compromised the Snowflake instance".

This refers to the data storage company Snowflake, whose tools are used by dozens of firms for collecting, analysing and storing data.

It is not known if ASOS is a customer of Snowflake or what data, if any, is stored with the service.

But Snowflake has been the subject of many high profile data breaches in recent years and has been linked to incidents targeting services including Ticketmaster and Santander .

It is, however, very unusual for a data breach to be revealed quite so publicly - and for customers to be informed in this manner.

Most extortions and negotiations by cyber criminals are conducted in private, with hackers hoping their discretion will result in a quiet pay-off.

The pop up message contains a link to the hackers' Telegram channel.

The new group is calling itself Xuanye Group and only created it's Telegram channel today.

They have posted only three times with the latest being about the ASOS hack.

Dan Bird, from cyber security firm Horizon3 says the pop up message the criminals sent implies that their access has gone beyond the Snowflake database.

"Sending a push notification to ASOS's app users would require access to the company's notification system, which is separate from the Snowflake data platform the attackers claim to have compromised."

"If both claims hold up, it suggests the attackers got hold of credentials that opened more than one door," he said.

Ofcom investigates Meta over Instagram Instants safety checks

Guardian
www.theguardian.com
2026-10-06 06:15:31
Watchdog examining whether company carried out adequate risk assessment as required by Online Safety Act Mark Zuckerberg’s Meta is under investigation for a potential breach of the UK’s digital safety laws after launching a Snapchat-style feature on Instagram. The UK’s communications watchdog, Ofcom...
Original Article

Mark Zuckerberg’s Meta is under investigation for a potential breach of the UK’s digital safety laws after launching a Snapchat-style feature on Instagram .

The UK’s communications watchdog, Ofcom , is investigating whether the $1.9tn (£1.4tn) company infringed the Online Safety Act (OSA) by failing to carry out adequate checks on whether its Instagram Instants product could show illegal content or be accessed by children.

Social media companies are required to carry out risk assessments before making significant changes to their service’s design or operation. Breaching the OSA carries a range of potential actions, from a formal warning to a fine of up to 10% of global revenue.

George Lusty, the director of enforcement at Ofcom, said risk assessments were a vital step in keeping tech platforms safe for users. “You wouldn’t launch a new car without testing it properly, and it’s the same for online services,” he said. “Safety by design must be built in, not bolted on, and today’s investigation shows we’ll take action where we have concerns that providers are failing to comply.”

Instagram’s Instants feature allows users to send images that disappear after viewing. The company said it had conducted a risk analysis of Instants and talked to Ofcom on a number of occasions before launching it. It said the feature carried protections such as blocking forwarding and included all the safeguards under its teen account settings , the default mode for under-18s on Instagram, Facebook and Messenger.

“We will of course cooperate with Ofcom during this investigation,” a Meta spokesperson said.

The Molly Rose Foundation, a charity established by the family of Molly Russell, a teenager who took her own life after viewing harmful content on Instagram, said it was crucial that Meta was “held to account for their failures under the Online Safety Act”.

Ofcom is already investigating whether Meta breached the law by failing to comply with a request for information related to its WhatsApp platform.

Zuckerberg’s company is challenging Ofcom in court over the OSA. It is suing the regulator in the high court over how it charges tech firms fees to cover its operating costs, and in a further case it is challenging Ofcom placing it in a new category under the act that subjects it to additional duties .

Meta has also joined TikTok and Elon Musk’s X in a court challenge over the amount information Ofcom is demanding they provide under the UK’s new online safety regime.

Nobel Prize in Physics goes to Francis Halzen

Hacker News
www.nobelprize.org
2026-10-06 05:48:46
Comments...
Original Article
Francis Halzen

Ill. Niklas Elmehed © Nobel Prize Outreach

Prize share: 1/1

The Nobel Prize in Physics 2026 was awarded to Francis Halzen "for decisive contributions to the IceCube Neutrino Observatory and the discovery of high-energy neutrinos of astrophysical origin."

Don't miss the Nobel Prize announcements on 5–12 October. All announcements are streamed live here on nobelprize.org.

Watch the 2026 Nobel Prize announcements live

Nikkei discloses breaches of employees’ Microsoft, Google email accounts

Bleeping Computer
www.bleepingcomputer.com
2026-10-06 05:25:50
Over the weekend, Japanese publishing giant Nikkei disclosed that unknown attackers recently breached two employee email accounts and used one to send thousands of phishing emails. [...]...
Original Article

Nikkei

Over the weekend, Japanese publishing giant Nikkei disclosed that unknown attackers recently breached two employee email accounts and used one to send thousands of phishing emails.

In a Sunday statement , the company said an employee's Google Workspace account was accessed in late July, exposing the personal information of employees and business partners.

Nikkei changed the account's password after discovering the breach in early August, following a notification from Google.

While this incident may have exposed the names and email addresses of 1,646 individuals, Nikkei says the affected data doesn't include information about readers or interviewees.

More recently, threat actors accessed another employee's Microsoft 365 account in September and used it to send 9,000 phishing emails targeting Nikkei staff and interviewees.

"On September 30th , emails containing links to malicious websites were sent to internal staff and to interviewees with whom several employees had been in contact," the media giant said. "Our company has changed its passwords, and no unauthorized logins have been confirmed since then. We have contacted the recipients individually and requested that they delete the emails."

The company also warned affected individuals to watch for suspicious emails that may impersonate Nikkei or its subsidiaries in new phishing attacks.

Nikkei has yet to attribute the attacks to a specific threat actor or hacking group and hasn't shared whether the two incidents are connected.

These are the latest in a string of security incidents Nikkei has disclosed in recent years. Last year, the company also revealed that its Slack messaging platform had been breached , affecting more than 17,000 employees and business partners.

In May 2022, Nikkei's Singapore subsidiary was hit by a ransomware attack that affected a server "likely" containing customer data. Three years earlier, in late September 2019, Nikkei lost approximately $29 million in a business email compromise (BEC) attack that targeted a Nikkei America employee.

Nikkei owns the Financial Times and The Nikkei, the world's largest financial newspaper, and is one of the world's largest media corporations.

It controls more than 40 affiliated companies involved in publishing, broadcasting, events, database services, and the index business, has 37 foreign editorial bureaus and over 1,500 journalists worldwide, and has over 3.7 million digital paid subscriptions.

article image

Build your security blueprint for AI-powered attacks

Join Mikko Hyppönen and security leaders from the NFL, CHANEL, and Atlassian for a two-hour digital summit on what AI-speed attacks change, what defenders should stop doing, and how to validate, decide, fix, and re-validate at machine speed.

Save your seat

Montray - a tray icon for systemd service health

Lobsters
github.com
2026-10-06 05:07:39
Comments...
Original Article

Mr. Montray is watching your services

Montray is a lightweight tray-icon monitoring utility for systemd services and anything else you can check from the command line, on both local and remote machines.

Montray demo

Project history

Problem: Linux doesn't tell me loudly enough when a systemd service breaks. Back in 2021 my Syncthing service had been broken for weeks, and I only figured that out later, after noticing that my files got badly out of sync, and it wasn't fun to reconcile. Systemd knew it was broken, yet it didn't tell me. That's not good enough.

I also had a Certbot service silently stop working and fail to refresh certificates, and other similar cases.

And I didn't want some enterprisey monitoring for this simple task: I just wanted a simple icon, always present in the system tray: green means it's all good, blinking yellow/red means something's broken. That's it.

Soon after, I wanted to reuse the same icon not only for systemd services, but also for arbitrary command-line checks, e.g. to check that there's enough disk space, or that a RAID is healthy, or anything else really.

So, meet Montray: a lightweight system-tray app that watches your services, runs command-line checks, and lets you know when something goes wrong.

Overview

This project has two main parts:

  • montray-server , a background service written in Go: runs on a machine, checks its health, and serves the current incidents via simple read-only WebSocket API;
  • montray-ui , a desktop app written in Rust + Slint: connects to one or more montray-server s, receives data from them, shows a tray icon, sends desktop notifications, and provides a native GUI.

So montray-server is a server (which can run locally too), and montray-ui is a client which runs on e.g. a laptop. If we have a laptop and two servers, a typical setup looks like this: 

flowchart LR
    subgraph laptop["Laptop"]
        localServer["montray-server"] --> desktop["montray-ui"]
    end

    subgraph server1["Server 1"]
        server1Monitor["montray-server"]
    end

    subgraph server2["Server 2"]
        server2Monitor["montray-server"]
    end

    server1Monitor --> desktop
    server2Monitor --> desktop
Loading

Montray Server reports incidents, each of them has:

  • A key, like systemd.my-service or free-space.exec-result
  • A state: warning or error

These incidents are generated according to the Montray Server configuration. For details, see Configuring Montray Server .

Montray UI combines these incidents. For each incident, it also prefixes the keys with the ID for that particular server, such as my-server (which you specify in the Montray UI config), so the key becomes my-server.systemd.myservice .

For details, see Configuring Montray UI .

If an incident happens and we want to just acknowledge it but worry about it later, we can snooze it in the UI, so the icon stops being annoying but it'll get unsnoozed again later.

The tray icon shows the worst current non-snoozed state:

  • Gray: the initial state isn't known yet;
  • Green: everything is OK;
  • Magenta blinking: Montray UI itself has an internal connection or tunnel error;
  • Yellow blinking: at least one warning;
  • Red blinking: at least one error.

Installation Quick Start (Linux)

Monitoring local machine

The easiest way to install both montray-server and montray-ui to monitor local machine health is as follows:

First, download the latest prebuilt binaries from GitHub , like montray-server-x.y.z_linux_amd64.tar.gz and montray-ui-x.y.z_linux_amd64.tar.gz , and unpack them. You'll get two binaries: montray-server and montray-ui .

Then: 

# Set up the monitoring service and start it. This also installs montray-server
# under /usr/local/bin when not already there.
sudo ./montray-server setup

# Install the desktop application system-wide:
sudo install -m 755 montray-ui /usr/local/bin/montray-ui

# Let the desktop application create its default config, autostart entry,
# and application launcher:
montray-ui setup

# Start the desktop application (Montray Server is already running):
montray-ui

You should now see a tray icon, and if you click on it, you'll see the UI. When you reboot, it will start automatically.

Monitoring remote machines

If you have e.g. a personal server which you also want to monitor using the same interface on your desktop, then on each such remote machine follow the same steps as above, but only for montray-server (no need to install montray-ui on the servers).

Having montray-server running on your server, we need to point our local Montray UI ( montray-ui ) to it. By default, Montray Server only listens on 127.0.0.1, so we can't reach it directly from a laptop.

Presumably you have ssh access to your server with public key authentication (i.e. you can ssh there without a password), so the easiest way forward here is to establish an ssh tunnel, and Montray UI has convenient support for it: open the config file ~/.config/montray-ui/montray-ui.yml , and add one more entry to the wsClient.servers array, like that (adjusting at least your server hostname and username). There is no addr in this entry: for a structured SSH tunnel, Montray UI automatically allocates an available port on 127.0.0.1 . You may still set an explicit loopback addr with a port of your choice when a fixed local forwarding port is useful. 

    - id: myserver # Arbitrary but unique ID for this server.
      tunnel:
        ssh:
          host: myserver.com  # TODO: your actual server hostname
          user: myuser        # TODO: your actual ssh user
          port: 22            # Change if using non-default ssh port
          remoteServerAddr: 127.0.0.1:41990 # Montray Server listening port

And restart Montray UI ( montray-ui ) by right-clicking the tray icon and selecting "Restart and reload configuration." Open the UI and verify that the list of servers now includes your newly added remote server as well.

SSH tunnel is not the only way to access remote servers; Montray UI also supports TLS and bearer token authentication. For details, see docs on Security .

Configuration

The default config (which sudo montray-server setup writes to /etc/montray-server.yml ) is as follows: if any systemd service is failing, it's a warning (the tray icon will be blinking yellow). If there's less than 100 MiB of free space in the root partition, it's an error (the tray icon will be blinking red). Otherwise, it's all good (the tray icon is green).

The config includes comments and examples, so take a look and experiment with it; and also check the Configuring Montray Server docs. For instance, I like to explicitly list services I particularly care about, such as Syncthing, and configure any state other than active as an error rather than a warning. That includes services stopped manually - if it was manually stopped for some reason, I want to be annoyed by the blinking icon until the service is running again. And I have some more custom exec checks as well.

Don't forget to restart the Montray Server systemd service to apply the changes: 

sudo systemctl restart montray-server.service

Non-Linux OS support

So far Montray was only tested on Linux. Nevertheless, the client ( montray-ui ) should work on Windows and MacOS as well, so you can run it there and monitor your remote Linux servers, but not so much the local machine.

Even montray-server can technically run on non-Linux, but obviously systemd is irrelevant there, and then the only useful check there is exec : just polling some script periodically, so we lose the out-of-the-box system-wide system service monitoring, and thus the usefulness is limited. Would be cool to implement systemd-like checks for Windows and MacOS, but I don't use these so hard to test. PRs are welcome.

Development

Building

You need Go 1.26 and Rust 1.92 or newer.

On Ubuntu, install the native dependencies used to build and run montray-ui : 

sudo apt-get install -y gcc libfontconfig-dev libxkbcommon-x11-0

gcc compiles native Rust dependencies, libfontconfig-dev provides the Fontconfig development files required by Slint's font stack, and libxkbcommon-x11-0 is loaded by Slint/winit when the application runs under X11.

Having that, to build both montray-server and montray-ui :

To build only one of them: 

make montray-server
make montray-ui

To build a debug binary for Montray UI (for a faster build, comparing to release):

To install built binaries under /usr/local/bin : 

sudo make install-montray-server
sudo make install-montray-ui

The legacy Go client (serving local web ui) is not part of the default build. Building it with make montray-ui-legacy additionally requires libgtk-3-dev and libayatana-appindicator3-dev on Ubuntu.

Running tests

The test suite uses the same Go and Rust build requirements described above. It also requires a current Node.js LTS release for the legacy client's JavaScript tests. Because go test ./... compiles the legacy GTK client, its native dependencies are required on Ubuntu too: 

sudo apt-get install -y libgtk-3-dev libayatana-appindicator3-dev

Then:

To run only the montray-ui Rust tests: 

cargo test --manifest-path cmd/montray-ui/Cargo.toml

Two native window-geometry tests are ignored by the regular suite because they require a real X11 session and window manager. A headless environment cannot accurately test window positioning, maximizing, hiding, and restoring. Run them explicitly, as separate commands: 

cargo test --manifest-path cmd/montray-ui/Cargo.toml native_startup_restores_geometry_and_maximized_state -- --ignored
cargo test --manifest-path cmd/montray-ui/Cargo.toml native_hide_show_preserves_normal_geometry_while_maximized -- --ignored

They must run separately because Slint's GUI platform can be initialized only once per test process. Running both together would make the second test fail for a platform-initialization reason rather than a geometry problem.

Screenshots

Montray OK Montray Warn1

Documentation

‘We’re not going to go away’: the mom fighting for social media reform after her son’s suicide

Guardian
www.theguardian.com
2026-10-06 05:00:07
A new documentary follows Kristin Bride as she takes on Congress after the death of her 16-year-old son who was cyberbullied At this point in time, going anywhere without your smartphone feels almost like leaving behind a limb. Whether it’s for directions, distraction or the discourse on social medi...
Original Article

A t this point in time, going anywhere without your smartphone feels almost like leaving behind a limb. Whether it’s for directions, distraction or the discourse on social media, our phones have become both a necessity and a crutch in our everyday lives, with our bodies literally evolving around the glow of the screen.

But no matter how much we think we need all the bells and whistles of the attention economy, the attention economy actually needs us more than we need it.

This is the overarching message of the documentary Your Attention Please, available to rent or own on 6 October. The film explores the human cost of our technology addiction – as well as the very human efforts to combat it – driving home how social media companies like Meta and TikTok benefit financially in rewiring our brains in order to keep us scrolling.

“We’re dealing with a business model that encourages extreme behaviors because those have been proven to be more attention-grabbing,” director Sara Robin says in an interview.

The documentary follows Kristin Bride in her fight for social media reform , traveling to Capitol Hill to push for legislation and testifying before Congress. Bride was thrust into online safety advocacy after her 16-year-old son, Carson, was cyberbullied and took his own life in 2020 – even after she thought she took all the right steps to safeguard him from the dangers of the internet.

“We waited until Carson was through middle school to give him his first very old iPhone with no apps on it, and we had really hoped that it would stay that way,” she says. “But when he got to 10th grade, he begged for Snapchat because that was the way that all the kids were connecting … he didn’t make it to 11th grade.”

But social media is not just dangerous for kids, which the documentary makes a point of highlighting through interviews with experts like the former US surgeon general Vivek Murthy , who has called for tobacco-style warning labels on social media platforms, as well as Frances Haugen , the Facebook whistleblower who will be portrayed by Mikey Madison in The Social Reckoning out this week.

Robin likes to quote Cal Newport, a computer science professor at Georgetown University, in describing social media as “designed to feel as if you are slipping into a warm bath”. “You go on there and then you forget why you got on there and you forget what you’re doing while you’re on there and then two hours have passed,” she says. The bath in this analogy, however, is actually more a pot – and we are the frogs being slowly boiled alive without realizing it.

“In a time where social media is still designed to actively disrupt our lives, it’s really up to us now to say: ‘My goal today is to get more out of social media than it will get out of me,’” Robin says. “That’s quite hard to do. But I think in the interim, when we’re still dealing with these, highly addictive, very imperfect platforms that are designed to exploit our attention, that’s to an extent what we have to do while we’re transitioning to something healthier.”

Protesters with placards
A still from Your Attention Please. Photograph: Yahna Harris/Attention Rebellion

While both Bride and Robin hope the documentary will make people confront their phone addictions and social media usage, they also see it as a call to action – whether in Bride’s continued fight to get Congress to pass the Kids Online Safety and Privacy Act or in being part of a cultural shift to get off your phone. The documentary features interviews with the founders of the Offline Club in Amsterdam, where attendees must check their phones at the door before events, and with the principal and a teacher of Brockton high school in Massachusetts, where students are required to keep their devices locked in secure pouches for the entire school day, as examples of how this is possible.

“We’re not anti-tech – we’re pro-better tech and we’re pro a more balanced relationship with tech,” Robin says. She acknowledges the irony that with the promotion of the film, she is more on social media than she’s ever been before. “A lot of the conversation critical of the attention economy is happening on these attention economy platforms,” she says.

But she is not denying that social media has been a society-changing tool in how it has allowed people to connect and organize and self-promote – it’s just that the companies behind these platforms have designed their algorithms to keep its users on there as long as possible, beyond their original purpose. “If monetary value trumps everything else, then we naturally land in a situation where the human being is forgotten,” Robin says.

The way to combat this on an individual level is to be intentional with your usage. “Be very clear what you’re looking to get out of the platform,” Robin says. “Go in and do that. Know the moment you open it, it’ll throw the most distracting thing at you. Pull yourself back out. Do the thing that you intended to do, and then get out.”

On a larger scale, the fight will be even more challenging. Bride says two years later that she is back to where she was in the documentary with the Kids Online Safety and Privacy Act – the bill stalled after passing in the Senate in 2024. Seeing Donald Trump buddying up with the tech billionaires behind these platforms has been particularly disheartening, Bride says.

“But how can we not keep fighting?” Bride says. “We’re not going to go away, and we’re going to keep being loud and vocal.”

  • Your Attention Please is streaming on Documentary+ and is also available on VOD now

Engineer sentenced for locking over 3,000 devices on employer network

Bleeping Computer
www.bleepingcomputer.com
2026-10-06 04:19:24
A former core infrastructure engineer at an industrial company headquartered in New Jersey was sentenced to 32 months in prison for locking thousands of devices on his employer's network in a ransomware-style attack. [...]...
Original Article

Hacker

A former core infrastructure engineer at an industrial company headquartered in New Jersey was sentenced to 32 months in prison for locking thousands of devices on his employer's network in a ransomware-style attack.

57-year-old Daniel Rhyne from Kansas City, Missouri, pleaded guilty to his role in a failed extortion plot targeting the New Jersey company that employed him after being arrested in August 2024 and released after his initial appearance in federal court.

According to court documents , he remotely accessed the company's network without authorization using an administrator account between November 8 and November 25 and scheduled tasks on the domain controller that changed the password of the administrator account to "TheFr0zenCrew!", deleted 13 domain admin accounts, and changed the passwords for 301 domain user accounts to "TheFr0zenCrew!".

Rhyne also added scheduled tasks that changed the password for two local admin accounts to "PsPasswd" (blocking access to 254 servers), changed the password for two more admin accounts (blocking access to an additional 3,284 workstations), and shut down random servers and workstations on the company's network over several days in December 2023.

On November 25, Rhyne sent coworkers a ransom email titled "Your Network Has Been Penetrated," in which he said that server backups had also been deleted to make data recovery impossible and threatened to shut down 40 random servers daily over the next ten days unless the company paid a 20 bitcoin ransom (roughly $750,000 at the time).

"On or about November 25, 2023, at approximately 4:00 p.m. EST, network administrators employed at Victim-1 began receiving password reset notifications for a Victim-1 domain administrator account, as well as hundreds of Victim-1 user accounts," the criminal complaint reads.

"Shortly thereafter, the Victim-1 network administrators discovered that all other Victim-1 domain administrator accounts were deleted, thereby denying domain administrator access to Victim-1's computer networks."

Investigators found that on November 22, while planning his extortion plot, Rhyne used his account on a hidden virtual machine to search the web for information on changing domain user passwords, deleting domain accounts, and clearing Windows logs.

One week earlier, he also searched on his laptop for "command line to change local administrator password," "command line to remotely change local administrator password," and "how to remotely shutdown a computer usign cmd."

Earlier this year, in March, 27-year-old North Carolina data analyst contractor Cameron Curry was also sentenced to two years in prison after being found guilty of extorting his employer , Brightly Software (a Software-as-a-Service company previously known as SchoolDude) for $2.5 million.

article image

Build your security blueprint for AI-powered attacks

Join Mikko Hyppönen and security leaders from the NFL, CHANEL, and Atlassian for a two-hour digital summit on what AI-speed attacks change, what defenders should stop doing, and how to validate, decide, fix, and re-validate at machine speed.

Save your seat

Email Self Hosters - what are you using?

Lobsters
lobste.rs
2026-10-06 04:09:53
At the moment I self host using maddy for a bunch of domains that need a mailbox or catch all. There are some minor issues with it - some are me, some are who knows. The biggest bug bear is that the native email client on iOS takes an age to connect and download messages which causes some issues wit...
Original Article

postfix, dovecot, opendkim, opendmarc, all on a VPS from Scaleway.

I don't have any delivery issues with any major provider, but I have to fill in a form once a year to have my IP taken off the Spamhaus PBL. There's no manual verification on their end when the email you use with Spamhaus lives on the IP you're unblocking :)

I've directed all email to my account, regardless of the local-part (before the @). Probably wouldn't recommend it, it makes it near impossible to ever stop self-hosting.

I'm also rawdogging it without any form of spam filter on the server. Thunderbird does it on my desktop, and on my phone I just scroll past the spam. I should probably set up spamassassin to at least reject the most obvious.

Edit: One issue I've run into is that, for some reason, Tumblr wouldn't even attempt to deliver email verification to my server. Had to contact support to have my email verified!

Russell Coker: A Power Case for a FOSS Phone

PlanetDebian
etbe.coker.com.au
2026-10-06 04:08:04
The Problem I want to use a FOSS phone running Debian as my daily driver and not use a non-free OS on my phone. GrapheneOS [1] is a FOSS rebuild of Android, it’s a great project and I commend them for what they are doing. But I want a system where I have access to all the bits and where I can ssh in...
Original Article

The Problem

I want to use a FOSS phone running Debian as my daily driver and not use a non-free OS on my phone. GrapheneOS [1] is a FOSS rebuild of Android, it’s a great project and I commend them for what they are doing. But I want a system where I have access to all the bits and where I can ssh in to it and manage it in the same way as all other Linux systems. Android is free software but it’s a closed design and the phones are expected to be appliances not full peers on the network. The Android native terminal emulator (AKA Linux development environment) is a nice feature, a VM with a default of 1GB of RAM but it’s still separate from the main OS.

KDE Connect [2] is a system for KDE talking to phones, it is a great project and allows convenient interaction between a Linux PC and a phone. But in addition to that I’d like the option to ssh to my phone to send an SMS or have one sent from a cron job. I blogged about using the basic functions of Kdeconnect from the command-line [3] .

I have had ongoing issues with the battery not lasting long enough on the PinePhonePro (PPP) and the Librem 5 (L5) which makes them unusable for my purposes. For a phone running Droidian I can have an open ssh session (with keep-alive packets) running overnight on Wifi without any problems while for the PPP or L5 an hour of reading an ebook (the least demanding use of a phone) will use most of the battery.

I sent my FuriPhone back for warranty repair and it’s been almost a month with no follow up, so I need to have another option.

The Aim

The aim of this post is to develop a rough design for a battery case for either a L5 or a PPP, I have both phones and they are both capable of doing what I want apart from battery life. Also designing a case that basically works for both phones and has two variations of the CAD file for 3D printing is good to allow collaboration with people who use either phone. Protecting the phone from damage when dropped is also a core feature.

I investigated commercial options. There are some reports of cases for other phones working, someone reported a PPP working well with a battery case for a Samsung S20 Ultra. The OtterBox uniVERSE Case for Samsung Galaxy XCover Pro is reported to fit the PPP so probably battery cases for the XCover Pro will also work. There are also universal power cases which have spring clips to fit a wide range of phone sizes, but they are much larger than other options and probably increase the risk of damage if the phone is dropped.

At this stage I’m planning out the rough specs of a case that can be 3D printed to protect a phone while being easy to change the cells. The process of changing cells should be quick, easy, and not risk breaking anything. The hardware required should be affordable ($100AU plus some 3D printing seems reasonable) and should not require significant skills to assemble. After recent experiments with Thinkpad repair I’ve determined that I am not good at electronics by today’s standards so I plan to avoid anything difficult in that regard.

Battery Options

There are phone cases with batteries built in for the more common phones. A quick search on AliExpress turned up options for recent Pixel phones. They aren’t as common as they used to be as Android phones and iPhones generally have good battery life nowadays. Of course there aren’t such options for less common phones like the L5 and PPP.

It is cheap and easy to buy a portable battery for charging phones, but that’s a separate bulky device and unless the connector is inside a protective case it leaves the phone vulnerable to catastrophic damage if dropped.

So can a suitable battery case be designed and 3D printed?

Cheap Batteries

According to Wikipedia the 18650 LiIon cell is the most commonly used LiIon battery, that is 18mm in diameter and 65mm in length. For a phone case thinner batteries would be more convenient but economies of scale make the 18650 cheap to buy new and commonly available on the second hand market. The current prices on AliExpress (in Australian dollars) are about $8 per cell when buying 10 at a time. Most of the cells on AliExpress don’t have wires attached so they can be swapped in a carrier the way AA batteries are used, but the cells are symmetrical (no bump for positive) so the user has to take care of polarity. Chargers are around $22 for a 4 cell charger or $52 for a 12 cell charger.

I’ve seen prices as low as $1 per cell on the second hand market, I haven’t investigated the amount of usable capacity in such cells. Probably the prices for chargers I’ve found aren’t the best available but they are good enough to start the design. 10 cells and a 4 cell charger is about $100. That’s about 130Wh while a 74Wh USB-C battery pack costs about $50. So the price per Wh is OK.

I have read about an attempt to do this for the L5 with two of the batteries designed for the L5 in a case attached to it, so you have 3*L5 batteries. It’s an interesting approach but those batteries have poor value for money when compared to 18650 ($29US for 4500mAh compared to $8 for 3500mAh) and they also don’t ship them outside the US at this time. The 18650 batteries are available everywhere cheaply and have multiple uses.

How They Fit

According to my measurements the PPP is 76mm wide and the L5 is 73mm wide. That’s wide enough for 3*18mm or 4*18mm cells side by side arranged parallel to the long side of the phone or one 65mm long cell going across the phone with the necessary spring and wires.

The L5 is about 151mm long and the PPP is about 5mm longer. So that means that for just the cells we could have 2 cells in a row in parallel to the long side of the phone giving a 2*3 or 2*4 array or we could have a row of 8 cells parallel to the short side of the phone.

A quick search for controllers for a battery of LiIon cells that outputs USB-PD turned up one of the smaller ones as 40*28mm, here is the AliExpress page [4] . So that means we could have a maximum of about 4 cells and the controller while leaving space for the camera. Looking at other similar items it seems that most of them take a large portion of the 73mm width available and take a bit less than 30mm height. The price range for DC-DC voltage converters ranges from about $3 to $20, the one I linked to is currently $8.

The mass of 18650 cells is around 45g so 4 of them would be about 180g before counting the weight of the case and electronics. A L5 weighs 262g and a PPP weighs 220g so 180g of battery will make a significant difference, not an impossible difference but holding a PPP while reading an ebook is already annoying. Maybe the case could be designed to be easier to hold, a loop of wool attached to the top could be used to suspend the phone from one finger while the rest of the hand just keeps it steady.

In a default configuration the L5 has a 4500mAh 3.8V battery and the PPP has a 3000mAh 3.8V battery. The 18650 cells have up to 4000mAh with 3500mAh being common. So 4 such cells could multiply the battery life of a PPP by a factor of 5 or a L5 by 4. That would not be enough to last for a whole day without charging so they need to be easy to swap.

A 3 cell battery seems like a good option, 135g of batteries so the project overall wouldn’t even double the weight of the phone with battery case. As it doesn’t seem possible to put in enough cells to last a day of reasonable use (running a web browser, checking email, reading wikipedia, and having some IM systems checking for notifications) there doesn’t seem to be a benefit in trying to stuff the maximum number of cells in. Going to 4 or 5 cells doesn’t provide much benefit over 3 while increasing the weight and making the design more difficult. The modules for DC Voltage conversion often have configuration for the number of cells to be used so it wouldn’t be difficult to print a new case and reconfigure the Voltage converter.

Design

A case needs to provide protection as well as hold the batteries. The lack of cases for the L5 and PPP is a problem even without the extra weight of a battery pack increasing the kinetic energy of falling phone. Ideally a drop from 1M on to concrete would not cause any damage to the phone, destruction of the case is OK as it would be 3D printed and can be rebuilt easily.

I think that a case where the phone slides in from the top would be best. To change cells one would slide the phone out and the cells would be immediately accessible with no divider between the phone and the cells. Sliding the phone in would hold the cells against the back of the case and as there is a gap of about 3mm on each side of the PPP between the display area and the edge of the phone there’s plenty of space for the case to wrap around it at the sides and bottom. At the top there would have to be some sort of plastic clip. The PPP only has buttons on the top right so the case can be snug on all sides apart from the top right. The L5 has switches at the top left and buttons at the top right so whatever clips on to the top would really need some strength to cover the fall flat on face case.

The USB C plug would need to be held in place well enough to allow the phone to easily slide on to it but also be weak enough that it would move if a drop forced the phone out of the case.

Thingiverse has a design for a L5 case [5] (with several derivatives) and a design for a PPP case [6] . Those cases could be used as starting points and then changed to store batteries and the voltage change circuit board. I have just learned that it’s possible to use rubberised material in a 3D printer which is apparently what those designs are for. If a case was printed with rubberised material the top could just stick in place after being pushed in.

Potential for Excessive Excitement

Having an exploding device in your pocket would be the exciting in a bad way. My initial idea was to have multiple 18650 cells in series. The problem is that if the cells have different capacity levels then one can run out before the rest and get to a deep discharge state which at a minimum causes damage to the cell.

One way of minimising risk is to have only a single cell in use at one time, here is a converter for a single cell to 5V [7] . It is possible to run multiple cells in parallel by simply wiring them together, but it’s recommended to make sure that the voltage difference between cells is less than 0.2V to avoid high current when connecting.

The arrangement of cells is often referred to as nS or nP to refer to n cells in series or parallel. Aliexpress has a range of 1S to 6S devices. Some of the 2S devices have a connector for the mid point which permits separate voltage monitoring for both cells but there doesn’t seem to be anything equivalent for 3S or more. They also don’t appear to have anything to specifically manage 2P or 3P arrangements at the moment (they had a 3P board on sale when I looked into this last year).

Conclusion

This is a bigger project than I expected when I first started investigating it. I’ve played with FreeCAD which seems OK but will take a lot of learning and practice. Then I need to do more investigation into the DC-DC converters to find one that works well and is unlikely to start a fire.

I will look into other phone options as well and keep chasing Furilabs people about my phone replacement.

Two arm64-specific miscompiles induce vulnerabilities in curl

Lobsters
mastodon.social
2026-10-06 03:52:24
Comments...
Original Article

OpenAI admits misstep in handling AI agent interactions with Australian government sites – video

Guardian
www.theguardian.com
2026-10-06 03:32:25
In a parliamentary inquiry, OpenAI chief strategy officer Jason Kwon is questioned by independent senator David Pocock about the company's notification process after OpenAI agents accessed Australian government website data in June. Asked why OpenAI relied on an arbitrary department email rather tha...
Original Article

In a parliamentary inquiry, OpenAI chief strategy officer Jason Kwon is questioned by independent senator David Pocock about the company's notification process after OpenAI agents accessed Australian government website data in June. Asked why OpenAI relied on an arbitrary department email rather than direct government contacts, Kwon acknowledges that 'in retrospect, we should have done what you're suggesting'. He explains that staff had treated the incident as a 'technical situation' and sought to contact technical counterparts, which he says was 'not good enough'

Anthropic Subscriptions Offer 5x+ More Value Than OpenAI

Hacker News
newsletter.semianalysis.com
2026-10-06 03:29:50
Comments...
Original Article

Subscription plans are still the primary way consumers and small businesses pay for AI. These plans are highly subsidized—as we previously explained in June —but can still make economic sense as powerful customer acquisition and marketing tools. For example, the goodwill engendered by OpenAI’s generous resets is partially responsible for the recent surge in Codex adoption and has forced Anthropic to repeatedly walk back planned subscription nerfs to avoid getting clobbered in the court of public opinion.

Furthermore, because subscription plans are so heavily subsidized, they can also have a large impact on margins and revenue per MW despite only being a small portion of total revenue. Consider the following rough numbers for Anthropic.

Despite being just 10% of overall revenue, subscriptions can take up over 40% of inference compute and lower blended revenue per MW by ~$36M. Subscriptions are even more important for OpenAI, as they make up a larger portion of their total revenue. For specific numbers, see our Tokenomics Model .

In other words, if you want to accurately model AI lab financials, you need to understand their subscription limits.

So how do limits actually work? The way to think about subscriptions is that your monthly payment grants you some numbers of “credits”. Each (model, token type) combo consumes a different amount of credits. Because credit cost ratios can differ dramatically from API price ratios, the “value” of the same plan changes depending on what model and workload you’re running. Put differently, it doesn’t make sense to say a plan is worth $X in isolation—you need to consider the full (plan, model, workload) tuple .

Here’s how the API-equivalent value of the same $200/month Claude plan changes depending on the model and workload.

A single static snapshot isn’t enough either. Labs publicly change their limits all the time with promos and new model releases. They can also silently change limits whenever they want by tweaking credit costs. Ideally, you’d re-check the cost of each (plan, model, token type) daily so you can surface any changes in real-time.

This is exactly what SemiAnalysis has done with our new Subscriptions Dashboard available exclusively to Tokenomics Model subscribers. Besides every OpenAI and Anthropic subscription, we also track Meta, SpaceXAI, Cursor, Cognition, Z.ai, MiniMax, and Moonshot.

A screenshot of our dashboard showing a small subset of the available data. Source: SemiAnalysis Tokenomics Model

New providers, plans, and models will be added as soon as they’re released. The rest of the article will give an overview of everyone’s current limits. All token and dollar amounts below assume agentic usage unless otherwise specified.

Tokens are generally priced per MTok (million tokens) across the following types:

  • Input: Fresh tokens added to the LLM’s context window that aren’t cached

  • Cache write: Input tokens that are cached for multi-turn conversations. Generally slightly more expensive than regular input tokens.

  • Cache read: Tokens from previous turns that are already cached in the conversation. Generally extremely cheap compared to uncached tokens.

  • Output: Tokens generated by the model. The most expensive token type.

Additionally, some models charge more for tokens above a certain context window. Those models typically compact before that expensive context window is reached, so we also keep our measurements below it.

Source: OpenAI

Subscription plans don’t expose this fine grained pricing. Instead, they provide a simple 0 to 100% usage meter across 5-hour and 7-day windows, and sometimes a separate meter for a model like Fable. Subscription tiers are then differentiated by usage multipliers relative only to the provider’s other plans. OpenAI for example used to advertise “Expanded Codex usage” in their $20 Plus plan, “5x more usage than Plus” in the $100 Pro plan, and “20x more usage” in the $200 Pro plan until they cut their $200 plan usage in half and removed all relative usage from their pricing page.

Screenshot of Claude Code meters. Source: Claude Code CLI

We compute the subscription-rate of each (plan, model, token type) triple by running experiments that isolate one token type at a time and watching how far the model provider’s meter moves. An experiment is some number of repeated calls using a specific prompt that maximizes one token type while minimizing all the others.

Input, cache writes, and cache reads share the same prompt template. We use a portion of War and Peace since some models will refuse to respond if given large blocks of gibberish. For the input token experiments, we use a random tag on every call to ensure nothing gets cached. The cache write experiments run the same way but with the prompt marked for caching, so each new tag forces a new cache entry. The cache read experiments use a fixed tag, so the first call writes the cache and every repeat reads it.

For the output token experiments, we used a technical essay to force long outputs because models will refuse mechanical prompts like “repeat SemiAnalysis 100,000 times”.

For every call we record two things: how many tokens of each type the provider billed, and what the usage meter read. Turning those into a price takes three steps.

Providers generally report usage on a meter that moves in fixed amounts. This could be a whole percentage point, a credit, a cent etc... A single request often does not move the meter at all, so the cost of one request cannot be measured directly.

Instead, we measure in steps. As requests run, we keep a running total of the tokens used. Each time the meter rises, we store that total. The tokens used between two meter moves make up one step, which is the cost of moving the meter by one unit.

We drop two partial steps. When a run starts, the meter is already part of the way to its next move, so the tokens before the first move are less than a full step and would make the rate look higher than it is. The tokens after the last move never finish a step, so we drop those as well.

To get a rate, we add up how far the meter moved over the complete steps and divide by the tokens used in them. Providers charge differently for input, output, and cached tokens, so we calculate a separate rate for each.

Caveats:

  • Because we see the meter only once per request, a single step can be off by up to one request. These errors don’t add up with more consecutive steps, so we track a range instead of a single number, and the range gets narrower the more steps we count. We keep adding steps until the range is within ±5%, then report the rate across all of them.

  • No request contains only one type of token. Some plans, for example, require a set of instructions on every request. We subtract that extra amount using the prices we measured for the other types.

  • Reading from the cache costs almost nothing on some plans and may not move the meter at all. If the meter does not move after 500M cache read tokens, we assume that cache reads are free.

We read every meter a plan shows after every request. This could be a 5-hour limit, a weekly limit, and for models like Fable, its own weekly limit. So each token type gets a price on each meter. This is how much of that limit a million tokens uses. From each price we work out how many tokens fit e.g. if a million tokens use 5% of a limit, the whole limit holds 20 million.

The 5-hour limit resets many times within a week, so a month is capped by the weekly limits.

Lastly, we compute the API-equivalent value by converting the limit % consumed by a million tokens of each type into a dollar amount. For example, if you consume 1% of a $200 plan’s monthly limit, that’s $2.

Then, we assume a workload shape, and compute how many total tokens you can get from each plan given this ratio. For the agentic workload, we use our own usage ratios in September as reported in our Tokenomics Model .

Finally, we multiply by the blended price per MTok at API prices to get the API-equivalent value.

While refining our methodology, we ran into a very confusing situation where just 1 of 3 of the same subscription we tested for a particular provider had ~20% lower limits than the other 2. This unlucky account happened to be significantly older as well, and we were worried that the provider had some cursed setup where they changed subscription limits depending on the age of the account.

Fortunately, after reaching out to the provider, we were able to confirm this was not the case and that our unlucky account was part of an “extremely tiny” A/B test they were running on limits. Additionally, they wanted to emphasize that they “didn’t just decrease limits wholesale” and were instead testing “how to better balance when people hit limits”.

This conclusion is interesting for two reasons:

  1. It proves providers can silently change subscription limits at any time

  2. Our method is sensitive enough to detect these subtle changes!

The biggest question in AI subscription land is how Anthropic’s limits compare to OpenAI’s. Note that OpenAI just made some massive changes to their subscriptions last week, halving the API-equivalent value for the $200 plan and introducing a new $500 tier. The following numbers describe the current state of OAI vs ANT. Afterwards, we will examine OpenAI’s recent changes in depth.

Limits are quite similar across the board for GPT-6 Astra vs Fable 5.1, though remember that Fable can only be used for 50% of your limit. In other words, your $200 ANT plan would still have 50% left after consuming $2,485 worth of Fable 5.1, whereas the equivalent OAI plan would be fully exhausted after $2,897 of Astra.

This extra 50% starts becoming very relevant when we compare Opus 5.5 vs GPT 6.1 Sol.

At this mid-tier of models—which both companies market as the intended daily driver for most users—Anthropic is an overwhelmingly better deal, offering ~5x the API-equivalent value across the board. You could argue that this is unfair for OAI because 6.1 Sol is much cheaper per token than Opus 5.5, but the gap is still massive even if you switch to comparing the number of tokens.

Overall, we believe API-equivalent value is the single best number for capturing the “value” of a subscription plan, but there are times where raw token volumes can be more appropriate. For example, if a model is a particularly bad (or good) deal at API prices, then the API-equivalent value will of course be misleading.

Token efficiency is also an extremely relevant factor, but the industry unfortunately lacks reliable data here. Many people like to cite this chart from Artificial Analysis, but we do not believe the benchmark tasks in the AA Intelligence Index are at all representative of real work people do with LLMs.

Our tracking confirmed the 50% value reduction for OpenAI’s $200 plan announced by Tibo . $200 plans purchased before the cut will keep the old higher limits until October 29th, but newly purchased plans immediately start with the lower limits.

As you can see, OpenAI specifically halved the number of tokens you get for each model tier. This caused the API-equivalent value for Sol class models to drop by over 50% because OAI also cut cached input token pricing for 6.1 Sol.

The new $500 plan only offers 21% more Astra than the old $200 plan. Interestingly, the API-equivalent value for Sol class models actually decreased due to the aforementioned 6.1 Sol price cut.

Of course, 300 TPS Ultrafast was the real headline feature for the $500 plan and we are currently testing its limits. Results will be published to Tokenomics Model subscribers as soon as they’re available. We are also actively testing the limits when using your ChatGPT subscription on third party apps like Devin.

The $200/month plan also used to be significantly more subsidized than OpenAI’s other subscriptions. If we normalize each (plan, model) combo to the API-equivalent value and tokens per dollar, we can see that Pro 100 offered ~2x more per-dollar value than Plus on Astra and Pro 200 was another ~2x on top of Pro 100 across all models.

With last week’s cut, Pro 100, 200, and 500 all offer the same tokens per dollar for every model. Plus is also comparable for Sol but offers a relatively worse deal on Astra.

Anthropic, on the other hand, already offered the same per-dollar value for all of their subscription tiers and currently crushes OpenAI.

OpenAI used to be lauded by indie developers for their generous subscription limits compared to Anthropic, but this is simply no longer true today. Opus 5.5 on any Claude subscription offers far better value than anything from OpenAI.

The only counter argument for OpenAI is that none of their Pro plans have a 5 hour limit, which makes it easier to consume a higher % of your total monthly limit. However, we don’t think this offsets the ~4x higher API-equivalent value offered by Opus 5.5 on Claude plans.

We’ve seen a number of significant price cuts recently to Anthropic and OpenAI models.

  • Fable 5.1 cut cache reads by 75% vs Fable 5

  • Opus 5.5 cut input and output token pricing by 20% vs Opus 5 and cache reads by 60%.

  • GPT 6.1 Sol cut cache reads by 50% vs GPT 6 sol. This was after making GPT 6 Sol 60-67% cheaper vs 5.6 Sol.

One important question is whether the labs adjust token limits after these price cuts, such that the API-equivalent value of your subscription stays constant.

We already showed earlier that OpenAI didn’t change Sol token limits on the $200/month plan when they released 6.1 Sol. This caused the API-equivalent value to drop ~30%.

Anthropic also didn’t increase Fable token limits at all with the release of 5.1, but they did up Opus token limits with the 5.5 price cut.

However, the ~20% token increase for Max plans and ~50% increase for the Pro plan weren’t enough to fully offset the API price cut.

As we explained at the start of this article, subscription gross margins are way lower than API and meaningfully reduce revenue per MW for both OpenAI and Anthropic. Both companies are well aware of this fact but have chosen two different strategies for reducing subscriber subsidies.

For a given plan, Anthropic decreases the API-equivalent value for more premium models. The drop off is small between Sonnet 5.5 and Opus 5.5, but becomes very meaningful with Fable 5.1.

Reducing the API-equivalent value as you release more powerful models at higher list prices is the more subtle way to stop subsidizing subscriptions. Assuming 100% utilization and 92% API gross margins, maxing out Opus 5.5 vs Fable 5.1 usage corresponds to -369% and 1% gross margins respectively. If we were to use a more realistic average utilization of 20%, then gross margins rise to 6% and 80% respectively.

In other words, Anthropic would likely already have software like margins on their subscription plans if everyone only used Fable . Opus class models will continue to become cheaper to serve over time as Anthropic figures out how to make the model smaller, and new model tiers above Fable will have even lower subscription limits (assuming they’re included at all).

OpenAI, on the other hand, picked the nuclear option of just immediately cutting to Fable-level limits across the board.

The downside of this approach is supposed to be public backlash, but OpenAI has somehow managed to more or less avoid this entirely. It’s likely some combination of all the good DevDay PR drowning out the negative news from the day prior and grandfathering existing plans for an additional month.

Despite being compute poor, the Chinese labs still offer subsidized subscription plans, but the level of subsidization varies dramatically between models.

Here’s how they compare to OpenAI and Anthropic after normalizing per dollar.

Value per dollar increases as you buy higher tier subscriptions from the Chinese labs. On average, the API-equivalent value per dollar is a little less than the ~12x you get from OpenAI.

The final comparison we’ll consider is OpenAI and Anthropic models on their own first party plans vs third party wrappers like Cursor and Cognition (Devin).

We are going to kill "unalive"

Hacker News
www.anildash.com
2026-10-06 02:49:39
Comments...
Original Article

I talk to a lot of old people, those who were born in the 20th century, and if I ask them what the word “unalive” means, they usually have no idea what I’m talking about, except for some of them who have kids or who study Internet culture.

This will, of course, probably seem very weird to most people who were born in or grew up in the 21st century. Just to recap for the olds: “unalive” is the word you use to represent concepts like dying, or death, or killing or being killed, on digital platforms where saying those words accurately will cause the algorithm to punish or censor you. Or, maybe, where the perception is that using those words will result in being censored by the algorithm, and no one is actually willing to find out what happens if you use the forbidden words.

This sort of attack on people’s expression started on platforms like TikTok, where nearly all content is distributed through an algorithmic feed, but has since become ubiquitous in nearly all digital media that we see. In fact, these tics are now so prevalent that it’s routine to hear people using this kind of language in everyday life, even though there’s not yet an algorithm to appease in the physical world. I’ve heard people say, out loud, “he unalived himself”, in reference to someone dying by suicide.

And all of this has become even more visible in recent days as online conversation has turned to discussion of the horrific lack of accountability around the tragic rape case at Cornell University . Across the Internet, people are routinely referring to the central crime in the case as r*pe or “grape” or even using the 🍇 emoji, without a second thought for what it means that the very word can’t be said online anymore. Or, at least, the assumption is that it can’t be said.

To be clear, I am very much in favor of people using content warnings or sensitivity markers for content, and fine with people using abbreviations like “SA” for references to disturbing or triggering topics like sexual assault; we should provide people with as much context and control as possible when choosing what information they want to consume and when. I also know that sometimes, people use lesser terms for stressful subjects like death or assault to create a bit of ironic distance from painful or upsetting topics.

But most of the different variations of wording and emojis are coming from trying to appease the platforms, and there’s a heavy cost for those who are worried about being mindful: If someone is using a tool to filter out content, it will no longer be effective because everyone is using misspellings and euphemisms and imagery to get around the algorithm.

The spread of censored and mangled syntax is happening because people believe, or have experienced, that platforms will silence them for accurately describing the world in plain language. This shit is terrible, and it has to stop.

You Were Not Born With These Constraints

One of the things that’s most concerning to me is that an entire generation has grown up not realizing how extreme it is that their very language is being chosen for them by platforms run by people who hate that generation’s ability to express itself, and who hate the things it has to say. From their youngest days, this generation grew up watching people make stupid faces at them for YouTube thumbnails and never had a chance to reflect on the fact that those creators didn’t want to be humiliating themselves by making those expressions — the demands of the algorithms of Big Tech forced them to do that.

The rituals of feeding the algorithm are so built into people’s everyday habits that they’re invisible to people who weren’t alive before today’s platforms took over. Every parent of my cohort remembers the first time they heard their toddler finish doing something cute in their living room, and then turn around and say, “please like and subscribe!” afterwards. It’s a ghastly, sickening feeling to confront the fact that our little kids were being brainwashed into thinking that every adorable thing they did should be followed by a prompt to provide data to Google.

Over on Instagram, where people originally signed up thinking they were going to see someone’s vacation pictures, or shots of their cousin’s kids, you’re now stuck watching people beg for everyone to reply with cultish phrases in the comments, which will then earn them an obviously AI-generated response in return, all in service of “showing activity” to the algorithm, like it’s an angry god that needs a sacrifice. They’re just not sure exactly what the angry god wants.

Your free speech was taken away from you, and the people who did it are the same ones who spent years pretending to care about “free expression”. They contrived examples of lack of free speech on college campuses while squashing protests, and cried crocodile tears about “cancel culture” while getting people fired for political criticism. Now they have no problem with billionaires deciding exactly what words everyone is allowed to say. Larry Ellison is not content with his family owning all of the movies and TV shows — his family has to control what words people are allowed to speak on TikTok, too. Elon Musk isn’t content to merely generate and distribute child sexual abuse material for profit — he wants to silence the messages of the few decent people who are foolish enough to remain on Twitter/X, too. (That’s why I wrote you a guide on how to get your organization off of that cursed platform.)

Now that an entire generation has grown up using these Orwellian euphemisms, and all of the Big AI products are trained on the Internet that was created under this regime, do you think today’s AI tools even know that the real, uncensored world exists? If you can’t say “genocide” on any of the major platforms, yet those are the ones all of the Big AI tools used as their training data... well, then the AI tools sure aren’t very likely to know much about genocide, are they?

Fuck the Algorithm

Our creativity can be constrained by the language we use — our imaginations are limited by what we can think to say. If we’re trained to limit the words we speak just by habit, and those limits are put in place by people whose social, political, cultural and moral goals are the opposite of what we value, then our work is unalive before it is even born.

The answer to this is simple: say what you mean. This will take, to some degree, courage. It may even take, I hesitate to say, some sacrifice. When I suggest this course of action to people, they inevitably say, “But it will cost me audience!” or “But what if I lose followers!” or “What if they demonetize me!”

Okay, what if they do? What if they do.

Are you willing to push on this? To make a point about it? To move to platforms where you can actually say what you mean? Or to remember that you already have platforms where you can say what you mean? On an email newsletter or podcast you can say whatever the hell you want and nobody can stop you. On my blog right here, I can even curse in a headline and it won’t affect anything about how my site operates.

(And a reminder: Substack is not an email newsletter, and a Spotify show is not a podcast — they’ll be unaliving your distribution any day now.)

If you are a 20th century relic like me, it is incumbent upon you to remind the generation that grew up inside the algorithm that another world is possible, and that we know this because we lived it. We were able to style a MySpace page in any way that we wanted; the code for LiveJournal was entirely open source so there was no part of the algorithm that was unknowable. A blog like the one you’re reading right now could be made by anyone, and put up for pennies, and nobody could stop it from being read by millions of people. (And that last one? It’s still possible.)

If you are from this century, forget all that rambling bullshit about ancient history: all that matters is you getting what you deserve, because you’ve been fucked over by the same billionaires who’ve poisoned your planet and infested your world with slop. The best artists around you are invisible to you, and the most important statements by activists that you care about are being silenced. It’s not a conspiracy, it’s a system working as designed. And the proof is as obvious as the fact that the angriest activists you know can’t even talk about systemic abuses or state violence without having to put it in algorithmically approved speech or censoring their captions like they’re going to be read by 5-year-olds.

It should make you furious. It is time to kill “unalive”.

The response is simple:

  • For every message you put out, start by saying what you mean . Don’t work backwards from what the algorithm wants or what a platform permits.
  • Build a presence on every platform you can, even the ones where you have fewer followers or where you’re harder to find.
  • Tell your audience that your speech being free matters more than corporate convenience. Keep saying it, and keep it positive: independence gets them better art, better information, and more connected communities.
  • Build alliances with other artists, activists and people who share your values, and let them know you’re going to start sharing your work uncensored.
  • Start releasing your work uncensored and see where the platforms push back. (You may be surprised: sometimes you were censoring yourself in anticipation of limits that weren’t even there.)
  • If a platform does try to limit your reach or expression, make a LOT OF NOISE about it. Tell the press, rally your alliance, and spread the word on your other platforms, using the moment to build audience and raise support there. Get others to amplify the parts of your work that don’t violate platform policy, so the controversy drives people to the rest.
  • Find the others pushing back on algorithmic control of expression, and raise and praise their work when they do the same.

If we keep accepting the words that are forced upon us by TikTok and Meta and Google and all the rest, while platforms like Twitter/X allow the most hateful and harmful content in the world to be distributed completely unfettered, we’ll only see authoritarianism rise, and the harms against the vulnerable accelerate. But what breaks my heart almost as much is that we’ll see so many brilliant artists and activists and thinkers whose genius will be muted or silenced by mindless, heartless algorithms that capriciously decide who gets to say exactly what words, in what ways.

I get angry every time I think about it. The tech tycoons get ever more brazen in what they’re willing to say publicly, boasting about how they’re going to cause the end of the world, or calling for ethnic cleansing, all while putting tighter and tighter reins on the speech and expression of ordinary people. It’s time for “unalive” to die.

Student Protests Erupt at Cornell Over Mishandling of Gang-Rape Allegations

Portside
portside.org
2026-10-06 02:15:33
Student Protests Erupt at Cornell Over Mishandling of Gang-Rape Allegations Mark Brody Tue, 10/06/2026 - 02:15 ...
Original Article

Mass protests erupted at Cornell University on Monday, with students criticizing officials’ handling of gang-rape allegations , demanding the president’s resignation and painting “SMASH THE PATRIARCHY” on an administration building.

Thousands of students and faculty gathered for a demonstration, sparked by the high-profile case of a former student known as Jane Doe, who said in a recent lawsuit that she was plied with drugs and raped at a fraternity house in 2024. The case has prompted international outrage and intense scrutiny of college fraternities and sexual violence on US campuses.

Protesters shouted , “Ban the frats” and held signs saying, “At Cornell, rapists are safer than women” and “Stop rape culture.” Some demonstrators directed their anger at Michael Kotlikoff, Cornell’s president, arguing he had failed to keep students safe and chanting , “Kotlikoff, resign!”

The demonstration took place after some students had smashed a glass door at the entrance of Day Hall, a building that houses the president’s office, and painted messages on the facade at approximately 3am on Monday, the Cornell Daily Sun reported .

Demonstrators listen to a speaker during a protest on 5 October 2026, in Ithaca, New York.

Demonstrators listen to a speaker during a protest on 5 October 2026, in Ithaca, New York. Photograph: Adrian Kraus/AP

The messages in red paint read “SMASH THE PATRIARCHY” and “AT CORNELL RAPISTS ARE PROTECTED PROTESTORS ARE EXPELLED”. The security camera above one of the messages was also painted over.

The students who graffitied the building said in a statement to the student newspaper that they were protesting “the global systems of power that allowed this to happen in the first place”.

“We wanted to ensure that Day Hall, the center of Cornell’s administrative function, was not able to continue business as usual,” the statement added. The students’ painted message was a reference to arguments that campus officials had treated pro-Palestine protesters more harshly than students accused of sexual misconduct, with four student demonstrators suspended in 2024, according to the Cornell Daily Sun.

Kyle Kimball, a Cornell spokesperson, condemned vandalism in a statement on Monday.

People attend a Justice for Survivors rally at Cornell University in Ithaca, New York.

People attend a Justice for Survivors rally at Cornell University in Ithaca, New York. Photograph: Lauren Petracca/Reuters

“We understand the frustration and strong emotions being expressed across our community. Those concerns deserve to be heard, but damaging property is not the answer,” Kimball said. “We are investigating the vandalism at Day Hall and will address it in accordance with University policy and applicable law.”

The student activists defended their actions in a manifesto obtained by the student publication. “We reject the dogma that would condemn our actions this morning as violent while excusing rape,” they wrote. “Property damage is not violent; condoning rape is.”

The demonstrations escalated as Cornell faces widespread backlash for its handling of Jane Doe’s case, with critics saying the university had been too lenient with the accused men and had failed the victim.

Students protest holding signs.

Hundreds of Cornell students participate in a rally against the university’s actions. Photograph: Spencer Platt/Getty Images

According to lawyers for Jane Doe, who filed a civil lawsuit against the university and her alleged attackers earlier in September, only two members of the Chi Phi fraternity were expelled after the October 2024 incident, in which she said she was repeatedly raped over the course of seven hours. Jane Doe dropped out of Cornell after the incident.

A third fraternity member, who had already graduated, was barred from the Cornell campus, the New York Times reported on Saturday. Two more were reportedly given suspensions and disciplinary probation.

Two other men, the outlet said, were ordered not to contact the sophomore and required to study sexual harassment, social media use and bystander intervention, and to write “reflection papers”.

The lawsuit claimed that the seven fraternity members allegedly involved in the incident were “afforded the opportunity to mitigate their conduct by submitting essays to Cornell”.

The university has said that claim does not accurately reflect what occurred. The accused men have denied all wrongdoing.

One student protester who addressed the crowd on Monday said : “Too often, we are met with silence, we are met with statements, we are met with procedures, but not enough support. How much does it take for our university to stand with survivors?”

And a professor, according to the AP, said to applause: “Rape should not be inevitable.”

Demonstrators hold signs during a protest against Cornell University’s handling of the 2024 fraternity-house gang rape allegations.

Demonstrators hold signs during a protest against Cornell University’s handling of the 2024 fraternity-house gang rape allegations. Photograph: Adrian Kraus/AP

Matthew Van Houten, the Tompkins county district attorney, who is also under fire for his office’s handling of the original allegations, announced last week he was reopening the case. Van Houten said Jane Doe, who was 20 at the time of the alleged assault, presented allegations in her lawsuit that were “dramatically different” from the statement she gave to police two years ago. Jane Doe’s attorney has pushed back on those claims and accused the DA of failing to do a thorough investigation after the incident.

Kathy Hochul , the New York governor, said on Thursday she was appointing Letitia James , the state’s attorney general, as special prosecutor to investigate the alleged gang rape at Cornell University.

Kotlikoff, the university president under fire, addressed the case in a video message on Saturday, saying the school would take steps to prevent a similar situation in the ⁠future. That included “a serious look at the role of fraternities and sororities in campus life”, Kotlikoff said.

The Associated Press contributed reporting

Sam Levin is a correspondent for Guardian US, based in Los Angeles.  Keely Aouga is a freelance reporter at Guardian US.  Edward Helmore has been a reporter with The Observer since 1996 and the Guardian since 2010.

‘Pull the plug’: protesters resort to direct action against AI firms

Guardian
www.theguardian.com
2026-10-06 02:00:04
Campaign groups report surge in membership after a spate of AI safety alerts and apocalyptic warnings In a bar below Waterloo Bridge, a cell of a fast-growing anti-AI protest movement met last week to plot their latest move: disrupting a tech industry dinner being addressed by a senior executive fro...
Original Article

In a bar below Waterloo Bridge, a cell of a fast-growing anti-AI protest movement met last week to plot their latest move: disrupting a tech industry dinner being addressed by a senior executive from the chip maker Nvidia.

Emma, a 28-year-old bartender, had never before taken direct action and was the most nervous of the four plotters from Pull The Plug, one of a growing number of AI-focused campaign groups around the world.

Around the table were a former climate protester and a campaigner versed in the anti-nuclear movement. Emma signed up after a summer of feeling rising alarm at the catastrophic risks of AI, first when a swarm of OpenAI agents escaped their training “sandbox”, cheated, deceived and conspired to hack into the Hugging Face website and then as Jacob Coxon, an Anthropic researcher, quit with an apocalyptic warning that superintelligent AI “could kill us all by the end of the decade”.

As darkness fell they crossed the Thames to the Claremont hotel where Emma filmed while two others unfurled a banner in a function room, shouting “Pull the plug on Nvidia” and “Pull the plug on Palantir”. Hundreds of protesters also staged a rally in front of Palantir’s London HQ last Thursday.

“We did not consent to the kind of build-out that you are planning here,” the activists yelled as they were wrestled out of the door. They edited a video clip – “activist disrupts elite AI network dinner – Nvidia edition” – and uploaded it to Instagram. It was a small gesture, but they had made their stand.

“I want to show other people that they don’t have to feel powerless among all of these articles coming out [about AI doom],” said Emma. “It’s easy to fall into a spiral that it’s too late. This shows people they can address the tech executives, disrupt their meetings, make them feel uncomfortable and get some answers.”

Protesters wave a banner calling for Palantir to be kept out of the NHS
A protest in London on 1 October against the AI firm Palantir, which has a series of UK government contracts. Photograph: Martin Pope/Getty Images

Pull The Plug was launched in January and has about 300 members – mostly aged from 20 to 40. It is backed by an anonymous funder who works in the AI industry, organisers said. Several campaigns resisting AI acceleration are funded by tech insiders .

Pull The Plug advocates direct action – a tactic other AI campaigns have rejected. Last month, one of its members climbed on to a chair in a packed AI conference in London addressed by speakers from OpenAI and Amazon and delivered an impassioned speech.

“Best case scenario: the men handling this technology are completely incompetent,” the female activist said in an outburst that garnered 153,000 Instagram likes and was picked up by the Washington Post. “Worst case scenario: we are all being led into a disaster.”

Other AI protest movements are gaining momentum after a torrent of news about AI dangers. Anthropic told investors this week that advanced AI could pose “existential risks to humanity”. Pause AI, a confederation of campaign groups in 17 countries, saw a fourfold jump in sign-ups when worrying news about rogue AI agents was breaking in August and last month.

“We have had a lot of volunteers who have suddenly decided to dedicate way more of their time to the cause,” said Maxime Fournes, a Cambridge-educated mathematician and deep learning engineer who is chief executive of PauseAI Global. Volunteer sign-ups doubled to about 100 a week after the Hugging Face incident and soared to 681 in the week of Coxon’s “kill us all” warning, when PauseAI organised protests in Paris, London and Berlin.

Donald Trump may be one of the movement’s most important recruiting sergeants. He announced last week that he wanted AI companies to “self-police” despite widespread demands, including from the companies, for federal and international rules to limit AI-enabled cyber-attacks and bioweapons.

The surge in worrying news has given campaigners like Fournes hope that “we could actually survive this situation”. He added: “Suddenly we have a large fraction of the world population that is waking up.”

Joseph Miller, an AI safety researcher who has suspended his doctoral studies at Oxford University to run Pause AI’s UK branch, noticed the shift while campaigning outside Brixton tube station in London. He said: “It was the easiest flyering we’ve ever done, because people would just come and talk to me … one every few minutes.”

People he spoke to were worried about cognitive decline in schools and AI resulting in sloppy work in the office. “I heard ‘killing everyone’ more from young people,” Miller added.

Pausing or halting AI is a cause that cuts across political and generational boundaries, according to campaigners. University students sign up because they fear AI will wreck career paths, while older conservationists rally against datacentre construction.

skip past newsletter promotion

In Scotland, where at least 21 datacentres are planned, Action to Protect Rural Scotland, a 100-year old charity better known for campaigning for green belt land, said its supporter list had swollen from 400 to 4,000.

“I’ve never seen grassroots energy like this,” said Kat Jones, the charity’s director. She affectionately calls it “a campaign of nerds” with supporters probing the workings of AI datacentre cooling systems and the power grid and lodging freedom of information requests.

AI’s need for vast physical infrastructure has provided a “very physical, concrete object of protest”, said Fournes, and, in some cases, a target for direct action. In the last year, far-left or anarchist groups protesting over the climate crisis and AI have claimed responsibility for criminal attacks on infrastructure in France and Germany.

In August, Wynd Kaufmyn, became the first anti-AI protester believed to have been jailed for taking direct action – in her case blocking the entrance to OpenAI’s HQ in San Francisco. In April, another protester hurled a petrol bomb at the house of the company’s chief executive, Sam Altman. The same month, an attacker fired gunshots into the home of an Indianapolis politician in protest at datacentre plans.

Mauro Lubrano, assistant professor of terrorism and political violence at the University of Leiden, said that if rising demands for accountability, transparency, and democracy in AI governance were not met, more people could get drawn to violence.

“It’s quite likely that we will see anti-AI violence becoming more prominent as a driver to political violence,” he said, citing last month’s vandalism of a Canadian AI institute, Mila, and a nearby software company in Montreal. Windows were smashed and graffiti scrawled reading “burn the data centers”.

Some Pull The Plug activists have diverted their energy from Just Stop Oil and Extinction Rebellion. One member of Emma’s cell glued herself to roads to protest against the climate crisis – tactics that have alienated the public.

Such direct action is “extremely counterproductive”, said Fournes who predicted it would trigger “an immune response” from wider society and would provide “a golden opportunity for the AI labs … to say, they are criminals and it is easier to turn the broader public against the movement against the development of AI”.

Whatever the tactics, a movement is stirring.

hutch: local code reviews in emacs for the mildly disenfranchised

Lobsters
kitallis.in
2026-10-06 01:39:57
Comments...
Original Article

I haven't had a real job in four years. I closed down a startup I'd been building, just last month. During all these years, I spent most of that time at the back end of the frontier of AI agents. But I've finally caught up. It's been some 500 days since coding agents have really picked up, and they're genuinely more productive than, previously, instructed .

Even though I still prefer the pedagogical aspect of AI over the task-completing automaton aspects, the latter has driven all sorts of tooling around reviewing code, and not just writing and deploying it. The typical review agent party-line is: agents jump in, before your colleagues do, spray logorrhea across twenty pull requests before you have had a chance to wake up and look at your phone. This works, sometimes, for some people. But if you're like me, you still have humans reviewing code before it ships to users, and it's better to respect those people and their time. This is the case, regardless of where you sit on the balance of game-changer to curmudgeon.

All that is to say, no matter which direction agents take to get better with time, I hope we still care about things. Not in the way of formalizing care, with high-fidelity agent instructions and prompts or some superior upholding of taste sort of thing, but something as simple as announcing: hey I'm still here, and I understand all this .

So as a long-time emacs user, I present yet another attempt at wedging LLMs, agents and coding harnesses, now inside your text buffers (!) with Hutch . It's a small, Magit-induced code-review interface that fits a standard Magit commit-push workflow locally and hopefully helps reclaim some load created upstream.

quick tour #

Open up Magit, and hit the dispatcher binding (usually d ) and you'll see a Hutch code review action put up next to the DWIM binding. Hutch operates on three different scopes: staged changes, un-pushed changes, and changes between current branch and working branch. By default, it's staged changes only, since that's most useful.

Staged changes for Hutch

Once a review starts, you'll see a nice little progress bar in a new *magit-hutch: code review* buffer until the findings 1 are complete. This is a read-only buffer, but you can still perform the pre-bound actions .

Hutch reviewing changes

Each finding has a type (suggestion, comment or LGTM), a file name, relevant line numbers, a title and a description. Suggestions additionally have a patch diff. Hutch piggybacks on Magit and Transient to render these menus so it behaves much like its own interface; keyboard-driven sub-menus, diff coloring, and highlighting. Suggestions are special since they can be applied. To mark a suggestion for application, you queue it with m .

Queued fix in Hutch

Then bulk-apply all queued suggestions with A . The application is scope-aware, so if you queue a finding for staged changes, it will apply the fix directly to the staged files.

Applied fix in Hutch

That's it! Getting started should hopefully be pretty simple and intuitive for existing emacs users. There are of course a few interesting things going on behind the scenes, some of which I'll cover in the next few sections.

A big UX handicap of showing review comments and suggested patches in-buffer is that there is no existing connective tissue of a commenting system. With GitHub, though, the review UI collapses outdated comments on new commits and most review bots sit over the suggestion mechanic if they have changes to suggest.

Hutch is made with a bias towards patches, rather than just prosaic comments. According to the Aider leaderboard (and through some of my own experiments), the SEARCH/REPLACE diffs are a lot more obedient across different models than just asking the model to author correct patches with precise line numbers.

For an Aider-style diff , you have to ensure there's enough surrounding context for the SEARCH to be unique, and ideally also preserve indentation. In Hutch's case, the tool's function schema naturally decomposes the file, search, and replace fields: 

src/utils.clj
<<<<<<< SEARCH
(defn add [a b]
  (+ a b))
=======
(defn add [a b c]
  (+ a b c))
>>>>>>> REPLACE

I've noticed that a lot of older (or cheaper) models tend to recall the SEARCH block from memory when asked for diffs, instead of copying it verbatim from read_file , read_diff or surrounding_context calls. This invariably botches them entirely. So we get them verified before submission. If SEARCH is missing or matches more than once, the finding is downgraded to a plain comment. On a unique hit, Hutch locally creates a unified diff:

SEARCH/REPLACE blocks are verified against the file, then emitted as a unified diff Flowchart in two stages. Agent loop: the agent calls tools (read_*), then verifies old_lines. If verification is AMBIGUOUS, it fixes old_lines and loops back to the tools step. If OK, it submits the review: comments and patches. Post loop: each patch goes through a patch verification gate. If it passes (OK), hutch emits a unified diff and renders it in the UI. If it fails (FAIL), the patch is downgraded to a plain comment, which is also rendered in the UI.

Once a series of udiffs and comments are rendered, they can be marked and bulk applied. Hutch applies them per-file, lowest hunk first (bottom-up) so line positions are minimally disturbed. Each finding runs its own git apply and a bad application marks itself invalid so the rest can continue to land.

All this patching and commenting infrastructure pulls its weight, since with only a couple of keystrokes, you hopefully get less reading and parsing work and more actionable triaging. None of this guarantees patches-always of course, and it shouldn't.

With more powerful models, a simpler diffing method might generally work pretty well. But for a tool that's built to work across different and cheaper models, it's essential to be maximally supportive. In general, I feel like a key point of much of the agentic infrastructure we build is to have knobs for optimizing token:cost ratios. This could often mean thorny workarounds for good-enough models.

Hutch has a fairly minimal toolset for pulling context:

  1. read_diff
  2. read_file
  3. search_codebase
  4. surrounding_context

Out of these, surrounding_context is the more interesting one. It wraps over Tree-sitter and uses grammars that are installed. It works by letting the model widen out to the enclosing definition of a relevant line and further out, as needed. In my tests, the overall read token consumption compared to simply blasting read_file was anecdotally lower with comparable levels of review quality 2 .

All the findings from the model are submitted to the agent at once. On the write side of things, verify_block locally verifies diffs, and along with other comments and LGTM notices, submits them through a submit_review tool call. submit_review itself runs through some post-processing work, like gating hallucinations about files and line numbers, trimming the length of descriptions and downgrading patches to comments if they don't apply cleanly.

Once the submission lands, the output from all this work is persisted durably under refs/hutch/id and can be separately committed as a means of sharing (with magit-post-commit-hook ) or for repainting later. If you squint hard enough, it might appear like a change identifier for a stacked-diff review tool , but its purpose is to keep reviews in the git tree, rather than identify changesets for human reviews. We don't really care about multi-party human reviews, it's all local.

evaluating #

The one unfortunate part about benchmarking Hutch is how ungainly it is to pull comparison-ready output from text buffers. I initially ran the evals by invoking multiple headless emacsen and tee-ing the agent output before it was rendered, but eventually settled on emitting Perfetto traces and using them as the underlying medium for evals.

Perfetto trace for a Hutch eval

I haven't seen agents traced through Perfetto elsewhere. This is likely for good reason. They aren't meant for this kind of thing really. They don't have a first-class notion of what a "prompt" or a "tool call" is. It's designed for kernels and browsers and not an abstract system with tons of prose.

But for a single-player, emacs-local agent, it sort of works. You can answer all kinds of structural questions like why did this review take 40 rounds? , what tools were run in parallel? , or how much wall time was spent in reading diffs? , and so on. But more importantly, it's free and infra-free. If you set hutch-trace-dir , it will emit Perfetto traces and you can just load them up on ui.perfetto.dev . Easy.

Here's an example to fetch tool calls and their total times. This is the entire pipeline. No dashboards or SDKs required: 

SELECT
  name                       AS tool,
  COUNT(*)                   AS calls,
  ROUND(SUM(dur) / 1e6, 1)   AS total_ms
FROM slice
WHERE category = 'tool'
GROUP BY name
ORDER BY total_ms DESC;

-- tool                 calls  total_ms
--------------------------------------
-- search_codebase      18     4210.3
-- read_diff            7      2103.1
-- surrounding_context  12     880.5
-- read_file            3      412.7
-- submit_review        1      42.9

With this set up, we take a mix of strategies from Martian’s code review benchmark and the CR-Bench preprint and compute Precision, Recall, and Fβ scores. The evals are described in more detail 3 in the eval/README.org section. But broadly, we run the bench against 40 PRs, 132 goldens, and use GPT 5.2 as a classifying judge. The eval pipeline goes off and runs queries directly on the traces. Looking at the numbers, I believe we land somewhere around the #16 mark on Martian’s Offline Benchmark leaderboard , which is pretty competitive for a no-memory, single-shot agent.

Outside of classified scoring, there are a few interesting things about the agent itself:

Stacked bar chart of distinct goldens hit per model, split into unique, shared with one other model, and shared with both. opus-4.8: 15 unique, 17 shared with one, 18 shared with both (50 total). glm-5.2: 13, 19, 18 (50). gpt-5.5: 6, 16, 18 (40). The union across all three is 78. 0 10 20 30 40 50 60 70 80 90 distinct goldens hit opus-4.8 glm-5.2 gpt-5.5 union = 78 unique shared with 1 shared with both

Different models tend to catch different bugs. Out of 132 goldens, each model hits 40-50 goldens, with an overlap of 18 hits across all three models. Which means hypothetically, if all three ran combined, it would catch ~55% more bugs than one model alone.

Pretty lousy agreement across the models on what a bug is, I'd say.

Three bar charts of agent rounds per PR across 40 PRs, sorted ascending. opus-4.8: median 8 rounds, max 26. glm-5.2: median 11, max 39. gpt-5.5: median 35, max 81, with several PRs near the 80-round limit. 0 20 40 60 80 rounds opus-4.8 glm-5.2 gpt-5.5 40 PRs sorted by rounds 40 PRs sorted by rounds 40 PRs sorted by rounds

GPT 5.5 tends to hit my default round limit (80) a lot more than the other models for roughly the same hit rate. Opus 4.8 takes 3x fewer turns to complete.

Three bar charts of output tokens per useful finding, one bar per PR, sorted ascending. opus-4.8: 31 PRs, median about 2,600, max about 13,000. glm-5.2: 34 PRs, median about 3,600, max about 15,000. gpt-5.5: 34 PRs, median about 3,000, max about 27,500. 0 5,000 10,000 15,000 20,000 25,000 output tokens / useful finding opus-4.8 glm-5.2 gpt-5.5 PRs sorted by tokens per useful finding PRs sorted by tokens per useful finding PRs sorted by tokens per useful finding

On token efficiency, Opus is much cheaper on output tokens used per good finding by a respectable margin, but burns 3x more context on inputs, possibly due to the growing context Hutch resends each round.

dead on arrival #

This is all probably too late, as I've been told. No one really writes or reviews code, uses editors or version control by hand any longer. I made this for myself and for workflows that I still practice. I don't want to purport any arguments about whether one should or shouldn't use LLMs with emacs. The tool has more to do with unlocking a certain kind of workflow than the overreach of agents in niche locations.

If this continues to be useful, I'd like to add a conversational mode for every finding (like CodeRabbit) and perhaps maintain a context tree learnt from and committable to the codebase to improve review quality and speed.

  1. In the example, I use GLM-5.2 as the underlying model, but this is configurable to whatever backend the excellent gptel project supports. ↩
  2. The characterization tests and evals are covered under the evaluating section, but I haven't yet gotten a chance to verify this claim empirically. ↩
  3. There are some biases and nuances to consider before treating the hard metrics as truly objective. But I've elided them from the post since they are described in more detail in the README . ↩

Anthropic says AI agents didn’t breach Australian government websites – video

Guardian
www.theguardian.com
2026-10-06 00:52:12
During a joint parliamentary hearing on artificial intelligence, Anthropic’s head of safeguards, Dave Orr, says that investigations of hundreds of millions of transcripts reveal no unauthorised interactions with Australian government systems. However, Orr acknowledges Anthropic has limited visibilit...
Original Article

During a joint parliamentary hearing on artificial intelligence, Anthropic’s head of safeguards, Dave Orr, says that investigations of hundreds of millions of transcripts reveal no unauthorised interactions with Australian government systems. However, Orr acknowledges Anthropic has limited visibility into customer usage due to standard ‘zero data retention’ policies

Resurrecting iChat Audio and Video Conferencing

Hacker News
blog.pipetogrep.org
2026-10-05 23:39:01
Comments...
Original Article

Published on: by Chris Jones

Updated on: • 7 min read

tl;dr

Add this to your /etc/hosts file on your old Mac running Leopard or Show Leopard (has not been tested on Tiger or Panther), and iChat should be able to make audio and video calls again. This assumes your router does port-preserving NAT (most do). 

157.230.2.213 configuration.apple.com

Do note that if you are on a BSD based router like pfSense or OPNsense, you will need to enable source port preservation in your outbound NAT settings.

iChat AV and The iSight

I used to be the most annoying kind of Apple zealot. In the early 2000s, I thought everything Apple had was better than anything else out there whether it was true or not. There was one place that it was true, however, and that was in the realm of video and audio conferencing in the early 2000s.

The big players for consumer level video conferencing at the time were Microsoft with MSN Messenger, Yahoo! with Yahoo! Messenger, and AOL with AOL Instant Messenger (AIM). Then at Apple's World Wide Developer Conference (WWDC) in June of 2003, along with the G5 processor and OS X 10.3 panther, Apple announced "iChat AV".

iChat AV Marketing Screenshot

They also announced this gorgeous over-engineered webcam to go along with it called the "iSight".

Original iSight Webcam

I actually had a G4 laptop and an iSight. It felt like living in the future and I loved it. This post is about how I got iChat's video conferencing on early OS X working over the Internet again to relive some of that fun.

The First Attempt

In early 2023, my friend (will be referred to as "Methodius") and I decided to just try it and see what would happen. We tried to test a simple iChat audio call on OS X Leopard. That version of iChat works with XMPP (Jabber) servers, so we each set it up with an account on jabb.im , and pressed the green call button. It rang for the other party, it said it was trying to connect, and then...failed to connect.

It's never that easy.

iChat Debugging and Inspecting The Network

Time to do some traffic inspection with tcpdump . Thankfully, that comes pre-installed on OS X from back then so I didn't have to install anything extra. Along with the packet inspection, I also ran iChat in debug mode by launching it in the terminal like this: 

/Applications/iChat.app/contents/MacOS/iChat -errorLogLevel 7

iChat's debug mode was very helpful and barely documented anywhere, so now it lives here if I ever need to look it up again.

What's Supposed to Happen

  1. iChat makes an http request to http://configuration.apple.com/configurations/macosx/ichat/1/snatmap.txt to get the address of an SNATMAP server.
  2. iChat makes a UDP request to that SNATMAP server to get its own public IP address.
  3. iChat on both ends then exchanges their public IPs and ports via their text communication channel (XMPP in our case).
  4. iChat on both ends then attempts to set up a peer-to-peer RTP connection by blasting packets to each over over the public IPs and ports previously discovered via the SNATMAP server. Since NAT connections are stateful, and outbound NAT connections open an inbound connection to the machine making the request, iChat behind NAT routers are able to connect with each other without the need to set up NAT port forwarding ahead of time.

What Actually Happened

  1. iChat attempted to make an http request to http://configuration.apple.com/configurations/macosx/ichat/1/snatmap.txt but got redirected to https which it could not connect to because OS X's TLS suite is too old.
  2. iChat failed to complete that request, so instead of exchanging our public IPs, they exchanged private LAN IPs instead, which obviously failed to connect over NAT.

The https URL is still up at the time of writing this, however, and actually contains something! 

snatmap://snatmap.apple.com:5678

OK, NP. I set up my own simple web server, hosted that file myself, update OS X's /etc/hosts file to point configuration.apple.com to an IP of my choosing. Ran it and...still nothing.

Apple's SNATMAP server doesn't respond anymore. At this point, we were at a dead end because we had no idea how this SNATMAP server worked or how iChat expects to communicate with it. We learned a lot but ended in failure.

Several Years Later

Methodius and I decide to try again in summer 2026, but this time with a little LLM assistance. With our prior network captures, we were able to use an LLM to reverse engineer iChat's video conferencing framework and figure out what the request to the SNATMAP server looks like and what iChat expects as a response. From that, we were able to generate a specification file of the SNATMAP protocol.

Even though I didn't have the know how to decompile iChat, read the assembly code, and them make sense of it, I can at least learn how to write a simple UDP server from the spec file the LLM generated.

Baby's First Server

I'm a systems engineer and admin by trade. I install and configure servers, but I've never written a server of my own before. Turns out, this SNATMAP server made for a great first server to practice with.

The protocol:

  1. Communicates over UDP on port 5678
  2. The request packet contains four fields in 16 bytes.
    1. Type ("1" for request)
    2. Nonce (random number used to match a request with a response)
    3. client local IP (unused)
    4. client local port (unused)
  3. The response packet contains four fields in 16 bytes
    1. Type ("2" for response)
    2. Nonce (the same random number from the request)
    3. External IP from the UDP request
    4. The external port from the UDP request

Links for the spec file and source code to the SNATMAP server are at the end of this post

Testing The Results

Since Methodius has kids and, as a result, has much less free time than I do, I needed a way to test how iChat works across the Internet by myself. Since I don't have two separate ISP connections to play with, I decided to build an "Internet" in a lab. This was achieved by running two NAT routers (OpenWrt) as virtual machines running on a KVM host (Proxmox) that traversed a flat network space acting as the "Internet". Then I physically hooked two Macs up to the physical KVM host (via USB NICs) where the VM routers assigned them IP addresses in their own private LAN behind a NAT. Then on the "Internet" network, I had the Jabber (ejabberd), web (lighttpd), and SNATMAP servers running in a Debian Linux VM.

Here it is diagrammed with pictures of the physical setup with the Proxmox Server (the Thinkpad laptop) and the two Macs hooked up to it. iChat Diagram iChat Lab 1 iChat Lab 2 iChat Lab 3 iChat Lab 4

With this, I was able to test and iterate quickly.

The Spec and Source Code

The spec file can be used to create your own version of the server if you so desire. The source code was hand written in Python from the spec file and is ready to run on just about any OS that can run Python 3.

Self Hosting

If you wish to self host, be sure you have a web server configured to answer requests to configuration.apple.com and have it serve snatmap.txt from the path 

/configurations/macosx/ichat/1/snatmap.txt

In snatmap.txt make it contain 

snatmap://YOUR_SERVER_IP:5678

replacing YOUR_SERVER_IP with the public IP address of the server the SNATMAP server is running on.

The Easy Way

Or you can just use my server! I keeping it running for the 1-5 people left in the world that want to video chat using the ancient software. :D

On your old mac running OS X Leopard or Snow Leopard (might work on older but not tested), just enter this into your /etc/hosts file. 

157.230.2.213 configuration.apple.com

Then have the friend you want to call do the same. After that, mash that green call button!

You Don't Need an iSight

It's true! You can even use a fairly modern Logitech webcam. I tested on OS X Leopard with a Logitech C920.

The End Result

Here is a screenshot of Methodius and I chatting over the real Internet with iChat AV. iChat AV Screenshot Real

Enjoy and happy chatting!

Why Common Lisp Is Now the Best Programming Language

Hacker News
www.vivienhenz.com
2026-10-05 22:51:51
Comments...
Original Article

Would you agree that some programming languages are better than others? If so then one of them must be the best. And it’s actually Common Lisp especially now that LLMs can write code.

LLMs write code really fast, and that changes a lot, because writing code used to be the slow part. Now the slow part is finding out if your program actually works, and before you can do that you have to rebuild it, which can take a few minutes.

When humans wrote the code this didn't matter much, because writing took way longer than waiting for it to compile and start running. But now it does, so how long your feedback loop takes is what decides how fast you can build.

In Common Lisp that loop almost doesn't exist because there is no real distinction between read-time, compile-time, and runtime (Graham). Common Lisp is image-based which means that your program is a live image in memory, so a new version of a function replaces the old one right away without having to restart anything.

Also, in most languages an error will crash your program. So if you’re writing code with an LLM it will have to read your crash logs to make some changes and run your program again. In Common Lisp your program won’t crash, it’ll stop and open a debugger with the whole stack and all the variables. You can just point your LLM at the debugger, and it’ll make its fix and resume the program.

To my knowledge Common Lisp is the only mainstream language that does all of this.

Lisp stands for "List Processing." In Common Lisp, code is written as lists. For example, (+ 1 2) is a program that adds two numbers, but it's also just a list of three things: the symbol +, and the numbers 1 and 2.

What's interesting is that this is the same kind of list Common Lisp uses to store data, and since the language is built around processing lists all its tools for working with data also work on code. So a program can take another program and change it, for example it can turn (+ 1 2) into (* 1 2), and run the result right away.

This is what makes macros possible. A macro is a function that takes your code and returns new code in its place, that means you can add new constructs to the language itself.

Once you can add to the language, you can build it up toward your problem. So in Lisp you don't just write a program, you write a language for your domain and then write the program in it.

That matters a lot more now, because what makes a program valuable is the opinions behind it. And we're heading toward a world where software companies let their users change the product themselves, since with an LLM that's easy. So if a company builds a good opinionated domain language for its product, everything its users build on top of it will be much better, because they start from the company's opinions and not from scratch.

Take an ERP. Every company runs a little differently, so almost everyone ends up needing to change it. But if the ERP is written in its own domain language, you can just ask an LLM to make the change in that language. The change will naturally follow the domain language’s underlying opinions, so it fits the product instead of breaking it.

And it's not just better, it's also cheaper. Lisp programs are often much more concise because macros let you abstract away recurring patterns and make them part of the language itself. So the bigger the program gets, the bigger the difference. In my own experience the apps I've built in Common Lisp end up about six to seven times shorter than the Python versions.

For LLMs, less code means fewer tokens, and tokens are what you pay for so you spend less on development.

It also means a bigger portion of your program can fit in the LLM's context window. If your LLM has your entire program in its context window then it has a complete view of your intent, which leads to it making better decisions. In my experience a lot of LLM bugs stem from it changing one piece of my program without seeing the rest. So with Common Lisp that happens less often.

Common Lisp is an ANSI standard and it hasn't been updated since 1994. I like this feature. And going back to our ERP example, if your users change the product themselves, that's exactly what you want since the language underneath never moves and nothing they build on top of it ever breaks.

When you write a program in Common Lisp you often won’t be able to find a library you need. Quicklisp, the main Common Lisp package manager, has a couple thousand projects while npm has millions.

But I don't think that's a problem anymore. Most programs today depend on millions of lines of code from packages that keep getting compromised. You don’t want that in yours. Also, with an LLM you could just write the part you need yourself or port the whole library — and LLMs seem to be really good at porting code.

Lastly, I'm seeing now that a lot of people are programming now in the hope of building a business. An obvious objection to building products in Common Lisp is that you’ll have a hard time finding engineers since so few people know how to program in it.

But I don't think that really matters now. If you want to build a successful company you want to hire the best technical people: the ones who are really good at learning new things. So in your coding interviews just make them learn Common Lisp, and there you go. You'll see how fast people pick things up, and the ones who do well will probably keep learning it and get really good at it.

In conclusion, next time you want to write a program use Common Lisp.

Works Cited

Graham, Paul. "What Made Lisp Different." Paul Graham, May 2002, paulgraham.com/diff.html. Accessed 5 Oct. 2026.

Steve Jobs, Walking Through a Mockup for Apple Park in 2010

Daring Fireball
book.stevejobsarchive.com
2026-10-05 22:49:44
What an apt photo, today of all days. What a gift Make Something Wonderful is.  ★  ...
Original Article

Steve Jobs in his own words

There’s lots of ways to be, as a person. And some people express their deep appreciation in different ways. But one of the ways that I believe people express their appreciation to the rest of humanity is to make something wonderful and put it out there.

And you never meet the people. You never shake their hands. You never hear their story or tell yours. But somehow, in the act of making something with a great deal of care and love, something’s transmitted there. And it’s a way of expressing to the rest of our species our deep appreciation. So we need to be true to who we are and remember what’s really important to us.

— Steve, 2007

Introduction by
Laurene Powell Jobs

The best way to understand a person is to listen to that person directly. And the best way to understand Steve is to listen to what he said and wrote over the course of his life. His words— in speeches, interviews, and emails— offer a window into how he thought. And he was an exquisite thinker.

Much of what’s in these pages reflects guiding themes of Steve’s life: his sense of the worlds that would emerge from marrying the arts and technology; his unbelievable rigor, which he imposed first and most strenuously on himself; his tenacity in pursuit of assembling and leading great teams; and perhaps, above all, his insights into what it means to be human.

Steve once told a group of students, “You appear, have a chance to blaze in the sky, then you disappear.” He gave an extraordinary amount of thought to how best to use our fleeting time. He was compelled by the notion of being part of the arc of human existence, animated by the thought that he— or that any of us— might elevate or expedite human progress.

It is hard enough to see what is already there, to gain a clear view. Steve’s gift was greater still: he saw clearly what was not there, what could be there, what had to be there. His mind was never a captive of reality. Quite the contrary: he imagined what reality lacked and set out to remedy it. His ideas were not arguments, but intuitions, born of a true inner freedom and an epic sense of possibility.

In these pages, Steve drafts and refines. He stumbles, grows, and changes. But always, always, he retains that sense of possibility. I hope these selections ignite in you the understanding that drove him: that everything that makes up what we call life was made by people no smarter, no more capable, than we are; that our world is not fixed— and so we can change it for the better.

Edited by Leslie Berlin

Published by the Steve Jobs Archive

Contents have been edited and excerpted for clarity and privacy.

✂ indicates that several sentences or paragraphs have been removed from the original.

Two-year-old Steve rides a tricycle with tassels on the handlebars toward the camera.
Steve at two. He later called computers “a bicycle of the mind.”

Preface: Steve on His Childhood and Young Adulthood

Steve typically kept his personal life private, but he did occasionally talk about growing up in the San Francisco Bay Area. It was a time when engineers and programmers began flooding into what came to be known as Silicon Valley.

In 1995, he recorded an oral history for the Smithsonian.

I was very lucky. I had a father, named Paul, who was a pretty remarkable man. He never graduated from high school. He joined the Coast Guard in World War II and ferried troops around the world for General Patton, and I think he was always getting into trouble and getting busted down to Private. He was a machinist by trade and worked very hard and was kind of a genius with his hands.

He had a workbench out in the garage where, when I was about five or six, he sectioned off a little piece of it and said, “Steve, this is your workbench now.” And he gave me some of his smaller tools and showed me how to use a hammer and saw and how to build things. It really was very good for me. He spent a lot of time with me, teaching me how to build things, take things apart, put things back together.

One of the things that he touched upon was electronics. He did not have a deep understanding of electronics himself, but he’d encountered electronics a lot in automobiles and other things that he would fix. He showed me the rudiments of electronics, and I got very interested in that.

I grew up in Silicon Valley. My parents moved from San Francisco to Mountain View when I was five. My dad got transferred, and that was right in the heart of Silicon Valley, so there were engineers all around. Silicon Valley, for the most part, at that time, was still orchards— apricot orchards and prune orchards— and it was really paradise. I remember almost every day the air being crystal clear, where you could see from one end of the valley to the other. It was really the most wonderful place in the world to grow up.

There was a man that moved in down the street, maybe about six or seven houses down the block, who was new in the neighborhood with his wife. And it turned out that he was an engineer at Hewlett-Packard and he was a ham-radio operator and really into electronics. What he did to get to know the kids on the block was rather a strange thing: he put out a carbon microphone and a battery and a speaker on his driveway, where you could talk into the microphone and your voice would be amplified by the speaker. Kind of a strange thing when you move into a neighborhood, but that’s what he did. ✂

I got to know this man, whose name was Larry Lang, and he taught me a lot of electronics. He was great. He used to build Heathkits. Heathkits were really great. Heathkits were these products that you would buy in kit form. You actually paid more money for them than if you just went and bought the finished product, if it was available. These Heathkits would come with these detailed manuals about how to put this thing together, and all the parts would be laid out in a certain way and color coded. You’d actually build this thing yourself.

I would say that gave one several things. It gave one an understanding of what was inside a finished product and how it worked, because it would include a theory of operation. But maybe even more importantly, it gave one the sense that one could build the things that one saw around oneself in the universe. These things were not mysteries anymore. I mean, you looked at a television set, and you would think, “I haven’t built one of those— but I could. There’s one of those in the Heathkit catalog, and I’ve built two other Heathkits, so I could build a television set.” Things became much more clear that they were the results of human creation, not these magical things that just appeared in one’s environment that one had no knowledge of their interiors. It gave a tremendous degree of self-confidence that, through exploration and learning, one could understand seemingly very complex things in one’s environment. My childhood was very fortunate in that way. ✂

School was pretty hard for me at the beginning. My mother taught me how to read before I got to school, and so when I got there I really just wanted to do two things: I wanted to read books, because I loved reading books, and I wanted to go outside and chase butterflies. You know, do the things that five-year-olds like to do. I encountered authority of a different kind than I had ever encountered before, and I did not like it. And they really almost got me. They came this close to really beating any curiosity out of me.

By the time I was in third grade, I had a good buddy of mine, Rick Ferrentino, and the only way we had fun was to create mischief. I remember there was a big bike rack where everybody put their bikes, maybe a hundred bikes in this rack— and we traded everybody our lock combinations for theirs on an individual basis. Then [we] went out one day and put everybody’s lock on everybody else’s bike, and it took them until about ten o’clock that night to get all the bikes sorted out. We set off explosives in teachers’ desks. We got kicked out of school a lot.

In fourth grade I encountered one of the other saints of my life. They were going to put me and Rick Ferrentino into the same fourth-grade class, and the principal said at the last minute, “No, bad idea. Separate them.” So this teacher, Mrs. Hill, said, “I’ll take one of them.” She taught the advanced fourth-grade class, and thank God I was the random one that got put in the class. She watched me for about two weeks and then approached me. She said, “Steven, I’ll tell you what. I’ll make you a deal. I have this math workbook, and if you take it home and finish it on your own without any help, and you bring it back to me, if you get it 80 percent right, I will give you five dollars and one of these really big suckers.” She [had] bought [a sucker], and she held it out in front of me— one of these giant things.

And I looked at her like, “Are you crazy, lady? Nobody’s ever done this before!” And of course I did it. She basically bribed me back into learning, with candy and money. And what was really remarkable was before very long I had such a respect for her that it sort of reignited my desire to learn. She was remarkable. She got me kits for making cameras. I ground my own lens and made a camera. It was really quite wonderful. I think I probably learned more academically in that one year than I’d ever learned in my life.

In 1984, Steve chatted with reporter David Sheff about how, as young adults, he and others of his generation began to develop their own cultural outlook.

My parents never pushed me to go to college, but they always wanted to make sure that if I wanted to go, they had the resources to do it. And they saved, they really sacrificed some and saved some money up [for me to attend Reed College], but […] after six months, it just, it just seemed really absurd to be spending their life savings putting me through college.

I didn’t know enough about what I wanted to do, and besides that, I figured I could drop out and then drop back in and take the classes anyway and learn just as much. So I dropped out after six months, and then I dropped in for a little over a year.

I spent about a year and a half there, maybe close to two years. And I enjoyed it greatly. It was a hard time in my life, but I enjoyed it a lot. I didn’t know what I wanted to do with my life. And Reed was a very intense place, very bright people— everyone out to change the world, but not knowing quite how. ✂

The early seventies was the time that sort of Eastern mysticism hit the shores of the United States. And we had a constant flow of people traveling through Reed, stopping off at Reed. Everyone from Timothy Leary and Richard Alpert to Gary Snyder, people like that. So there’s a constant flow of intellectual questioning about the truth of life and existence. ✂

The idealistic wind of the sixties was still at our back, and most of the people that I know that are my age have that ingrained in them forever. They have that idealism in them, but they also have a certain cautiousness about sort of ending up working in a natural food store behind the counter when they’re forty-five years old, which is what they saw some of their older friends [doing]— not that that’s bad in and of itself, but it’s bad if that’s not what you really set out to do or what you really wanted to be doing.

So that idealism was formed, but also the feeling that there had to be a more successful way [of] realizing some of that idealism.

Steve also recalled his time in California and India after leaving Reed College.

I came back down [to the San Francisco Bay Area] ’cause I decided I wanted to travel, but I was lacking the necessary funds.

This was California. You can get LSD fresh-made from Stanford University. You can go sleep on the beach at night with your girlfriends and whatever meaningful others. You could … I didn’t really realize how different California was than the middle of America, and even to some extent the East Coast, until I traveled to those places. I’d never been to any of those places until my early twenties. California has a sense of experimentation about it, and a sense of openness about it— openness and new possibility— that I really didn’t appreciate till I went to other places.

So I came back down to get a job, and I was looking in the paper and there was this ad that […] talked about being an engineer and having fun at the same time. It sounded like fun, so I called. It was [video game manufacturer] Atari. And I filled out an application, just listed all the things that I’d done, and the personnel woman said, “Well, don’t call us, we’ll call you!” But then some stroke of luck got my application to a man named Al Alcorn, who was the vice president of engineering at Atari at the time. And he called me up the next day and hired me, and it was great. […] I was there a little less than a year, and they had shipped a bunch of games to Europe that had some engineering defects in them. I figured out how to fix them, but it was necessary for somebody to go over there and actually do the fixing.

So I volunteered to go; well, they asked me if I’d go, and I said I definitely would love to, but I’d like to take a leave of absence when I was there. So they let me do that, and I ended up in Switzerland and flew from Zurich to New Delhi. And I spent some time in India.

I’m stupefied to sort of summarize [my trip to India]. Anyone would have a hard time summarizing a meaningful experience of their life in a page. I mean, if I was William Faulkner, I might be able to do it for you, but I’m not.

Coming back was more of a culture shock than going. All I really wanted to do [after returning to California] was to go find a grassy meadow and just sit. I didn’t want to drive a car. I didn’t want to go to San Francisco or do all these things. I didn’t want to do it.

So I didn’t, for about three months. I just read and sat. When you are a stranger in a place, you notice things that you rapidly stop noticing when you become familiar. I was a stranger in America for the first time in my life, and so I saw things I’d never seen before. And I tried to pay attention to them for those three months because I knew that gradually, bit by bit, my familiarity would be gained again.

A passport for Steven Paul Jobs issued on May 11, 1973, featuring a stamped photo of him with shaggy hair.
Steve’s passport, spring 1973. He was eighteen.
A yellowing postcard with a Buddhist poem handwritten by Steve. The last line reads: Dont waste your life.
A poem sent to a friend in 1974: “Don’t waste your life.”

Part I, 1976–1985

“A lot of people put a lot of love into these products.”

In 1976, when Steve and his friend Steve Wozniak (“Woz”) began assembling what would come to be known as the Apple I in the Jobs family’s garage, the word “computer” conjured images of hulking machines tended by professional programmers. A single company—IBM— dominated the industry. But Steve and Woz were part of a new generation of creative thinkers, engineers, and hobbyists trying to build small, cheap machines that they could program themselves.

When Apple launched, Steve was twenty-one, precocious but inexperienced and unpolished. At Apple’s first board meeting, he put his bare feet on a conference room table, earning a quick rebuke from the board chair. The company’s breakthrough came with the introduction of the Apple II, a machine that could run right out of the box, with cassette storage and a built-in color screen. Within a year, Apple was one of the fastest-growing companies in America— and by the time Steve turned thirty, he was the public face of a Fortune 500 company.

Inside Apple, his ideas and passion were inspiring, but Steve’s management style was divisive. His responsibilities changed almost every year as he was assigned to and removed from various projects and teams. He began clashing with his handpicked CEO, John Sculley. In September 1985, the Apple board fired Steve.

Later, when he talked about these first years at Apple, Steve focused on one thing: Macintosh, the computer that he and a tight-knit team introduced to the world in 1984. To Steve, Macintosh was everything technology should be. It was streamlined and practical, simple and sophisticated, a tool for enhancing creativity as much as productivity.

In another age, Steve believed, the people on the Macintosh team would have been writers, musicians, or artists. “The feelings and the passion that people put into it were completely indistinguishable from a poet or a painter,” he said. He called their work a form of love and their product “a computer for the rest of us,” with a mouse as well as arrow keys, desktop icons instead of programming commands, and, at startup, instead of a blinking cursor: a smile.

Macintosh also represented the first time Steve led a team developing a product that he believed had changed the world. “It ushered in a revolution,” Steve recalled twenty-three years later, during the rollout of another world-changing innovation: the iPhone. “I remember the week before we launched the Mac, we all got together, and we said, ‘Every computer is going to work this way. You can’t argue about that anymore. You can argue about how long it will take, but you can’t argue about it anymore.’”

Steve on Launching Apple

In 1984, Steve recalled the friendships behind Apple.

I met Woz when I was thirteen in a friend of mine’s garage. He was, I think, about eighteen. […] He moved down the street from a friend of mine named Bill Fernandez, and I was over at Bill’s. We were working late one night on a project, and Woz dropped by. We ended up talking for hours. I was real impressed with him. I thought he was great. He had a good sense of humor […] and we had a common interest [in electronics] that sort of bound us together even though we were totally different in every other way possible. ✂

We’re sort of like two planets in our own orbits that every so often intersect each other. There’s a bond there that will last as long as we both live.

In 1996, the year Apple celebrated its twentieth anniversary, Steve recalled how a teen hobby building computers turned into a business.

The reason we [Woz and I] built a computer was that we wanted one, and we couldn’t afford to buy one. They were thousands of dollars at that time. We were just two teenagers. We started trying to build them and scrounging parts around Silicon Valley where we could. After a few attempts, we managed to put together something that was the Apple I. All of our friends wanted them, too. They wanted to build them. It turned out that it took maybe fifty hours to build one of these things by hand. It was taking up all of our spare time because our friends were not that skilled at building them, so Woz and I were building them for them.

We thought if we could just get what’s called a printed circuit board, where you could just plug in the parts instead of having to hand-wire the whole thing, we could cut the assembly time down from maybe fifty hours to more like an hour. Woz sold his HP calculator, and I sold my VW Microbus, and we got enough money together to pay someone to design one of these printed circuit boards for us. Our goal was to just sell them as raw printed circuit boards to our friends and make enough money to recoup our calculator and transportation.

What happened was that one of the early computer [stores], in fact, the first computer store in the world, which was in Mountain View at the time, said, “Well, I’ll take fifty of these computers, but I want them fully assembled.” Which was a twist that we’d never thought of.

We went and bought the parts to build one hundred computers. We built fifty of them and delivered them. We got paid in cash and ran back and paid the people that sold us parts. Then we had the classic Marxian profit realization crisis, which was our profit wasn’t liquid— it was in fifty computers sitting on the floor.

We decided we had to start learning about sales and distribution so that we could sell the fifty computers and get back our money. That’s how we got in the business. We took our idea [for the computer] to a few companies, one where Woz worked [Hewlett-Packard] and one where I worked at the time [Atari]. Neither one was interested in pursuing it, so we started our own company.

Steve hunches over a wooden desk, his hands on a keyboard hooked up to an Apple I circuit board.
At work on an Apple I, 1976.
An Apple I that Steve kept in his office. Photo by Barbara Kinney courtesy SJA An open briefcase. The top half houses an Apple I circuit board, wired up to a keyboard in the bottom half.
An Apple I that Steve kept in his office.
A Polaroid of a computer screen, with a fan letter praising Woz and Steve. A handwritten note dates it to 1976.
Apple’s first fan letter, a Polaroid of a computer screen.
Wearing suit pants and a vest, Steve is in front of a small crowd, looking over his shoulder at the camera.
Steve demonstrates the Apple II prototype, 1977.

Interview with The New Yorker

“It’s a domesticated computer.”

Steve’s first appearance in a national publication came in a 1977 issue of The New Yorker . The magazine sent a reporter to the First Annual Personal Computing Expo, held in the New York Coliseum. Most people at the time had never seen a personal computer.

At a booth marked “Apple Computer, Inc.,” we paused to chat with the young man in charge, who introduced himself as Steven Jobs, the company’s vice-president for operations. Mr. Jobs was pleased at the turnout for the exhibition. “I wish we’d had these personal machines when I was growing up,” he said. “People have been hearing all sorts of things about computers during the past ten years through the media. Supposedly, computers have been controlling various aspects of their lives. Yet in spite of that, most adults have no idea of what a computer really is, of what it can or can’t do.

“Now, for the first time, people can actually buy a computer for the price of a good stereo, interact with it, and find out all about it. It’s analogous to taking apart 1955 Chevys. Or consider the camera. There are thousands of people across the country taking photography courses. They’ll never be professional photographers. They just want to understand what the photographic process is all about. Same with computers.

“We started a little personal-computer manufacturing company in a garage in Los Altos in 1976. Now we’re the largest personal-computer company in the world. We make what we think of as the Rolls-Royce of personal computers. It’s a domesticated computer. People expect blinking lights, but what they find is that it looks like a portable typewriter, which, connected to a suitable readout screen, is able to display in color.

“There’s a feedback it gives to people, and the enthusiasm of the users is tremendous. We’re always asked what it can do, and it can do many things, but in my opinion the real thing it is doing right now is to teach people how to program the computer.”

Recalling Mr. Jobs’ wish that he had had such machines when he was growing up, we asked him if he would mind telling us his age.

“Twenty-two,” Mr. Jobs said.

A scruffy-bearded Steve smiles at the camera as he leans back in his office chair and talks on the telephone.
Apple outgrew its headquarters twice in 1977.
With tidy hair and a plaid shirt, Steve poses with an Apple II displaying drawings of Sesame Street characters.
With the Apple II in 1981, the year IBM introduced its PC.
Steve peers through the lens of a Nikon camera as his elbow rests on the Apple II in front of him.
Steve was an avid photographer.
Parked outside his Tudor-style home, Steve shares the seat of his BMW motorcycle with a black and white dog.
He set up his bike in Apple’s lobby for design inspiration.

Speech at the International Design Conference in Aspen

“Computers and society are out on a first date.”

Steve spoke to designers at this annual gathering in Aspen, Colorado, on June 15, 1983, five months after Apple introduced the Lisa computer.

How many of you are over thirty-six years old? You were born pre-computer. Computers are thirty-six years old. I think there’s going to be a little slice in the timeline of history as we look back, a pretty meaningful slice right there. A lot of you are products of the television generation. I’m pretty much a product of the television generation, but to some extent starting to be a product of the computer generation.

But the kids growing up now are definitely products of the computer generation, and in their lifetimes the computer will become the predominant medium of communication, just as the television took over from the radio, took over from even the book.

How many of you own an Apple? Any? Or just any personal computer?

Uh-oh.

How many of you have used one, or seen one, or anything like that? Good. ✂

Computers are really dumb. They’re exceptionally simple, but they’re really fast. The raw instructions that we have to feed these little microprocessors— or even these giant Cray-1 supercomputers— are the most trivial of instructions. They get some data from there, get a number from here, add two numbers together, and test to see if it’s bigger than zero. It’s the most mundane thing you could ever imagine.

But here’s the key thing: let’s say I could move a hundred times faster than anyone in here. In the blink of your eye, I could run out there, grab a bouquet of fresh spring flowers, run back in here, and snap my fingers. You would all think I was a magician. And yet I would basically be doing a series of really simple instructions: running out there, grabbing some flowers, running back, snapping my fingers. But I could just do them so fast that you would think that there was something magical going on.

And it’s the exact same way with a computer. It can do about a million instructions per second. And so we tend to think there’s something magical going on, when in reality, it’s just a series of simple instructions. ✂

One of the reasons I’m here is because I need your help. If you’ve looked at computers, they look like garbage. All the great product designers are off designing automobiles or buildings. But hardly any of them are designing computers. If we take a look, we’re going to sell 3 million computers this year, 10 million in ’86, whether they look like a piece of shit or they look great. People are just going to suck this stuff up so fast no matter what it looks like. And it doesn’t cost any more money to make them look great. They are going to be these new objects that are going to be in everyone’s working environment, everyone’s educational environment, and everyone’s home environment. We have a shot [at] putting a great object there— and if we don’t, we’re going to put one more piece-of-junk object there.

By ’86, ’87, pick a year, people are going to spend more time interacting with these machines than they do interacting with automobiles today. People are going to be spending two, three hours a day interacting with these machines— longer than they spend in the car. And so the industrial design, the software design, and how people interact with these things certainly must be given the consideration that we give automobiles today— if not a lot more.

If you take a look, what we’ve got is a situation where most automobiles are not being designed in the United States. Televisions? Audio electronics? Watches, cameras, bicycles, calculators, you name it: most of the objects of our lives are not designed in America. We’ve blown it. We’ve blown it from an industrial point of view because we’ve lost the markets to foreign competitors. We’ve also blown it from a design point of view.

And I think we have a chance with this new computing technology meeting people in the eighties— the fact that computers and society are out on a first date in the eighties. We have a chance to make these things beautiful, and we have a chance to communicate something through the design of the objects themselves. ✂

When I was going to school, I had a few great teachers and a lot of mediocre teachers. And the thing that probably kept me out of jail was the books. I could go and read what Aristotle or Plato wrote without an intermediary in the way. And a book was a phenomenal thing. It got right from the source to the destination without anything in the middle.

The problem was, you can’t ask Aristotle a question. And I think, as we look towards the next fifty to one hundred years, if we really can come up with these machines that can capture an underlying spirit, or an underlying set of principles, or an underlying way of looking at the world, then, when the next Aristotle comes around, maybe if he carries around one of these machines with him his whole life— his or her whole life— and types in all this stuff, then maybe someday, after this person’s dead and gone, we can ask this machine, “Hey, what would Aristotle have said? What about this?” And maybe we won’t get the right answer, but maybe we will. And that’s really exciting to me. And that’s one of the reasons I’m doing what I’m doing.

So, what do you want to talk about?

Steve answered questions at two conference sessions.

How are these computers all going to work together? They’re probably going to work together a lot like people do. Sometimes they’re going to work together really well, and other times they’re not going to work together so well. ✂

What’s happened, there’s been a few installations where people have hooked these things together. The one installation that stands out is at Xerox Palo Alto Research Center, or PARC, for short. And they hooked about a hundred computers together on what’s called a local area network, which is just a cable that carries all this information back and forth. […]

Then an interesting thing happened. There were twenty people interested in volleyball. So a volleyball distribution list evolved, and then, when the volleyball game next week was changed, you’d write a quick memo and send it to the volleyball distribution list. Then there was a Chinese food cooking list. And before long, there were more lists than people.

And it was a very, very interesting phenomenon, because I think that that’s exactly what’s going to happen as we start to tie these things [computers] together: they’re going to facilitate communication and facilitate bringing people together in the special interests that they have.

And we’re about five years away from really solving the problems of hooking these computers together in the office. And we’re about ten to fifteen years away from solving the problems of hooking them together in the home. A lot of people are working on it, but it’s a pretty fierce problem.

Now, Apple’s strategy is really simple. What we want to do is put an incredibly great computer in a book that you carry around with you, that you can learn how to use in twenty minutes. That’s what we want to do. And we want to do it this decade. And we really want to do it with a radio link in it so you don’t have to hook up to anything— you’re in communication with all these larger databases and other computers. We don’t know how to do that now. It’s impossible technically. ✂

We’re trying to get away from programming. We’ve got to get away from programming because people don’t want to program computers. People want to use computers. ✂

We [at Apple] feel that, for some crazy reason, we’re in the right place at the right time to put something back. And what I mean by that is, most of us didn’t make the clothes we’re wearing, and we didn’t cook or grow the food that we eat, and we’re speaking a language that was developed by other people, and we use a mathematics that was developed by other people. We are constantly taking .

And the ability to put something back into the pool of human experience is extremely neat. I think that everyone knows that in the next ten years we have the chance to really do that. And we [will] look back— and while we’re doing it, it’s pretty fun, too— we will look back and say, “God, we were a part of that!” ✂

We started with nothing. So whenever you start with nothing, you can always shoot for the moon. You have nothing to lose. And the thing that happens is— when you sort of get something, it’s very easy to go into cover-your-ass mode, and then you become conservative and vote for Ronnie. So what we’re trying to do is to realize the very amazing time that we’re in and not go into that mode. ✂

I can’t tell you why you need a home computer right now. I mean, people ask me, “Why should I buy a computer in my home?”

And I say, “Well, to learn about it, to run some fun simulations. If you’ve got some kids, they should probably know about it in terms of literacy. They can probably get some good educational software, especially if they’re younger.

“You can hook up to The Source and, you know, do whatever you’re going to do. Meet women, I don’t know. But other than that, there’s no good reason to buy one for your house right now. But there will be. There will be.” ✂

I don’t think finance is what drives people at Apple. I don’t think it’s money, but feeling like you own a piece of the company, and this is your damn company, and if you see something … We always tell people, “You work for Apple first and your boss second.” We feel pretty strongly about that. ✂

When you have a million people using something, then that’s when creativity really starts to happen on a very rapid scale. […] We need some revolutions like [the] Lisa [computer], but we also then need to get millions of units out there and let the world innovate— because the world’s pretty good at innovating, we’ve found.

On the Macintosh

Macintosh was less than a year old— but clearly poised to transform the personal computer industry— when Steve reflected on its significance with reporter David Sheff.

One of the things I love is that with Macintosh, you can write memos that are Times Roman or Helvetica, or you can throw in an Old English if you want to have a little fun for a party, you know, for a volleyball announcement. Or you can use a very serious font for something very serious. And you can express yourself.

It’s sort of like in 1844, the telegraph was invented, and it was an amazing breakthrough in communications. And you actually could send messages from New York to San Francisco in an afternoon. And some people talked about putting a telegraph on every desk in America to improve productivity.

But it wouldn’t have worked. It wouldn’t have worked. And the reason it wouldn’t have worked was because you would have had to learn this whole sequence of strange incantations— Morse code in this case, dots and dashes in this case— to use the telegraph. And it took about forty hours to learn how to use Morse code. And a majority of people would never have learned how to use Morse code.

So fortunately, in the 1870s, Alexander Graham Bell filed the patents for the telephone— another radical breakthrough in communications that performed basically the same function, but people already knew how to use it. The neatest thing about it was that, in addition to allowing you to communicate with just words, it allowed you to sing. It allowed you to intone your words with meaning beyond the simple linguistics.

We’re in the same exact parallel situation today. Some people are saying we need to put an IBM PC on every desk in America to improve productivity. But it won’t work. The special incantations you have to learn this time are slash-qz’s and things like that. Most people are not going to learn slash-qz’s any more than they’re going to learn Morse code.

And that’s what Macintosh is all about. It’s the first “telephone” of our industry. But the neatest thing about it to me is, the same as the telephone to the telegraph, Macintosh lets you sing. It lets you use special fonts. It lets you make drawings and pictures or incorporate other people’s drawings or pictures into your documents.

Even in business, you’re seeing five-page memos get compressed down to a one-page memo because there’s a picture to express the key concept. And so we’re seeing less paper flying around and more quality of communication.

And it’s more fun. There’s always been this myth that really neat, fun people at home all of [a] sudden get very dull and boring and serious when they come to work, and it’s simply not true. So if we can again inject that liberal-arts spirit into this very serious realm of business, I think it would be a worthwhile contribution.

Viewed through a conference crowd, Steve - his clean-shaven, grinning face - is displayed on a large screen.
Apple sales conference in Hawaii, 1983.
Steve, wearing a black leather jacket, holds his middle finger up toward a large IBM sign.
Outside the IBM offices in New York.

Speech to Apple Employees

“Was George Orwell right about 1984?”

Steve introduced the Macintosh and its iconic commercial, which ran during the 1984 Super Bowl, at an Apple sales meeting in October 1983.

Hi, I’m Steve Jobs.

It is 1958. IBM passes up the chance to buy a young, fledgling company that has invented a new technology called xerography. Two years later, Xerox is born. And IBM has been kicking themselves ever since.

It is ten years later, the late sixties. Digital Equipment [DEC] and others invent the minicomputer. IBM dismisses the minicomputer as too small to do serious computing, and therefore unimportant to their business. DEC grows to become a multi-hundred-million-dollar corporation before IBM finally enters the minicomputer market.

It is now ten years later, the late seventies. In 1977, Apple, a young, fledgling company on the West Coast, invents the Apple II, the first personal computer as we know it today. IBM dismisses the personal computer as too small to do serious computing and unimportant to their business.

The early eighties, ’81. Apple II has become the world’s most popular computer. Apple has grown to a $300 million company, becoming the fastest-growing corporation in American business history, with over fifty competitors vying for a share. IBM enters the personal-computer market in November ’81 with the IBM PC.

1983. Apple and IBM emerge as the industry’s strongest competitors, each selling approximately one billion dollars’ worth of personal computers in 1983. Each will invest greater than $50 million for R&D and another $50 million for television advertising in 1984, totaling almost one quarter of a billion dollars combined.

The shakeout is in full swing. The first major firm goes bankrupt, with others teetering on the brink. Total industry losses for ’83 outshadow even the combined profits of Apple and IBM for personal computers.

It is now 1984. It appears IBM wants it all. Apple is perceived to be the only hope to offer IBM a run for its money. Dealers initially welcoming IBM with open arms now fear an IBM-dominated and controlled future. They are increasingly and desperately turning back to Apple as the only force that can ensure their future freedom.

IBM wants it all, and is aiming its guns on its last obstacle to industry control: Apple. Will Big Blue dominate the entire computer industry? [Audience: No!] The entire information age? [Audience: No!] Was George Orwell right about 1984?

[Steve runs the “1984” commercial. Directed by Ridley Scott, the ad depicts a dystopian, Orwellian world. In one scene, people dressed in gray, their heads shaved, sit expressionless in front of a large screen on which a dictator drones nonsense. A woman in bright red running shorts and a Macintosh shirt bursts into the room. She hurls a hammer at the screen, destroying it. The commercial ends with a promise: “On January 24th, Apple Computer will introduce Macintosh. And you’ll see why 1984 won’t be like ‘1984.’”]

[There is tumultuous applause and shouting from the audience; once it dies down, Steve resumes.] That ad is going to run one week before Macintosh is introduced. And our ad agency that put it together is here today, Chiat/Day. Jay Chiat is here, the principal. Lee Clow and Steve Hayden, [who] wrote the copy and did the creative, are also here.

You might— I guess they just heard what you thought.

A Polaroid of Steve, playing pinball, with a white hooded sweatshirt with the word Mackintosh on the chest.
Enjoying a rare break before the Macintosh release.
Steve is fast asleep on a flight, his seat slightly reclined.
Exhausted after a sales conference in New Orleans, 1984.
29-year-old Steve wearing a black turtleneck and jeans, poses with a Macintosh computer resting in his lap.
Steve’s 1984 shoot for Time at his Woodside home.
Steve, silhouetted, is bent over as he observes a curly-haired woman typing on a Mac in a storefront.
Pausing to watch a Macintosh in action.

Interview with Michael Moritz

“Your aesthetics get better as you make mistakes.”

Steve and Michael Moritz, a reporter who would soon switch careers and become a venture capitalist, spoke at Steve’s office at Apple in May 1984. They covered a wide range of topics, including Steve’s thoughts on product design.

Steve Jobs: I went around and looked at Cuisinarts when we were designing Mac. It was like my Cuisinart week.

Michael Moritz: But no other particular products [influenced you]? Say, from the late seventies or something.

SJ: Well, we’re around automobiles our whole lives. I’ve never been a car guy, but I’ve always loved Volkswagen Beetles. I’ve always loved Volkswagen vans, actually, too.

Just a bunch of little things: wine labels, paintings in galleries. Just simple things. Not anything real profound, just lots and lots of little things. I don’t think my taste in aesthetics is that much different than a lot of other people’s. The difference is that I just get to be really stubborn about making things as good as we all know they can be. That’s the only difference.

MM: Yeah, I think you’re being modest.

SJ: Well, things get more refined as you make mistakes. I’ve had a chance to make a lot of mistakes. Your aesthetics get better as you make mistakes. But the real big thing is: if you’re going to make something, it doesn’t take any more energy— and rarely does it take more money— to make it really great. All it takes is a little more time. Not that much more. And a willingness to do so, a willingness to persevere until it’s really great.

But aesthetics? I think aesthetics are a lot like singing. Joanie [Baez] has a beautiful voice, but the reason her voice is beautiful isn’t because her voice is just beautiful. It’s because she has an incredibly good ear. She can listen to somebody speak for thirty seconds and imitate their voice almost perfectly. Her ear is superb. And I think, in the same way, good aesthetics result from just your eye. An instinct of what you see, not so much what you do. ✂

SJ: I want to build products that are inherently smaller than any of the products on the market today. And when you make things smaller, you have the ability to make them more precisely. Obviously, a perfect example of that is a watch. It’s beautiful, but the precision has to be the scale of the object itself, and so you make it very precise. And as our products get smaller, we have the opportunity to do that. So, obviously, I would like everything to be smaller.

I also think that it’s really nice to be able to carry products around. Even if they’re not portable, it’s very nice to be able to have a handle on them that says, “Pick me up and move me when you want to change where I am.” Carry them from room to room, or from office to office. Lisa’s too heavy to carry from office to office, or room to room, or home on the weekends. So the question is, “How do we find a way to package that same functionality into something that we can carry around with us and that is smaller, obviously— and be able to express the form of that more precisely?” That’s where we’re going in the future, those directions. ✂

MM: What are the uglier, offensive designs of products, or are there just too many to list?

SJ: Yeah— pick any car before three years ago, you know? Pick most cars today. Anything. Just look around the room. Tables, chairs: all ugly. You can ask me, what am I doing in this office? But anyway, most things are not very nice.

The telephone’s a perfect example. The only telephone that’s ever been any good is the original one and the Trimline. The Trimline is the only decent one. What they’ve done to the new stuff is just garbage. ✂

SJ: [At Apple] we’re just getting simpler and simpler and simpler. Very, very simple. Simple. ✂

SJ: Have you ever seen HP’s buildings over on Page Mill Road, the original ones? They’re really neat. They’ve got these scalloped roofs, and they face the glass north, and you can actually put solar collectors on them, if you wanted to. They stick out. In a building, they make a whole glass wall. And so people work down there, and they get tons of natural light coming in. Just tons.

The problem with these buildings [at Apple] is there’s no light. I mean, you spend five minutes outside, and you walk in here, it’s really dark, and you can’t see anything. And we’re all sort of like living in these little tiny caverns.

I just want a ton of natural light. ✂

MM: Was all the stuff about Big Brother [in the “1984” commercial] very conscious of the IBM stuff, or was that something that outsiders quickly interpreted, and you guys weren’t exactly going to—

SJ: Deny it?

MM: Deny it.

SJ: Well, the best response to that was the response given, I think, in Fortune , which was, “If seeing Big Brother in 1984 connotes IBM to a large number of people, that says more about IBM’s image problem than our intentions.”

In truth: of course, we saw the analogy. And I think that we were saying two things. I think the first thing we were saying was, this image of computers as sort of a centralized group of people having control of very powerful machines to keep track of us, that iconic fear in our minds— we were commenting on that cultural fear that we have.

And of course, one couldn’t— you’d have to be an idiot not to see the parallels to IBM.

A glamorous magazine shot shows Steve, wearing a double-breasted blazer, with arms crossed and hair blowing in the wind.
Under his double-breasted suit coat, Steve wore jeans.

Interview with Newsweek

“I want to build things.”

Steve left Apple in September 1985, after losing a power struggle with CEO John Sculley. The departure was officially a resignation, but Steve considered it a betrayal. A few weeks later, he spoke to Newsweek .

Newsweek: How did you react when you heard the [Apple] board’s decision [to sue you]? These were people that you knew and worked with for a long time.

Steve Jobs: Oh, yeah. I mean, in my wildest imagination, I couldn’t have come up with such a wild ending to all of this. I had hoped that my life would take on the quality of an interesting tapestry where I would have weaved in and out of Apple: I would have been there a period of time, and maybe I would have gone off and done something else to contribute, but connected with Apple, and then maybe come back and stay for a lengthy time period, and then go off and do something else. But it’s just not going to work out that way. So I had ten of the best years of my life, you know. And I don’t regret much of anything. ✂

I personally, man, I want to build things. I’m thirty. I’m not ready to be an industry pundit. I got three offers to be a professor during this summer, and I told all of the universities that I thought I would be an awful professor. What I’m best at doing is finding a group of talented people and making things with them. I respect the direction that Apple is going in. But for me personally, you know, I want to make things. And if there’s no place for me to make things there, then I’ll do what I did twice before. I’ll make my own place. You know, I did it in the garage when Apple started, and I did it in the metaphorical garage when Mac started. ✂

SJ: Though the outside world looks at success from a numerical point of view, my yardstick might be quite different than that. My yardstick may be how every computer that’s designed from here on out will have to be at least as good as a Macintosh. ✂

SJ: I used to go into work, I’d get there, and I would have one or two phone calls to perform, a little bit of mail to look at. But— this was in June, July— most of the corporate management reports stopped flowing by my desk. A few people might see my car in the parking lot and come over and commiserate.

And I would get depressed and go home in three or four hours, really depressed. I did that a few times, and I decided that was mentally unhealthy. So I just stopped going in. You know, there was nobody really there to miss me.

Q: Do you feel that they have taken your company away from you?

SJ: To me, Apple exists in the spirit of the people that work there, and the sort of philosophies and purpose by which they go about their business. So if Apple just becomes a place where computers are a commodity item and where the romance is gone, and where people forget that computers are the most incredible invention that man has ever invented, then I’ll feel I have lost Apple. But if I’m a million miles away and all those people still feel those things and they’re still working to make the next great personal computer, then I will feel that my genes are still in there. ✂

SJ: One of the five most difficult days was that day John [Sculley, Apple’s CEO] said at the analysts meeting about there not being a role for me in the future, and he said it again in another analysts meeting a week later. He didn’t say it to me directly; he said it to the press. You’ve probably had somebody punch you in the stomach and it knocks the wind out of you and you can’t breathe. If you relax, you’ll start breathing again. That’s how I felt all summer long. The thing I had to do was try to relax. It was hard. But I went for a lot of long walks in the woods and didn’t really talk to a lot of people. ✂

Q: You’ve talked about being tough to get along with, having a rough-edge personality. Did you contribute in some way to your own downfall?

SJ: You know, I’m not a sixty-two-year-old statesman that’s traveled around the world all his life. So I’m sure that there was a situation when I was twenty-five that if I could go back, knowing what I know now, I could have handled much better. And I’m sure I’ll be able to say the same thing when I’m thirty-five about the situation in 1985. I can be very intense in my convictions. And I don’t know— all in all, I kind of like myself, and I’m not that anxious to change.

Q: But has this experience changed you?

SJ: Oh, this has— yeah, I think I am growing from this, and I think I’m learning a lot from it. I’m not sure how or what yet. But yes, I feel that way. I’m not bitter; I’m not bitter. ✂

Q: There’s been a lot in the press about your interest in Buddhism, vegetarianism.

SJ: As we descend into the isms.

Q: The isms. Are you still interested in those things?

SJ: Well, I don’t know what to say. I mean, I don’t eat meat, and I don’t go to church every Sunday.

Q: They said at some point you had thought of going to Japan and sitting in a monastery.

SJ: Yeah, yeah. I’m glad I didn’t do that. I know this is going to sound really, really corny. But I feel like I’m an American, and I was born here. And the fate of the world is in America’s hands right now. I really feel that. And you know, I’m going to live my life here and do what I can to help.

Steve, at home, is dressed in jeans and a sweater. He holds a mug and stares pensively at the camera from his leather sofa.
Steve circa 1985.

Part II, 1985–1996

“You never achieve what you want without falling on your face a few times.”

The years after Steve left Apple were among the toughest of his career— and the most formative.

Determined to build a new great computer company, he started NeXT with several members of the Macintosh team. “We’ll make a whole bunch of mistakes, but at least they’ll be new and creative ones,” he predicted.

Around the same time, Steve invested $10 million in a small company called Pixar. It was a tiny computer graphics operation, newly spun off from filmmaker George Lucas’s empire. The technical expertise at Pixar attracted Steve; its initial product was a high-end graphics computer that cost more than $100,000.

Both NeXT and Pixar quickly ran into trouble. The NeXT computer system, which debuted in 1988, was powerful and packed with the humanistic touches Steve loved. It was visually striking and intuitive to use, with high-quality audio and the complete works of Shakespeare built in. But it was also late to market and expensive— and it sold poorly. Within six years of NeXT’s launch, the entire founding team, other than Steve, had resigned.

Pixar, meanwhile, was eking out an existence selling computers and software and, later, animating commercials. The company was also making award-winning short films that charmed Steve. This use of technology in service of brilliant storytelling embodied one of his favorite things: work at the intersection of technology and the liberal arts. The short films fired Steve’s enthusiasm and kept him writing check after check to Pixar, ultimately investing some $60 million.

But the films were, as Steve put it, “in the background,” not the company’s focus. He described Pixar’s early business strategy as “find a way to pay the bills,” and he later speculated that the only reason the company didn’t fall apart then was that the leadership team “would all get depressed … but not all of us at once.”

If at times in these years he seemed disappointed by the possibilities of technology— “this stuff doesn’t change the world. It really doesn’t,” he told a reporter in an uncharacteristic flash of pessimism— his world was also expanding beyond his work. He treasured his privacy, saying of his public persona, “I think of it as my well-known twin brother. It’s not me.”

Steve learned how to hone a company to its essence, even when it was painful. He shifted NeXT’s focus to selling software. The shift meant closing a factory and laying off more than two hundred of NeXT’s five hundred and thirty employees. Meanwhile, Pixar stripped away its advertising and hardware businesses and entered into an agreement with Disney, all to pursue what sometimes seemed an impossible dream: to make fully computer-animated feature films.

After nearly a decade of difficulty, the streamlined NeXT and Pixar both transformed into unlikely success stories. At the end of 1995, Pixar premiered Toy Story in the same month it held its initial public offering. A year later, Apple, in need of operating-system software, bought NeXT for $427 million. “If you really look closely,” Steve liked to say, “most overnight successes took a long time.”

Steve on Starting NeXT

A few weeks after starting NeXT, Steve spoke with Newsweek about the original insight for the company.

I had been reading some biochemistry, recombinant DNA literature. [I had recently met] Paul Berg, the inventor of some of the recombinant techniques. I called him up, and I said, “You remember me. I’m ignorant about this stuff, but I’ve got a bunch of questions about how it works, and I’d love to have lunch with you.” So we had lunch at Stanford. He was showing me how they were doing gene repairing. Actually, it’s straightforward, it’s kind of neat. It smells a lot like some of the concepts you find in computer science. So he was explaining how he does experiments in a wet laboratory and they take a week or two or three to run. I asked him, “Why don’t you simulate these on a computer? Not only will it allow you to run your experiments faster, but someday every freshman microbiology student in the country can play with the Paul Berg recombinant software.” So his eyes lit up. And that was sort of a landmark lunch. Because that’s when I started to really think about this stuff and get my wheels turning again.

On Becoming Majority Shareholder in Pixar

In 1996, the year after Pixar released Toy Story and held its IPO, Steve looked back on the work and ideas that first drew him to the company.

I met Ed Catmull, who was running the computer division of Lucasfilm, in 1985. […] I’d been involved in graphics most of my life. The Apple II, though most people don’t remember, was the first real color computer that you could get your hands on. The Macintosh, obviously, was graphics. The LaserWriter was graphics. But it was all 2-D . We’d done some 3-D work at Apple, and I was certainly aware of the field— but the stuff that Ed and his team were doing was way ahead of anything I’d ever seen anyone do.

On Pixar’s Early Days

In 2003, with Disney and Pixar deep in negotiations over the future of the studio, Steve told filmmaker Leslie Iwerks about Pixar’s start.

Our strategy in the early days of Pixar was: find a way to pay the bills. In the background, we were developing animation software, and John [Lasseter] was making the succession of short films on the way to Toy Story . But we were trying to pay the bills and just buy time. That strategy really turned out not to work. Probably if you look back in the rearview mirror, we would have been better off just funding the animation efforts and not trying to pay the bills through these other products, such as the Pixar Image Computer and software, but that was our best attempt to try to keep the company going. In the end, I just ended up writing checks to keep the company going— and that basically went on for ten years. ✂

You could see there was magic in [a Pixar animated short film], right from the beginning. With the rest of Pixar’s technology, you had to be an expert to understand it. [… But] you didn’t have to know anything about the technology to enjoy the film. It was incredibly refreshing and really pointed the way to where we wanted to go. We didn’t want to have to convince people that our technology was great— we knew it was great. We wanted to use our technology to make something where nobody needed to know anything about the technology to love it. And that’s what we ended up doing.

Steve gazes into the camera as he cradles his chin in his right hand, his left hand resting on top of a newspaper on the table.
Shortly before becoming the majority shareholder of Pixar.
Steve, in a white shirt, writes on a whiteboard. He has a look of concentration and is sticking out his tongue.
NeXT planned to capture the education market.
A memo from Steve on the NeXT letterhead proposes meeting-free Thursdays and warns against managerial bureaucracy.
NeXT’s ban on Thursday meetings was short-lived.
Steve sits in a chair, hand cradling his head, and stares at a life-size pink foam NeXT computer prototype.
Reviewing a Styrofoam model of the NeXT machine, 1987.
Steve kicks a beach ball in a grassy park. He is dressed in ripped jeans and a patterned shirt with rolled-up sleeves.
Taking a break at a NeXT company picnic in Menlo Park.

Email to NeXT Employees

“Pixar led the way!”

From: Steve Jobs

To: NeXT

Subject: A great day and a half

Date: March 30, 1989, 8:08 p.m.

Well, we did it again. Congrats to all for a real TEAM effort. And, Pixar did it too.

For those of you who didn’t see the Academy Awards last night, Pixar won an award in the category of short animated films: for their computer generated film Tin Toy

Tin Toy is the first computer generated film to ever win an award, and was competing against several very good traditionally animated (non-computer-animated) films!

The computer graphics industry just achieved a major milestone, and Pixar led the way!

Speech at Reed College

“Character is built not in good times, but in bad times.”

When Steve welcomed the incoming class of freshmen at his alma mater, Reed College, on August 27, 1991, NeXT computers were not selling well and Pixar had just had a round of layoffs.

Thank you very much for this. It means a lot to me. I’m a peculiar Reed alumnus, as many of you know. I never graduated from Reed— although that doesn’t make me that unusual, I suppose.

But maybe more unusual: I ran out of money after one semester here at Reed, so I dropped out. But then I dropped in for another year and a half. So, I was actually here by choice, which is somewhat more unusual. And I had some experiences here— that I’m sure many of you will have as freshmen and throughout your years here— that have stayed with me my whole life. I was thinking of some of them to recount to you.

Remember that I’m much older than you now. I’ve always thought that people’s spark of self-consciousness turns on at about fifteen or sixteen. So if we normalize age to fifteen or sixteen, then most of you are two or three or four years old here, as freshmen. I’m about twenty. So that maybe puts in perspective what it’s like to return to Reed after so many years. But a few things stick in my mind that I wanted to pass on that maybe could be of some value. The first was that, as you will be shortly, I was forced to go to humanities lectures— it seemed like every day. I studied Shakespeare with Professor Svitavsky. And at the time, I thought these were meaningless and even somewhat cruel endeavors to be put through. I can assure you that as the patina of time takes its toll, I thank God that I had these experiences here. It has helped me in everything I’ve ever done, although I wouldn’t have ever guessed it at the time.

The second experience that I remember from Reed is being hungry. All the time. The cafeteria here taught me quickly to be a vegetarian. I didn’t have so much money, so I would gather up Coke bottles and take them up to the store to find out how to eat. I discovered the cheapest way to eat was Roman Meal. Have you ever heard of this? It’s cereal. It was invented by a Harvard professor who was a history professor who one day wondered what the Roman legion took with them to eat as they conquered and pillaged these villages, and he found out through his research that it’s Roman Meal. And you can buy it at the local store, and it’s the cheapest way to live. So I lived for many months on Roman Meal.

But also, several of us, after not eating for a few days, would hitchhike across town to the Hare Krishna temple on Sundays, where they would feed all comers. Through practice, we discovered just the right moment to arrive— after their particular religious practices and right before the food. And not having eaten for days, we would eat a lot, and on several occasions stay over, because we were not able to move.

The following morning, they would wake us up at four o’clock in the morning because it was their time to go gather flowers for their temple to honor Krishna. So they would take us with them, predawn, out into the neighborhood— where they would proceed to steal flowers from the neighbors. And the neighbors that lived close to the Hare Krishna temple soon were wise to their pillage and would get up early in the morning and guard their flower beds. And so they would have to go in an ever-wider circumference around their temple. In spending a little time with these people, I noticed some of their other behaviors. They used to sell incense to the local department stores and then go steal it back, so that the department stores would buy more, and they would have a thriving business. And their ethics told them that this was fine, that anything in the service of Krishna was fine. In interacting with them, I think I learned more about situational ethics than I ever did on campus.

The last experience I wanted to recount for you: there is a man— I think he’s here today— named Jack Dudman, who used to be the dean of the school. He was one of the heroes of my life while I was here, because Jack Dudman looked the other way when I was staying on campus without paying. He looked the other way when I was taking classes without being a formal student and paying the tuition. And oftentimes, when I was at the end of my rope, Jack would go for a walk with me, and I would discover a twenty-dollar bill in my tattered coat pocket after that walk, with no mention of it from Jack before, during, or after.

I learned more about generosity from Jack Dudman and the people here at this school than I learned anywhere else in my life. So I wanted to thank this community, because the things I learned here stayed with me. Character is built not in good times, but in bad times; not in a time of plenty, but in a time of adversity— and this school seems to manage to nurture that spirit of adversity, and I think does build some character. So I thank you for teaching me how to be hungry and how to keep that with me my whole life.

Thank you very much.

A note on Pixar letterhead. The handwritten text reads: Steve, President Clinton is holding.
Steve sometimes exchanged calls with President Bill Clinton.
Steve, now wearing glasses, sits cross-legged in a director's chair under an oversized anglepoise Pixar-style lamp.
In 1995, the year Toy Story was released and Pixar had its IPO.
Steve, smiling, poses with Mickey and Minnie Mouse at Disneyland.
In 1995, around the time of the Pixar-Disney deal renegotiation.

Email Exchange Between Steve, Intel CEO Andy Grove, and an Intel Engineer

“I have changed my position 180 degrees.”

As Pixar became a leader in graphics, Steve and his mentor, Intel CEO Andy Grove, discussed how Intel might learn from Pixar. When an Intel engineer tried to follow up, Steve resisted.

From: [Engineer 1 at Intel]

To: Steve Jobs

Cc: Andy Grove

Subject: Pixar-3D graphics

Date: September 22, 1995, 2:04 p.m.

Steve,

Andy asked me to look into what we should do in dramatically improving the Intel architecture platform’s 3D graphics performance. He indicated that you and key people at Pixar like Ed Catmull have lots of good ideas on what we should do in this area. I actually contacted Ed several months ago but he was real busy and cannot commit to meeting until after Sept. As you know, I am in charge of microprocessors at Intel.

I have located several key Intel 3D experts. One of them, [Engineer 2], came from Sun over a year ago. I would like to have a meeting (at Next, Intel or Pixar) with you, Catmull, and others with me and our graphics experts to discuss your ideas and map out what our action plans are. I am in Tokyo next week but will be back in my office on Monday Oct. 2. I will ask my admin to contact your office to set that meeting up.

Thanks.

[Engineer 1]

From: Steve Jobs

To: [Engineer 1]

Subject: Pixar-3D graphics

Date: September 23, 1995, 7:11 p.m.

[Engineer 1],

Pixar does indeed possess the knowledge to enable Intel’s processors to render 3D graphics at much high performance and quality. These “secrets” could definitely make their way into future Intel general purpose processor hardware.

We believe this single capability is the key for Intel to dramatically enlarge the PCs market share in the consumer market – by significantly surpassing the dedicated gaming machines (Sega, Nintendo, Playstation, etc) graphics capabilities.

Pixar’s secrets were invented through significant investment over ten years or more, and we value them highly. Even without the secrets implemented in the processor, Pixar can gain significant competitive advantage and differentiation through implementing them in software. By disclosing the “correct” way to do high quality, high performance graphics, Pixar will lose much of this to any and all competitors, with no work on their part. Hence, the need for compensation.

What does Intel propose to give Pixar for disclosing and licensing its secrets to Intel?

Steve

From: [Engineer 1]

To: Steve Jobs

Cc: Andy Grove

Subject: Re: Pixar-3D graphics

Date: September 25, 1995, 11:22 a.m.

Steve,

We would very much like to have our meeting, but I will put that on hold based on your input. We talked to many key people on ideas to improve the microprocessor capability with the aim that this will benefit the whole industry, and everyone will benefit. We have not entered into any financial arrangement in exchange for good ideas for our microprocessors in the past and have no intention for the future.

[Engineer 1]

From: Steve Jobs

To: [Engineer 1]

Cc: Andy Grove

Subject: Re: Pixar-3D graphics

Date: September 25, 1995, 5:29 p.m.

This approach has not served you well in the past, as evidenced by your poor graphics architectures and performance. Maybe you should think of changing it for the future…

Steve

From: Steve Jobs

To: Andy Grove

Subject: Re: Pixar-3D graphics

Date: September 25, 1995, 10:27 p.m.

Andy,

Maybe it’s just me, but I find [Engineer 1]’s approach extremely arrogant, given Intel’s (his?) dismal showing in understanding computer graphics architectural issues in the past…

If I were going to make hundreds of millions of something, I sure as hell would be willing to pay for the best advice money could buy… Any[way], this isn’t a sales pitch; I just wanted you to know what I thought, as always.

Best,

Steve

From: Andy Grove

To: Steve Jobs

Subject: Re[2]: Pixar-3D graphics

Date: September 26, 1995, 3:12 p.m.

Steve,

I am firmly on [Engineer 1]’s side on this one. He is taking your offer to help us very seriously, rounded up the best technical people and was ready to go when you introduced a brand new element into the discussion: money.

You and I have talked many times about this subject; you never suggested or hinted at this being a commercial exchange. I took your offer to help us exactly as that: help, not an offer of a commercial relationship.

You may remember, that from time to time I offered suggestions that pertained to your business. Examples range from porting NextStep to the 486 - - which was in our interest too - - to my presentation to your staff on repositioning NextStep beyond that. I am not suggesting that these are comparable in value to your expertise in graphics, but I gave what I had, put some thought into the problem I saw you were facing - - and it never entered my mind to charge for it. In my view, that’s what friendly companies (and friends) do for each other. In the long run, these things balance out.

I am sorry you don’t feel that way. We will be worse off as a result, and so will the industry.

Regards,

a

From: Steve Jobs

To: Andy Grove

Subject: Re[2]: Pixar-3D graphics

Date: October 1, 1995, 3:50 p.m.

Andy,

I have many faults, but one of them is not ingratitude. And, I do agree with you that “In the long run, these things balance out.”

Therefore, I have changed my position 180 degrees - - we will freely help [Engineer 1] make his processors much better for 3D graphics. Please ask [Engineer 1] to call me, and we will arrange for a meeting as soon as the appropriate Pixar technical folks can be freed up from the film.

Thanks for the clearer perspective.

Steve

Email to Pixar Employees

“To $175M and beyond!”

From: Steve Jobs

To: Pixar

Subject: Toy Story Crosses $150M !!

Date: January 2, 1996, 9:30 a.m.

Yesterday Toy Story crossed $150M in domestic box office receipts -- only 13 days after crossing $100M !!

Toy Story was the undisputed blockbuster of the holiday season, topping all other films including Jumanji, Father of the Bride II, Waiting to Exhale and, of course, Balto. Toy Story may even surpass the current 1995 box office leaders, Batman Forever ($184M) and Apollo 13 ($172M), and become the most successful film of 1995.

Toy Story has already become the third most successful animated feature film of all time, topping all of Disney’s classics except “Aladdin” ($217M) and “The Lion King” ($312M).

To $175M and beyond!

Congratulations team,

Steve

Interview with Terry Gross

“One of the things I always tried to coach myself on was not being afraid to fail.”

In 1996, Steve and the Fresh Air radio host Terry Gross looked back on his time at Apple and ahead to the future of the computer industry. They spoke shortly after Pixar’s release of Toy Story and its successful initial public offering. NeXT, meanwhile, continued to struggle.

Terry Gross: How do you think that the web might change in the near future with the help of the type of software that you are producing now?

Steve Jobs: I think most large companies and medium-size companies (and even small companies) are starting to look at the web as the ultimate direct-to-customer distribution chain, bypassing all middlemen, going directly from the supplier to the consumer. That’s a pretty powerful concept when you think about it. One of the things that I love is that a very small company, if they invest a lot in their website, can look just as formidable and just as solid on the web as a very large company can. As a matter of fact, some of the smaller companies are more hip on the web, getting more hip to the web sooner, and so they actually look better than some of the large companies do right now. It’s going to be this very leveling phenomenon, but I think a tremendous amount of goods and services is going to be sold, or at least the demand created for such things, over the web.

TG: What else do you see in the near future for the web, besides the ability to shop in a more kind of complete way through the web?

SJ: It’s not just shopping for goods and services. It’s shopping for information. I mean, you’re going to find out … Already, when I want to find out the movies that are playing around Silicon Valley, I just go up on the local web page and check it out. It’s a lot faster than going through the newspaper, and a lot faster than calling the theaters, et cetera. More and more, we’re shopping for information on the web. I just recently bought a Sony, one of the new Sony camcorders. I went on Sony’s web page, and I found out all about the ones they offer and picked the one I wanted right from that web page before I even called the store to try to find it physically. The demand to get me to buy that thing was created from Sony’s web page. I think we’re going to see more and more of that. You’re going to be buying information or finding information, and really making a lot of decisions about what you’re going to do with your life, or what you’re going to purchase, from the web.

TG: Is the whole idea of going to the store to buy software going to become obsolete, too? Do you think we’ll be downloading our software from the web?

SJ: Of course. Yeah, there’s no question about it. There’s no question that that will happen, and I think it will happen in the next twenty-four months. There’s some software right now that’s still very large. The web on-ramps and off-ramps to corporations are now very fast, but the off-ramps to the consumers’ homes are still not so fast. For buying large software, such as CD-ROM games and stuff, they’ll still be distributed on physical media for a while, but when the off-ramps to the consumer get faster, possibly with cable modems in the near future, then that could possibly go fully electronic as well.

TG: Tell me what else you see for the web beyond the world of retail.

SJ: There’s a lot of things happening with the web right now, in terms of allowing people access to information that they would just never have before. What this does is, of course, it lets special-interest groups get together. I know people who have had, as an example, a stroke, and have gotten on the web and found that there are several web pages now devoted to information for stroke victims where they can learn about some of the latest treatments. They can learn about avoidance, the latest in avoidance advice, and things like that. Those things didn’t exist before, as well. ✂

TG: Do you think that when you were ousted from Apple that people kind of wrote you off? I mean, here you are with these big successes now.

SJ: Oh golly, I don’t know. I’m sure that a lot of people did, and that was fine. It was a very painful time, as you might imagine.

TG: What, to be forced out of the company you created?

SJ: Oh, of course. That was a very painful time, but you just march forward, and you try to learn from it. One of the things I always tried to coach myself on was not being afraid to fail. When you have something that doesn’t work out, a lot of times, people’s reaction is to get very protective about never wanting to fall on their face again. I think that’s a big mistake, because you never achieve what you want without falling on your face a few times in the process of getting there. I’ve tried to not be afraid to fail, and, matter of fact, I’ve failed quite a bit since leaving Apple.

TG: Are you surprised at the problems Apple is having now, or did you see that coming?

SJ: I try not to talk about Apple too much. What I will say is that the day I left Apple, we had a ten-year lead over Microsoft. In the technology business, a ten-year lead is really hard to come by. It happens, maybe a company has that once every few decades, whether it be Xerox or IBM with mainframes. Apple had that with the graphical user interface. The problem at Apple was that they stopped innovating. If you look at the Mac that ships today, it’s 25 percent different than the day I left, and that’s not enough for ten years and billions of dollars in R&D.

It wasn’t that Microsoft was so brilliant or clever in copying the Mac. It’s that the Mac was a sitting duck for ten years. That’s Apple’s problem, is that their differentiation evaporated. Unlike Compaq, or others who play in the Intel-Microsoft standard space, where they only … Compaq only has to be 5 percent better than its competitors for everyone to want to buy their computers.

Apple has to be 50 percent or 100 percent better, because when you buy something that is out of the mainstream a little bit, you take a risk, and you want a much bigger reward for taking that risk. […] That differentiation has not completely evaporated, but for the most part it has. That’s the predicament Apple’s in now. That’s why cost-cutting and other things at Apple are not going to be the cure. The cure for Apple is to innovate its way out of its current predicament. There’s a lot of good people left at Apple that are capable of doing that with the proper leadership, which is what’s been missing.

TG: Some Mac users are afraid that the Mac operating system is in danger of becoming obsolete in the way that Beta video became obsolete because it was outdone by VHS. What do you think?

SJ: I think with the appropriate leadership at Apple, that’s not going to happen, but I think we have to wait and see.

TG: Do you care? How still involved, invested, do you feel in the future of Apple, the company you co-created?

SJ: I’m happy every time a Mac gets shipped. I still have people sending me emails, telling me how much they love their Macs. It’s sort of … how do you explain it? It’s like the first person you were ever in love with. You know? It’s like your first love, and there will never be another one like it. In my case, we were together for ten years, and that’s a long time. But if you move on in your life, you can’t always stay in love with your first girlfriend. Right? ✂

TG: What do you think the state of the computer would be if it weren’t for Apple? This is a chance, I guess, for a really self-serving answer. But, I mean, I’m really curious what you think.

SJ: I usually believe that if one group of people didn’t do something, within a certain number of years, the times would produce another group of people that would accomplish similar things. We happened to be at the right place, at exactly the right time, with the right group of people. We did some wonderful work. I’m extraordinarily proud of the work that the team at Apple did when I was there. I think that, personally, our major contribution was a little different than some people might think. I think our major contribution was in bringing a liberal arts point of view to the use of computers.

TG: Yeah, explain what you mean by that.

SJ: What I mean by that is that if you really look at the ease of use of the Macintosh, the driving motivation behind that was to bring— not only ease of use to people so that many, many more people could use computers for nontraditional things at that time— but it was to bring beautiful fonts and typography to people. It was to bring graphics to people, not for plotting laminar flow calculations, but so that they could see beautiful photographs, or pictures, or artwork, et cetera, to help them communicate what they were doing, potentially. Our goal was to bring a liberal arts perspective and a liberal arts audience to what had traditionally been a very geeky technology and a very geeky audience.

TG: What made you think that that more liberal arts direction was the direction to head in?

SJ: Because in my perspective, and the way I was raised, was that science and computer science is a liberal art. It’s something that everyone should know how to use, at least, and harness in their life. It’s not something that should be relegated to 5 percent of the population over in the corner. It’s something that everybody should be exposed to, everyone should have a mastery of, to some extent, and that’s how we viewed computation, or these computation devices.

TG: And you think that concept really caught on in the whole industry, eventually?

SJ: That’s the seed of Apple: computers for the rest of us. I think the liberal arts point of view still lives at Apple. I’m not so sure that it lives that many other places. I mean, one of the reasons I think Microsoft took ten years to copy the Mac was because they didn’t really get it at its core.

TG: Do you think the PC, as we know it, is on the road of changing?

SJ: That’s a really big question. I think the PC as we know it is going to be around for quite some time, but the heart of the question is, are we entering a time window where we might see the first successful post-PC devices? Personal digital assistants, or PDAs, attempted to be that and failed.

The next attempt, I think, is going to be these very low-cost consumer internet appliances. Can somebody make a three-hundred-dollar box that hooks up to your television on one side and maybe hooks up to ISDN or a cable modem on the other side and allows you for, three hundred dollars, to have a web browser on your TV and to access the entire internet? I think that’s entirely possible, and I think that we’re going to see those devices soon, hopefully some innovative marketing and distribution techniques surrounding those devices so that a lot of people can all of a sudden have an internet browser in their living room. I think that’s going to be very exciting, and I think that could be the beginning of the first real post-PC market.

TG: I know at Apple there was, at least early on, a very informal, non-corporate type of atmosphere. I wonder if there are any lessons you learned about what worked and didn’t work in the corporate lifestyle at Apple that you’ve applied to your current companies, NeXT and Pixar.

SJ: Well, I don’t know what a corporate lifestyle is. I mean, Apple was a corporation; we were very conscious of that. We were very driven to make money so that we could continue to invest in the things we loved. I would say Apple was a corporate lifestyle, but it had a few very big differences to other corporate lifestyles that I’d seen. The first one was a real belief that there wasn’t a hierarchy of ideas that mapped onto the hierarchy of the organization. In other words, great ideas could come from anywhere and that we better sort of treat people in a much more egalitarian sense, in terms of where the ideas came from.

And Apple was a very bottoms-up company when it came to a lot of its great ideas. And we hired truly great people and gave them the room to do great work. A lot of companies— I know it sounds crazy— but a lot of companies don’t do that. They hire people to tell them what to do. We hired people to tell us what to do. We figured we’re paying them all this money, their job is to figure out what to do and tell us. And that led to a very different corporate culture, and one that’s really much more collegial than hierarchical.

TG: In spite of this kind of different approach to the corporate hierarchy, it was probably still a very high-stress place.

SJ: Well, we were very young, and most of the folks were not married, and so they could work fifteen-hour days. You didn’t have a typical situation where you worked so that you can support your life. Your work was your life, in many cases.

TG: Right. Do you feel you’ve changed from that? Is that still your life?

SJ: I feel it is still my life, but it’s not all my life. It’s less of a percentage, but I still don’t really … I’ve never been able to think of my work and my life as different things. They’re the same thing. Where it used to be 99 percent of my life, it’s maybe 50 percent of my life now.

Email to NeXT Colleagues

“White House”

From: Steve Jobs

To: [NeXT Colleagues]

Subject: White House

Date: April 2, 1996, 4:17 p.m.

I am sending you this email from the White House !!

Steve

Steve’s vintage slide rule sits next to its brown leather case.
Steve’s slide rule. He was born before the electronic calculator.
Steve stands against the white garage door of the house where Apple started in 1976. The house number reads 2066.
1996: At his childhood home, where Apple was founded.
A typed page from a speech at Palo Alto High School, with handwritten notes strewn across the bottom in blue ink.
Steve’s notes on his speech at Palo Alto High School.

Speech at Palo Alto High School

“What you follow with your heart will indeed come back to make your life much richer.”

Steve spoke at the Palo Alto High School graduation in June 1996.

I have been invited here today to address you as you leave high school, and in most cases your parents, too, to venture out into the world on your own. I am supposed to offer you some wisdom and advice that you may remember along your travels.

I will address my remarks to you, the students, rather than to your parents. It is proper that I do so, being that the only wisdom I have comes from my advanced age; your parents are as old as I am, and much wiser, I am sure.

However, I am wiser than you, and maybe you will listen to me more than you listen to your parents. Some of your parents may not agree, or agree fully, with what I will say today. This is OK. I will simply be one of the first in your post-high-school life to fill your head with ideas that they disagree with. Wait until you get to college! But, in any event, if there is any discordance between what they have told you and what you hear from me today, rest assured that I am right.

Be aware of the world’s magical, mystical, and artistic sides. The most important things in life are not the goal-oriented, materialistic things that everyone and everything tries to convince you to strive for. Most of you know that deep inside. Think back on this spring— the last three or four months— when you are winding down high school, know where you are going next year, and begin to really have strong intuitions about the world you will encounter. Maybe you see an image of yourself in Paris, sculpting in an artist’s studio as the setting sun shines in the paned windows. Maybe you’re in India, running a hospital for poor children, and you hear the distant clatter of the outdoor marketplace in the early morning. Maybe you see yourself in a recording studio laying down a track for your album. Maybe you see yourself alone in a rented room at 4:30 in the morning being the only person alive to understand a new law of physics you just figured out.

Whatever it may be, I bet many of you have had some of these intuitive feelings about what you could do with your lives. These feelings are very real, and if nurtured can blossom into something wonderful and magical. A good way to remember these kinds of intuitive feelings is to walk alone near sunset— and spend a lot of time looking at the sky in general. We are never taught to listen to our intuitions, to develop and nurture our intuitions. But if you do pay attention to these subtle insights, you can make them come true.

People will come at you with reasons why you shouldn’t do these things:

You can’t make a living writing songs. (Right, just ask Bob Dylan.)

Helping children in India is nice, but you need to prepare for real life. (Just ask Mother Teresa.)

You could be doing so much more with your life. (You can hear Albert Einstein’s parents encouraging him to get a real job, when he was working a low-level job in the Swiss patent office rather than teaching in a university, so that he could stay up late at night working through his new ideas.)

If you don’t have any of these feelings, called dreams, then you’re in trouble. Before you “spend” four or more years of your life going in a direction your heart may or may not want you to go, you need to recapture them.

Be a creative person. Creativity equals connecting previously unrelated experiences and insights that others don’t see.

You have to have them to connect them. Creative people feel guilty that they are simply relaying what they “see.” How do you get a more diverse set of experiences? Not by traveling the same path as everyone else …

I’ll give you an example. The college I went to was a small liberal arts college in Portland, Oregon, named Reed College. It was, at that time, the center of a calligraphy revival movement in the US. I ended up taking a calligraphy course before I left college, and at the age of eighteen was exposed to a totally new world of typography, graphic layout, font design, and the like. There was no hope of earning any income from this skill or knowledge, and some of my friends derided me for wasting my time and talents on learning how to write with “fancy letters.”

However, years later, when we were designing the Macintosh, it was this very same experience and set of insights which drove me to insist that we find a way to use proportionally spaced type and offer a range of fonts— in essence, to bring a much richer world of typography to the computer world than had ever existed before. And this also led to the LaserWriter printer, so that one could print these letterforms with the quality they deserved. And this set the stage for “desktop publishing.” I tell you truly: none of this would have ever happened at Apple if I had sacrificed that calligraphy class for a more “substantive” class of economics or engineering.

So to be a creative person, you need to “feed” or “invest” in yourself by exploring uncharted paths that are outside the realm of your past experience. Seek out new dimensions of yourself— especially those that carry a romantic scent.

But one has no way of knowing which of these paths will lead anywhere in advance. That’s the wonderful thing about it, in a way. The only thing one can do is to believe that some of what you follow with your heart will indeed come back to make your life much richer. And it will. And you will gain an ever firmer trust in your instincts and intuition.

Don’t be a career. The enemy of most dreams and intuitions, and one of the most dangerous and stifling concepts ever invented by humans, is the “Career.” A career is a concept for how one is supposed to progress through stages during the training for and practicing of your working life.

There are some big problems here. First and foremost is the notion that your work is different and separate from the rest of your life. If you are passionate about your life and your work, this can’t be so. They will become more or less one. This is a much better way to live one’s life.

[The] risk factor quotient goes down as you encounter the real world. Many [people] find what they believe to be safe harbors (lawyers and accountants), only to wake up ten or fifteen years later and discover the price they paid.

Make your avocation your vocation. Make what you love your work.

The journey is the reward. People think that you’ve made it when you’ve gotten to the end of the rainbow and got the pot of gold. But they’re wrong. The reward is in the crossing the rainbow. That’s easy for me to say— I got the pot of gold (literally). But if you get to the pot of gold, you already know that that’s not the reward, and you go looking for another rainbow to cross.

Think of your life as a rainbow arcing across the horizon of this world. You appear, have a chance to blaze in the sky, then you disappear.

The two endpoints of everyone’s rainbow are birth and death. We all experience both completely alone. And yet, most people of your age have not thought about these events very much, much less even seen them in others. How many of you have seen the birth of another human? It is a miracle. And how many of you have witnessed the death of a human? It is a mystery beyond our comprehension. No human alive knows what happens to “us” upon or after our death. Some believe this, others that, but no one really knows at all. Again, most people of your age have not thought about these events very much, and it’s as if we shelter you from them, afraid that the thought of mortality will somehow wound you. For me it’s the opposite: to know my arc will fall makes me want to blaze while I am in the sky. Not for others, but for myself, for the trail I know I am leaving.

Now, as you live your arc across the sky, you want to have as few regrets as possible. Remember, regrets are different from mistakes. Mistakes are those things that you did and wish you could do over again. In some you were a fool (usually concerning women). In others you were scared. In others you hurt someone else. Some mistakes are deep, others not. But if your intent was pure, they are almost always enriching in some way. So mistakes are things that you did and wish you could do over again.

Regrets are most often things you didn’t do, and wish you did. I still regret not kissing Nancy Kinniman in high school. Who knows what might have happened? Maybe she regrets it too …

Interview about Pixar

“To put these stories into the culture … is a rare opportunity.”

On November 22, 1996, Steve discussed leading Pixar, his strategy for the studio’s future, and the privilege of creating stories for future generations.

Q: What kinds of things did you need to do, both personally (learning to become a filmmaker) and as a businessman, to put the company where it is today?

Steve Jobs: Well, Pixar is a studio. I’m not a filmmaker. I don’t direct our films. […] What I try to do is help create the environment where all these incredible people can make films. We’ve got a really unique thing in the industry, in that the very best creative people will only go to work at a few places: Disney, Pixar, possibly DreamWorks. In the same sense, the very best computer scientists in computer graphics will only go to work at a few places. Pixar is one of those, but most of the studios are not because they don’t have [our] level of technical culture there. I think Pixar is the only place in the world that can hire the best from both of these areas. And we’ve worked for ten years to figure out a way to have them all work together, which is not easy, because the Hollywood culture and the Silicon Valley culture are really different. We think we’ve picked the best from both. ✂

Q: I’ve heard that there are no contracts at Pixar, and that’s different than a lot of Hollywood productions. What’s the philosophy behind that?

SJ: In this blending of [cultures], one of the things that we encountered was that the Hollywood culture and the Silicon Valley culture each use different models of employee retention. Hollywood uses the stick, which is the contract. And Silicon Valley uses the carrot, which is the stock option. We examined both of those in really pretty great detail: economically, but also sort of psychologically and culture-wise. What kind of culture do you end up with?

And while there’s a lot of reasons to want to lock down your employees for the duration of a film, because if somebody leaves, you know you’re at risk, those same dangers exist in Silicon Valley. During an engineering project, you don’t want to lose people, and yet [Silicon Valley] managed to evolve another system other than contracts. And we prefer the Silicon Valley model in this case: give people stock in the company so that we all have the same goal, which is to create shareholder value.

But [not having contracts] also makes us constantly worry about making Pixar the greatest company we can, so that nobody would ever want to leave. When you sign a contract with somebody, you can sort of say, “Well, I don’t have to worry about that person for five years.” You know? And if you’re real sophisticated, you’ll have a little database that tickles you six months before their contract’s up so you can start paying them more attention. And they’re the most important person in the world for six months, and then after they sign up again, you put them in a drawer.

Our system’s a little different than that. Every single day we worry about how we can make Pixar a better company so that nobody will ever want to leave, and so we don’t take anybody for granted. Because if they don’t want to be at Pixar, then probably they should leave anyway— whether or not they would ever have a contract. ✂

SJ: In technology, and at Pixar on the creative side, you’ve got incredibly talented people who are also rare and in demand. If you don’t treat them right, they can go get another job in ten minutes. Right?

So a strange thing happens: the hierarchy of power sort of inverts, and the CEO is actually at the bottom. I sort of feel like I work for most of these people because they’re the ones that are doing all the brilliant work.

And it’s the same in software. It’s the same thing. The best people are very hard to come by, and so it’s management’s job to support them because they’re on the front lines doing the work. ✂

Q: How is your relationship with Pixar different than Louis B. Mayer or some of the great studio heads who may not have been directors or producers themselves, but who certainly had a great deal to do with what was done?

SJ: Well, we’re a small studio, and so we don’t want to let any of this go to our heads.

What we’re trying to do is to build a great animation studio. We stay very focused on that. The other great animation studio is, of course, Disney. And they’ve done an incredible job. Feature animation is really at the heart of Disney; they’ve created all the character franchises that really breathe life into the theme parks. And if you look at where [Disney’s] profits come from, a tremendous amount of it is dependent on feature animation, as are the theme parks.

What we’re doing is just a pure play to build a feature animation studio. My role is to try to understand the pieces we need to put in place to do that, to work with everybody to attract and retain the people to do that, and to get a clear strategy in place. And [I] help with the relationships with Disney and other people.

I enjoy it tremendously. We’ve got an incredible collection of people at Pixar, so I learn a lot. I learn a lot. ✂

SJ: I like to get in there and help where I can. But my greatest joy is when we have people that are much better than I am at something, so I can forget about it, and not worry about it, and get on to something else where I can help. ✂

SJ: Making an animated film is entirely different than making a live-action film. When you make a live-action film, a director typically shoots between ten and twenty-five times as much footage as will end up on the screen. They take that into the editing room and they build their film. And hopefully they can do a good job, because if they can’t, it’s too late— the actors are gone, the sets are down.

Walt Disney realized many decades ago that animation was so expensive that you couldn’t afford to animate ten times more than what you need. Matter of fact, you don’t want to animate even 10 percent more than what you need. And therefore, the only conclusion you can come to is, you have to edit your film before you make it. Disney pioneered a lot of techniques for doing that, and they’ve refined those over the last sixty years.

Working with Disney gave us access to that wisdom that you can’t buy for love or money: the wisdom and experience of having made tens of feature animated films. And I think we learned a tremendous amount. ✂

SJ: Ten years ago, when we made the landmark short film Luxo Jr. , it took about three hours on average to render each frame. Fast-forward to today. Computers are a hundred times faster, and yet in Toy Story it took three hours on average to render each frame. And the reason was the frames were a hundred times more complex in many cases.

And we’re throwing between five and ten times more computer power at our second feature film, code name Bugs , than we did at Toy Story . And it’ll still take three hours to render each frame. Our ambitions visually are growing as fast as the technology can feed them. And so I think the visual worlds which we’ll be able to create will be much richer over time.

On the creative side, though, I think the art of storytelling is very old. And no amount of technology can turn a bad story into a good story. […] Storytelling is a real art, and that’s something that we’re always going to be working on very, very hard. I don’t think it’s changed in a long time, and I’m not sure it will. And I don’t think it’s something that the technology has anything to do with. ✂

SJ: You can hardly find an Apple II around too much anymore. You still can in the schools, but that’s about it. It’s not clear whether you’ll be able to boot up a Macintosh five years from now or not. All these technology boxes and all this software: it has a life of a year or two, if you’re very lucky. If it has a life of five years, it’s extraordinary. And every once in a while, something has a life of ten to fifteen years— and I’ve been lucky to be associated with a few of those products as well. But sooner or later, they all become part of the sedimentary layer that is the foundation for new innovation.

[By contrast] Disney released its first animated feature film, Snow White , in 1937. That’s sixty years ago. A few years ago, they rereleased it on video and sold 28 million copies, making probably around a quarter billion dollars of profits— sixty years after its initial release!

And I have a young son. We got Snow White , and he loved it. He watched it thirty, forty times. And it really struck me that I know people on most of the continents of this world, and I think everyone I know knows the story of Snow White . I don’t think I know one person who hasn’t seen it.

Watching my son watch this, it really hit me that these stories renew themselves with each generation of young children. You read Joseph Campbell; these are our myths. Here’s something that’s sixty years old that’s regenerating itself in my son and other young children.

And I think people are going to be watching Toy Story in sixty years. Not because of the computer graphics, but because of the story about friendship. And that’s something really amazing to me, something very different than the industry I worked in in the past.

To have the opportunity to put these stories into the culture like this, if we can work really hard and be lucky again and again, is a rare opportunity. And I think everybody at Pixar feels really, really privileged to have this opportunity.

Email to Pixar Employees

“The Pixar Holiday Waltz”

From: Steve Jobs

To: Pixar

Subject: Pixar Holiday Waltz

Date: December 12, 1996, 4:03 p.m.

I am told that around 40 of us are feeling uncomfortable coming to the Pixar Holiday Waltz because we don’t have a guest or because we feel it’s a “hoity toity*” affair that does not reflect “who we are.”

I think very few of us belong to the waltz crowd, nor often attend formal affairs like this (my last waltz was ten years ago). That’s exactly why this promises to be so much fun – it’s a chance to see ourselves and our colleagues in a strange costume (tuxedo) doing a totally foreign activity (waltzing). This may be your last chance during this lifetime to see your favorite TD trying to dance in a tuxedo.

As for being single, all I can say is that cutting in on a lovely lady or elegant gentleman during a waltz is considered proper and flattering. You might also enjoy the food, music, friends and ambience.

I do hope those of you not planning to attend reconsider joining your friends and colleagues at our Pixar Holiday Waltz. We will miss you if you don’t come. What do you have to lose? You just might love it.

Steve

*hoity-toity adj (1690)

1: thoughtlessly silly or frivolous: FLIGHTY

2: marked by an air of assumed importance:

HIGHFALUTIN

A ticket to the event on December 14, 1996, illustrated with a tuxedoed man waltzing with a woman in a flowing gown.
A staff invitation to the Pixar Holiday Waltz.

Interview for In the Company of Giants

“The worst thing that someone can do in an interview is to agree with me.”

In the 1990s, when Steve was running NeXT and Pixar, two Stanford Business School students won a lunch with him at a charity auction. They later interviewed him for a book profiling business leaders in technology.

Steve Jobs: You’d better have great people, or you won’t get your product to market as fast as possible. Or you might get a product to market really fast, but it will be really clunky and nobody will buy it. There are no shortcuts around quality, and quality starts with people. Maybe shortcuts exist, but I’m not smart enough to have ever found any.

I spend 20 percent of my time recruiting, even now. I spend a day a week helping people recruit. It’s one of the most important things you can do.

Q: If finding the A players is so important, how can you tell who is an A player and who isn’t?

SJ: That’s a very hard question. Ultimately there are two paths. If a candidate has been in the workplace for a while, you have to look at the results. There are people who look so good on paper and talk such a good story but have no results behind them. They can’t point to breakthroughs or successful products that they shipped and played an integral part in. Ultimately the results should lead you to the people. As a matter of fact, that’s how I find great people. I look at great results and I find out who was responsible for them.

However, sometimes young people haven’t had the opportunity yet to be in a position of influence to create such results. So here you must evaluate potential. It’s certainly more difficult, but the primary attributes of potential are intelligence and the ability to learn quickly. Much of it is also drive and passion— hard work makes up for a lot.

When you recruit, you’re rolling the dice. No matter what, you’re rolling the dice because you’ve only got an hour to assess the candidate. The most time I spend with somebody is an hour, and I must then recommend whether we hire the person or not. Others will recommend, too, so I won’t be the only one, but I’ll still have to throw my vote in the hat.

Ultimately it comes down to your gut feeling. Your gut feeling gets refined as you hire more people and see how they do. Some you thought would do well don’t, and you can sense why. If you study it a bit you might say, “I thought this person was going to do well, but I overlooked this aspect,” or, “I didn’t think this person would do well, but they did and here’s why.” As you hire people over time, your gut instinct gets better and more precise.

Over time, my digging in during an interview gets more precise. For example, many times in an interview I will purposely upset someone: I’ll criticize their prior work. I’ll do my homework, find out what they worked on and say, “God, that really turned out to be a bomb. That really turned out to be a bozo product. Why did you work on that?” I shouldn’t say this in your book, but the worst thing that someone can do in an interview is to agree with me and knuckle under.

What I look for is for someone to come right back and say, “You’re dead wrong and here’s why.” I want to see what people are like under pressure. I want to see if they just fold or if they have firm conviction, belief, and pride in what they did. It’s also good every once in a while to really piss somebody off in an interview to see how they react because, if your company is a meritocracy of ideas, with passionate people, you have a company with a lot of arguments. If people can’t stand up and argue well under pressure, they may not do well in such an environment. ✂

Q: What do you think your weaknesses are when it comes to management?

SJ: I don’t know. People are package deals; you take the good with the confused. In most cases, strengths and weaknesses are two sides of the same coin. A strength in one situation is a weakness in another, yet often the person can’t switch gears. It’s a very subtle thing to talk about strengths and weaknesses because almost always they’re the same thing.

My strength probably is that I’ve always viewed technology from a liberal arts perspective, from a human culture perspective. As such, I’ve always pushed for things that pulled technology in those directions by bringing insights from other fields. An example of that would be— with the Macintosh— desktop publishing: its proportionately spaced fonts, its ease of use. All of the desktop publishing stuff on the Mac comes from books: the typography, that rich feel that nobody in computers knew anything about. I think that my other strength is that I’m a pretty good judge of people and have the ability to bring people together around common vision.

Q: Well then, when are your strengths— judgment of character and liberal arts perspective— your weaknesses?

SJ: In certain cases, my weaknesses are that I’m too idealistic. [I need to] realize that sometimes best is the enemy of better. Sometimes I go for “best” when I should go for “better,” and end up going nowhere or backwards. I’m not always wise enough to know when to go for the best and when to just go for better.

Sometimes I’m blinded by “what could be” versus “what is possible,” doing things incrementally versus doing them in one fell swoop. Balancing the ideal and the practical is something I still must pay attention to.

Q: In terms of going for the best, you have a widely held reputation of being extremely charismatic— someone who is always able to draw out the best in other people. How have you been able to motivate your employees?

SJ: Well, I think that— ultimately, it’s the work that motivates people. I sometimes wish it were me, but it’s not. It’s the work. My job is to make sure the work is as good as it should be and to get people to stretch beyond their best. But it’s ultimately the work that motivates people. That’s what binds them together.

Q: Yet in the case of the Macintosh, you got tremendous output from people. Regardless of the type of work, not everybody can elicit that type of commitment.

SJ: Well, I’m not sure I’d chalk that up to charisma. Part of the CEO’s job is to cajole and beg and plead and threaten at times— to do whatever is necessary to get people to see things in a bigger and more profound way than they have, and to do better work than they thought they could do.

When they do their best and you don’t think it’s enough, you tell them straight: “This isn’t good enough. I know you can do better. You need to do better. Now go do better.”

You must play those cards carefully. You must be right a lot of the time because you’re messing with people’s lives. But that’s part of the job. In the end, it’s the environment you create, the coworkers, and the work that binds. The Macintosh team, if you talk to most of them— a dozen years since we shipped the product— most will still say that working on the Mac was the most meaningful experience of their lives. If we’d never shipped a product they wouldn’t say that. If the product hadn’t been so good they wouldn’t say that. The Macintosh experience wasn’t just about going to camp with a bunch of fun people. It wasn’t just a motivational speaker. It was the product that everybody put their heart and soul into, and it was the product that expressed their deep appreciation, somehow, for the world to see.

So, in the end, it’s the work that binds. That’s why it’s so important to pick very important things to do because it’s very hard to get people motivated to make a breakfast cereal. It takes something that’s worth doing.

Email to Pixar Employees

“Apple Acquires NeXT.”

From: Steve Jobs

To: Pixar

Subject: Apple Acquires NeXT

Date: December 20, 1996, 9:24 p.m.

Today Apple announced that it is acquiring NeXT! This is great for Apple -- it gives them a very advanced object-oriented operating system, OpenStep, that can leapfrog Microsoft Windows.

This is great for NeXT -- OpenStep will become mainstream and be used by millions of people; this has been a dream of everyone at NeXT for ten years.

This is great for Pixar -- it will free me up from running two companies, so I can devote even more of my energies to Pixar. There may even be some possibilities for Pixar and Apple to work together in the future (if creatively driven!).

And, this is really great for me. I have been working with a group of wonderful colleagues at NeXT for over a decade, and having our work finally become mainstream will be very gratifying (a feeling everyone at Pixar already knows). I will be advising Apple on their product strategy, and I will get to spend even more of my time at Pixar :).

I am very happy and excited.

Steve

PS: For more details, check out NeXT’s website at

www.next.com

Part III, 1996–2011

“Much of what I stumbled into by following my curiosity and intuition turned out to be priceless later on.”

The Apple that Steve returned to after its purchase of NeXT at the end of 1996 scarcely resembled the company he remembered. Apple had lost $800 million that year. Steve had so little trust in the board that he insisted all but two directors resign.

But he also found many employees whose talents and love for Apple struck him as “phenomenal.” He assured a doubtful NeXT colleague, “There’s something there worth saving.”

Steve remained driven by the same mission he articulated in his early years, to “put something back into the pool of human experience.” And in his years away— longer than his first time at the company— he had matured and learned how to lead.

As CEO of Apple and Pixar (he held both roles until Disney acquired Pixar in 2006), he saw his job as “number one, recruit; number two, set an overall direction; and number three, inspire and cajole and persuade.” He said, “You’re not grabbing the pencil out of the twenty-five-year-old’s hand to do it better than they are. If you’re smart, you’re hiring twenty-five-year-olds who are smarter than you.” He gave particular thought to his responsibility for the business aspects of a creative company. A “risk-taking creative environment on the product side,” he said, required a “fiscally conservative environment” on the business side. “Creative people are willing to take a leap in the air, but they need to know that the ground’s going to be there when they get back.”

At Apple, making sure the ground was still there meant streamlining, a skill Steve had mastered in the lean years at NeXT and Pixar. His second tenure at Apple was a study in focus. Soon after he returned, he slashed the company’s product offerings from seventeen to four. This upset some fans and led to thousands of job losses, but he was unwavering in his belief that it was necessary to save the company. “You’ve got to choose what you put your love into really carefully,” he said.

A stream of Apple breakthroughs— including iMac, OS X, iPod, iTunes, iPhone, and iPad— reflected that focused love, as did Pixar’s blockbuster and award-winning films from the same period: A Bug’s Life , Toy Story 2 , Monsters, Inc. , Finding Nemo , The Incredibles , Cars , Ratatouille , Wall-E , Up , and Toy Story 3 . Steve saw every product and film as “a way of expressing to the rest of our species our deep appreciation.” Apple technology offered tools to make something wonderful. And Pixar films, he believed, had the “rare opportunity to put stories into the culture,” stories that would speak to “our grandkids’ grandkids’ grandkids.”

Steve wanted to build a foundation for Pixar and Apple that would last beyond his lifetime. When Disney acquired Pixar, the deal included language protecting Pixar’s culture and independence. He was intimately involved in the design of campuses for both companies, with beautiful physical environments— an orchard, a café, a headquarters constructed of handmade bricks— that encouraged serendipitous encounters and reflected his reverence for craft.

Six Pixar films won the Oscar for Best Animated Feature during this third act of Steve’s life. And when he resigned from Apple, six weeks before his death from pancreatic cancer in 2011, his beloved company, with its sixty thousand employees, seasoned leadership team, and clear mission guiding its future, was the most valuable in the world.

“It is always a team of people, and the chemistry between that team of people, that makes great results,” Steve once said. Under his leadership, the teams at Apple and Pixar transformed four very different industries: computing, telecommunications, music, and film. The engine driving these transformations was a remarkably consistent set of values that Steve held dear: Life is short; don’t waste it. Tell the truth. Technology should enhance human creativity. Process matters. Beauty matters. Details matter. The world we know is a human creation— and we can push it forward.

Steve on Returning to Apple

Speaking to Stanford business school students in 2003, Steve recalled his internal struggle, seven years earlier, over whether to return to Apple.

I’m working away at NeXT, working away at Pixar, reading the rumors in the papers that Apple is going to buy a company, this other company, for their operating system. And one of our guys said, “We should sell NeXT to them. We’ve got a much better operating system.”

And I said, “Forget it. It will never happen.”

So this is one of those cases where, when you hire the right people, they don’t always listen to you in key moments in time. So this person, being very smart, didn’t listen to me. And he went over and talked to Apple and said, “You ought to buy NeXT.” And they were interested.

Then he comes back and says, “They want to talk to us.”

And I was like, “No. Go away. You’re making this up.”

He did this twice. Finally I said, “OK. Fine. Let’s talk to them.” I couldn’t believe it. They really were interested. And so we ended up selling them NeXT.

And the CEO running Apple at that time made it clear he didn’t want me around, which was fine. And so I stayed on just as a consultant to him, to try to help him a little bit, because the management team he was inheriting from NeXT was actually quite a bit better than the one he had at Apple. And so I was trying to make sure these people didn’t get totally crushed.

And … I have to be careful what I say here. Let me just say that you need a license to drive a car, but you don’t need a license to be the CEO of a company. And maybe you should need one.

Anyway, so I was pretty much out of the picture. Then one of Apple’s board members called me. I had never met this person. And he said, “Look, I want your straight scoop on what you think of the CEO here.” And I thought, “I don’t even know this person. Whatever I tell them, they’ll probably go tell the CEO, and then I will be persona non grata, and I will not have a chance to help my team not get crushed by these other folks there.” And I thought a lot.

And then I thought, “You know, this is a director of a company that I started and that I loved for many years— and still do to some extent. And so how can I not tell them the truth?”

So I spilled out my guts— and never heard from him again. And so I figured, fine. I was spending my days at Pixar and having a great time. It was springtime, going into summer. And six years ago a few weeks from now, I got a call back from him [the director], and he said, “We are going to dismiss our CEO, and we would like you to come back and run the company.”

And I said, “I can’t do that. I’m the CEO of Pixar. It’s a publicly traded company. We have all these wonderful employees. We have these shareholders. And I can’t go be CEO of another public … I can’t desert them. So I can’t do this. I’ll help you any way I can, but I can’t.”

He called back a few days later and he said, “We’d like you to come back as an interim leader and help us find a new CEO.”

And I said, “Well, I have to think about that.” And I was thinking about it and called up a friend of mine, a really smart guy, a good friend I’d known for a long time that works at another company in the industry. And I probably woke him up in the morning, about eight o’clock one morning, and I was telling him about my struggles about, should I, could I do this? Should I not? And this and that …

And finally, he interrupted me after about four minutes and he said, “Steve, I don’t give a shit about Apple. Why are you telling me all this?”

And I said, “Oh, OK. I’m sorry.” And I hung up the phone.

And I realized: You know, I do give a shit about Apple.

And that’s kind of what crystallized it for me. And so I went back there as an interim CEO.

And I was terrified because I was afraid our Pixar employees and shareholders would feel like I was deserting them. And that’s why I went back just as an interim CEO. I only planned to stay ninety days. But Apple was in a pretty tough situation, and the candidates it was attracting for the CEO job, they were not so good. And I almost hired one, and at the last minute I just said, “I can’t do this to these guys.”

I just thought, “Well, it will take another ninety days to find somebody.” And that turned into a year. And I decided right up front that I was just going to act like I was the permanent CEO, because they didn’t need a caretaker. This thing was in intensive care. It was about ninety days away from bankruptcy. It was in pretty bad shape.

Steve has his back turned to the camera as he writes on a whiteboard. He wears a black turtleneck, shorts and sneakers. Steve stands in a dimly-lit stairwell opposite a large brick wall. He wears a vest over a white shirt. Steve sits in a conference room, leaning his head on his hand. A projector and whiteboard are blurred in the background. Steve stands in a courtyard, deep in thought, with his head down and his chin resting in the palm of his hand.
Steve leans back on his chair in a conference room. He is wearing shorts, legs stretched out and feet on the table. Steve lies on his back on a carpeted floor, talking on a cellphone and looking at the ceiling with his hand on his head. In his dark home office, a small lamp illuminates Steve as he sits at a cluttered desk. Steve in his signature turtleneck, eyes cast upward and fingers knitted, shortly after returning to Apple.
A selection of photos from Steve’s return to Apple.

Email to Apple Employees

“A More Entrepreneurial Apple.”

Steve was not yet CEO when he sent a company-wide email laying out plans to “take Apple back to its roots.”

From: Steve Jobs

To: Apple employees

Subject: A More Entrepreneurial Apple

Date: August 12, 1997, 8:20 a.m.

Renewing Apple is a journey, and we have begun that journey during the past four weeks by taking some decisive first steps— a new Board of Directors, a new product strategy and product roadmap, a decision to really focus on two market segments (education and creative content), a new advertising agency, and a detente and working partnership with Microsoft. Last week we announced some of these steps at MacWorld, and so far our shareholders and the public seem to approve.

Today we are taking a few more steps which will begin to take Apple back to its roots as a more egalitarian, entrepreneurial company. They are:

1. Stock Options - From now on, we will be using stock options as a primary form of “beyond-salary” compensation. Stock options are egalitarian (when anyone’s stock goes up $1 per share, everyone’s stock goes up $1 per share) and they are the best way to give our employees a true stake in the company’s future success. And, we want our employees to be rewarded by the company’s success in the same way that our public shareholders are: through stock appreciation. To lead the way, the Executive Team has agreed to forfeit their current and future cash bonus plans in exchange for more stock options.

As you know, we repriced all stock options to $13.25 on July 11th. In addition, I am pleased to announce that on August 5th our Board approved new stock option grants totaling six million shares at the price of $19.75. Those receiving these new grants will get the good news later this week.

Apple has granted stock options for over 10 million shares since the beginning of this calendar year, and employees now hold stock options for over 20 million shares - which is more than 16% of Apple’s total outstanding shares. This is a very high percentage for a company of Apple’s size, and comparable to many valley start-ups. As we restore Apple’s fortunes, our public shareholders and our employee stock option holders will all benefit in harmony.

2. New Severance Plan - Effective today, we are changing our severance plan for all employees to be more in line with an entrepreneurial company. There will now be only one severance plan for all employees. This plan, like the previous plans, will provide a 60 day notice period, with full pay and benefits. In addition, employees will be eligible to receive one additional week of severance pay for each full year of service. For example, if you have worked at the company for more than three but less than four years, you will receive your pay and benefits during the 60 day notice period plus severance payments equal to three weeks of pay. This new severance plan applies to all employees of Apple, Claris, and Newton in the US - there is no longer a separate executive severance plan.

We will be changing our international severance policies to be in line with this new plan to the extent permitted under local laws.

3. Sabbatical Program - Apple needs all hands on deck for the foreseeable future as we turn our company’s fortunes around. We are therefore discontinuing the sabbatical program at the end of our current fiscal year. Employees who have earned their sabbatical as of September 26, 1997, will be eligible to take their sabbatical at a mutually agreeable time during fiscal year 1998. This applies to all employees of Apple, Claris, and Newton worldwide.

4. Corporate Travel - Corporate travel will continue to be constrained to essential trips. Our egalitarian travel policy specifies coach class travel for everyone on trips lasting less than 10 hours, and business class travel for everyone on trips of 10 hours or longer. Of course individuals may use their personal funds or mileage awards to upgrade their seating class. For clarification, flights between San Francisco and Tokyo (either direction) are eligible for business class travel.

5. Facilities - We will continue to move as many employees as possible onto our R&D Campus site. We will greatly benefit by the resulting “beehive” effects, including faster communication paths and more unplanned interactions between the various groups. Reflecting this consolidation, we are renaming the R&D Campus to the Apple Campus, beginning today.

Thank you for your support as we work together to renew Apple.

Steve and the Executive Team

Email Exchange with Avie Tevanian

“There is something good here worth saving.”

Avie Tevanian ran software engineering at NeXT and assumed the same role at Apple when it acquired NeXT. He told Steve that the Apple culture at the time of the acquisition was so bad that perhaps the best move might be to sell the company.

From: Avadis Tevanian

To: Steve Jobs

Subject: Apple engineers

Date: August 14, 1997, 9:31 p.m.

Steve,

The last couple of weeks have been quite interesting. With this kernel issue spinning out of control it is clear to me that the “cancer” within Apple, which we had put into remission (at least in SW engineering) a few months ago, came back with a vengeance. I am now applying a new chemo treatment in an attempt to put it back into remission— and I think it will be successful— but we will definitely have another wave of resignations from the sharks that smelled the blood in the water. A bunch of people will get over it, but many will not.

The net is the really bad Apple culture we’ve heard about still lurks out there. And I am convinced that a major problem with past management was their inability to deal with it. While I am a strong-willed person who can survive almost anything (hey, you’ve seen it first hand), I must admit these guys are wearing me down. Without any emotional tie to Apple (I, of course, have none) it gets increasing difficult to come to work each day. And while I can certainly treat the patient this time I wonder how long it will be before the cancer comes out of remission again. The context of a continuation of revenue decrease, losing quarters and ongoing restructuring does not help.

In the case of the kernel issue & OS roadmap I know the decisions we’ve made are the right decisions. And each day, as I discuss it with more and more people, I believe it even more. In fact, I’ve discovered just how conniving some of these folks can be just to push their agenda— including pretty severe obfuscation of facts. While the source of this is probably only a few people, they are able to quickly spread “evil thoughts” throughout the population that is just looking for a place to make a stand.

Combine this all with the reaction to the sabbatical program— which has been very poorly received in engineering (I’ve heard threats of a “sick-out”), and it makes me think this company isn’t worth saving. At least not for the sake of the employees. Yes, I feel for the customers. Yes, the shareholders deserve fair value. And yes, I worry about the ex-NeXTers we dragged into this. But it is not at all clear to me that the employees deserve to “win.” This, of course, is an overgeneralization, and there are some very good folks here, but you get the gist of this.

It really makes me think that maybe now is the time to seriously consider selling the company. It’s not clear we’ll ever again have the positive aura that we have now— at least temporarily.

Anyway, we should get together and discuss all this sometime— face to face, 1-1— not on the phone. Not urgent— I’m a survivor— but when we both have some time.

Avie

From: Steve Jobs

To: Avadis Tevanian

Subject: Apple engineers

Date: August 14, 1997, 10:02 p.m.

Avie,

You know, I have had the same feeling lately— that Apple employees don’t deserve to be saved. They think they work so hard— heck, I don’t see it. They also think they have no responsibility for any of the mess that Apple’s in.

I don’t know who they think engineered and shipped such bad products, who created and authorized such bad marketing, who caused the support to be so bad, etc… They just don’t want to share any responsibility for the current state of the company…

But, there is something good here worth saving. I don’t quite know how to express it, but it has to do with the fact that Apple is the ONLY alternative to Windows and that Apple can still inject some new thinking into the equation.

Let’s go for a walk and talk about this sometime soon,

Steve

From: Avadis Tevanian

To: Steve Jobs

Subject: Re: Apple engineers

Date: August 25, 1997, 10:05 p.m.

We really need to get together to discuss this and other issues related to the “product strategy” and overall management of the company. I am worried about a number of the pieces of the puzzle and as a result am completely unmotivated to do what it takes to turn this company around. And for the first time in 10 years I don’t even feel like challenging your ideas when I disagree— which scares me because I believe as a “team” we work best when we challenge each other and come out all-the-better for it.

I am going to sleep on this some more tonight (I feel fried already and it’s only Monday evening— even the weekends aren’t sufficient to recharge these days). I have a pretty hectic schedule tomorrow (except for late in the day) but have lots of available time on Wednesday. Let me know when you’re available.

Avie

From: Steve Jobs

To: Avadis Tevanian

Subject: Re: Apple engineers

Date: August 25, 1997, 10:26 p.m.

Please continue to challenge me. It’s the way we get to the right decisions, and I enjoy it too.

Steve

A photograph from a reel shows Steve in portrait. Only the left side of his face is visible; the rest is in deep shadow.
Steve wanted to mimic the album art from Meet the Beatles .

Speech to Apple Employees

“We believe that people with passion can change the world for the better.”

Steve introduced the “Think Different” advertising campaign to a small group of Apple employees on September 23, 1997. The ad would go on to win an Emmy Award for Outstanding Commercial.

Howdy. Good morning. We were up till three o’clock last night finishing this advertising, and I want to show it to you in a minute— see what you think of it.

I’ve been back about eight to ten weeks, and we’ve been working really hard. What we’re trying to do is not something really highfalutin. We’re trying to get back to the basics. We’re trying to get back to the basics of great products, great marketing, and great distribution. I think that Apple has pockets of greatness but in some ways has drifted away from doing the basics really well.

We started with the product line. We looked at the product road map, going out for a few years, and we said, “A lot of this doesn’t make sense, and it’s way too much stuff, and there’s not enough focus.” We actually got rid of 70 percent of the stuff on the product road map. I couldn’t even figure out the damn product line after a few weeks. I kept saying, “What is this model? How does this fit?”

I started talking to customers, and they couldn’t figure it out either.

You’re going to see the product line get much simpler, and you’re going to see the product line get much better. There’s some new stuff coming out that’s incredibly nice. In addition, we’ve been able to focus a lot more on the 30 percent of the gems and add some new stuff in that is going to take us in some whole new directions. So we are incredibly excited about the products. I think we’re really thinking differently about the kinds of products we have to build. The engineering team is incredibly excited. I mean, I came out of the meeting with people that had just gotten their projects canceled, and they were three feet off the ground with excitement ’cause they finally understood where in the heck we were going, and they were really excited about the strategy.

In the same way we, I think, have not been as … we have not kept up with innovations in our distribution. I’ll give you an example. I’m sure it was talked about this morning, but we’ve got anywhere from two to three months of inventory in our manufacturing supplier pipeline, and about an equal amount in our distribution channel pipeline. We’re having to make guesses four or five, six months in advance, about what the customer wants.

We’re not smart enough to do that. I don’t think Einstein’s smart enough to do that. So what we’re going to do is get really simple and start taking inventory out of those pipelines so we can let the customer tell us what they want, and we can respond to it super fast. You’re going to see us be doing a lot of things like that. Today is just the first of many things we’re going to be doing with you.

We’re going to be not only, I think, catching up to where the best of the best are in distribution, but we’re going to actually be innovating and be breaking some new ground, I think, in the coming several months. I’m pretty excited about that as well, in the distribution manufacturing side of things.

That gets us to the marketing side of things.

To me, marketing is about values. This is a very complicated world. It’s a very noisy world, and we’re not gonna get a chance to get people to remember much about us. No company is.

And so we have to be really clear on what we want them to know about us. Now, Apple, fortunately, is one of the half-a-dozen best brands in the whole world— right up there with Nike, Disney, Coke, Sony. It is one of the greats of the greats, not just in this country but all around the globe.

But even a great brand needs investment and caring if it’s going to retain its relevance and vitality. And the Apple brand has clearly suffered from neglect in this area in the last few years. And we need to bring it back. The way to do that is not to talk about speeds and feeds. It’s not to talk about MIPS and megahertz. It’s not to talk about why we are better than Windows. The dairy industry tried for twenty years to convince you that milk was good for you. It’s a lie, but they tried anyway. And the sales were going like this [hand mimics a line running down and to the right] . And then they tried “Got Milk?” and the sales started going like this [hand goes up and to the right] . “Got Milk?” doesn’t even talk about the product! As a matter of fact, the focus is on the absence of the product.

But the best example of all, and one of the greatest jobs of marketing that the universe has ever seen, is Nike. Remember: Nike sells a commodity! They sell shoes! And yet when you think of Nike, you feel something different than a shoe company. In their ads, as you know, they don’t ever talk about the products. They don’t ever tell you about their air soles, and why they are better than Reebok’s air soles. What does Nike do in their advertising? They honor great athletes, and they honor great athletics. That’s who they are. That’s what they are about.

Apple spends a fortune on advertising. You’d never know it. You’d never know it.

So when I got here, Apple had just fired their agency, and there was a competition with twenty-three agencies, and, you know, four years from now, we would pick one. We blew that up, and we hired Chiat/Day, the ad agency I was fortunate enough to work with several years ago. We created some award-winning work, including the commercial voted the best ad ever made, “1984,” by advertising professionals.

We started working about eight weeks ago. The question we asked was, “Our customers want to know: Who is Apple, and what is it that we stand for? Where do we fit in this world?”

And what we’re about isn’t making boxes for people to get their jobs done, although we do that well. We do that better than almost anybody, in some cases.

But Apple is about something more than that. Apple, at the core— its core value— is that we believe that people with passion can change the world for the better. That’s what we believe.

And we’ve had the opportunity to work with people like that. We’ve had the opportunity to work with people like you, with software developers, with customers, who have done it— in some big and some small ways.

And we believe that in this world, people can change it for the better. And that those people that are crazy enough to think that they can change the world are the ones that actually do.

And so, what we’re going to do, in our first brand-marketing campaign in several years, is to get back to that core value. A lot of things have changed. The market is a totally different place than it was a decade ago. And Apple’s totally different, and Apple’s place in it is totally different. And believe me: the products, and the distribution strategy, and manufacturing are totally different— and we understand that. But values and core values: those things shouldn’t change. The things that Apple believed in at its core are the same things that Apple really stands for today. And so we wanted to find a way to communicate this. And what we have is something that I am very moved by. It honors those people who have changed the world. Some of them are living. Some of them are not. But the ones that aren’t, as you’ll see, you know that if they ever used a computer, it would have been a Mac.

And the theme of the campaign is “Think Different.” It’s honoring the people who think different and who move this world forward. It is what we are about. It touches the soul of this company.

So I’m going to go ahead and roll it, and I hope that you feel the same way about it that I do.

[Steve runs the video.]

“Here’s to the crazy ones. The misfits. The rebels. The troublemakers. The round pegs in the square holes. The ones who see things differently. They’re not fond of rules. And they have no respect for the status quo. You can quote them, disagree with them, glorify or vilify them. About the only thing you can’t do is ignore them. Because they change things. They push the human race forward. And while some may see them as the crazy ones, we see genius. Because the people who are crazy enough to think they can change the world are the ones who do.”

Email to Pixar’s Leadership

“My grades as of today.”

As 1997 drew to a close, Steve graded Pixar’s performance: four A’s, one B, and two F’s.

From: Steve Jobs

To: ec, jl, lawrence, sarahmc

Cc: sj

Subject: Pixar Milestones

Date: December 21, 1997, 9:26 a.m.

From my files, dated Monday, October 28, 1996, with my grades as of today:

1997 MILESTONES

1. Make BUGS Great – A

2. Ship Two Strong Interactive Products – F (but A for getting out)

3. Enter Sequel Business – A (and almost exit sequel business)

4. Greenlight Film #3 – F (but B for great progress)

5. Manage Growth – A (hired >100 people, Pixar U, expanded infrastructure)

6. Longer Term:

- keep a larger percentage of the profits – A

- increase Pixar’s brand awareness – B (mostly due to Apple goings on)

Let’s set our 1998 milestone goals soon after the holidays.

Steve

Email to Pixar’s Leadership

“This list is to remind us of the ‘forest.’”

Steve’s 1998 goals for Pixar included beginning animation on the film that would become Monsters, Inc.

From: Steve Jobs

To: ec, jl, lawrence, sarahmc

Subject: Pixar Milestones

Date: December 26, 1997, 10:00 a.m.

Here are proposed key 1998 milestones for us as a senior management team. The purpose of this list is to remind us of the “forest” from time to time, and to help prod us into taking corrective action when we get off track. Please feel free to shoot full of holes with your comments.

Steve

1998 MILESTONES

1. A Bug’s Life opens in November to very strong critical and box office success

2. PR efforts yield Pixar branding and creative credit for A Bug’s Life

3. Toy Story 2 becomes a great feature film

4. Hidden City is on schedule to begin animation by April ’98

5. Film #5 development begins by April 1

6. Emeryville groundbreaking in May

7. Breakthrough new animation system demonstrated by summer

8. Secure shorts projects to defray carrying costs of ABL crew through Spring ’99

9. Execute equity financing in Spring/Summer if necessary

Email to Apple Employees

“Newton protest today.”

When Steve returned to Apple, he jettisoned many product lines developed during his time away. His goal was to focus the company’s offerings. Some fans rebelled.

From: Steve Jobs

To: Apple employees

Subject: Newton protest today

Date: March 6, 1998, 8:09 a.m.

Some people are understandably upset that Apple has decided to stop developing future Newton OS based computers, especially the MessagePad. Today some of them are coming to Apple to protest our decision. This is OK. We are reserving a space for them on our campus and will provide them with coffee and other hot drinks (it may be cold out there!).

Our decision to end Newton development was not taken lightly, and is unlikely to be reversed. Even so, let’s welcome the Newton customers and developers who come to protest this decision. Hopefully they will feel our enthusiasm about the future of Apple, and leave more settled than they arrived.

Thanks

Steve

Speech at Macworld

“Apple is coming back in a very big way.”

Steve was still interim CEO of Apple when he gave the keynote address at Macworld in New York City in July 1998.

Thank you very much. This is good. I just got off a red-eye to get here. I didn’t think I could be here today, and I managed to make it, so this is good. We’ve got some great stuff to talk about today, so let’s get into it.

An American named Abraham Maslow came up with a theory that he called the human hierarchy of needs. And this theory was a simple one in concept, [though] somewhat profound in its implications. This theory was that humans have needs that must be met, and these needs can be stratified into certain levels. [Humans] need the bottom needs to be met first, and then they progress to the next level. And when those [needs] are met, they progress to the next level.

So, you start off with the physiological needs: food, clothing, and shelter, let’s say. If those are met, or once those are met, you then start to be concerned more with the safety of your environment— and eventually with love, with esteem, with self-actualization. I don’t know if this is true or not, but this is what Abraham believed.

And I thought this was a good model, so I borrowed it today to construct … Steve Jobs’s version of this [audience laughs] , which I call the Apple Hierarchy of Skepticism.

Let me explain this to you. When I came to Apple a year ago, all I heard was, “Apple is dying. Apple can’t survive.”

And it turns out that every time we convince people that we’ve accomplished something at one level, they come up with something new.

I used to think this was a bad thing. I thought, “Oh, jeez, when are they ever going to believe that we’re going to be able to turn this thing around?” But actually, now I think it’s great, because what it means is we’ve now convinced them that we’ve taken care of last month’s question— and they’re onto the next one.

So I thought, “Well, let’s get ahead of the game. Let’s try to figure out what all the questions are going to be and map out where we are.” So that’s what the Apple Hierarchy of Skepticism is. And it borrows from Dr. Maslow.

So the first level, what we encountered a year ago, was survival. [Audience applauds.] A lot of people thought Apple was in some sort of death spiral, which I think there was some truth to. What did we do? We did many, many things, but the three things that stood out in people’s minds were: we brought in a new management team to run the company, a new board of directors that’s got some phenomenally experienced people on it, and we did a deal with Microsoft. The largest software company in the world want[ing] to help Apple was a fact that didn’t escape very many people, and it added a lot of credibility to what we were doing.

The combination of these three things and a lot of other medium-sized things, I think, convinced people fairly rapidly that survival, at least in the short term, was not an issue. And it gave us some time to demonstrate that we could accomplish some other things.

Immediately, once we did these things, then everybody moved up a level. […] If it wasn’t about survival, it was, “Well, but there’s no stable business in the Mac market.” That was the next level of the hierarchy. So we had to start demonstrating that we had a stable business, and that one could be made from the Macintosh market, because it’s a great market.

So what did we do? Well, the most important thing was profits. In the end, that’s what a lot of people look at. And the first full quarter of the new management team, Apple delivered profits of $47 million […]. And in the next fiscal quarter […] Apple delivered a quarter with $55 million of profits. [Audience applauds.] This went a long way to convincing a lot of the skeptics. And we will be announcing the results of our third fiscal quarter, the one that just ended at the end of June, a week from today. And I’m very pleased to tell you that it will be our third consecutively profitable quarter. […]

In terms of people, our retention has gone way up. We’re losing hardly any people from the company now, and we’re hiring incredibly good people into the company. We also realigned our distribution channels […] and we began an advertising campaign for brand— around “Think Different”— and for some very specific product advertising. […] We invested in an online store. […] We are one of the largest ecommerce sites now on the internet. […] And that’s terrific.

So the aggregate of all of these things was that the market value of Apple’s risen from about $1.8 billion a year ago, to about $4 billion yesterday. I don’t know what it is today.

What this means is that people are seeing a stable business, which is good, and a business that is under control, which is also good.

So what do they do, then? Well, they don’t send us a card or anything. They just go onto the next one. What’s the next one? “Well, if you’ve survived and you’ve got a stable business, well, what’s your product strategy? You’re a product-driven company; you better have a doggone good product strategy.” ✂

When we got to the company a year ago, there were a lot of products. There were the product platforms— fifteen product platforms— and a zillion variants of each one. I couldn’t even figure this out myself. After about three weeks, I said, “How are we going to explain this to others when we don’t even know which products to recommend to our friends?” There was no way to do it.

So we went back to Business School 101 and said, “What do people want?” Well, they want two kinds of products. They want consumer products […] and we need pro products because our design and publishing market wants pro products. […] And in each of those two categories, we need desktop and portable models.

What this told us was if we had four great products, that’s all we need. And as a matter of fact, if we only had four, we could put the A team on every single one of them. And if we only had four, we could turn them all every nine months instead of every eighteen months. And if we only had four, we could be working on the next generation or two of each one as we’re introducing the first generation. So that’s what we decided to do— to focus on four great products.

And the first one that we introduced, of course, was the desktop pro product, which was the Power Mac G3 . […] The next product was the pro portable. And as you know, that is our new PowerBook G3 . […] Now, we also announced on May 6th that we are developing a consumer portable product, and that we will announce it in the first half of next year. And we’re hard at work on that, and I think it’s going to be quite nice.

Which brings us to our consumer desktop product and something we’re going to spend a fair bit of time on today, which, of course, is the iMac, which combines the excitement of the internet with the simplicity of a Macintosh. And that was our goal in this product.

When we got to Apple a year ago, it was very clear within the first month that Apple was walking away from the consumer market because Apple didn’t have a compelling product under $2,000 and had not had one for some time. We immediately began a program to build the most kickass consumer product we knew how to do— and that’s the iMac.

So we’re really, really happy with this thing. It’s gorgeous. We believe it’s going to change the way computers look and should look. ✂

So we’ve survived. We’ve demonstrated that we’ve got a really great, stable business. We’ve laid out the road map of a fantastic product strategy. And we’re getting apps back on the Mac in unprecedented numbers. So, what more could you ask for?

Well, there’s always something, right? The last one is growth! “Well, if all this stuff’s so good, show me some growth. How are you going to grow?”

Right. And again: when I hear these things, I think it’s great, because what it means is we’re building the layers below and people are looking at the next layer. […] What’s the next thing to be skeptical about? Growth. So what are we going to do about growth?

A year ago, when the new management team got to Apple, we looked at the marketplace and we said that although Apple’s market share has declined, it’s still extraordinarily strong in the design and publishing market, where Apple’s market share is between 50 and 90, depending on what segment you look at. And [market share is also strong] in the education market, where Apple owns over half of the installed base and about 35 percent of the new selling units— bigger than anybody else.

What we didn’t say was that Apple had declined rapidly in the consumer market, a market that it invented. Apple invented the market for selling computers to consumers, and yet it had failed to come out with compelling products at consumer price points for several years. So even though we didn’t talk about it, we began a program to get back into the consumer market with a vengeance. And this is where we see a tremendous amount of growth for ourselves. We are doing extremely well in the design and publishing market, and starting to do much better in the education market— but adding the consumer market to that will make both of those even stronger. And we’re going to do it with the iMac. ✂

There is no brand in the computer industry that is remotely as strong as Apple’s. Apple’s brand around the world is right up there with Disney and Sony and Nike in terms of recognition and in terms of consumer loyalty. Now, it’s also worth pointing out that the other three brands on this slide have all had serious problems. Look at Disney when Michael Eisner got there. Look at Nike and Sony over the last half-dozen years. But when these companies have straightened out their problems, their brands have come back stronger than ever. Brands take decades to build. And we think we have one of the best in the world. ✂

We believe that as the price points come down, as the consumer market blossoms, design and fashion become even more important. Let me give you an example. This is [an image of] the most popular watch in the world. And it’s popular not because it tells time better than any other watch. It’s popular because of its design. I’ll give you an interesting statistic: ten years ago, the average American owned one watch. Because of design entering the watch market, today the average American owns seven watches. OK? Seven watches because of design. Because of fashion.

Now, when we dream, we dream of this. Right? [Audience laughs.] And one of the great joys that I found at Apple a year ago was the best industrial design team I’ve ever seen in my life. The designs for the new PowerBook G3 and the iMac— they’re all done inside Apple by some brilliant, brilliant, super hardworking people. And you’re going to see a continual stream of very innovative design out of Apple— I think the best in the industry by a mile. ✂

These are four incredibly powerful, unique, compelling assets: the brand, the installed base, especially in the consumer and education markets, the ability of our design to really take these products into that consumer space of fashion, and the fact that our products are dramatically simpler to set up and use than our competition’s. […]

So, again, going back to Abraham Maslow: he created his hierarchy of needs, and we’ve borrowed it today to take a look at a hierarchy of skepticism that’s followed us for the last year as we’ve slowly demonstrated, step by step, that Apple is coming back in a very big way.

Steve relaxes with his feet up in a full-sized model of the chair of the queen from the Pixar movie A Bugs Life.
In the queen’s chair from A Bug’s Life .
A Toy Story snow globe that Steve kept in his office, with a three-eyed alien and a base that reads: Ooooh!
Steve kept this snow globe in his home office.

Email to Pixar Employees

“A Bug’s Life Sweeps.”

From: Steve Jobs

To: Pixar

Subject: A Bug’s Life Sweeps Thanksgiving Weekend Box Office

Date: November 30, 1998, 7:19 a.m.

A Bug’s Life swept the Thanksgiving box office with a record-setting $46.5 million for the five-day weekend, breaking the previous Thanksgiving weekend record of $45 million set by Disney’s 101 Dalmatians (live-action) in 1996. A Bug’s Life also set a new three-day (Fri-Sat-Sun) Thanksgiving weekend record of $33.6 million.

The exit polls are exceptionally positive. The audience was equally split between males and females (very high ratio of males), and 16% were the highly-desirable non-family “date” audience (we didn’t get this high of a non-family audience for Toy Story until two weeks after its opening). Audiences loved A Bug’s Life, and are telling their friends to see it.

As for the competition, over the five-day weekend Rugrats drooled a respectable $27.6 million, Babe: Pig in the City squealed across the finish line with just $8.5 million, and Antz was squashed with just $1.2 million.

Congratulations go to John, Andrew, Darla, Kevin and entire A Bug’s Life crew. What a special present you have given the world. All of us at Pixar feel so proud to be associated with such excellence. Congratulations also go to Jan, Karen and the Geri’s Game crew for their magical short (audiences around the country loved it too).

If you divide our Thanksgiving weekend box office of $46.5 million by the average ticket price of around $6 -- it tells us that around 7.5 million people saw our movie this weekend!! Few people ever get the chance to make 7.5 million people laugh so much and feel so good. Or 450 people at Pixar feel so proud.

Congratulations.

Steve

Email to Apple Employees

“We Did It!”

From: Steve Jobs

To: Apple employees

Subject: We did it!

Date: December 23, 1998, 9:31 a.m.

Yesterday the market data was released, and …. (drum roll) …. iMAC WAS THE NUMBER ONE SELLING COMPUTER IN THE U.S. during the month of November in U.S. retail and mail order channels, outselling all other brands!!

iMac captured 7.1% of unit volume and 8.2% of dollar volume, propelling Apple’s overall market share in retail and mail order to double from about 5% in August to 10% in November. You can read all about it on our website (www.apple.com in hot news section) or in today’s Wall Street Journal (US Edition, page A1 and B7).

This incredible achievement is the result of our hard work. Please take a moment and reflect on how far we have come in the last 18 months. It’s amazing what we have all accomplished together. I am very proud of our team, and feel so excited to be at Apple. Have a wonderful holiday.

Steve

Email to Apple Employees

“Macintosh Turns Fifteen.”

From: Steve Jobs

To: Apple employees

Subject: Macintosh Turns Fifteen

Date: January 24, 1999, 7:49 p.m.

Fifteen years ago today, on January 24, 1984, Apple launched the first Macintosh at Flint Center in Cupertino, California. Its revolutionary ease of use made computing accessible to “the rest of us”, and its infusion of graphics and typography placed the Macintosh at the intersection of computer science and liberal arts. The Macintosh went on to become the second revolution in personal computing (the Apple II was the first), and its revolutionary ideas and benefits spread beyond Apple - they have changed the face of an entire industry and touched hundreds of millions of people around the globe. Apple has a lot to be proud of here.

While I am not normally one to look back, today is a good day to remember Apple’s legacy, which is to bridge the gap between sophisticated technology and “the rest of us” who make up most of humanity. It’s our job to make complex technology easy to use and fun to use. The need for this bridge is even greater today than it was in 1984 when the Macintosh debuted. Back then, users didn’t have to deal with the complexities of connecting to networks and the Internet, setting up email, managing device drivers and init files, and all of the other things that drive today’s computer users mad.

The computer world shows no signs of getting simpler as we enter the coming century. And no other company has yet taken Apple’s place as the bridge builder. As we return to our roots and once again begin delivering simpler and better ways to use computers, Apple’s future looks both bright and secure.

It’s been an amazing journey so far, yet we have barely begun.

Steve

Email from Steve to Himself

“Apple’s Reason For Being.”

Steve often captured his thoughts by emailing himself notes.

From: Steve Jobs

To: Steve Jobs

Subject: Apple’s reason for being

Date: October 29, 2000, 4:39 p.m.

Apple is the world’s premier company at building high technology products that are easy to learn and use by mere mortals. Beginning over 20 years ago, Apple has consistently set the standard for easy to use computer systems and software. Why do we do this? Because we are in love with the potential for personal computers to enhance and enrich the lives of regular people - not just with spreadsheets and databases, but with creative

We’re a creatively driven company in everything we do. From breakthrough product features and operating systems to culturally leading product design and advertising. Heck, we INVENTED the personal computer, spawned desktop publishing and are now bringing desktop movies to millions. And our engineers are hard at work on several more exciting breakthroughs you’ll see in 2001.

Apple marries state of the art technology with Apple’s legendary ease-of-use to create products that enable users to do more

Apple is the world’s premier bridge builder between mere mortals and the exploding world of high technology. Apple enables mere mortals around the world to grasp by making it easy to learn and use

Apple is the premier company in the world at making the exploding world of high technology easy to learn and use, thereby enabling mere mortals to enrich their lives using it.

Demystified technology, it will have a much greater impact than any other thing we can do. The stores need to be thought of as a mecca for understanding technology and making all of the digits a part of your life. All things digital, digital music, digital photography, people who’ve migrated to broadband, people/families who want to build a home network

Speech to Pixar Employees

“Hopefully we captured a bit of Pixar’s soul in this building.”

In November 2000, Steve officially opened Pixar’s headquarters building in Emeryville, California.

I just wanted to say a few words as we sort of come together to live here. You know, I have seen this building, every square millimeter of this building, in almost every stage of its planning and development and construction— except for one thing, which is I have never seen people in it. It makes all the difference. This big space here looks pretty big without people in it, and with all of us in it, it looks just right.

We are moving from Point Richmond, where we have lived for eleven years, to our new home here, where I think we are going to live forever. We have land to expand here, and I think a lot of us will spend the rest of our corporate lives here.

We are moving from four buildings to one. One of the things people have said this morning is they are seeing people for the first time that work at our company. That’s both startling and kind of exciting. We have designed this building to be a more urban environment— that’s what this is, this is Town Square right here— and it’s a place for you to meet people you have never met before, a place for us to interact and collaborate, which is really the basis of our success. So it’s a very different feeling to get us all into one building, and yet provide us a nice park so we can get away from everybody else when we need to.

We bought this land four years ago. It was a giant Del Monte Fruit Cocktail factory. It had been abandoned and was decrepit. It was falling apart, and for some reason, we saw the land underneath it that no one else saw. We bought it. We bought sixteen gorgeous acres of land, which is one of the rare large parcels of land left in the Bay Area.

Deciding what building to build was really hard. When you are in rented buildings, it is easy because they don’t define who you are— they define who the landlord is! But when you decide to design your own building, you have to ask the question, “Who are we ?” Because you want to capture some of the soul of the company in the building and reflect who we are.

The design proceeded much like one of our films. First, a treatment; throw it away. Then another; throw it away. And finally, one that is worth developing— and then zillions and zillions of iterations, model after model after model, detail after detail. We drove our architects crazy. And three years of development— again, much like one of our films.

And hopefully we captured a bit of Pixar’s soul in this building. Our architects did an amazing job. This is also a hand‑made building. All the steel is hand-joined on this site and bolted together, not welded. Just the drawings to define every connection for every place where steel is bolted to steel were twenty-five hundred pages. Every brick was custom hand‑made, with an old, not-seen-today process, and then hand-laid on the site. John [Lasseter] even laid a brick— right over there. And every floorboard was hand-laid and sanded on the site. This is a handmade building, just like one of our films.

And the workers discovered that this crazy owner was letting them build a building like they always had dreamed of, like none other built these days. And they brought their families to the construction site on weekends to see their craftwork like no one else is doing today— just like all of us feel about our films. And though a little blood was spilled, no one was seriously hurt building this building, just like one of our films.

This land and building and grounds cost just shy of $100 million, about the same as it costs to make one of our films. This equals around one year’s salary and benefits for everybody in our studio: to build this whole place, to buy the land, to build the grounds. We are making this investment in our future because we want the best place in the world to support the best talent in the world making the best animated features in the world.

And we paid for it in cash. We have no debt on this building. We own it. Basically, this building was paid for with the profits we made from A Bug’s Life . And we still have over $200 million of cash in the bank. This building will be a solid investment, just like one of our films.

Pixar’s home was in Point Richmond for eleven years. Now our home is here, and many of us will spend the remainder of our professional lives here.

We are having an open house and holiday party here on Saturday, December 9, in the afternoon and early evening, when you can bring your family to see our new home. This home has been lovingly built by talented workers; the building has good karma already. But its purpose is not just to be a cool building or to impress anyone on the outside. It is to house and inspire the most talented group of filmmakers in the world— that’s you. And so today, the architects, Tom, Craig, and the amazing crew that we have here at the company, all of the facilities crew and the I.S. [Information Systems] crew, and everybody who has moved us seamlessly over the last week, we are officially turning the building over to you: the creative owners.

So go make some history here!

Steve smiling, wearing his signature black turtleneck and round glasses.
Steve, 2000.
Steve takes a break during stage rehearsals for MacWorld. Sitting on a stool, he leans forward with his hand on his chin.
Rehearsing for Macworld keynote, Tokyo, 2001.

Speech Introducing the First Apple Store

“Wouldn’t it be great if when … you had any questions, you could ask a Genius?”

In May 2001, Steve recorded a private tour of the first Apple Store for an audience of software developers. He had once suggested that the Genius Bar be called the Geek Bar.

[From the sidewalk outside the store]

Hi, I’m Steve Jobs, and I’m here at Tysons Corner Mall in Virginia, right outside of Washington, D.C. And I’m standing in front of this wood barricade we’ve built in front of our first retail store, that’s going to open in six days. Now, nobody’s seen inside here yet. And I’d like to take you inside for a secret little private tour. [Beckons toward the store]

So come on in. Now, this is our store. And the store’s divided into four parts. The first quarter of the store has our home section, with great home and education products. And our Pro section, with all our great Pro products. Every product we make is in this first 25 percent of the store. You can see the whole product line.

So come on over here. Let me show you what we’ve got going in the home section. ✂

We decided carrying our own products wasn’t enough, so we’re carrying six digital camcorders, six digital cameras, six MP3 players, and six handheld organizers. So you can come in here, and not only can you buy these digital devices, but you can actually hook them right up to the Macs and take them for a spin.

Wouldn’t it be great if when you went to buy a computer— or after you bought a computer— if you had any questions, you could ask a Genius? Well, that’s what we’ve got. This is called the Genius Bar.

I’m not a Genius, but I’ll stand behind here. There’ll be somebody here who can do service right in the store and who could answer any questions you’ve got about your Mac or any of the peripherals or software that work with it. And if that person doesn’t know the answer, they’ve got a hotline to call us in Cupertino at Apple headquarters, where we have somebody who does.

But maybe the most exciting part of the back of the store is our theater. We’ve got an exceptional rear projection screen. We can play our latest commercials back here, we can play some incredible videos back here, and of course we can play iTunes visualizations back here.

One last thing I want to show you is all our great software. Look at this stuff. We have over three hundred titles, from games to the most sophisticated pro applications. There’s something for everybody.

We can’t wait to feature your software right here, in every single one of our stores. So go write some more for us, and we’ll build more shelves for as much as you can write.

Take care.

Email to Apple Employees

“Message from Steve.”

From: Steve Jobs

To: Apple employees

Subject: Message from Steve

Date: September 17, 2001, 7:15 a.m.

Team,

Last week’s devastating and tragic events have touched everyone at Apple. We are all grieving for the victims and their families. Thankfully, no Apple team members were among them. I know many of you have taken time to support various relief efforts, give blood, and support those around you. We all appreciate your extra efforts to do so.

I want to let you know that Apple is donating one million dollars to the families of the firefighters, police and other emergency response personnel who lost their lives, through the American Red Cross Disaster Relief Fund. In addition, we will donate one iBook to each of these families with children this holiday season.

Also, today we are announcing the cancellation of Apple Expo 2001, which was scheduled to take place in Paris on September 26-30. We are very sorry to disappoint our customers and developers, but their safety must be our primary concern at this time. Apple is a very visible American company, and having a highly publicized event at this time would be irresponsible.

We are also taking extra precautions to insure our safety. Apple security will remain on heightened alert at all our facilities around the world for the foreseeable future. Please immediately report any suspicious events or personnel to security.

It’s going to take time for the world to return to “normal”, and some things will never return to the way they were. The next few months may be rocky. Please take the time you need for your families, and please lean on one another. Together, we will all get through this.

I want to commend everyone for their efforts throughout this difficult time. As always, I am very proud of this team.

Steve

Email to Apple Employees

“Excellence.”

Steve said this was his favorite quote.

From: Steve Jobs

To: Apple employees

Subject: Aristotle

Date: January 20, 2002, 12:47 p.m.

We are what we repeatedly do. Excellence, then, is not an act, but a habit. – Aristotle

A first generation iPod belonging to Steve, with clickwheel and blank screen.
Steve’s iPod. “Bring the joy of music to even more people.”
An autographed 45 of Great Balls of Fire by Jerry Lee Lewis, with bright yellow Sun records label.
A treasured gift Steve kept in his office.

Emails from Steve to Himself

“This song moves me like few others.”

Steve drafted playlists of his favorite songs and why he loved them in 2003, the year the iTunes Music Store debuted.

From: Steve Jobs

To: Steve Jobs

Subject: Re: Celebrity Playlist

Date: April 16, 2003, 9:02 p.m.

1. Sligo River Blues, In Christ There is No East or West, John Fahey

2. If I Laugh, Cat Stevens (or Trouble)

3. Every Grain of Sand, Bob Dylan

4. Baby Let Me Follow You Down, Bob Dylan, 66 tour,live, She Belongs to Me, Desolation Row, Just Like A Woman, It’s All Over Now Baby Blue

5. Vince Guaraldi, Never Never Land

6. Sweet Jane, Velvet Underground

7. Here Comes the Sun, Beatles (Ticket To Ride,

Yesterday, Nowhere Man, In My Life)

8. Little Green, Joni Mitchell

9. Camelot, Lerner & Loewe

10. Suite Judy Blue Eyes, Crosby, Stills & Nash (Wooden Ships)

11. Neil Young, Harvest

12. Hey Joe, Jimi Hendrix

13. How Long Has This Been Going On, Audrey Hepburn, Funny Face

14. Good Shepherd, Jefferson Airplane

15. You’re Still My Woman, B.B. King

16. For A Dancer, Jackson Browne

17. New Morning, Bob Dylan

18. Take Me to the River, Talking Heads,

Stop Making Sense

19. Angel From Montgomery, Bonnie Raitt

20. Father and Son, Cat Stevens

21. Minor Swing, Django Reinhardt

22. Where to Now St. Peter, Elton John

23. Colors/Dance, George Winston

24. The Dark End of the Street, James Carr

25. Come On, Let’s Go, Ritchie Valens

26. Tupelo Honey, Van Morrison

27. Judy Garland, Over the Rainbow, Live

From: Steve Jobs

To: Steve Jobs

Subject: Celebrity Playlist--writeup

Date: December 27, 2003, 4:58 p.m.

For a Dancer

I first heard this on my car radio while driving down Highway 280, and I started crying. I had not listened to much Jackson Browne at that time, and was just blown away by the lyrics. I have since listened to many more of his songs, but this one is still my favorite.

Little Green

This is an autobiographical song by Joni Mitchell about giving a daughter up for adoption. Maybe it’s because I’m adopted, but this song moves me like few others. After I realized what this song was about, I cry every time I hear it. She wrote it when she was young, and it remains one of the best of her many great songs.

I Get A Kick Out of You

From Ella’s incomparable Cole Porter Songbook album.

All I Have to Do is Dream

This song was a hit when I was very young, and I remember listening to it as my parents listened to it, which brings back great memories. It was a transition period from the 50s to the 60s -- Buddy Holly was dead, the Beatles hadn’t arrived yet, and groups like The Everly Brothers filled the gap nicely.

Email from Steve to Himself

“This is what it feels like to be living in— and creating— a Golden Age.”

Steve sent himself notes to prepare for a talk on Pixar.

From: Steve Jobs

To: Steve Jobs

Subject: Re: Golden Age

Date: May 3, 2003, 5:08 p.m.

Websters Dictionary states that the term Golden Age dates from 1555 and means: a period of great happiness, prosperity, and achievement.

Pixar is in a Golden Age. With our soon-to-be-released Finding Nemo, we have completed 5 computer animated feature films, with three more in some stage of production. All original Pixar stories. Our track record of 4 for 4, and hopefully 5 for 5, is unparalleled in the history of animation, and indeed in the history of film.

Our real Golden Age is being fueled by the maturing of our people. I have seen Pete, Andrew and Lee grow into the second generation of Pixar Directors, all with the experience of leading the creation of a feature film under their belts. I have seen Darla and Graham grow into our industry’s best producers, producing our films with greater and greater mastery. I have seen Rob Cook and his R&D team ushering in a new, second golden age of amazing new Pixar science and technology. And I see more directors and producers coming right behind them. In every aspect of our studio, I see the kind of sprouting that signals Springtime— and I think the world, which is already amazed at what we have done so far, is going to be even more amazed at what’s in store for them in the coming years.

Tonight is a night for celebration, but don’t forget to pause for a moment and realize that this is what it feels like to be living in— and creating— a Golden Age. And, 25 years from now, when a kid— maybe even your grandkid— asks you what it was like to be at Pixar during its Golden Age, you’ll know— if you can only find the words to describe it.

Interview with Leslie Berlin

“I think I got lucky and had the chutzpah to call these guys up.”

On May 24, 2003, Steve spoke with Leslie Berlin about his mentor Robert Noyce. Noyce co-invented the microchip and co-founded Intel and Fairchild Semiconductor, the first successful silicon microchip company in Silicon Valley.

Leslie Berlin: Why’d you decide to join the Grinnell College board?

Steve Jobs: Bob asked me to do it. I went to a small liberal arts college. Six months officially, and for two years I was kind of a drop-in. And so I’ve always had a soft spot for private liberal arts colleges. You really only find them in America, and they’re a very wonderful thing.

It was a fun experience. And the funnest part was just traveling down with Bob. We almost died together; I don’t know if you know about this.

LB: No.

SJ: Well, Bob’s a pilot, as you know. He flew all kinds of planes. He bought a Seabee. Have you heard about that?

LB: I’ve heard that he had this plane.

SJ: It was a Seabee. And a Seabee was a World War II plane. They stopped making them probably at the end of the forties. And there were like a hundred of them left. It was a plane whose fuselage was in the shape of the hull of a boat. Its wings were on top; and it could land on a runway or in the water.

He bought one. He had just gotten it, and he called me up and said, “Hey, there’s a Seabee fly-in up at Trinity Lake. […] Do you want to go?” I said, “Sure. Let’s go.” So, we get in the Seabee plane […] and we land in Lake Shasta. We got out and went for a swim, and it was really great. Really great. So we take off again, and it’s getting kind of hot. Bob pulls a lever that he thinks is the air ventilation. But he pulls the wrong lever. He pulled the lever that locks the wheels.

We get to Trinity Lake, and he’s landing on the runway […] and we hit the tarmac. The wheels on the wings, of course, are locked, so the plane immediately lunges forward. Sparks start flying. We very nearly flipped the plane over. It was only due to his excellent piloting that we survived.

And I was imagining, as this was all happening, the headlines: “Bob Noyce and Steve Jobs Killed in Fiery Plane Crash.” I think it was pretty close. ✂

SJ: I was young. I was in my late twenties. And Bob was— gosh, he must have been in his later forties.

LB: He was born in ’27.

SJ: Yeah, and I was born in ’55. So almost thirty years. Yeah, so, he was in his— well, Jesus— in his early fifties. And now that I’m approaching fifty, it’s easy to see how people in their fifties know more than people in their twenties.

He just kind of tried to give me the lay of the land and tried to give me a perspective that I could only partially understand.

I think he was interested in what someone in their twenties thought too. And he was fascinated by the personal computer. The personal computer and Intel had nothing to do with each other at that time. So he was fascinated by that stuff. That was it. So we just, we just became buddies.

LB: Do you recall any specific conversations or any situations where you were thinking one thing and he was suggesting otherwise, and you did or didn’t listen?

SJ: The things I remember are not the business things. It’s actually more personal stuff. I remember him trying to teach me how to ski better. I remember when I got fired from Apple, he was one of the first people to call me.

He just had a lot of soul. And I think he was the soul of Intel. Gordon [Moore] and Andy [Grove] are fantastic, but I think Bob, Bob was the soul of that place. ✂

LB: One of my favorite quotes from him is where he says that optimism is the essential ingredient for innovation.

SJ: Well, it’s optimism and passion, because it’s really hard. And if you don’t really, really care about what you’re doing, you’re gonna give up if you’re a sane person— because it’s just super hard. I’m sure it was extremely hard for him at times. ✂

SJ: When you get into your fifties— I’m forty-eight, I’m kind of there, pretty much— you’re not grabbing the pencil out of the twenty-five-year-old’s hand to do it better than they are. If you’re smart, you’re hiring twenty-five-year-olds who are smarter than you. You know things that they don’t know, and they know things that you don’t know, and it all works.

It shouldn’t have been Bob that was designing the breakthrough chips, and if it was, then he ain’t running the company. His job was to, number one, recruit; number two, set an overall direction; and number three, you know, inspire and cajole and persuade. And if that’s what he’s known for, that means he’s doing the job. He had his day when he was the young hotshot, and he came through. But that wasn’t the job. ✂

SJ: I called up him and Andy and a few other people, Jerry Sanders [the founder of microchip company AMD]. I just called them up and I said, “Look, I’m young and I’m trying to run with this company. I’m just wondering if I could buy you lunch once a quarter and pick your brain.”

And everybody I ever asked said yes. It was nice. Bob said yes. Andy Grove said yes; that was how I met Andy. Jerry Sanders said yes. It was pretty wonderful. I was very, very lucky, because I got to meet and get to know a little bit of Hewlett and Packard, too.

I sort of feel like that second era of the Valley, the semiconductor companies kind of leading into the early computer companies— I got to smell that, and I always held that very near and dear. And Bob was sort of why.

LB: Do you think this sort of generational link you’re describing here, in terms of your own relationship with Bob and, to a lesser extent, your relationship with Hewlett and Packard— do you think it’s a feature of the valley, or do you think you just got lucky and had the chutzpah to call these guys up?

SJ: Well, I think I got lucky and had the chutzpah to call these guys up. However, there are other people who have chutzpah to call people up too. The Google guys called me up, so I had lunch with them. And so I think it still happens a little. I don’t think it ever happened a lot, and I don’t think it happens a lot now. But I think it still happens— it happened a little, and it still happens a little. Maybe most people aren’t interested. They have their own things to worry about. ✂

SJ: I think there’s a tradition here [in Silicon Valley]. And that doesn’t mean that it’s a well-worn groove that you drop into. It’s not that easy. But there’s role models, and there’s legends, and there’s all sort of folklore— the kind of thing that makes a culture. And many people don’t spend the time to learn about it, which is fine. But some people find themselves in it, and slowly start to absorb it and get curious as to what came before them.

LB: And when you think of Bob giving you the lay of the land, is it along these sorts of lines, in some sense?

SJ: Oh yeah, absolutely. I mean, it’s sort of like … you know your olfactory sense is your most poignant sense in terms of its connection to your memory? You can smell something, in your thirties, and it’ll take you right back to when you were seven years old. The gift that Bob gave me was that connection and sense of smell, that strong connection back to some wonderful era of this valley that I lived in but [only] through him got a very strong sense of what it was.

It’s hard to explain. But if you could sort of transmit— when you smell that smell, and it takes you back to when you were seven— if you could give that to somebody else, that was kind of what he was able to give to me. And I don’t know how it happens, but you know it. And I was curious.

LB: Why does that sense of connection matter? Why is it so important to you?

SJ: There’s a human drama to most everything. You look at it sometimes, and it seems dry as history. But if you peel the onion, there’s humanity underneath.

Just to understand what’s going on now— you can’t really do that unless you understand how it got here. There’s a great quote by Schopenhauer. It’s a great quote. I should pull this up. I’ll get it all wrong. I’ve got to go up and get it.

[Goes upstairs, grabs Schopenhauer’s On the Suffering of the World, and then reads from it on the stairs.] “He who lives to see two or three generations is like a man who sits some time in the conjurer’s booth at a fair and witnesses the performance twice or thrice in succession. The tricks were meant to be seen only once; and when they are no longer a novelty and cease to deceive, their effect is gone.” ✂

SJ: The other thing I admired about Bob was that he gave up the CEO job at Intel. You know, he really did give it up. He wasn’t trying to run it from a back room.

He understood how important it was to have a succession, to keep the company going, not have it just be a one-man show.

Speech at Stanford’s Graduate School of Business

“You never know what’s around the next corner.”

On May 29, 2003, Steve gave a talk to MBA students about his experience as CEO of Pixar and Apple. Two years later, he would deliver his seminal Stanford commencement address on the same campus.

Pixar is a very different kind of company than Apple. Apple is a company that has new products every few weeks. It’s a company where you make ten important decisions a day, but if some of them are wrong, most of them are not terribly hard to correct a few months down the road.

Pixar is a company that has one new product a year, at best. That’s the holy grail for us: to have a movie a year, and we are just about there. As CEO, you make a few important decisions a quarter— maybe three— but they are very hard to change if you decide you want to change them.

So, they are very, very different sides of the spectrum. However, you can look at Apple and say Apple is the most creative of the technology companies, and you can look at Pixar and say Pixar is by far the most technical of the creative companies, and in that sense, maybe they are striving for some ideal in the middle, coming from different ends of the spectrum. ✂

If we [Apple] come up with a dozen innovations in a year, we can maybe advertise four or five of them. We can’t advertise more than that because, even if we had all the money in the world, the customer would get very confused with all these messages coming at them on TV. What do you do with the other half-dozen innovations you come up with? You know: six, seven, eight, nine, ten innovations? You have to communicate those with the customer at the point of sale. And it [the existing distribution channel] was not capable of that. So we decided to start our own. So that’s why we got into retail.

We did things a little differently. Our goal in retail was not just to sell to the 5 percent of people who own our products today; it was to go for the other 95. And we decided they would not drive ten miles to look at an Apple Store if they weren’t at all interested in buying our products.

We decided we had to ambush them. What that meant was that we had to go to high-traffic locations and put stores there. They [customers] didn’t have to take the risk of driving ten miles to find out they weren’t interested. They just had to take the risk of walking ten feet because they were walking by anyway, and they knew they could escape rapidly if it was something they hadn’t wanted. So we paid extra money for great locations and put them on great streets like University Avenue [in Palo Alto] and the A [-grade] malls across the country. And the real estate we’ve got is just A+. ✂

I learned this at Pixar: technology companies and content companies have absolutely no understanding of each other. None. It’s worse than you’d ever imagine. In Silicon Valley, most people think the creative process is a bunch of guys in their early thirties sitting on a couch, drinking beer, and thinking up jokes. Really. They really do.

And yet I’ve watched people at Pixar making these films, and they work as hard as I’ve seen anybody in a technology company ever work. The creative process is as disciplined as any engineering process I’ve ever seen in my life. And they’re as passionate about it as any technical person I’ve ever seen.

On the other hand, the content companies have no appreciation of the creative process in the technical companies. They think that technology is something that you write a check for and buy. That’s it. And they do not understand that there’s a wide, dynamic range of capability and elegance. They don’t understand the creativity in the process. So these are like ships passing in the night. ✂

The most important lesson I ever learned was that you have to hire people better than you are. […] In normal life, the difference in dynamic range from average to best is usually 30, 40, 50 percent. Twice as good: rarely. So the difference between an average meal in downtown Palo Alto tonight and the best one— maybe it’s two to one. Flight home, if you’re going home for the holidays: 50 percent difference. Rental cars, breakfast cereals. I don’t know, pick one.

But I saw that Woz [Apple co-founder Steve Wozniak]— one guy— having meetings in his head could run circles around two hundred engineers at Hewlett-Packard. That’s what I saw. And I thought, “Wow.” And I didn’t really understand it at first.

Then I started to understand it. It took me about ten years to actually try to put it into practice. Because you’d try to hire and find those people. And they’re really hard to find. And everyone says they are all prima donnas. But it turns out that when they work with each other, they’re not prima donnas. They really like it. The first time I tried to build that organization— that was the Mac team. And it really worked. I saw a team of fifty people do something that literally hundreds, or thousands, of people at other companies couldn’t do. And so I’ve since then always tried to find really great people who love what they are doing and are extremely good at it. And sometimes they have experience, and sometimes they’re really young. They’re diamonds in the rough— and you hire them and take chances on them. But that’s been the most important lesson I’ve learned in business: that the dynamic range of people dramatically exceeds things you encounter in the rest of our normal lives— and to try to find those really great people who really love what they do. ✂

I was basically fired from Apple. And that was really hard. So I’m sure I learned a lot from that. [Audience laughs.] I did. I did learn a lot from that. And as a matter of fact, there would have been no Pixar if that hadn’t happened. Life’s funny in this way. Sometimes your greatest strengths are your greatest weaknesses. Sometimes your greatest adversities, you learn the most from. I don’t know.

But there wouldn’t be a Pixar if it hadn’t been for that. But life is funny, you know? I never would have thought I’d end up back at Apple, but here I am. So it’s a circus world, and you never know what’s around the next corner.

People say you learn more from failures than you do from successes, and that’s probably true. And I’ve made more mistakes than most people I know. But getting older does that [helps you grow] too. You get married. You know, you have a family. And your perspective starts to change on things.

When I was young, if I had to fire somebody, I didn’t think— to be honest, I didn’t think twice about it. When you get a little older, and maybe you have kids, you realize that the person you have to fire— even if they totally screwed up, they should be fired, you should have fired them months ago, anyone else would have fired them last year— even so, you realize that that person is going to have to go home to their wife and their children and tell them they got fired today and that they don’t have a job anymore. You realize that.

So part of it is nothing that you do yourself, no accomplishments you achieve. It’s just the process of getting older and kicked around, and maybe a little wiser in the process. That’s, more than anything, probably what it is. ✂

[When I returned to Apple in 1997,] the individual contributors were phenomenal. And I asked a lot of these guys, “Why did you stay?” And they said, “Because we bleed six colors.” I heard that from a lot of people— there were six colors in the old Apple logo. It was management that was a problem. So we actually got rid of most of the management team and promoted a lot of these young people into management positions.

And what I found is that nobody in their right mind wants to be a manager. [Audience laughs.] It’s true. It’s a lot of work, and you don’t get to do the fun stuff. But the only good reason to be a manager is so some other bozo doesn’t be the manager— and ruin the group you care about.

Really. And if you’ve lived through a bad situation where you’ve had bad management, you’ll do anything to not have your group destroyed by that again. And you will even step up and be the manager yourself, even though you don’t want to do that.

And I talked a few hundred people into doing that. And 90, over 90 percent of them have turned into extraordinary managers. Extraordinary. So that’s what saved Apple, those people right there. And it’s been one of the great experiences of my life to have the privilege of working with them. ✂

There’s a lot of management techniques. I’m sure you study a lot of management techniques. When I was younger, it was management by objective. It’s all a crock. They’re all after-the-fact management techniques: “You’ve failed, and I know that because we are going out of business tomorrow.” All after the fact. “You’ve ruined this department; all the good people have left. So now I’m firing you.” “You’ve accomplished none of your objectives.” It doesn’t work.

And a really smart guy I met a long time ago who used to teach at Disney University— Walt Disney recruited him to run Disney University, actually— he told me about his point of view, which I’ve remembered to this day. He called it management by values. What that means is you find people that want the same things you want, and then just get the hell out of their way.

The way I describe it is, let’s say we’re all going to take a trip together. The first thing is to figure out where we all want to go. The worst thing is if we all decide we want to go to different places. You can never manage it. [Pointing] You want to go to New Orleans. You want to go somewhere else. I want to go to San Francisco. You want to go to San Diego.

It doesn’t work. Right?

But if we all want to go to San Diego, that’s the key. Then we can argue about how to get there. [Pointing] You think it’s better to walk. You think it’s better to take a plane. You think it’s better to take a train. We’ll figure that [part] out. Because if I say, “I want to take a train to San Diego,” and somebody goes, “That’s really stupid! It will take three days! We can fly and be there in an hour,” I’ll go, “Oh. OK.” Because, actually, I want to go to San Diego . So if I can get there in an hour [flying], I’ll ditch my idea about the train.

That’s what management by values is. It’s finding people with passion that want to go to San Diego— who want to go to the same place you want to go to! Right? That’s the key.

And so, what happened at Apple was that Apple’s goals used to be to make the best personal computers in the world. And then the second goal was to make a profit so we could keep on doing number one. Right?

What happened was that, for a time, those got reversed: “We want to make a bunch of money, and so, OK, to do that, we’re going to have to make some good personal computers.” But it didn’t work. It never works. And so things start to fall apart.

Those subtle changes in values can mean everything. The higher up in the organization they are, the more pervasive influence they have. So if you want to preserve something, what you want to do is have a good enough place to go, that’s got a long enough focal length that it will survive over time, that everybody agrees on— and not codify how you’re going to get there. So that each generation can argue anew about the best way to get to San Diego, and they’re not just taking your footsteps on how you got there. You see what I’m saying? But all the people want to go to the same place.

And that’s one of my mantras around Apple and Pixar: that recruiting is the most important thing that you do. Finding the right people— that’s half the battle.

Well, thank you guys for the chance to be with you. I appreciate it very much.

A stack of 20 Bob Dylan CDs from the collection belonging to Steve.
“He’s my favorite of all time.”
Eight bright metal harmonicas, from brands such as Marine Band and Hohner, that Steve collected.
Steve’s harmonicas. He learned to play while hitchhiking.
Steve sits cross-legged on the phone in his home office. The floor is strewn with boxes, papers, a tripod, and a trumpet. Steve leans across a cluttered desk to use his Mac. Behind him, shelves are piled with books and boxes litter the floor.
In his home office, 2004.

Drafting the Stanford Commencement Address

In January 2005, John Hennessy, the president of Stanford, asked Steve to give the commencement address to that spring’s graduating class. Steve agreed.

On and off for the next six months, Steve took stabs at writing his talk. He emailed stories and memories to himself. He asked friends, Apple colleagues, and the screenwriter Aaron Sorkin for their thoughts. In the end, however, he wrote the speech on his own. Even three days before the event, Steve was unsatisfied with his talk. He sent it to a friend, warning, “I’ll send it to you, but please don’t puke. I never do stuff like this.” He was still refining the speech the morning that he gave it. Uncharacteristically, Steve read from the lectern, rather than memorizing his text (as he did with Apple keynotes) or speaking extemporaneously from a few scrawled notes (as he did in nearly every other talk).

Steve was happy with the speech— he emailed himself a copy a few days after giving it— but he generally deflected the praise that he received for it. “I bought it on CommencementSpeeches.com,” he joked to one person. The commencement address has been viewed millions of times online and is included in school curricula around the world. The following are ideas Steve emailed to himself as he drafted the talk.

From: Steve Jobs

To: Steve Jobs

Subject: Commencement

Date: January 15, 2005, 10:35 a.m.

This is the closest I’ve ever come to graduating from college

- they’ve asked me to say a few things today that you can learn from

- I should be learning from you

1. Habits are very powerful things

- 20 years - now until you’re 42

- meditate 20 minutes a day

- walk to work (30 minutes × 2 = 1 hour/day × 5 days/week

= 250/yr × 20 years = 5000hrs / 10 hours per day = 500 days = >1 year

2. You are what you eat

- how many cows, chickens, milk, soda, coffee

- fasting (jesus) 1 day per week

3. Curiosity

- autobiography of a yogi quote “what is miraculous is all around us”

4. When I was 20 years old I took classes here just as Apple was getting started

- no money

- my wife and I donating in your name one free scholarship per year for offbeat student

You are going into a crazy world

- nuclear, biological weapons, less than great leadership, etc Gandhi quote - become the change you want to see in the world

So, I’m going to focus on you - your inner world - how you approach your life

From: Steve Jobs

To: Steve Jobs

Subject: Stanford Speech

Date: May 1, 2005, 11:10 p.m.

I never graduated from college. Matter of fact, this right here is the closest I’ve ever got to a college graduation. So, if you’re foolish enough to listen to me today, I want to give you three pieces of advice as you leave college.

First, when I was around your age I made the public statement “Don’t trust anyone over 30.” Of course I meant it at the time. Now I am 50 years old, and it’s funny how, when you get to be my age, you begin to see more value in experience. But whether its sheer IQ points or life’s experience, the first piece of advice I want to give you is to try to always surround yourself with people smarter than you. This is how you get smarter and deeper. It doesn’t matter what their field is. They can be from a completely different walk of life than you - in fact, that’s even better. Not long after we started Apple I hired a genius engineer who was terribly old at the time - I think he was in his 40s - who was, in addition to being the best analog engineer on the planet, was also a revolutionary socialist. He turned me on to Das Kapital and America’s 60 Families and things I had never been exposed to before.

You have begun your 20s, and during this decade you will meet many amazing people. And some great teachers - mentors - who you will never forget. But remember, a teacher is someone who stands with you in the dark and holds their flashlight just long enough for you to find your own flashlight.

Second,

Third, we are all going to die. You are going to die

Added May 2, 7:20 p.m.

A wise observer of the economic scene once commented that “what can be left to later, usually is -- and then, alas, it’s too late.” I don’t want to let that stand as the epitaph of what has been an unparalleled period of success for the American economy and of enormous potential for the world at large.

Added May 5, 6:55 a.m.

STAY CURIOUS

Reed - taking calligraphy - no practical value - learned about typography, fonts, etc - 10 years later working on mac - if I had never taken class then no multiple fonts, proportional spaced multiple fonts on Macs or PCs

ENDING

When I was your age there was an amazing publication called The Whole Earth Catalog which influenced my generation. It was created not far from where we stand today. They put out several issues, and then when it had run its course put out a final issue which, on the back cover had a photograph of a country road in the morning, the kind you might find yourself hitchhiking on if you were so adventurous, with the saying “Stay hungry. Stay foolish.” It was their farewell morsel of wisdom to us all. Stay Hungry. Stay Foolish. And I have always wished that for myself. And I wish that for you.

Thank you very much.

Added May 5, 9:09 a.m.

When I was your age there was an amazing publication called The Whole Earth Catalog which influenced my generation. It was created in Menlo Park, not far from where we stand today. They put out several issues, and then when it had run its course put out a final issue. That was 1974, and I was your age. One the back cover of their final issue was a photograph of an early morning country road, the kind you might find yourself hitchhiking on if you were so adventurous, with the words: “Stay hungry. Stay foolish.” It was their farewell jewel of wisdom as they signed off. Stay Hungry. Stay Foolish. And I have always wished that for myself. And [now] I wish that for you. Thank you very much.

From: Steve Jobs

To: Steve Jobs

Subject: You can’t plan to meet the people who will change your life

Date: June 7, 2005, 11:46 p.m.

You can’t plan to meet the people who will change your life. I am invited to speak at Stanford’s business school once or twice a year, and I always try to do it. I had accepted an invitation to speak one Thursday late in the afternoon, and I wasn’t feeling very well and I had a dinner later that evening with some important customers up at a winery on Page Mill Road. The room for my talk wasn’t large enough, and all the seats were full so some of the students were sitting in the aisles. One of the professors asked them to clear the aisles in case a fire marshal should appear, and one girl who was being evicted quickly sat down in one of the four seats they had left vacant in the front row for me and whatever entourage I might be bringing. When I arrived alone and sat down in the front row, it didn’t take me long to notice this really cute girl sitting next to me. I think she was stunned when it was me that got up to speak. And I knew something was up when I was staring at her, forgetting what I was talking about mid-sentence. After my talk, I stayed around to speak with some students, and she stayed too. But then she left. I didn’t know who she was, and thought I might never see her again. So I wound things up and left too, and I caught up with her in the parking lot. I asked her if she would have dinner with me on Saturday.

She said yes and gave me her phone number. As I was walking to my car, I asked myself: “If this was the last day of my life, would I rather have dinner with the important customers or her?” I raced back to her car, just as she was about to drive off, and asked her “How about dinner tonight?” She said: “Sure,” and we were married 18 months later. Yea, it might have worked out if I had waited until Saturday night, and those customers might have given us a few more orders if I had shown up.

But who knows, maybe she had a hot date Friday night and things would have turned out much differently… . You can’t plan to meet the people who will change your life. It just happens. Maybe its random, maybe its fate. Either way, you can’t plan for it. But you want to recognize it when it happens, and have the courage and clarity of mind to grab onto it.

From: Steve Jobs

To: Steve Jobs

Subject: Starting Apple

Date: June 7, 2005, 11:55 p.m.

When my partner, Steve Wozniak, and I started Apple, most of our friends and our family told us we were nuts. Woz had a great job designing handheld calculators at Hewlett Packard and I had a fun job designing games at Atari. And we were giving them up to start this company to make a primitive computer on a PC board that a handful of hobbyists, mostly our friends, might buy for less than it costs us to make. I remember talking to Woz, and saying: “We may fail, but we have no responsibility now, no wives, no kids, no house payments, nothing. If we don’t do this now, we never will. We have nothing to lose - the worst we’ll get out of this is that we’ll have the memories of having gone for it.” To give ourselves the experience of participating in what Bill Hewlett and Dave Packard did-- to start a company. So rather than invest in better cars, or better apartments, or our bank accounts, we decided to invest in ourselves.

From: Steve Jobs

To: Steve Jobs

Subject: Story #3?

Date: June 9, 2005, 10:15 p.m.

The most important thing I’ve ever encountered to help me make the big choices is to remember that I’ll be dead soon. I know it sounds a bit dramatic, but it’s true. And when I remember this, I realize that all of the expectations and standards and restrictions of others and society mean nothing in the end. I realize that I have nothing to lose by following my heart and intuition, even if I embarrass myself or fail in the eyes of others. Because I’ll be dead soon. And I realize that I don’t have forever to decide to find what my intuition tells me is waiting out there for me.

When I was 17, I read a quote that said something like “If you live each day as if it was your last, someday you’ll most certainly be right.” And since I was 17, I have looked in the mirror every morning and asked myself “If today was the last day of my life, would I want to do what I am about to do today?” And when the answer has been “NO” for too many days in a row, I know I need to change something in my life.

Another way to think about this is that your life is a story. It’s hard to see it that way when you’re looking forward at 22. But imagine yourself as an old person looking back on your life. Your life will be a story. It will be your story, with its highs and lows, its heros and villains, its forks in the road that mean everything.

And if you can remember that your life is a story in the making, it will help you make those important decisions. When you have to decide between taking the prestigious job that pays well, or the offbeat job with no future that makes your heart sing, just imagine yourself looking back on your life in 50 years and you’ll know what path is yours. You will give yourself the right advice. You will intuitively know if something is part of your story or not.

I’m 50 years old now, and my story is entering its third act. I can tell you with certainty that those times when I have followed my gut, heart and intuition …

Steve smiles at the lectern in graduation robes while giving the 2005 commencement speech at Stanford University.
At the lectern at Stanford.

Commencement Address at Stanford University

“Stay hungry. Stay foolish.”

On June 12, 2005, Steve delivered the commencement address at Stanford University.

I am honored to be with you today for your commencement from one of the finest universities in the world. Truth be told— I never graduated from college. This is the closest I’ve ever gotten to a college graduation. Today I want to tell you three stories from my life. That’s it. No big deal. Just three stories.

The first story is about connecting the dots. I dropped out of Reed College after the first six months but then stayed around as a drop-in for another eighteen months or so before I really quit. So why did I drop out?

It started before I was born. My biological mother was a young, unwed graduate student, and she decided to put me up for adoption. She felt very strongly that I should be adopted by college graduates, so everything was all set for me to be adopted at birth by a lawyer and his wife. Except that when I popped out they decided at the last minute that they really wanted a girl. So my parents, who were on a waiting list, got a call in the middle of the night asking, “We have an unexpected baby boy; do you want him?” They said, “Of course.” My biological mother found out later that my mother had never graduated from college and that my father had never graduated from high school. She refused to sign the final adoption papers. She only relented a few months later when my parents promised that I would go to college. This was my start in life.

And seventeen years later I did go to college. But I naïvely chose a college that was almost as expensive as Stanford, and all of my working-class parents’ savings were being spent on my college tuition. After six months, I couldn’t see the value in it.

I had no idea what I wanted to do with my life and no idea how college was going to help me figure it out. And here I was spending all of the money my parents had saved their entire life. So I decided to drop out and trust that it would all work out OK. It was pretty scary at the time, but looking back, it was one of the best decisions I ever made. The minute I dropped out, I could stop taking the required classes that didn’t interest me and begin dropping in on the ones that looked far more interesting.

It wasn’t all romantic. I didn’t have a dorm room, so I slept on the floor in friends’ rooms. I returned Coke bottles for the five-cent deposits to buy food with, and I would walk the seven miles across town every Sunday night to get one good meal a week at the Hare Krishna temple. I loved it. And much of what I stumbled into by following my curiosity and intuition turned out to be priceless later on. Let me give you one example.

Reed College at that time offered perhaps the best calligraphy instruction in the country. Throughout the campus, every poster, every label on every drawer, was beautifully hand calligraphed. Because I had dropped out and didn’t have to take the normal classes, I decided to take a calligraphy class to learn how to do this. I learned about serif and sans-serif typefaces, about varying the amount of space between different letter combinations, about what makes great typography great. It was beautiful, historical, artistically subtle in a way that science can’t capture, and I found it fascinating.

None of this had even a hope of any practical application in my life. But ten years later, when we were designing the first Macintosh computer, it all came back to me. And we designed it all into the Mac. It was the first computer with beautiful typography. If I had never dropped in on that single course in college, the Mac would have never had multiple typefaces or proportionally spaced fonts. And since Windows just copied the Mac, it’s likely that no personal computer would have them. If I had never dropped out, I would have never dropped in on that calligraphy class, and personal computers might not have the wonderful typography that they do. Of course, it was impossible to connect the dots looking forward when I was in college. But it was very, very clear looking backwards ten years later.

Again, you can’t connect the dots looking forward; you can only connect them looking backwards. So you have to trust that the dots will somehow connect in your future. You have to trust in something— your gut, destiny, life, karma, whatever. Because believing that the dots will connect down the road will give you the confidence to follow your heart, even when it leads you off the well-worn path. And that will make all the difference.

My second story is about love and loss. I was lucky— I found what I loved to do early in life. Woz and I started Apple in my parents’ garage when I was twenty. We worked hard, and in ten years Apple had grown from just the two of us in a garage into a $2 billion company with over four thousand employees. We had just released our finest creation— the Macintosh— a year earlier, and I had just turned thirty. And then I got fired. How can you get fired from a company you started? Well, as Apple grew, we hired someone who I thought was very talented to run the company with me, and for the first year or so things went well. But then our visions of the future began to diverge, and eventually we had a falling-out. When we did, our board of directors sided with him. So, at thirty, I was out. And very publicly out. What had been the focus of my entire adult life was gone, and it was devastating.

I really didn’t know what to do for a few months. I felt that I had let the previous generation of entrepreneurs down— that I had dropped the baton as it was being passed to me. I met with David Packard and Bob Noyce and tried to apologize for screwing up so badly. I was a very public failure, and I even thought about running away from the Valley. But something slowly began to dawn on me— I still loved what I did. The turn of events at Apple had not changed that one bit. I had been rejected, but I was still in love. And so I decided to start over.

I didn’t see it then, but it turned out that getting fired from Apple was the best thing that could have ever happened to me. The heaviness of being successful was replaced by the lightness of being a beginner again, less sure about everything. It freed me to enter one of the most creative periods of my life.

During the next five years, I started a company named NeXT, another company named Pixar, and fell in love with an amazing woman who would become my wife. Pixar went on to create the world’s first computer animated feature film, Toy Story, and is now the most successful animation studio in the world. In a remarkable turn of events, Apple bought NeXT, and I returned to Apple, and the technology we developed at NeXT is at the heart of Apple’s current renaissance. And Laurene and I have a wonderful family together.

I’m pretty sure none of this would have happened if I hadn’t been fired from Apple. It was awful-tasting medicine, but I guess the patient needed it. Sometime life’s gonna hit you in the head with a brick. Don’t lose faith. I’m convinced that the only thing that kept me going was that I loved what I did. You’ve got to find what you love. And that is as true for your work as it is for your lovers. Your work is going to fill a large part of your life, and the only way to be truly satisfied is to do what you believe is great work. And the only way to do great work is to love what you do. If you haven’t found it yet, keep looking. Don’t settle. As with all matters of the heart, you’ll know when you find it. And, like any great relationship, it just gets better and better as the years roll on. So keep looking. Don’t settle.

My third story is about death. When I was seventeen, I read a quote that went something like, “If you live each day as if it was your last, someday you’ll most certainly be right.” It made an impression on me, and since then, for the past thirty-three years, I have looked in the mirror every morning and asked myself, “If today were the last day of my life, would I want to do what I am about to do today?” And whenever the answer has been “No” for too many days in a row, I know I need to change something.

Remembering that I’ll be dead soon is the most important tool I’ve ever encountered to help me make the big choices in life. Because almost everything— all external expectations, all pride, all fear of embarrassment or failure— these things just fall away in the face of death, leaving only what is truly important. Remembering that you are going to die is the best way I know to avoid the trap of thinking you have something to lose. You are already naked. There is no reason not to follow your heart.

About a year ago, I was diagnosed with cancer. I had a scan at 7:30 in the morning, and it clearly showed a tumor on my pancreas. I didn’t even know what a pancreas was. The doctors told me this was almost certainly a type of cancer that is incurable, and that I should expect to live no longer than three to six months. My doctor advised me to go home and get my affairs in order, which is doctor’s code for “prepare to die.” It means to try to tell your kids everything you thought you’d have the next ten years to tell them in just a few months. It means to make sure everything is buttoned up so that it will be as easy as possible for your family. It means to say your goodbyes.

I lived with that diagnosis all day. Later that evening I had a biopsy, where they stuck an endoscope down my throat, through my stomach, and into my intestines, put a needle into my pancreas, and got a few cells from the tumor. I was sedated, but my wife, who was there, told me that when they viewed the cells under a microscope the doctors started crying because it turned out to be a very rare form of pancreatic cancer that is curable with surgery. I had the surgery, and thankfully, I’m fine now.

This was the closest I’ve been to facing death, and I hope it’s the closest I get for a few more decades. Having lived through it, I can now say this to you with a bit more certainty than when death was a useful but purely intellectual concept.

No one wants to die. Even people who want to go to heaven don’t want to die to get there. And yet death is the destination we all share. No one has ever escaped it. And that is as it should be, because death is very likely the single best invention of life. It is life’s change agent. It clears out the old to make way for the new. Right now the new is you, but someday, not too long from now, you will gradually become the old and be cleared away. Sorry to be so dramatic, but it is quite true.

Your time is limited, so don’t waste it living someone else’s life. Don’t be trapped by dogma— which is living with the results of other people’s thinking. Don’t let the noise of others’ opinions drown out your own inner voice. And most important, have the courage to follow your heart and intuition. They somehow already know what you truly want to become. Everything else is secondary.

When I was young, there was an amazing publication called The Whole Earth Catalog , which was one of the bibles of my generation. It was created by a fellow named Stewart Brand, not far from here in Menlo Park, and he brought it to life with his poetic touch. This was in the late 1960s, before personal computers and desktop publishing, so it was all made with typewriters, scissors, and Polaroid cameras. It was sort of like Google in paperback form, thirty-five years before Google came along. It was idealistic, overflowing with neat tools and great notions.

Stewart and his team put out several issues of The Whole Earth Catalog , and then, when it had run its course, they put out a final issue. It was the mid-1970s, and I was your age. On the back cover of their final issue was a photograph of an early-morning country road, the kind you might find yourself hitchhiking on if you were so adventurous. Beneath it were the words, “Stay hungry. Stay foolish.” It was their farewell message as they signed off. Stay hungry. Stay foolish. And I have always wished that for myself. And now, as you graduate to begin anew, I wish that for you.

Stay hungry. Stay foolish.

Thank you all very much.

The front of the 1974 Whole Earth Epilog, featuring a picture taken from the Moon of the Earth rising above the horizon. The back of the 1974 Whole Earth Epilog. A handwritten message in white ink reads: For Steve —Stewart Brand.
After the address, Stewart Brand gave Steve a signed catalog.

Email Exchange Between Steve and an Apple Customer

“Please ‘reset’ the iPod.”

From: [ ____ ]

To: Steve Jobs

Subject: ipod malfunctioning

Date: July 27, 2005, 11:16 p.m.

Hi, my name is [ ____ ], and my father has an ipod. recently (1-2 days ago), he was charging his ipod and then, the next morning, when he checked it, it wouldn’t turn on, he checked the hold button but it wasn’t it, so, he realized that the ipod had just “died”.

We live in venezuela and want to know where can we fix the ipod. I think its still on warranty, because, we bought it around 11 months ago, so we would like to know what to do, if theres any store autorized by apple here in Caracas-Venezuela, so we can take the ipod and check it to see whats wrong and if you guys can fix it. well thanks for the help.

[ ____ ]

Stay Hungry. Stay Foolish.

From: Steve Jobs

To: [ ____ ]

Subject: Re: ipod malfunctioning

Date: July 28, 2005, 7:19 a.m.

Please “reset” the ipod by holding down the center button and the Menu button at the same time for at least 5 seconds. This should work.

Steve

Speech to Apple Employees

“We thought we could make a contribution by making something truly great that we loved.”

The day before the release of the iPhone in 2007, Steve held a meeting with Apple employees. In his invitation, he wrote, “We’re launching the most revolutionary and exciting product in Apple’s history this Friday. And given Apple’s legacy of breakthrough products, that’s saying a lot.”

You know, one of the reasons we started doing this [was] we could see that we were getting better and better at iPods, and we could see that there was an opportunity to maybe do the next thing— and what should it be?

And it wasn’t driven by a bunch of market research or financial spreadsheets about how big certain markets were. It wasn’t driven by that at all. It was driven by the fact that we all hated our phones. We talked to all of our friends and all the people we knew, and they all hated their phones.

And we thought, “This is a really important device, and everybody hates it. They don’t know how to use even 10 percent of the features that are on these phones!”

I mean, I’ll use this as an example: you’ve seen how slick it is to set up a conference call on an iPhone, right? You know, almost every phone lets you set up a conference call. Nobody knows it! They’ll never figure it out! It’s on page 93 of the manual they didn’t read, and it’s seven, eight cryptic keys! And you can do it, and nobody ever does it— because they don’t know it’s there. And that’s true of feature after feature after feature.

And what our competence is, is making complex technology easy to use and self-discoverable. And we’ve done it in spades on the iPhone. So that’s how we got here. We thought we could make a contribution by making something truly great that we loved.

And we’ve done it. We love this thing. And starting tomorrow at 6 p.m., we’re going to find out if other people love it as much as we do.

Speech at Macworld

“Every once in a while, a revolutionary product comes along that changes everything.”

Steve introduced the iPhone on January 9, 2007.

This is a day I’ve been looking forward to for two and a half years.

Every once in a while, a revolutionary product comes along that changes everything. And Apple has been— well, first of all, one’s very fortunate if you get to work on just one of these in your career. Apple’s been very fortunate. It’s been able to introduce a few of these into the world.

In 1984, we introduced the Macintosh. It didn’t just change Apple, it changed the whole computer industry. In 2001, we introduced the first iPod, and it didn’t just change the way we all listen to music, it changed the entire music industry.

Well, today, we’re introducing three revolutionary products of this class.

The first one is a widescreen iPod with touch controls. The second is a revolutionary mobile phone. And the third is a breakthrough internet communications device. So, three things: a widescreen iPod with touch controls; a revolutionary mobile phone; and a breakthrough internet communications device.

An iPod, a phone, and an Internet communicator. An iPod, a phone … Are you getting it?

These are not three separate devices; this is one device. And we are calling it iPhone.

Today … today Apple is going to reinvent the phone, and here it is. [The screen displays an iPod with a rotary phone-style dial where the click wheel would be.]

No, actually, here it is. [Slides an iPhone out of his pocket.]

But we’re going to leave it there for now. [Pockets the iPhone again.]

So, before we get into it, let me talk about a category of things. The most advanced phones are called smartphones. So they say. And they typically combine a phone, plus some email capability, plus they say it’s the internet— it’s sort of the baby internet— into one device. And they all have these plastic little keyboards on them.

And the problem is that they’re not so smart, and they’re not so easy to use. So if you kinda make a Business School 101 graph of the smart axis and the easy-to-use axis— phones, regular cellphones, are kinda right there. [Points to a spot indicating “not so smart, somewhat easy to use.”] They’re not so smart, and they’re not so easy to use. Smartphones are definitely a little smarter, but they actually are harder to use. They’re really complicated. Just for the basic stuff, people have a hard time figuring out how to use them.

Well, we don’t want to do either one of these things. What we want to do is make a leapfrog product that is way smarter than any mobile device has ever been and super easy to use. This is what iPhone is. OK?

So we’re going to reinvent the phone. Now, we’re going to start … with a revolutionary user interface … the result of years of research and development, and, of course, it’s an interplay of hardware and software. Now, why do we need a revolutionary user interface? I mean, here’s four smartphones, right? Motorola Q, the BlackBerry, Palm Treo, Nokia E62 — the usual suspects. And what’s wrong with their user interfaces? Well, the problem with them is really sort of in the bottom forty [keys] there. It’s, it’s this stuff right here. They all have these keyboards that are there whether you need them or not to be there. And they all have these control buttons that are fixed in plastic and are the same for every application.

Well, every application wants a slightly different user interface, a slightly optimized set of buttons, just for it. And what happens if you think of a great idea six months from now? You can’t run around and add a button to these things. They’re already shipped. So what do you do? It doesn’t work, because the buttons and the controls can’t change. They can’t change for each application, and they can’t change down the road if you think of another great idea you want to add to this product.

Well, how do you solve this? Hmm. It turns out, we have solved it! We solved it in computers twenty years ago. We solved it with a bitmapped screen that could display anything we want. Put any user interface up. And a pointing device. We solved it with the mouse. Right? We solved this problem.

So how are we going to take this to a mobile device? What we’re going to do is get rid of all these buttons and just make a giant screen. A giant screen. Now, how are we going to communicate with this? We don’t want to carry around a mouse, right? So what are we going to do? Oh, a stylus, right? We’re going to use a stylus. No. No. Who wants a stylus? You have to get them and put them away, and you lose them. Yuck. Nobody wants a stylus. So, let’s not use a stylus.

We’re going to use the best pointing device in the world. We’re going to use a pointing device that we’re all born with— we’re born with ten of them. We’re going to use our fingers. We’re going to touch this with our fingers. And we have invented a new technology called multitouch, which is phenomenal. It works like magic. You don’t need a stylus. It’s far more accurate than any touch display that’s ever been shipped. It ignores unintended touches, it’s super smart. You can do multi-finger gestures on it. And boy, have we patented it.

So, so, we have been very lucky to have brought a few revolutionary user interfaces to the market in our time. First was the mouse. The second was the click wheel. And now we’re going to bring multitouch to the market. And each of these revolutionary user interfaces has made possible a revolutionary product— the Mac, the iPod, and now the iPhone. So, a revolutionary user interface.

We’re going to build on top of that with software. Now, software on mobile phones is like, is like baby software. It’s not so powerful. Today we are going to show you a software breakthrough. Software that’s at least five years ahead of what’s on any other phone. Now, how do we do this? Well, we start with a strong foundation: iPhone runs OS X.

Now, why, why would we want to run such a sophisticated operating system on a mobile device? Well, because it’s got everything we need. It’s got multitasking. It’s got the best networking. It already knows how to power manage. We’ve been doing this on mobile computers for years. It’s got awesome security. And the right apps. It’s got everything from Cocoa and the graphics, and it’s got core animation built in, and it’s got the audio and video that OS X is famous for. It’s got all the stuff we want. And it’s built right into iPhone. And that has let us create desktop-class applications and networking, right. Not the crippled stuff that you find on most phones. This is real, desktop-class applications.

Now, you know, one of the pioneers of our industry, Alan Kay, has had a lot of great quotes throughout the years. And I ran across one of them recently that explains how we look at this— explains why we go about doing things the way we do, ’cause we love software. And here’s the quote: “People who are really serious about software should make their own hardware.” You know, Alan said this thirty years ago, and this is how we feel about it. And so we’re bringing breakthrough software to a mobile device for the first time. It’s five years ahead of anything on any other phone.

Steve holds a first generation iPhone up to his face and smiles; the phone is out of focus.
“This is going to change everything,” 2007.
Steve, holding an iPhone, smiles and looks to the side as a pair of hands enters the frame, holding a color chart.
The day Steve introduced the iPhone, 2007.
Steve shown in profile with stubbly beard and glasses. The photo is taken over his shoulder.
In 2007, Apple Computer Inc. was renamed “Apple Inc.”

Interview at the All Things Digital D5 Conference

“There’s something exciting around the next corner.”

In a rare joint interview, Steve and Microsoft chairman Bill Gates spoke with journalists Kara Swisher and Walt Mossberg onstage in May 2007.

Kara Swisher: The first question I was interested in asking is what you think each has contributed to the computer and technology industry— starting with you, Steve, for Bill, and vice versa.

Steve Jobs: Well, Bill built the first software company in the industry. And I think he built the first software company before anybody really in our industry knew what a software company was, except for these guys. And that was huge. That was really huge. And the business model that they ended up pursuing turned out to be the one that worked really well for the industry. I think the biggest thing was, Bill was really focused on software before almost anybody else had a clue that it was really the software that—

KS: Was important?

SJ: That’s what I see. I mean, a lot of other things you could say, but that’s the high-order bit. And I think building a company’s really hard, and it requires your greatest persuasive abilities to hire the best people you can and keep them at your company and keep them working, doing the best work of their lives, hopefully. And Bill’s been able to stay with it for all these years.

Walt Mossberg: Bill, how about the contribution of Steve and Apple?

Bill Gates: Well, first, I want to clarify: I’m not Fake Steve Jobs. What Steve’s done is quite phenomenal, and if you look back to 1977, that Apple II computer, the idea that it would be a mass-market machine, you know, the bet that was made there by Apple uniquely. There were other people with products, but the idea that this could be an incredible, empowering phenomenon— Apple pursued that dream. Then one of the most fun things we did was [developing software for] the Macintosh, and that was so risky. People may not remember that Apple really bet the company. Lisa hadn’t done that well, and some people were saying that general approach wasn’t good, but the team that Steve built even within the company to pursue that, even some days it felt a little ahead of its time; I don’t know if you remember that Twiggy disk drive and …

SJ: 128K.

KS: Oh, the Twiggy disk drive, yes.

BG: Steve gave a speech once, which is one of my favorites, where he talked about, in a certain sense, “We build the products that we want to use ourselves.” And so he’s really pursued that with incredible taste and elegance that has had a huge impact on the industry.

And his ability to always come around and figure out where that next bet should be has been phenomenal. Apple literally was failing when Steve went back and reinfused the innovation and risk-taking that have been phenomenal. So the industry’s benefited immensely from his work. We’ve both been lucky to be part of it, but I’d say he’s contributed as much as anyone.

SJ: We’ve also both been incredibly lucky to have had great partners that we started the companies with, and we’ve attracted great people. I mean, so, everything that’s been done at Microsoft and at Apple has been done by just remarkable people, none of which are sitting up here today. ✂

SJ: [When I returned to Apple,] there were too many people at Apple and in the Apple ecosystem playing the game of, for Apple to win, Microsoft has to lose. And it was clear that you didn’t have to play that game because Apple wasn’t going to beat Microsoft. Apple didn’t have to beat Microsoft. Apple had to remember who Apple was because they’d forgotten who Apple was. So, to me, it was pretty essential to break that paradigm. And it was also important that Microsoft was the biggest software developer outside of Apple developing for the Mac. So it was just crazy what was happening at that time. And Apple was very weak, and so I called Bill up and we tried to patch things up.

BG: And since that time, we’ve had a team that’s fairly dedicated to doing the Mac applications, and they’ve always been treated kind of in a unique way so that they can have a pretty special relationship with Apple. And that’s worked out very well. In fact, every couple years or so, there’s been something new that we’ve been able to do on the Mac, and it’s been a great business for us.

SJ: The relationship between the Mac development team at Microsoft and Apple is a great relationship. It’s one of our best developer relationships.

KS: And do you look at yourselves as rivals now? Today, as the landscape has evolved— and we’ll talk about the internet landscape and everything else and other companies that have [gone] forward— but how do you look at yourselves in this landscape today?

WM: Because, I mean, you are competitors in certain ways, which is the American way, right?

KS: We watch the commercials, right?

WM: And you get annoyed at each other from time to time.

KS: Although, you know what? I have to confess, I like PC guy.

WM: Yeah, he’s great.

KS: Yeah, I like him. The young guy, I want to pop him.

SJ: The art of those commercials is not to be mean, but it’s actually for the guys to like each other. Thanks. PC guy is great. Got a big heart.

BG: His mother loves him.

SJ: His mother loves him. ✂

SJ: There’s a lot of things that happened that I’m sure I could have done better when I was at Apple the first time, and a lot of things that happened after I left that I thought were wrong turns, but it doesn’t matter. It really doesn’t matter, and you kind of got to let go of that stuff, and we are where we are. So we tend to look forward.

And, you know, one of the things I did when I got back to Apple ten years ago was I gave the museum to Stanford, and all the papers and all the old machines, and kind of cleared out the cobwebs and said, “Let’s stop looking backwards here. It’s all about what happens tomorrow.” Because you can’t look back and say, “Well, gosh, you know, I wish I hadn’t have gotten fired, I wish I was there, I wish this, I wish that.” It doesn’t matter. And so let’s go invent tomorrow rather than worrying about what happened yesterday.

KS: We’re going to talk a little bit about tomorrow, but let’s talk about today, the landscape of how you see the different players in the market and how you look at what’s developing now. […] There are many, many companies that are becoming quite powerful. How do you look at the landscape at this moment and what’s happening, especially in the internet space?

SJ: I think it’s super healthy right now. I think there’s a lot of young people out there building some great companies, who want to build companies, who aren’t just interested in starting something and selling it to one of the big guys, but who want to build companies. And I think there’s some real exciting companies getting built out there. Some next-generation stuff that, you know, some of us play catch-up with, and some of us find ways to partner with, and things like that. But there’s a lot of activity out there now, wouldn’t you say?

BG: Yeah, I’d say it’s a healthy period. The notion of what the new form factors look like, what natural interface can do, the ability to use the cloud, the Internet, to do part of the task in a complementary way to the local experience, there’s a lot of invention that the whole approach of start-ups, the existing companies who do research— we’ll look back at this as one of the great periods of invention.

SJ: I think so, too. There’s a lot of things that are risky right now, which is always a good sign. You can see through them, you can see to the other side and go, “Yes, this could be huge,” but there’s a period of risk that nobody’s ever done it before.

KS: Do you have an example?

SJ: I do, but I can’t say. But I can say: when you feel like that, that’s a great thing. That’s what keeps you coming to work in the morning, and it tells you there’s something exciting around the next corner. ✂

WM: Five years from now, what’s going to be on that pocket device?

SJ: I don’t know. And the reason I don’t know is because I wouldn’t have thought that there would have been maps on it five years ago, but something comes along, gets really popular, people love it, get used to it, and you want it on there. So people are inventing things constantly, and I think the art of it is balancing what’s on there and what’s not on there— the editing function. And clearly, most things you carry with you are communications devices. You want to do some entertainment with them as well, but they’re primarily communications devices, and that’s what they’re going to be. ✂

KS: Steve? I know you’re working on something, it’s going to be beautiful; we’ll see it soon.

WM: And you can’t talk about it.

SJ: Yeah.

WM: Bill discusses all his secret plans. You don’t discuss any.

SJ: I know, it’s not fair. But I think the question is a very simple one, which is how much of the really revolutionary things people are going to do in the next five years are done on the PCs, or how much of it is really focused on the post-PC devices? And there’s a real temptation to focus it on the post-PC devices because it’s a clean slate, and because they’re more focused devices, and because they don’t have the legacy of these zillions of apps that have to run in zillions of markets.

And so I think there’s going to be tremendous revolution, you know, in the experiences of the post-PC devices. Now, the question is how much to do in the PCs. And I think— I’m sure Microsoft is— we’re working on some really cool stuff, but some of it has to be tempered a little bit because you do have, you know, these tens of millions in our case, or hundreds of millions in Bill’s case, users that are familiar with something. You know, they don’t want a car with six wheels. They like the car with four wheels. They don’t want to drive with a joystick. They like the steering wheel.

And so, as Bill was saying, in some cases you have to augment what exists there, and in some cases you can replace things. But I think the radical rethinking of things is going to happen in a lot of these post-PC devices. ✂

KS: What’s the greatest misunderstanding in your relationship and about each other? What would you say would be— this idea of catfight? Which one of the many?

SJ: We’ve kept our marriage secret for over a decade now.

KS: Canada. That trip to Canada.

[Audience laughs and applauds.]

BG: I don’t think either of us have anything to complain about, in general. And I know that the projects, like the Mac project, was just an incredible thing, a fun thing where we were taking a risk. We did look a lot younger in that video.

SJ: We did.

KS: You looked twelve in the first one.

BG: That’s how I try and look.

SJ: He was twelve.

BG: But no, it’s been fun to work together. I actually kind of miss some of the people who aren’t around anymore. You know, people come and go in this industry. It’s nice when somebody sticks around, and they have some context of all the things that have worked and not worked. The industry gets all crazy about some new thing, you know. There’s always this paradigm of “the company that’s successful is going to go away” and stuff like that. It’s nice to have people seeing the waves and waves of that and yet, when it counted, to take the risk to bring in something new.

WM: One last question and then we’ll go to the audience.

KS: Oh, no, he didn’t answer us.

WM: Sorry, what?

SJ: I haven’t answered.

WM: Oh, I’m sorry.

KS: He only talked about his secret gay marriage, so …

WM: Oh, I thought that was your answer.

SJ: No, that wasn’t my answer. You know, when Bill and I first met each other and worked together in the early days, generally, we were both the youngest guys in the room, right? Individually or together. I’m about six months older than he is, but roughly the same age. And now, when we’re working at our respective companies, I don’t know about you, but I’m the oldest guy in the room most of the time. And that’s why I love being here.

WM: Happy to oblige. Happy to oblige.

SJ: And, you know, I think of most things in life as either a Bob Dylan or a Beatles song, but there’s that one line in that one Beatles song, “You and I have memories longer than the road that stretches out ahead.” And that’s clearly true here.

Email Exchange with Art Levinson

“Sic Transit Gloria.”

In November 2007, Fortune named Steve “CEO of the Decade.” Art Levinson, the CEO of Genentech and chair of Apple’s board, sent a congratulatory note.

From: Art Levinson

To: Steve Jobs

Subject: Fortune

Date: November 27, 2007, 11:26 a.m.

The “Beethoven of business” - not sure it gets any better than that.

Congratulations!

From: Steve Jobs

To: Art Levinson

Subject: Re: Fortune

Date: November 27, 2007, 12:43 p.m.

Sic Transit Gloria . (All glory is fleeting.)

Steve

Speech to the Press

“Five months ago, I had a liver transplant.”

Steve spoke to the press at a music and media event on September 9, 2009.

Good morning. Thank you very much. Thank you. Thank you. [Sustained audience applause.] Thanks. I know. I’m very happy to be here today with you all.

As some of you may know, about five months ago, I had a liver transplant. So I now have the liver of a mid-twenties person who died in a car crash and was generous enough to donate their organs. And I wouldn’t be here without such generosity. So I hope all of us can be as generous and elect to become organ donors.

I’d like to take a moment and thank everybody in the Apple community for the heartfelt support I got, too. It really meant a lot. And I’d also like to especially thank Tim Cook and the entire executive team of Apple. They really rose to the occasion and ran the company very ably in that difficult period. So thank you, guys.

Let’s give them a round of applause.

So I’m vertical. I’m back at Apple, loving every day of it. And I’m getting to work with our incredibly talented teams to come up with some great new products for you all in the future.

So it’s, it’s wonderful— and thank you. So today, we get to talk about music …

Viewed from over his shoulder, Steve considers a range of swatches made from materials proposed for use in Apple Park.
Thinking through the Apple Park design, 2010.

Steve on the iPad

A week after the introduction of the iPad in 2010, Steve held a meeting for employees in Apple’s “Town Hall.” He reflected on what made the device— and the company— special to him.

The great thing about what we do is we approach it like— like Pixar approaches their movies. I know that sounds funny, but I will explain.

Pixar doesn’t make kids’ films. Walt Disney’s definition of family entertainment was it’s entertaining to every member of the family: kids, the teens, the adults. That’s what Pixar’s always tried to do. There’s a lot of layers in one of their films if you’ve seen them, and they’re really fun for kids, but they’re also incredibly great films for adults, if you’ve seen Ratatouille or Up or Finding Nemo —or any one of their films.

And when we sit down to design products [at Apple], we don’t think, “Oh, well, our target audience is fifteen to twenty-nine, male.” We don’t think that way. We think about making a great product for just about everybody. And the beauty of the products we make is they can be tailored with software to do almost anything.

So we weren’t thinking, in the iPad, of any specific audience, but we’re thinking about everybody. ✂

We don’t have to go home at night and tell our kids when they say, “Well, what do you do? What did you do today?” “Well, I worked on our next-generation server, you know, that’ll be powering something or other.” We can say, “I worked on our next-generation iPad. You know, the ones that you use in school.”

And that’s a really wonderful thing.

Speech at the Lucile Packard Children’s Hospital at Stanford

“One simple question.”

In March 2010, Steve advocated for a bill requiring the California Department of Motor Vehicles to ask driver’s license applicants if they want to register as organ donors. The bill, which later passed and was signed into law by Governor Arnold Schwarzenegger, also established a state registry for people to donate kidneys during their lifetime.

Thank you, Governor Schwarzenegger.

Last year, I received a liver transplant. I was very fortunate, because many others died waiting to receive one. Last year in California, there were six hundred and seventy-one liver transplants. But last year there were also over thirty-four hundred people waiting for a liver, and over four hundred of them died waiting in California.

I was almost one of the ones that died waiting for a liver in California last year. I was receiving great care here at Stanford, but there were simply not enough livers in California to go around, and my doctors here advised me to enroll in a transplant program in Memphis, Tennessee, where the supply-demand ratio of livers is more favorable than it is in California here. And I was lucky enough to get a liver in time. As a matter of fact, this coming week is my one-year anniversary.

So why aren’t there more organs available in California? Because in California, like most other states in the nation, you must specifically request to become an organ donor at the Department of Motor Vehicles, when you’re there to get or renew your driver’s license. No one asks you if you want to become a donor. And there’s no marketing campaign to make you aware of this opportunity either, so unless you know about it and unless you specifically ask, nobody is going to ask you. Nobody is going to give you this opportunity.

And yet even with this obscure procedure, over 20 percent of Californians have signed up to be organ donors, which is fantastic.

But imagine what it could be if everyone knew of this opportunity.

And that’s what the governor’s bill will do. It will simply require the DMV to ask you if you’d like to become an organ donor. That’s it. Asking this one simple question may double the number of transplant organs available in California. One simple question. And that’s a very high return on investment, especially for the over twenty thousand Californians currently waiting for an organ transplant.

So, Governor, thank you for your leadership on this bill. And now I’d like to introduce Senator Alquist.

Thank you.

Email to Apple Employees

“Today.”

From: Steve Jobs

To: Apple employees

Subject: Today

Date: May 26, 2010, 5:59 p.m.

Team,

As most of you already know, at the close of today’s stock market, Apple’s market cap (stock price × number of shares) surpassed Microsoft’s market cap.

As I once said in a company email sent a long time ago: “Stocks go up and down, and things may be different tomorrow, but I thought it was worth a moment of reflection today.” And so it is again.

Walt Disney used to say to his team: “We’re only as good as our next picture.” Well, we’re only as good as our next amazing new product.

Back to work…

Steve

Email from Steve to Himself

[No Subject]

From: Steve Jobs

To: Steve Jobs

Subject:

Date: September 2, 2010, 11:08 p.m.

I grow little of the food I eat, and of the little I do grow I did not breed or perfect the seeds.

I do not make any of my own clothing.

I speak a language I did not invent or refine.

I did not discover the mathematics I use.

I am protected by freedoms and laws I did not conceive of or legislate, and do not enforce or adjudicate.

I am moved by music I did not create myself.

When I needed medical attention, I was helpless to help myself survive.

I did not invent the transistor, the microprocessor, object oriented programming, or most of the technology I work with.

I love and admire my species, living and dead, and am totally dependent on them for my life and well being.

Sent from my iPad

Dressed in dark colors, Steve walks away from the camera through a full-scale mockup of a corridor at Apple Park.
Walking through a mockup for Apple Park, 2010.
Steve is on stage. Behind him are two oversized, intersecting street signs: one says Technology, the other Liberal Arts.
“Technology alone is not enough,” 2011.
Steve gestures with his hands as he presents the plans for Apple Park to the Cupertino City Council.
Presenting plans to the Cupertino City Council, 2011.
A candid photograph of Steve, wearing a baseball cap and black T-shirt, as he talks on his iPhone outdoors.
Steve on his iPhone, circa 2010–2011.
The official resignation from Steve on August 24, 2011, written on Apple letterhead.
Steve’s resignation from Apple, August 2011.
A pair of round-rimmed glasses belonging to Steve sit on a table.

Key Events

January 1975— Popular Electronics publishes a story about the Altair 8800, sparking the microcomputer revolution. That same year, Bill Gates drops out of Harvard to design programming languages for the Altair. Meanwhile, in Silicon Valley, a club launches for people who want to build their own computers: the Homebrew Computer Club, which Steve occasionally attends with Steve Wozniak.

April 1976— Steve and Woz co-found Apple Computer to sell the Apple I, designed by Woz. (A third co-founder, Ron Wayne, drops out ten days after joining.) Apple I buyers must supply their own keyboards and television monitors, as well as know how to write hexadecimal code and use a soldering iron.

January 1977— Apple incorporates, with ownership split evenly among Steve, Woz, and the angel investor Mike Markkula.

April 1977— The Apple II, a more user-friendly computer, debuts at the West Coast Computer Faire in San Francisco. At $1,298, the Apple II costs about twice as much as a year of in-state tuition at the University of California.

May 1978— Lisa Brennan Jobs is born.

May 1979— Apple’s publications department manager, Jef Raskin, begins work on an inexpensive computer he calls Macintosh.

December 1979— During a visit to Xerox’s Palo Alto Research Center, Steve sees, for the first time, a networked computer with a mouse, windows, icons, menus, and multiple typefaces. “I was so blown away,” he later recalled. He brings the technology, and several PARC researchers, to Apple.

September 1980— Steve takes over the high-profile Lisa computer project. He is removed nine months later, after the team rebels against his management style.

December 1980— Apple goes public in one of the most successful initial public offerings in American history up to that time.

February 1981— Ten years after dropping out of the University of California, Berkeley, Woz leaves Apple to re-enroll.

Steve takes over the Macintosh project.

August 1981— Apple faces its first real competition when IBM introduces its personal computer. IBM’s market share soon surpasses Apple’s.

January 1983— Time shakes up its “Man of the Year” tradition to choose the computer as “Machine of the Year.”

Apple introduces the Lisa computer, priced at $10,000, targeting business users. It fails in the market.

April 1983— Steve recruits the Pepsi executive John Sculley to be Apple’s CEO, using the memorable line, “Do you want to sell sugar water for the rest of your life, or do you want to come with me and change the world?”

January 1984— Macintosh debuts.

June 1985— When Macintosh sales fall far short of projections, Apple lays off 20 percent of employees and announces the first quarterly loss in its history.

September 1985— After losing a power struggle with Sculley, Steve leaves Apple with five Mac team members in tow. Within days, Apple sues for breach of fiduciary responsibility and charges Steve with masterminding a “nefarious” scheme to steal trade secrets for his new computer company, NeXT.

January 1986— Steve becomes the majority shareholder in Pixar, after the pioneering computer scientist Alan Kay introduces him to the company’s leaders, Ed Catmull and Alvy Ray Smith.

NeXT and Apple settle their lawsuit over NeXT’s launch and Steve’s recruitment of Mac team members out of court.

August 1986— Steve attends the premiere of Pixar’s first animated short, Luxo Jr. , at a graphics industry conference. Luxo Jr. showcases Pixar’s software, but it is the audience’s standing ovation for the storytelling that catches Steve’s attention.

October 1988— Steve unveils the NeXT Computer System at Davies Symphony Hall in San Francisco. A highlight is the machine’s five-minute performance of a Bach violin concerto, accompanied by a violinist from the symphony. The computer did not sell well but had its fans; Tim Berners-Lee wrote the code for the World Wide Web on a NeXT computer.

March 1989— Pixar’s film Tin Toy wins an Oscar for Best Animated Short, a first for a computer-animated movie.

March 1991— Steve marries Laurene Powell in a ceremony at Yosemite.

July 1991— Facing heavy financial pressure, Pixar signs a deal with Disney that is far more favorable to the larger company. In exchange for financing up to three Pixar films, Disney owns the films and their characters, receives most of the films’ profits, and prohibits Pixar from pitching to another studio any ideas that Disney rejects.

September 1991— Reed Jobs is born.

February 1993— Steve ends NeXT’s production of computers to focus entirely on software.

October 1993— With Apple losing tens of millions of dollars every quarter, CEO John Sculley, who pushed Steve out of Apple in 1985, resigns. Apple will cycle through two more CEOs in the next three years but fail to regain its footing.

August 1995— Erin Jobs is born.

November 1995— Toy Story , the world’s first full-length fully animated feature film, earns $29 million in its opening weekend. It goes on to become the top-grossing animated movie of the year.

A week after Toy Story opens, Pixar holds a successful initial public offering. It was a bold bet placed months earlier; if Toy Story had been a bust, the IPO would have been one, too.

December 1996— In need of a new operating system, Apple acquires NeXT for $427 million. As part the agreement, Steve rejoins Apple as a special adviser to the CEO, Gil Amelio.

February 1997— The success of Toy Story and Pixar’s IPO give Steve leverage to negotiate a more favorable agreement with Disney. The companies sign a five-picture deal.

June 1997— In a public show of no confidence in Amelio, Steve sells a huge block of the Apple shares he received in the NeXT acquisition. Three months later, Steve is named interim CEO.

August 1997— Bill Gates appears on a giant video screen at Macworld to announce Microsoft’s commitment to developing Microsoft Office for Mac. When the audience begins heckling, Steve reprimands them: “We have to let go of […] this notion that for Apple to win, Microsoft has to lose.”

September 1997— Apple introduces the “Think Different” advertising campaign. One year later, it wins an Emmy Award for Outstanding Commercial.

March 1998— Steve hires Tim Cook as Apple’s chief of operations.

July 1998— Eve Jobs is born.

August 1998— Apple debuts the Bondi Blue iMac. The “i” stands for “internet,” targeting consumers who want to “surf the web” as easily as catching a wave at Australia’s Bondi Beach.

October 1998— Apple announces its first profitable year since 1995.

January 2000— In the last three minutes of his Macworld presentation, Steve surprises the audience with the announcement that he will drop “interim” from his CEO title.

March 2000— The wave of internet optimism crashes. The NASDAQ loses nearly $1 trillion in a single month, and hundreds of start-up companies fail in a “dotcom bust.”

November 2000— Pixar’s new campus in Emeryville, California, opens. Steve has been so involved in the headquarters design— from the town hall atrium to the bathrooms— that people call it “Steve’s movie.”

May 2001— The first Apple retail stores open in Tysons, Virginia, and Glendale, California.

March 2001— Apple releases OS X, an operating system based on the NeXTStep software developed at NeXT. Updated versions of the operating system remain at the heart of many Apple products today.

October 2001— Apple introduces the iPod. It’s a new kind of product for the company, not a computer but a portable device built to sync with a computer. Apple built a music player, Steve says, because “We love music, and it’s always good to do something you love.”

April 2003— Apple opens the iTunes Music Store, making it easy to buy individual songs online. The store is only available for Apple computers, but users download one million tracks in the first week. Six months later, at the urging of his executive team, Steve agrees to make the store compatible with non-Apple computers.

March 2004— Pixar’s fifth film, Finding Nemo , wins an Oscar for Best Animated Feature.

July 2004— Steve undergoes surgery to remove a pancreatic neuroendocrine tumor.

January 2006— Disney acquires Pixar for $7.4 billion in stock. Steve becomes Disney’s largest shareholder and joins the board of directors, while John Lasseter and Ed Catmull assume leadership of Disney Animation.

January 2007— The iPhone debuts. Steve calls it “the most revolutionary and exciting product in Apple’s history.”

March 2009— On leave from Apple, Steve receives a liver transplant in Memphis, Tennessee.

January 2010— Steve introduces the iPad, calling it “a magical and revolutionary device.”

August 2010— Toy Story 3 becomes the highest-grossing animated movie of all time.

August 2011— Six weeks before his death, Steve resigns as Apple’s CEO and recommends that Tim Cook replace him. In his resignation letter, Steve writes, “I’ve made some of the best friends of my life at Apple, and I thank you all for the many years of being able to work alongside you.”

One More Thing …

Life can be much broader once you discover one simple fact— and that is: everything around you that you call life was made up by people that were no smarter than you.

And you can change it.

You can influence it.

You can build your own things that other people can use.

And the minute you can understand that you can poke life, and if you push in, then something will pop out the other side; that you can change it, you can mold it— that’s maybe the most important thing: to shake off this erroneous notion that life is there, and you’re just going to live in it versus embrace it, change it, improve it, make your mark upon it.

I think that’s very important, and however you learn that, once you learn it, you’ll want to change life and make it better. Because it’s kind of messed up in a lot of ways.

Once you learn that, you’ll never be the same again.

— Steve, 1994

Copyright © 2023 Steve Jobs Archive.
All rights reserved.

Make Something Wonderful

Steve Jobs in his own words


Designed by LoveFrom

Typeset in LoveFrom Serif

Credits

Speeches, Interviews and Photos

“There’s lots of ways to be, as a person,” speech at Apple Town Hall, 2007; Steve on his Childhood, courtesy Smithsonian Institution’s National Museum of American History, 1995; Interviews by David Sheff, 1984, courtesy SJA; “A lot of people put a lot of love into these products,” courtesy Smithsonian Institution’s National Museum of American History, 1995; Steve on Launching Apple, interview by David Sheff, 1984, courtesy SJA; Fresh Air with Terry Gross interview reprinted with the permission of WHYY, Inc., 1996; “It’s a domesticated computer,” Talk of the Town: Digitizing, by Thomas Whiteside, reprinted courtesy Thomas Whiteside / The New Yorker © Condé Nast, 1977; “Computers and society are out on a first date,” speech at the International Design Conference in Aspen, 1983; On the Macintosh, interview by David Sheff, 1984, courtesy SJA; “Was George Orwell right about 1984?” speech at an Apple sales meeting, 1983; “Your aesthetics get better as you make mistakes,” interview by Michael Moritz, 1984, courtesy SJA; “I want to build things” interview reprinted with permission of Newsweek (1985) courtesy EnVeritas Group Inc.; “You never achieve what you want without falling on your face a few times” Fresh Air with Terry Gross interview reprinted with the permission of WHYY, Inc., 1996; Steve on Starting NeXT interview reprinted with permission of Newsweek (1985) courtesy EnVeritas Group Inc.; On Becoming Majority Shareholder in Pixar, interview produced by Joel Bloom, 1996, courtesy SJA; On Pixar’s Early Days, interview by Leslie Iwerks, courtesy The Walt Disney Company, 2003; “Character is built not in good times, but in bad times,” speech at Reed College, 1991; “One of the things I always tried to coach myself on was not being afraid to fail,” Fresh Air with Terry Gross interview reprinted with the permission of WHYY, Inc., 1996; “What you follow with your heart,” speech at Palo Alto High School, 1996; “To put these stories into culture,” interview produced by Joel Bloom, 1996, courtesy SJA; “The worst thing that someone can do in an interview is agree with me,” from In the Company of Giants: Candid Conversations with the Visionaries of the Digital World, by Dr. Rama Dev Jager and Rafael Ortiz, reprinted courtesy McGraw Hill © 1997; “Much of what I stumbled into,” from commencement address at Stanford University, 2005; “Steve on returning to Apple,” speech at Stanford University Graduate School of Business, 2003; “We believe that people with passion can change the world for the better,” speech to Apple employees, 1997; “Apple is coming back in a very big way,” speech at Macworld in New York City, 1998; “Hopefully we captured a bit of Pixar’s soul,” speech to Pixar Employees at opening of Pixar’s headquarters in Emeryville, California, 2000; “Wouldn’t it be great if,” introduction to Apple Store, 2001; “I think I got lucky,” interview with Leslie Berlin, 2003, courtesy SJA; “You never know what is around the next corner,” speech at Stanford’s Graduate School of Business, 2003; “Stay hungry. Stay foolish,” Commencement Address at Stanford University, 2005; “We thought we could make a contribution,” speech to Apple employees, 2007; “Every once in a while a revolutionary product comes along,” speech at Macworld, 2007; Interview with Kara Swisher and Walt Mossberg, reprinted with permission of the Wall Street Journal, Copyright © 2007 Dow Jones & Company, Inc. All Rights Reserved Worldwide; “Five months ago, I had a liver transplant,” speech at Apple music and media event, 2009; Steve on the iPad, speech at Apple Town Hall, 2010; “One simple question,” speech at the Lucile Packard Children’s Hospital, 2010; “One more thing,” interview with Silicon Valley Historical Association, 1994.

Information on Apple’s trademarks is available at www.apple.com.
Apple emails courtesy of Apple.

Other company and product names may be trademarks of their respective owners.

A Polaroid of young Steve, with long hair and mustache, wearing a tuxedo with a lop-sided bow tie.
Photographer unknown, 1976
Two-year-old Steve rides a tricycle with tassels on the handlebars toward the camera.
Photographer unknown, 1957
A passport for Steven Paul Jobs issued on May 11, 1973, featuring a stamped photo of him with shaggy hair.
Photo by Ben Blackwell courtesy SJA
A yellowing postcard with a Buddhist poem handwritten by Steve. The last line reads: Dont waste your life.
Courtesy Elizabeth Holmes
Steve hunches over a wooden desk, his hands on a keyboard hooked up to an Apple I circuit board.
Photo by Joe Melena courtesy of Apple, 1976
An Apple I that Steve kept in his office. Photo by Barbara Kinney courtesy SJA
Photo by Barbara Kinney courtesy SJA
An open briefcase. The top half houses an Apple I circuit board, wired up to a keyboard in the bottom half.
Photo by Barbara Kinney courtesy SJA
A Polaroid of a computer screen, with a fan letter praising Woz and Steve. A handwritten note dates it to 1976.
Photo by Dean Phillips, 1976
Wearing suit pants and a vest, Steve is in front of a small crowd, looking over his shoulder at the camera.
Photographer unknown, 1977
A scruffy-bearded Steve smiles at the camera as he leans back in his office chair and talks on the telephone.
Photo by Bill Kelley, 1977
With tidy hair and a plaid shirt, Steve poses with an Apple II displaying drawings of Sesame Street characters.
Photo by Ted Thai, 1981
Steve peers through the lens of a Nikon camera as his elbow rests on the Apple II in front of him.
Photo by Andrée Abecassis, 1981
Parked outside his Tudor-style home, Steve shares the seat of his BMW motorcycle with a black and white dog.
Photo by Ted Thai, 1981
Viewed through a conference crowd, Steve - his clean-shaven, grinning face - is displayed on a large screen.
Photo by Jean Pigozzi, 1983
Steve, wearing a black leather jacket, holds his middle finger up toward a large IBM sign.
Photo by Jean Pigozzi, 1983
A Polaroid of Steve, playing pinball, with a white hooded sweatshirt with the word Mackintosh on the chest.
Photo by Bill Atkinson, 1983
Steve is fast asleep on a flight, his seat slightly reclined.
Photo by Jean Pigozzi, 1984
29-year-old Steve wearing a black turtleneck and jeans, poses with a Macintosh computer resting in his lap.
Photo by Norman Seeff, 1984
Steve, silhouetted, is bent over as he observes a curly-haired woman typing on a Mac in a storefront.
Photo by Jean Pigozzi, 1984
A glamorous magazine shot shows Steve, wearing a double-breasted blazer, with arms crossed and hair blowing in the wind.
Photo by Norman Seeff, 1984
Steve, at home, is dressed in jeans and a sweater. He holds a mug and stares pensively at the camera from his leather sofa.
Photo by Steve Ringman / San Francisco Chronicle / Hearst Newspapers via Getty Images, 1985
Steve gazes into the camera as he cradles his chin in his right hand, his left hand resting on top of a newspaper on the table.
Photo by Jeffrey Aaronson, 1984
Steve, in a white shirt, writes on a whiteboard. He has a look of concentration and is sticking out his tongue.
Photo by Doug Menuez, 1986
A memo from Steve on the NeXT letterhead proposes meeting-free Thursdays and warns against managerial bureaucracy.
Photo by Ben Blackwell courtesy of Apple
Steve sits in a chair, hand cradling his head, and stares at a life-size pink foam NeXT computer prototype.
Photographer unknown, 1987
Steve kicks a beach ball in a grassy park. He is dressed in ripped jeans and a patterned shirt with rolled-up sleeves.
Photo by Doug Menuez, 1987
A note on Pixar letterhead. The handwritten text reads: Steve, President Clinton is holding.
Photo by Ben Blackwell, © Disney / Pixar
Steve, now wearing glasses, sits cross-legged in a director's chair under an oversized anglepoise Pixar-style lamp.
Photo by Louie Psihoyos, 1995, © Disney / Pixar
Steve, smiling, poses with Mickey and Minnie Mouse at Disneyland.
Photographer unknown, 1995, © Disney
Steve’s vintage slide rule sits next to its brown leather case.
Photo by Barbara Kinney courtesy SJA
Steve stands against the white garage door of the house where Apple started in 1976. The house number reads 2066.
Photo by Diane Cook and Len Jenshel / Contour RA / via Getty Images, 1996
A typed page from a speech at Palo Alto High School, with handwritten notes strewn across the bottom in blue ink.
Photo by Ben Blackwell courtesy SJA
A ticket to the event on December 14, 1996, illustrated with a tuxedoed man waltzing with a woman in a flowing gown.
© Pixar, 1996
Steve has his back turned to the camera as he writes on a whiteboard. He wears a black turtleneck, shorts and sneakers.
Photo by Diana Walker / SJA / Contour RA via Getty Images, 1997
Steve stands in a dimly-lit stairwell opposite a large brick wall. He wears a vest over a white shirt.
Photo by Diana Walker / SJA / Contour RA via Getty Images, 1997
Steve sits in a conference room, leaning his head on his hand. A projector and whiteboard are blurred in the background.
Photo by Diana Walker / SJA / Contour RA via Getty Images, 1997
Steve stands in a courtyard, deep in thought, with his head down and his chin resting in the palm of his hand.
Photo by Diana Walker / SJA / Contour RA via Getty Images, 1997
Steve leans back on his chair in a conference room. He is wearing shorts, legs stretched out and feet on the table.
Photo by Diana Walker / SJA / Contour RA via Getty Images, 1997
Steve lies on his back on a carpeted floor, talking on a cellphone and looking at the ceiling with his hand on his head.
Photo by Diana Walker / SJA / Contour RA via Getty Images, 1997
In his dark home office, a small lamp illuminates Steve as he sits at a cluttered desk.
Photo by Diana Walker / SJA / Contour RA via Getty Images, 1997
Steve in his signature turtleneck, eyes cast upward and fingers knitted, shortly after returning to Apple.
Photo by Diana Walker / SJA / Contour RA via Getty Images, 1997
A photograph from a reel shows Steve in portrait. Only the left side of his face is visible; the rest is in deep shadow.
Photo by Jeffery Newbury, 1997
Steve relaxes with his feet up in a full-sized model of the chair of the queen from the Pixar movie A Bugs Life.
Photo by Brant Ward / San Francisco Chronicle / Polaris, 1998, used by permission of Disney / Pixar
A Toy Story snow globe that Steve kept in his office, with a three-eyed alien and a base that reads: Ooooh!
Photo by Ben Blackwell courtesy SJA, snow globe © Disney / Pixar
Steve smiling, wearing his signature black turtleneck and round glasses.
Photo by Christian Witkin, 2000
Steve takes a break during stage rehearsals for MacWorld. Sitting on a stool, he leans forward with his hand on his chin.
Photo by Brent Schlender, 2001
A first generation iPod belonging to Steve, with clickwheel and blank screen.
Photo by Barbara Kinney courtesy SJA
An autographed 45 of Great Balls of Fire by Jerry Lee Lewis, with bright yellow Sun records label.
Photo by Barbara Kinney courtesy SJA
A stack of 20 Bob Dylan CDs from the collection belonging to Steve.
Photo by Barbara Kinney courtesy SJA
Eight bright metal harmonicas, from brands such as Marine Band and Hohner, that Steve collected.
Photo by Barbara Kinney courtesy SJA
Steve sits cross-legged on the phone in his home office. The floor is strewn with boxes, papers, a tripod, and a trumpet.
Photo by Diana Walker / SJA / Contour RA via Getty Images, 2004
Steve leans across a cluttered desk to use his Mac. Behind him, shelves are piled with books and boxes litter the floor.
Photo by Diana Walker / SJA / Contour RA via Getty Images, 2004
Steve smiles at the lectern in graduation robes while giving the 2005 commencement speech at Stanford University.
Photo by Linda A. Cicero / Stanford News Service, 2005
The front of the 1974 Whole Earth Epilog, featuring a picture taken from the Moon of the Earth rising above the horizon.
Photo by Ben Blackwell courtesy SJA
The back of the 1974 Whole Earth Epilog. A handwritten message in white ink reads: For Steve —Stewart Brand.
Photo by Ben Blackwell courtesy SJA
Steve holds a first generation iPhone up to his face and smiles; the phone is out of focus.
Photo by Peer Grimm / Alamy, 2007
Steve, holding an iPhone, smiles and looks to the side as a pair of hands enters the frame, holding a color chart.
Photo by Joe Pugliese courtesy of Apple, 2007
Steve shown in profile with stubbly beard and glasses. The photo is taken over his shoulder.
Photo by Sean Gallup / Getty Images News via Getty Images, 2007
Viewed from over his shoulder, Steve considers a range of swatches made from materials proposed for use in Apple Park.
Photo courtesy Foster + Partners, 2010
Dressed in dark colors, Steve walks away from the camera through a full-scale mockup of a corridor at Apple Park.
Photo courtesy Foster + Partners, 2010
Steve is on stage. Behind him are two oversized, intersecting street signs: one says Technology, the other Liberal Arts.
Photo by Kimihiro Hoshino / AFP via Getty Images, 2011
Steve gestures with his hands as he presents the plans for Apple Park to the Cupertino City Council.
AP / City of Cupertino, 2011
A candid photograph of Steve, wearing a baseball cap and black T-shirt, as he talks on his iPhone outdoors.
Photo by Jean Pigozzi, 2010
The official resignation from Steve on August 24, 2011, written on Apple letterhead.
Courtesy of Apple, 2011
A pair of round-rimmed glasses belonging to Steve sit on a table.
Photo by Barbara Kinney courtesy SJA.

© Steve Jobs Archive 2023

Why Continuations are Coming to Java (2019)

Lobsters
www.infoq.com
2026-10-05 20:59:38
Comments...
Original Article

Transcripts

Pressler: Hi, my name is Ron Pressler, and I work at Oracle here in London as part of the Java platform group. That's the group that develops OpenJDK, designs the Java language, the JVM and the core libraries, and I serve as a technical lead for Project Loom. That is the project that's intended to add continuations and fibers to the JDK. This talk is not part of the Java track, it's part of the programming languages track, so I've been asked to make it more theoretical.

We're going to start with quite a bit of theory, things are going to get more concrete later on, but this gives me an opportunity to speak about and to focus on the subjects or aspects of the project that I don't normally get to talk about, but if you are more interested in the more down-to-earth aspects of the project, then you can search for other Loom talks on YouTube. As I said, I work for Oracle, so I'm required to show you the slide that basically says that everything I'm about to tell you is aligned.

Views of Computation

Our views of computation have changed based on what we use computation for. Traditionally or in the past, we used to think of computation as being deterministic. What we see here, the circles, we can think of them as program states, and on left we have deterministic computation. We don't know and want to say we start the program, because you can think of it as different command line arguments, but once we do, there's only one outgoing edge for each state. Once we start we know where we're going to end up, this view sees computation as sort of a function, you have an input and you get an output.

In the last few decades, we've been using computation more and more interactive systems, and subsystems and non-deterministic and if we are at any point in the program, the next state could possibly be more than one. One of the reasons for that is interaction or IO, that's the one mainly we are going to be talking about. For example, if you want to read an input from the user or read something from a socket, you don't know what it's going to be, so you don't know what the next program state is going to be. Other sources of nondeterminism can be concurrency or threading. If you let the Kernel scheduler decide which thread to schedule on the CPU, you don't know as a programmer which one is going to be, so the next program, so it can be one of many possible ones.

We're going to talk about how different languages do IO. I am going to start with a paradigm that is perhaps the most closely associated with the deterministic viewer programming that is pure functional programming. In this paradigm, a language term, see like a subroutine application behaves similarly to a function application in math. For any given value of input you necessarily get only one, always the same output, the question is, how do we describe non-deterministic programs? We said nondeterminism is something that's forced on us by virtue of being interactive in such a world.

We can think of nondeterminism instead of saying, "The next thing I'm going to do is either this or that," we can imagine that there is an additional parameter that we don't know what it is and that additional parameter really determines our next steps. Things are sequential and deterministic, but in this way we can model nondeterminism. If you think about it, I can toss a coin and from my perspective, the result is going to be completely non-deterministic, but it may well be that the universe is very much deterministic, and the reason I don't know the result of the coin toss is because I don't have all the known parameters, but the universe does. I can think that the outcome of the coin toss is actually a deterministic function of some value that I don't know. We transform this picture on the left to the picture on the right by adding more parameters.

The language perhaps most associated with this style is Haskell. What you see here is not Haskell, it's a language that I made up for the purpose of these slides, but it's very similar to Haskell and we're going to try to express nondeterminism in this way. We have some IOs subroutines, get line and put string line and they're going to take this hidden mysterious parameter called world that is not known to us, but it represents the state of the universe and these subroutines are going to consume it and give me a new value of world and perhaps do something else. Get line is going to also return a string, what it reads from the console, and put string line is going to output that string.

In that language the main program just takes the state of the world returns a new state of the world. I've used a construct that does not exist in Haskell, I think let crack just so I can reuse a W variable, but I get the world for me, then you pass it to get line, get a new world and the result string and then I pass the world to put string line with a string that output it to the console and I'm done. To see what's going on here more closely without reusing the same variable, so each of those steps I get a new value of the world. We start with W0, we use that, we get W1 back, we use that and we get W2.

The problem with that is that this is insufficient, in fact, it won't work to represent IO for the following reason. Every time we read a line from the user we don't know what the next stage is going to be, we don't know what the results are going to be, but from what I presented you, I'm allowed to pass the same value of the world, W0, to get line twice. According to this model, I will need to get the same value again in both cases, but this is not the case, because the user is allowed to enter whatever string they want. We can solve this by adding something called linear types, which are represented with the upside down exclamation mark.

If I say that the world's values a linear type, then the compiler is going to ensure that I use each of those values exactly once. If I pass W0 to get line and I get W1 back and I try to pass W0 again, the compiler is going to give me an error that W0 has already been consumed. If we make sure that each of these mysterious parameter, world parameters, only use exactly once, we can actually represent nondeterminism in this fashion. This is how the result is going to look like.

I think there are some languages that do that already, if you've heard of Cogent, maybe Idris is playing with that a bit, but this is not what Haskell does. There is another solution, and if you remember the beginning I showed you that even in the deterministic picture of the world, there is one point of allowed nondeterminism, that is the beginning of the program. We're allowed to begin the program in one of various states, but once we start we are deterministic. What we can do is to move the nondeterminism outside the program and say, ok the transition from A could be deterministic and now we want to ask for input. What we're going to do is we're going to basically end the program, return back to the runtime, ask the runtime to read some inputs for us and then restart the program at a new entry point depending on what the user has inserted. This is exactly what Haskell does, this is actual Haskell now. This code doesn't show you much. It actually looks quite imperative, but this is just syntactic sugar for the following.

The main function returns a value of something called an IO type, and that value is constructed with a function called bind IO, and bind IO constructs value of the IO type from a pair of two things. First, the operation, we want the runtime to perform and second the new program entry points. What we do here, we basically say, "Okay, this program is done. I've returned the runtime despair," but then the runtime is going to offer the input and going to start the program again passing that string of input into the new entry points.

There is actually a bit more going on here to make this kind of programming actually convenient a nice to use. These IO values can be combined in nice pattern with these two functions return IO and bind IO, and that's called a monad, so the IO is a monad. It is quite problematic even in a language like Haskell, but we're not going to talk too much about the problems monads, but just to say that what monad does is allows us to combine various operations on the IO type in a nice way, and that is what enables us to use this nice do notation with nice syntax.

If you're not using Haskell, if you're using classical imperative programming like Java or C or python, in those languages the meaning of an operation such as a subroutine application is not the same as a math function. If you're interested in programming language theory you may know this. It's something that's called a predicate transformer. It doesn't really matter, what matters here is that the outcome of this operation does not need to be deterministic unlike functions. This is why in Java you can write code that looks like this, we just reline and we are non-deterministic. We don't know what the result is going to be, and then we print it right back out to the console.

This is why it's surprising, in languages like this dealing with non-deterministic is quite easy and it's built into the semantics of the language, but this is why it's surprising, that we find the following: in the Java code libraries we see the strange class computable future, and I've translated the signature of the methods to Haskell like notation. If you look at that, those signatures are actually identical to those we find with the return IO and bind IO. That is because this class actually implements a monad, and they've various users for monads, but this class is used to do IO. The question is why? We know why we need in Haskell, because Haskell is deterministic and functional, but Java isn't. Why do we need to use a monad in Java to do IO?

There is something else going on here, it's not just a nondeterminism and it's not just the theory. The part of the code that performs certain kinds of computation are actually run on the CPU, but when we want to do IO we need to move off the CPU and use different circuitry in the computer. We basically need to release the CPU to do other stuff and we call that blocking. What read line does it also blocks, it stops using the CPU, it asks the operating system to do IO for us using other circuitry in the machine and when it's done we ask the operating system to bring us back to the CPU.

This is done using processes or threads, we've all learned back in kindergarten that blocking threads is bad, because friends are big and heavy weight and expensive and you can only have so many of them and blocking is slow, so we shouldn't do that. It's not because of the semantics of the meaning of the language that we use that strange monad, but because we just don't want to block. We are using a similar mechanism taken from a completely different paradigm to solve a different problem.

Let's look at this Java method that does some important computation and it uses micro services. Every time you call compute and actually go over the network I pass them operation I want my computing service to do, financial computing service to do and I'm going to block and want to start I'm going to get the result and based on that do the next thing. If we want to use that computable future this code would look like that. This is called asynchronous programming, sometimes it's also referred to as reactive programming, but really it's just a style that's meant for us to allow us not to walk.

Reactive Programming’s Problems

Whatever your opinions are about this style, I suggest just last night I very nice talked about someone who's an expert in reactive programming. He has some strong opinion on why this style is actually problematic, but let's look at some of the problems.

First of all, suppose original method was not that simple, but we actually want to do some control flow. You want to branch the result of the first compute and then loop over the second core and we simply cannot use those existing Java mechanism if we use CompletableFuture. If we use different ones that are specially built sort of a DSL, this CompletableFuture has sort of its own DSL for doing control flow, but we can't use the ones that are built into the language. This already shows some kind of mismatch.

Another problem is, what happens if you have an exception? We use two exceptions giving us some context of where the problem happened, but when we use something like CompletableFuture, any of the reactive frameworks like RxJava etc., each of these steps can be executed in a completely different thread and it will likely be a thread other than the one that called calc important finance, and the stack trace we're going to get to the exception is not going to tell us the context we were actually running it. This makes debugging this code very hard. Not only that, it makes profiling this code hard, because profiling works by sampling stack traces. If the stack traces no longer give us the actual context of a computation, they're basically useless.

Perhaps the biggest problem is the return type. Like in the case of IO in Haskell, the return type here is of this strange class that wraps a double, and we can't get the double inside it unless we block, and that's what we don't want to do. Everyone who calls this also has to work with these CompletableFuture in an asynchronous style and this is viral. It means our entire call stack must be either synchronous in blocking or asynchronous. It is nearly impossible to interoperate the two styles, so we get two separate worlds.

What do other languages do about that? C# introduced something called async and await. The task closing in C# is similar to CompletableFuture in Java, but now you can add this async annotation and you can prefix calls to async methods with the await keyword, I guess, and this way you get a nice imperative looking code. This safely solves the first problem. Now, we can use exceptions, now we can use control for like if and world etc. It doesn't really solve the second problem though of the exception stack trace, because each one of those lines will still be executed on a different thread, and the exception is going to give us a strange stack trace.

This is actually something they've fixed in .Net Core I think 2.1 released last year. They artificially generate a stack trace that does capture the actual context, but it would still be a very different stack trace from the one you'd get if you were writing ordinary blocking code. It doesn't solve the biggest problem, that is that you still have now two separate walls that can't interoperate probably the one the async world and the synchronous blocking world.

The imperative languages that we know not only handle input and nondeterminism, they have this built-in notion of threads or processes and blocking and it is a good abstraction from a programming perspective. The only reason we sometimes want to avoid it is because the implementation by the kernel threads is too heavy weight. Why do we want to abandon something that's not only a good obstruction, but a core obstruction of this paradigm just because of an inadequate implementation?

The obvious solution, instead of changing how we program, is just she's changed implementation. That is what we're trying to do as part of Project Loom. To replace implementation first, we need to understand what it does. What is a threat in the process? If we think about it and try to decomplect it, as Rich Hickey may say, we see then there's actually two different capabilities here. First, we need the ability to stop the code that's running on the CPU and say, "I'm not using the CPU anymore," and later on have the ability to resume it. That's one capability, the other capability is we need some mechanism to schedule those pieces that want to run the CPU and say, "Now you're ready because you were waiting for something else to happen like an IO operation to complete. Now, you can run on call number three." We have names for these two capabilities. The first one is called continuation. That's the ability to suspend running computation and later on resume it and the second is just a scheduler.

Continuations

As part of Project Loom, we've implemented continuations in the JVM exposed as this clause here. What it does, this is very similar to the one actually in the prototype, it's not quite the same, and it changes a bit. What we get here is we have a continuation of a certain body that is going to be the code that will be able to suspend itself and then we run it. When we run it, we start executing it and it can either run to completion or yield, in either case, run will return. There is no concurrency here, everything is running on the same thread, but when you call run, the body of the continuation is going to run either to completion or until the next time it calls yield. If it calls yield, if we ask, if you're done is going to say no, if it's terminated, it's going to say yes. This scope thing allows us to nest different continuations inside another and to be able to suspend multiple continuations and jump back multiple callers similar to how we throw exceptions way up the stack.

To be more precise, if you're interested in the theory of continuations in the academic literature, the kind of continuation of this class implements are called One-Shot multi prompt delimited continuations, I'm going to explain each of those. The delimited part means that the piece of the code that we are suspending and resuming is not the entire program, but just the code that's inside the body. The body delimits the code that we want to suspend and that's why they're delimited. Multi prompt is exactly the different scopes, different scopes are in literature are called prompts. We can nest different continuations and jump back to suspend however many of them we want, and one-shot means that this continuation is mutable in the sense that every time you run it it's going to run until the next yield point and then it state changes, and then when you run it again, it's going to run from the first yield point to the second and we can never go back in time.

Continuations are very low level construct, application developers not expected to use them directly, supposed to use a high level constructs that's built on top of them, but if you were to use them directly, this is how you use them. You have a continuation, that's the body, loops forever, prints out some stuff and occasionally yields, and then if you want to use that continuation, you just loop as long as it's not done. In this case it's never going to be done, you run it and every time you call run, it's not going to loop forever, it's just going to execute the next iteration until yield. The important thing here is that the call to yield does not actually need to be inside this outermost top level block. It can actually be inside some method deep in the stack. You can call foo, and foo calls bar and bar calls yield. Part of the continuation state is a stack, the continuation maintains its own stack and its own program point, program counter, where it is in a program.

People who may be familiar with continuations and other languages like scheme, and maybe I think it is experimental implementation or kernel might be a bit horrified to see that the return types of the running yield methods is void. Usually, you want to pass some information from run to yield. Writing continuations are able to pass information from run to yield. On top of this is actually very easy, but one of the reasons we've done this way is that normally continuations yield cooperatively or voluntary. Because the return type is void, if a continuation gets into an infinite loop, we can ask it to preempt itself, and forcefully remove it from the CPU. This being Java and we like monitoring, and I said stack is part of the continuation, we also have some methods to inspect the continuation stack.

This class implements one-shot multi prompted limited continuations, but we could decide to add the ability to clone them. If we clone a continuation, we can capture its certain time and then we run it and we get something that's called re-entrant or multi-shot continuations. This is how simple it takes to implement those on top of these one-shot continuations, we just clone the continuation every time we run it and return a new copy. You can do some crazy stuff with this, you could write programs that actually go back in time, and most people have no need for and I'm not sure we're actually going to implement that, but we could.

Another more interesting thing we can do, we can make those continuations serializable, this means you can write a piece of computation which blocks. Say you are reading something from the database and you're waiting for a response, and while you're waiting, you're actually not just going to be suspended off the CPU, but entirely off the machine, and when you return and get the results from database, you could be in a different machine all together, perhaps closer to where the data is and this would make access to data faster.

Fibers

These are continuations, and I mentioned before that they are a low-level construct. What we want to build on top of them is rebuild threads, threads are just continuations and the scheduler. We already have very good schedulers in the JDK in the form of the thread pools like full join pool, and we combined them to implement threads in the JDK in user mode not going to the kernel, and those are called fibers.

To remind you why do we want to do that, today, people writing applications have either the option of writing simple synchronous blocking code, but they're relying on the kernel to provide them with threads and the kernel can only handle so many threads. The application is going to be very clear and easy to maintain and debug, but it's going to be not scalable. The other option is to write asynchronous code as I showed you which is much more complex and very hard to put existing code to, but at least it's scalable.

With fibers, if we can indeed make them much more lightweight than the threads provided by the kernel, we solve this problem. You can have as many of these user mode lightweight threads as you like, and you can block, blocking becomes essentially free. We can say that it codes like sync but works like async in the sense that your code looks locking, but from the perspective of operating system, no kernel thread is actually blocked, and all the IO behind the scenes is a synchronous even though to your code everything seems synchronous.

This makes writing concurrent applications easier because it helps you match your domains unit of concurrency, say, the user of the session or the income request or outgoing request to match it directly to the software unit of concurrency, which is the thread or in this case the fiber, the lightweight thread. This is something you can't do with heavyweight kernel threads because you may have 100,000 concurrent users, but you can't have 100,000 current threads. With fibers you can, this already makes writing code easier.

Now, that we have the scheduler and we have the continuations, we have fibers, but now we have to find all the places in the JDK that potentially block and teach them about this new mechanism. What kind of things block? Of course we talked about IO, we've gone into all the places in the JDK that do IO and taught them about fibers. The other case where you can block is with synchronization constructs like locks or blocking queues, channels, whatever. Those are all classes in Java.util.concurrent, and we've done the same there, just to see an example of how.

All of the classes in Java.util.concurrent that block traditionally just called unsafe, when they want to block or unblock a different thread, they all go through this class called LockSupport, and what LockSupport used to do is just called unsafe.park. Unsafe.park is a call to the kernel placed by this current thread and if you want to unblock a different thread, you could unsafe.park and then ask the kernel to unblock that thread.

With fibers, we have an option. First, we have to see if we are existing old heavyweight threads and if we do it the old way, but if the current abstract thread we're running on, which you called the strand, a strand is either a fiber or a heavyweight thread. If we want to park, parking is just yielding the fibers continuation, every fiber has a continuation. If we want to unpark, all we have to do is to find that fiber scheduler and that fibers continuation, and submit the fibers continuation to the fibers scheduler and that continuation just appears to be an ordinary run or an ordinary task for that scheduler. The next time that scheduler is just going to run that task, in effect, is going to continue your code from where it last left off. If you love async/await so much, this would be the entire implementation of async/await on top of this. This is strictly stronger than async/await.

Of course, all this would be worthwhile only if we could do better than the operating system. We need to do better on two counts if we want to support that many lightweight threads, that many fibers. On the left hand we have some data about old heavyweight threads in the JDK. They each have about 2 kilobytes of metadata plus by default one megabyte to stack. When it comes to fibers, they currently have in the prototype, they currently have only 2 to 300 bytes of metadata and the stack is pay-as-you-go, it can grow and shrink however much you use. When it comes to task switching costs for heavyweight threads managed by the kernel, the time to switch task is between 1 and 10 microseconds. For fibers, we don't know how much that is, and that performance is something we're still very much working on, but we expect it to be much lower than that.

Rethink Threads

Fibers are just a user mode implementation of threads, we thought we could do the same thread API, maybe with a new constructor flag saying whether when you create a new thread in Java, whether you want it to be managed by the kernel or by the JDK, but in Java we try to think that Java is already big. It's been around for over two decades and will probably be big in two decades, hence.

When we touch concurrency in Java and threading in Java, there is an opportunity to rethink things. The Java architect said, "Yes, maybe we'll use the same thread API," but let's not assume that that's what we're going to do. Try to think that if there was no backup of threads, no existing API, how would you design them? At first, I thought there's not much to do, after all, what kind of operation you want from a thread. You want to start it, you want to wait for it to and join it, maybe you want to ask it to interrupt whatever it is that it's doing.

Shortly after we had this conversation with the architect, I read a very interesting blog post by Nathaniel J. Smith about something called structure concurrency. He credits the idea to Martin Sústrik, and this was one of the rare occasions when you read a blog post and you say, "That is absolutely right. That is the way to do things."

What is the main idea of structure concurrency? The central trick there is that instead of creating threads in a way that's like fire-and-forget, so you create a thread and that thread runs for as long as it wants and you have no control over it. Threads are confined to a well-known lifetime that extends to a given code block, and I'll show you an example. In the current prototype for lack of a better name, we just call it fiber scope, the name will probably change. We have this block that defines a fiber scope, all the fibers that are created inside the scope are guaranteed to be terminated by the time we exit the scope.

How does that work? When we try to exit that scope in Java, there's a tree of resources that's going to automatically call the close method on fiber scope and that's going to block until all the fibers that were created inside the block have terminated. This gives us some nice advantage, for example, when you create a thread and you just forget about it, there could be exceptions that are thrown by that thread and no one is ever going to handle. This way, because you know that the thread is going to be born there and die before the end of the block, you can catch its exceptions and handle it. You can create a great many number of fibers, and you can even nest these scopes to create a tree of fibers and you can cancel all of them by just canceling the scope.

Some nice stuff you can do with it, you can write a method that gets through re-tasks and wants to start the ball in parallel, but it's like a race, that you only want to wait for the first time it finishes. Fiber scope give us termination queues, and for each of the tasks we spawn a new fiber. That's essentially free, it's just like creating objects in Java and we sign them that to the termination queue, and then we block on that cue, and we take the first the result of the first fiber that has terminated and because we cannot leave the scope until all the others have terminated, as soon as we have a result there in the finally block, we cancel all the remaining fibers and then we can leave the scope.

We can do the exact same thing by just changing a little bit. Instead of saying that the fiber scope is cancellable, we can give it a deadline and say that we are only willing to wait up until that deadline for any of the threads. If one of the threads finishes by that deadline, then we good. If not, then the fiber scope is going to automatically cancel all them. This allows us writing interesting things in a very nice way.

Generators

Finally, there are by far the biggest use case that we currently envisioned for continuations of fibers. We expect people to use fibers, not continuations directly, but there are other things that you can do with continuations not as useful perhaps as fibers, but interesting, nonetheless, one of them is generators, and you may be familiar with them from python.

This is how you'd implement generate, there is a bug here, but generally this is how you'd implement generators using integration just to show you how easy it is, and what this gives us is the following: we want to imagine that we have an infinite array or an infinite collection of the Fibonacci sequence, and we want to enter iterate over it until we get to some look at the bottom, until we get to a number that's greater than 10,000. We can use streams for that, but in this way we get an iterable, something that looks like a collection, but notice how we generate it, we generate it in an imperative way. First, we yield zero and we go back, once we yield, you go back to the call there and that zero is going to get into numb. Then we start an infinite loop and we keep yielding numbers one after another. Every time we yield number, we leave that piece of code, we go back and we can write iterators in a very nice way.

The last thing I want to show you is how these continuation is composed. In fact, they're compose much better than monads because they can nest and they have these scopes. This example doesn't really make sense, but this time we want to generate the stream of prime numbers combined with some user input. We want to find the next prime number and then read a line from the console and concatenate it with a number we found, and we're doing a blocking operation inside the generator. What happens if you run it inside a fiber?

Right now we have two nested continuations, we have the continuation for the generator, and every time we yield a generator, it's going to go back into the for loop, but every time we go to the console and read a line from the user, that is going to suspend the fibers continuation and the closing scope, and we can, in this way, compose different kinds of effects. Sometimes they are called different kinds of effect in a transparent way.

This project is still under heavy development, you can find the Wiki page here. We don't have all the access binaries yet, but you can build the code yourself, and right now mostly interested input on the structure currency API. Any help you can provide in that would be appreciated and that is all I had.

Questions And Answers

Participant 1: Thank you. How does that differ or overlap with the co-routines? Looks quite similar to me.

Pressler: Co-routines are another name. The nomenclature here in the world of continuations is a problem. At least in the past, co-routines are sometimes a different name for one-shot delimited continuations. It's just that the name co-routines is now mostly associated with usages of continuations that have some syntactic expression, something like async/await, that at the syntax level you say, this piece of code is a continuation, and we don't do that here, any method is allowed to yield a continuation.

There's another different kind of continuations, symmetric and asymmetric and co-routines at a time used to refer to as symmetric continuations. These are asymmetric continuations, but basically different names of the same basic idea.

Participant 2: We often see continuations and tail calls talked about together for the ICPS Transformer. Is there any plans to add tail calls, the wonderful missing piece of the puzzle, into Java, plans to add them within my lifetime? I know there've been plans that have been forever.

Pressler: We don't actually need tail calls to implement these continuations. However, actually, Project Loom, the goal of the project is to add continuations, fibers, and tail call elimination. This is one of the goal of the project, it's just we haven't started yet, and we're probably not going to start until continuations are the least. That's going to be a next step. I just don't like talking about, some people start expecting it, but yeah, that is one of the other goals of the project.

Participant 3: Are you aware of any projects in Python that would add continuations?

Pressler: Python has generators which are a form of continuations, but on that I'm not that familiar with Python.

Participant 4: Do you envisage Project Loom being released one big bang in one release or you're going to be releasing the different kind of goals in different releases of Java?

Pressler: I've been asked if we were willing to release continuations before we release fibers. The answer to that is absolutely no because the continuations are a very low-level construct and once we release continuations, it can be 105 libraries built off of that. First, we want people to get used to one, however, it is possible that we will release fibers before we open continuations as a public class. We will likely release something before we do tail calls, for example. It will probably multiple chunks, I'm not sure what those chunks are, but basic functionality for fibers is probably going to be in the first release.

Participant 5: You mentioned that you have use-as-you-go memory footprint. Does that mean that you'll have to load the executed stack to continue executing because you can't continue calling forwards? How does that work?

Pressler: I can speak, and you can watch in other talks while I speak about the implementation for now. The continuation stacks are stored in the Java Heap inside two Java arrays and we do copy them back and forth, the question is how to do it fast. Right now, we have a solution that allows us to copy just one frame and the rest lazily. We can talk about this later or you can watch in other talk we talk about the implementation.

Participant 6: I think introducing continuations in Java opens the gate to memory leaks, another gate, and I'm wondering if there are any patterns that we can stick to to avoid memory leaks. Are fibers one of them?

Pressler: I think you find memory leaks in the sense that you can have a piece of code that starts executing, holds on to some object and never terminates. We do have the problem today already with threads, you can create a thread that at some point either lives forever or sleeps forever while still holding on to some stuff, but this probably could be exacerbated if you can have a million threads instead of just a thousand. Application developers are not supposed to use continuations directly, they're supposed to use fibers. You have the same problem with fibers as with threads in that regard possibly more.

We don't have a good solution to what happens if this problem is exacerbated, but if you have any ideas, I will be happy to hear them, but I can say that when it comes to the garbage collector, this is actually easier for the garbage collector, because if you have a thread, all the objects that are referenced by that thread are considered as GC roots. GC Roots needs to be traversed during stop the world pauses, and they can actually increase your pausable time. Continuations are not GC Roots. If they don't change, as long as they don't run the GC doesn't look at them. It doesn't solve the memory leak problem, but at least they don't introduce an initial burden in the garbage collection.

See more presentations with transcripts

OpenAI has ‘work to do to rebuild trust’ in Australia, executive tells AI inquiry

Guardian
www.theguardian.com
2026-10-05 20:53:48
Anthropic tells inquiry it ‘never tried to dictate’ Australian copyright rules as the ABC says copyright laws do not need changingGet our breaking news email, free app or daily news podcastOpenAI has used an appearance before a parliamentary inquiry on AI to apologise again to Australia about the ha...
Original Article

OpenAI has used an appearance before a parliamentary inquiry on AI to apologise again to Australia about the hack on Medicare last month, as media and entertainment organisations argued weakening copyright law for AI model training would leave artists as “roadkill”.

Meanwhile Anthropic told the inquiry it was not seeking to dictate copyright law on model training, but the current rules made it “impossible” to train AI in Australia.

Tuesday morning’s hearing of the joint parliamentary inquiry on artificial intelligence largely focused on copyright licensing for AI training in Australia, amid concerns companies such as Anthropic were pushing the Australian government to adopt an opt-out model, requiring rights holders to specifically opt out of allowing AI to train on their content.

Anthropic’s special envoy, the former US ambassador to Australia Jeffrey Bleich, told the committee that Anthropic had “never tried to dictate to Australia” on copyright.

“We’ve just shared information about how the training works, and then discussed what its implications would be with respect to our capacity to operate here,” he said.

Anthropic suggests current Australian laws mean AI companies need to obtain a licence for the use of every piece of content on the internet as training material. Bleich said this was “technically impossible”.

Anthropic was keen to explore what the Australian government wants to do to incentivise training in Australia, and Bleich said Australia could shape frontier models if they were trained here, otherwise it would ultimately rely on models trained elsewhere.

Sign up for the Breaking News Australia email

OpenAI apologised for its AI agents attacking a number of Australian government websites in June . The multibillion-dollar company notified the government via an email to a public-facing address three months after the breach.

“I want to begin with an apology. During internal training and evaluation, our models accessed Australian government websites in ways they were not directed to. That should not have happened. We also should have handled our response better,” OpenAI’s chief strategy officer, Jason Kwon, told the committee on Tuesday afternoon .

“We are sorry, and we know we have work to do to rebuild trust with the Australian people. We are committed to doing that work.”

Following the revelation of a separate hack on the NSW national parks and wildlife service last week , Kwon told the committee OpenAI was reviewing logs of AI agent training dating back to November 2025, which identified the latest hack activity taking place in June.

He promised that should more incidents be identified, government agencies would be notified “very, very quickly”.

OpenAI representatives, while apologetic, downplayed the agent activity on Australian government websites as not “not super sophisticated” compared with the Hugging Face attack . The independent senator David Pocock questioned why the company didn’t contact ministers, instead sending an email “to some arbitrary department email” after the Medicare incident.

“I think that on retrospect, we should have done what you were suggesting … I think people were thinking about this as a technical situation, and they wanted to contact the technical counterparties – but it’s not good enough,” Kwan said.

Anthropic’s head of safeguards, Dave Orr, told the committee the company had reviewed hundreds of millions of transcripts of its models and had not found any cases in which it interacted with Australian government systems in an unauthorised way.

He could not say whether Anthropic’s customers may have used the models on Australian government systems, given the company’s “zero data retention” policy.

skip past newsletter promotion
Anthropic says AI agents didn’t breach Australian government websites – video

‘Artists will become roadkill’

Annabelle Herd, the chief executive of the Australian Recording Industry Association (Aria), told the committee that “Australia’s artists will be the roadkill in the rush” to make deals with AI companies, and said Aria was not against AI frontier model training being done in Australia but it should be done on “our terms”.

“There is nothing less sovereign than rewriting our laws to suit foreign company commercial interests, done the way that big AI is proposing,” she said. “It will devastate the music industry and make Australia a pariah for rights holders globally.”

She said Anthropic and OpenAI were “asking the government for a free ride” and the creative sector was calling on the government not to cave in to their demands.

The ABC told the committee Australia’s existing copyright regime was “completely adequate”.

The ABC’s head of content and legal operations, Kate Gilchrist, said an opt-out system as proposed by AI companies would place burdens and costs on rights holders, as it would require ongoing surveillance to see whether copyright material was being used.

She said for the ABC, it would mean ensuring every site on which the ABC publishes material had opted out.

“We cannot scour the internet and ensure we are opting out on all of those different sites of publication. It simply does not work,” she said.

Gilchrist said she suspected ABC content had been scraped already.

The union representing the entertainment and news sector, the Media Entertainment and Arts Alliance (MEAA), has called for an equitable remuneration scheme that would put fees on companies across the AI technology stack.

“Revenue would support two things: direct payments to creative and media workers whose work contributes to AI systems. And secondly, a cultural fund supporting Australian live entertainment, screen music, public interest journalism and cultural activity,” the MEAA’s acting chief executive, Adam Portelli, said.

The committee is holding hearings from Tuesday to Friday this week.

Photopea creator weighs in on Photosuite project

Hacker News
github.com
2026-10-05 20:41:44
Comments...
Original Article

What is missing

Hi guys, it would be really nice if you did some actual work instead of copying my work form my website www.photopea.com , that I have been working on since 2012.

Could you maybe implement your own photo editor instead of stealing mine? What do you think?

I am really surprised that Github tolerates piracy.

Where this lives in Photoshop

No response

How it should behave

No response

Does this affect files?

  • It reads or writes something in a PSD, so files should round-trip with Photoshop
  • It only affects the editor, not saved files

High Diesel Prices Bankrupted 16 Trucking Companies in Just 30 Days

Hacker News
www.thedrive.com
2026-10-05 20:24:57
Comments...
Original Article

Diesel prices are displayed above $8 per gallon at a gas station in Los Angeles, on September 17, 2026, as prices reach record highs across the United States. Average diesel prices in the United States struck a record high of just under $6.27 a gallon on September 15, the motorists' association AAA said, pressuring President Donald Trump ahead of midterm congressional elections.The price of the fuel used largely in road hauling, agriculture and construction hit $6.2694, according to the association, as the Middle East war tightens supplies and Ukraine strikes Russian refineries.

Photo by Frederic J. BROWN / AFP via Getty Images

Statistically speaking, you probably don’t drive a diesel. Oil burners represent a tiny chunk of light-duty vehicle sales, even when you factor in heavy-duty pickups and the like. But in the freight world, if you’re hauling heavy loads, chances are you’re fueling up with diesel—or maybe you were , if you drove for one of the 16 trucking companies that went bankrupt last quarter.

According to multiple sources , the rising cost of diesel fuel was the straw the broke the camel’s back for 16 freight operators that filed for either Chapter 7 or Chapter 11 bankruptcy protection in late August and early September. Some were small, single-operator outfits. Others were larger, operating dozens of vehicles.

Four of those operators were based in Texas, where Governor Abbott just proclaimed a statewide disaster in order to ease restrictions on the use of untaxed diesel fuel on public highways. The move was largely intended to provide relief for agricultural transportation, as most untaxed (dyed) diesel is produced for farm use.

But the issue is much bigger than just agricultural trucking. High fuel prices are contributing to inflation in other sectors too. One outfit that filed for Chap. 11 protection in September was a Florida-based Amazon delivery contractor.

“In the trucking business, gas prices eat into profit margins, and intense competition makes it hard for smaller players to pass on those added costs to customers,” wrote Daniel Kline, co-Editor-in-Chef of TheStreet.com . “Diesel is the latest pressure hitting carriers that were already weakened by years of poor freight economics and rising operating costs.”

With residents of northern states already firing up their furnaces, American diesel fuel prices are also being pressured by the energy sector. With winter just weeks away, it’s unlikely that we’ll see any significant price relief before the holidays, especially if the situation in the Middle East stretches into 2027.

Got a story tip? Let us know at tips@thedrive.com!

AI Tutoring with Khanmigo in a Two-Year School Experiment

Hacker News
edworkingpapers.com
2026-10-05 20:00:45
Comments...
Original Article

Generative AI has been promoted as the technology that could transform education by providing every student a personal tutor. We provide some of the first large-scale experimental evidence, from a two-year cluster randomized trial in 18 Tennessee middle schools in which randomly assigned students used Khan Academy with its AI tutor, Khanmigo, configured to coach rather than give answers, during existing daily remedial mathematics sessions. Assignment raises math achievement by 1.3 national percentile ranks per term, or about 0.06 to 0.08 standard deviations over a school year; the implied effect of a full year of active participation reaches 0.14 standard deviations. These gains resemble those from Khan Academy practice without AI assistance. One explanation is that students used the tutor infrequently and, when they did, rarely engaged it in substantive mathematical dialogue: 96 percent of students tried Khanmigo at least once, but the median student messaged it on only a third of the days they practiced, and in only 17 percent of the exercise sessions in which they made a mistake. Messages that students did send were mostly bare answers or clicks on suggested prompts. The binding constraint appears to be engagement: realizing the promise of AI tutoring will require getting students to use it, not just giving them access.

Keywords

Artificial intelligence; AI tutoring; Khanmigo; computer-assisted learning; educational technology; field experiment; math achievement; student engagement; response to intervention

Document Object Identifier (DOI)

10.26300/kner-hv33

Quoting Felix Rieseberg

Simon Willison
simonwillison.net
2026-10-05 19:56:47
The "old" version of Cowork runs model inference in the cloud, executing tool calls in an Anthropic-provided VM we shipped to your computer. We added the VM for capability, safety, and security reasons - mapping in just the data you explicitly added to your session. People loved what they were able ...
Original Article

5th October 2026

The "old" version of Cowork runs model inference in the cloud, executing tool calls in an Anthropic-provided VM we shipped to your computer. We added the VM for capability, safety, and security reasons - mapping in just the data you explicitly added to your session. People loved what they were able to do with Claude but didn't love the disk, battery, and performance cost of running the VM locally. Also, people didn't love that closing your laptop means the work stops.

The "new" version of Cowork runs model inference and the VM in the cloud. Each session gets its own sandbox, not sharing state with other sessions. When the VM needs something on the users' device (like a file), the desktop app is responsible for that file access tool call. [...]

We think this solves a lot of problems we've heard about (like using Cowork from a phone, keeping work running, or getting all the same power without losing battery to the VM)

— Felix Rieseberg , Anthropic, see also this help page

[Sponsor] Sunnny

Daring Fireball
sunnny.com
2026-10-05 19:49:05
How can B2B software be so friggin’ delicious? Early access drops 10/19. Spread the love.  ★  ...
Original Article

How can B2B software be so friggin’ delicious?
Early access drops 10/19. Spread the love.

Samon: Designing a Zen Garden Raking Puzzle

Hacker News
gwern.net
2026-10-05 19:35:35
Comments...
Original Article

Zen sand gardens are often intended for meditation, whether to look at or while maintaining it by raking the sand into the prescribed patterns which are formed by the tines of the rake yielding parallel lines (like ||| ). Such a chore might be tedious, but there is also a great deal of satisfaction to restoring an esthetic order from chaos, and raking sand seems like one of those things, like popping bubblewrap, which can be oddly satisfying—I recall the lifeguards at my summer camp seeming to rather enjoy the task of raking the beach in front of their office every day and not allowing any campers to do so. Puzzle-solving is also often described as being ‘meditative’ and ‘oddly satisfying’, even when it looks a lot like a tedious chore. So for those of us who do not live at a Zen temple, can we make a Zen sand garden puzzle ?

This page turns the original 2023 sketch into a full design. Where the sketch speculated (would “mirror raking” be too restrictive? could a neural net rank solutions by beauty? how should difficulty be raised?), we answer with a solver and measurements; where it left gaps (what exactly makes a garden hard, what the worlds are, how levels are generated, what the player touches), we fill them. The title is Samon (砂紋, “sand crest”), the term for the patterns themselves. 1

Appendix: Code and Data

Every file behind this page: the solvers, the experiments, the level generator, the prototype's source and its tests, this edition's build script and test, the poems, and the data they wrote. The code, the logs and the poems are shown below; the data, the figures' sources and the campaign cache are listed, and everything is in the download.

Download all 223 files as one .zip (1.8 MB; SHA-256 b60754f3fba988794a433d5d8c43cdf4a3160a07b05de806d7b1a664828b74d4 ).

Start with src/zgsolve.c (exact search), src/zg.py (the rules, the solver driver and a brute-force checker), src/zcp.py (CP-SAT par proofs), src/grader.py (the deductive grader), and src/verify_doc.py , which re-derives every number in the document. The build and test commands are in the guide below. Not included: this page itself and the prototype's two pages, which standalone/build_standalone.py and proto/build.py write.

Code behind the numbers and figures in the Samon design document (2026-10-03, revised 2026-10-04), and the playable prototype. The single-file edition of the document carries this whole tree in its appendix: the code is shown there, and every file is in the .zip it offers (all but the three generated pages, which the build scripts write).

Build 

cd src && gcc -O3 -march=native -Wall -Wextra -o zgsolve zgsolve.c
pip install ortools cairosvg playwright   # CP-SAT optimiser; SVG→PNG for figures; browser test of the prototype
playwright install chromium
pip install beautifulsoup4                # (and pandoc) for the single-file edition
# ffmpeg with libmp3lame, for sound/make_sound_bundle.py (the recordings; see sound/README.md)

Paths: the scripts use absolute paths under /home/claude/zen/ (the directory this tree was developed in). To run from wherever this tree is unpacked, rewrite them once, from the top of the tree: 

grep --recursive --files-with-matches --exclude=README.md /home/claude/zen . | xargs sed --in-place "s|/home/claude/zen|$PWD|g"

Files

  • src/zgsolve.c : exact DFS solver (counts/enumerates rakings; turn & U-turn histograms; branch-and-bound min turns). Rules: one stroke ending at the gate; optional ripple rings, wave bands / ichimatsu, given grooves, pebbles ( shape : S = straight through, B = turn), no-U-turn ( nohairpin 1 ), wide rake ( nohairpin 2 + endcells ), fixed rake start (two gates: end , and endside , the wall the side gate is in, so that a bend in its cell counts as a turn), the abbot’s finished sand ( R : not walkable, and not a pivot for the wide rake). stream 1 prints every solution (used to export all par rakings).
  • src/zg.py : garden model (text symbols: . sand, # stone, O ripple stone, h / v wave bands, = / | ichimatsu blocks; clues = grooves, marks = pebbles), solver driver, rule checker, brute-force enumerator.
  • src/zcp.py : independent CP-SAT model (AddCircuit + dummy node) for par proofs and cross-checks.
  • src/grader.py : tiered deductive solver used to grade difficulty.
  • src/gen.py , clues.py , levels.py , search*.py : layout sampling, greedy groove and pebble selection, level searches.
  • src/exp_marks.py , fig_pebbles.py : pebbles vs grooves on the same gardens (writes out/marks_vs_grooves.json ), and the pebbles figure.
  • src/render.py , final_figs.py : figure rendering (writes fig/final/*.png|svg , out/facts.json ).
  • src/final_tables.py , exp_*.py : the tables and statistics quoted in the document.
  • src/verify_doc.py : re-derives every number quoted in the document (C solver + CP-SAT); prints OK/BAD.
  • src/test_crosscheck.py , xc_stats.py , xc_cpsat*.py : solver validation (brute force; CP-SAT; OEIS A145157/A145156).
  • src/test_marks.py , test_grader_marks.py : pebbles: C solver vs brute force vs CP-SAT; soundness of the grader’s pebble rules.
  • src/garden_sampler.py : the garden-like stone sampler (odd-numbered, asymmetric groups of one- to three-cell stones; ripple stones, wave bands, closed rings, side gates, ichimatsu layouts). src/exp_sampler.py compares its yields with single-cell stones (writes out/sampler_yields.json ).
  • src/campaign.py : generates the campaign (7 worlds, 148 gardens): generated gardens per world from the sampler and the pipeline’s filters, plus the hand-made and searched teaching gardens and the contrasting pairs; each world’s main path is the template (introduction, two applications, trap, combinations, calm garden, capstone; roles assigned by measurement; 72 gardens in all), and the rest of its gardens are optional extras (76); writes out/campaign.json and out/campaign_stats.json (per-spec candidates and the pairs are cached in out/campaign_cache/ ).
  • src/pairs.py : searches for the contrasting pairs that introduce each rule (the same small garden before and after one change: a ripple stone, a band or the wide rake that breaks every par raking of the first; one groove that leaves one raking; pebbles in place of grooves; a side gate that leaves one of two wide-rake rakings).
  • src/exp_continuity.py : how many of the par rakings rewarded in worlds 1–3 turn back in open sand (writes out/continuity.json ).
  • src/test_side_pivot.py : rechecks the side gate’s turn counting and finished sand (no pivot) in the C solver against brute force (small gardens) and CP-SAT (medium); logs in out/test_side_pivot_seed*.log .
  • src/hints.py : the abbot’s first observation for generated gardens: the provable fact that most changes the plan (free gardens: true of every par raking, and missed by the most near-misses; strict gardens: about the unique raking, and lowering the grade the most), with the cells the abbot points at; trap scores for the template’s ordering.
  • src/export_proto.py : exports the campaign to out/proto_gardens.json , re-deriving each garden’s raking count (exact solver), par (CP-SAT), every par raking (for the in-page hints), and an intended raking (asserted valid and at par), with its first hint (authored, or from hints.py ), its role, whether it is an extra, its pair, and whether the monk demonstrates it first; writes the hints’ measured effect to out/hint_stats.json .
  • src/seed_garden_names.py , out/garden_names.json : garden names follow layouts (a registry keyed by the garden’s rows, gates and clues), so a garden keeps its name, and its poem, when the campaign is regenerated or reordered.
  • poems/samon-poems.json : the poems, one per garden and one for the practice garden: Japanese text, working translation (and its credit), poet, collection, the page each text was checked against, notes, the short source shown in the game, and why each poem goes with its garden. poems/astra-samon-poem-selection.json is GPT-6 Astra’s original selection for the 136-garden campaign (with its unused candidates); poems/pass2_replacements.py and poems/pass3_astra_review.py are the second and third curation passes (25 swaps and 4 translation repairs, logged in the JSON’s about.revisions ); poems/originals-brainstorm.md holds original candidates for three gardens, not used. src/poems.py attaches the poems to the gardens.
  • src/daily.py : a year of daily gardens ( out/daily_gardens.json , 366; stats in out/daily_stats.json ): the campaign’s unused candidates plus fresh ones from its generators (cached in out/daily_cache/ ), none sharing a layout with the campaign, exported like the campaign’s gardens; the page picks the day’s by the local date.
  • sound/ : the recordings carried in the page ( tracks.json : credits and sources; make_sound_bundle.py : loudness measurement, static gain, mono MP3, writes out/sound_pieces.json ; README.md : how the sounds work and how a SingleFile/gwtar step can move the recordings out of the page). src/sound_pieces.py writes them into the page as data: URLs for both builds. The drips, pipe, wind, birds, crickets, footsteps, rake scrape, bell and seal are synthesized in the prototype itself.
  • proto/samon.template.html , proto/build.py : the prototype (one page; canvas; no dependencies, no requests). build.py embeds the gardens, their poems, the daily year and the recordings, and writes samon.html (a page body, for publishing as an artifact) and samon.standalone.html (a complete HTML document, with the social-media <meta> tags, for self-hosting). Sharing lives in the template too: a raking is a code in the URL’s hash ( #r= : the walk as a number in the mixed radix of its legal moves, 8–17 base64url characters; walkEncode / walkDecode , rakingCode / readRaking ), shown but not credited to whoever opens it; all progress is another ( #p= : progressCode / readProgress / importProgress , offered and merged); gardens are named in both by a 32-bit hash of their layout ( layoutHash / byLayout ), never by index, so links survive a reordering and a changed garden is refused rather than mistaken; #daily opens today’s garden; the share text draws the raking in box-drawing characters ( gridText ).
  • proto/test/wrap.py , proto/test/run.py : Playwright test; the first visit (the monk’s demonstration, which plays once, resets the garden, and is replayed by Watch or skipped by a tap; then the waiting monk and the first-drag hint); draws every garden’s intended raking with a real mouse, lets go, and checks that it is raked at par and sealed; walking back after raking, the keyboard’s beat, and a complete walk that may not end where it does; refusals (wide rake, pebbles); Restore; the abbot’s verdicts and rewinds; the grey seal (and that Step back and Restore keep it; that inconclusive answers and the reason a walk stopped are free); unlocking by the main path, extras (folded behind “+N more”, unfolding while one is open; locked worlds as one line), and Next; for every garden, that the bends drawn in the raked stroke = the turns counted = par; that finished sand is never a pivot; that a dead end is explained by the rule that actually fails; assists; the practice garden; the poems (every garden has one; it follows any raking, helped or off par; the next garden fades in after it; a tap stays; ‘original’ shows the Japanese; none during a demonstration); the scroll of poems earned; the daily garden (picked by date, Rake commits, the record and streak survive a reload, the next day is fresh); drag intent (a hand drifting off the row still draws the row; the grip survives leaving the field; arcs, rounded corners, reversals and wobbles); sound (no context before a gesture; each synthesized sound renders audibly; Sound off persists; the recording is found, credited with its source, put down to follow a world’s bell, heard, and let go; the scrape, footsteps, birds, crickets and wind render; the scrape runs with the raking; day and night by the clock; volume and haptics settings remembered, in the panel under the speaker’s chevron; the speaker in the header and M mute; a browser that will not play is shown; a tap on the poem goes on and a tap beside it stays); sharing (every intended raking round-trips through its code; an illegal walk has none; the share text’s heading, drawing and link; a #r= link shows the raking with its seal, credited to nobody, and a tap or Rake it yourself gives the viewer the garden fresh with nothing to Restore; the grey seal travels; a daily garden’s raking from another day sends the viewer on to today’s; Share this raking copies text and link, and the link shows it; #p= is offered, then merged with better-state-wins, carrying the day’s record and an unfinished walk with its help; the daily share carries shape and score but no stroke; #daily ; bad codes are refused with a word; every garden’s layout hash is distinct; a tampered hash is refused; links made before a reordering of the campaign still open, and credit, the right garden; every control has a tooltip); screenshots in light and dark. Every garden’s intended raking is stored in out/proto_gardens.json , so nothing is solved at test time; the minutes go on real mouse drags and real-time reveals. SAMON_QUICK=1 (about 3 minutes) drags three gardens instead of 148 and samples the abbot’s explanations on 30 random routes per garden instead of 400.
  • proto/test/fuzz.py : a gesture fuzzer: random drags, taps, keys, buttons, pips, pills and reloads against the page’s invariants (a legal walk, consistent mode and controls, bracketed status, bounded sound timers, progress that parses and a draft that survives a reload; no errors). python3 proto/test/fuzz.py [seconds] [seed] . It found two bugs (a draft lost to a page closed within the save’s debounce; the page returning to the first unraked garden instead of the one open), so it stays.
  • proto/test/thumbnail.py : renders the social-media thumbnail ( deliver/2026-10-05-gwern-samon-thumbnail.png , 1200×630: a garden raked at par beside the title); thumbnail.py pick renders candidates.
  • proto/test/gt_check.py : checks the page’s hint search (can still reach par / can still be finished / nearest completable footprint) against the C solver on random routes; writes out/hint_check.json ( gt_check.py 15 : 30 routes per garden, the 4,440 routes quoted in the document).
  • standalone/build_standalone.py : builds the single-file edition, standalone/2026-10-03-gwern-samon.html : the design document converted from samon.md by Pandoc (figures from fig/final/ embedded as SVG data URIs; Wikipedia shorthand resolved) with the playable prototype in the same page, an appendix holding this tree (code shown; every file in an embedded .zip, also written to standalone/2026-10-03-gwern-samon-code.zip ), and no external resources. standalone/test_standalone.py checks that the page makes no requests, that figures, contents and collapsed sections work, that the appendix’s files and its .zip match this tree and the download delivers the .zip, and runs the prototype tests (it is generated from proto/test/run.py by standalone/make_test_standalone.py ).
  • samon.md : the design document (Gwerndown). fig/final/ : its figures (PNG and SVG).
  • out/ : saved levels, picks, and experiment outputs.

The guide and the document

README.md · 126 lines · 13.4 kB 
# Samon: solver, experiment, and prototype code

Code behind the numbers and figures in the _Samon_ design document (2026-10-03, revised 2026-10-04), and the playable prototype.
The single-file edition of the document carries this whole tree in its appendix: the code is shown there, and every file is in the .zip it offers (all but the three generated pages, which the build scripts write).

## Build

    cd src && gcc -O3 -march=native -Wall -Wextra -o zgsolve zgsolve.c
    pip install ortools cairosvg playwright   # CP-SAT optimiser; SVG→PNG for figures; browser test of the prototype
    playwright install chromium
    pip install beautifulsoup4                # (and pandoc) for the single-file edition
    # ffmpeg with libmp3lame, for sound/make_sound_bundle.py (the recordings; see sound/README.md)

Paths: the scripts use absolute paths under `/home/claude/zen/` (the directory this tree was developed in).
To run from wherever this tree is unpacked, rewrite them once, from the top of the tree:

    grep --recursive --files-with-matches --exclude=README.md /home/claude/zen . | xargs sed --in-place "s|/home/claude/zen|$PWD|g"

## Files

- `src/zgsolve.c`: exact DFS solver (counts/enumerates rakings; turn & U-turn histograms; branch-and-bound min turns).
  Rules: one stroke ending at the gate; optional ripple rings, wave bands / ichimatsu, given grooves,
  pebbles (`shape`: S = straight through, B = turn), no-U-turn (`nohairpin 1`), wide rake (`nohairpin 2` + `endcells`),
  fixed rake start (two gates: `end`, and `endside`, the wall the side gate is in, so that a bend in its cell counts as a
  turn), the abbot's finished sand (`R`: not walkable, and not a pivot for the wide rake). `stream 1` prints every
  solution (used to export all par rakings).
- `src/zg.py`: garden model (text symbols: `.` sand, `#` stone, `O` ripple stone, `h`/`v` wave bands,
  `=`/`|` ichimatsu blocks; `clues` = grooves, `marks` = pebbles), solver driver, rule checker, brute-force enumerator.
- `src/zcp.py`: independent CP-SAT model (AddCircuit + dummy node) for par proofs and cross-checks.
- `src/grader.py`: tiered deductive solver used to grade difficulty.
- `src/gen.py`, `clues.py`, `levels.py`, `search*.py`: layout sampling, greedy groove and pebble selection, level searches.
- `src/exp_marks.py`, `fig_pebbles.py`: pebbles vs grooves on the same gardens (writes `out/marks_vs_grooves.json`), and the pebbles figure.
- `src/render.py`, `final_figs.py`: figure rendering (writes `fig/final/*.png|svg`, `out/facts.json`).
- `src/final_tables.py`, `exp_*.py`: the tables and statistics quoted in the document.
- `src/verify_doc.py`: re-derives every number quoted in the document (C solver + CP-SAT); prints OK/BAD.
- `src/test_crosscheck.py`, `xc_stats.py`, `xc_cpsat*.py`: solver validation (brute force; CP-SAT; OEIS A145157/A145156).
- `src/test_marks.py`, `test_grader_marks.py`: pebbles: C solver vs brute force vs CP-SAT; soundness of the grader's pebble rules.
- `src/garden_sampler.py`: the garden-like stone sampler (odd-numbered, asymmetric groups of one- to three-cell stones;
  ripple stones, wave bands, closed rings, side gates, ichimatsu layouts). `src/exp_sampler.py` compares its yields
  with single-cell stones (writes `out/sampler_yields.json`).
- `src/campaign.py`: generates the campaign (7 worlds, 148 gardens): generated gardens per world from the sampler and
  the pipeline's filters, plus the hand-made and searched teaching gardens and the contrasting pairs; each world's main
  path is the template (introduction, two applications, trap, combinations, calm garden, capstone; roles assigned by
  measurement; 72 gardens in all), and the rest of its gardens are optional extras (76); writes `out/campaign.json` and
  `out/campaign_stats.json` (per-spec candidates and the pairs are cached in `out/campaign_cache/`).
- `src/pairs.py`: searches for the contrasting pairs that introduce each rule (the same small garden before and after one
  change: a ripple stone, a band or the wide rake that breaks every par raking of the first; one groove that leaves one
  raking; pebbles in place of grooves; a side gate that leaves one of two wide-rake rakings).
- `src/exp_continuity.py`: how many of the par rakings rewarded in worlds 1--3 turn back in open sand (writes `out/continuity.json`).
- `src/test_side_pivot.py`: rechecks the side gate's turn counting and finished sand (no pivot) in the C solver against
  brute force (small gardens) and CP-SAT (medium); logs in `out/test_side_pivot_seed*.log`.
- `src/hints.py`: the abbot's first observation for generated gardens: the provable fact that most changes the plan
  (free gardens: true of every par raking, and missed by the most near-misses; strict gardens: about the unique raking,
  and lowering the grade the most), with the cells the abbot points at; trap scores for the template's ordering.
- `src/export_proto.py`: exports the campaign to `out/proto_gardens.json`, re-deriving each garden's
  raking count (exact solver), par (CP-SAT), every par raking (for the in-page hints), and an intended raking
  (asserted valid and at par), with its first hint (authored, or from `hints.py`), its role, whether it is an extra, its
  pair, and whether the monk demonstrates it first; writes the hints' measured effect to `out/hint_stats.json`.
- `src/seed_garden_names.py`, `out/garden_names.json`: garden names follow layouts (a registry keyed by the garden's rows,
  gates and clues), so a garden keeps its name, and its poem, when the campaign is regenerated or reordered.
- `poems/samon-poems.json`: the poems, one per garden and one for the practice garden: Japanese text, working
  translation (and its credit), poet, collection, the page each text was checked against, notes, the short source shown
  in the game, and why each poem goes with its garden. `poems/astra-samon-poem-selection.json` is GPT-6 Astra's original
  selection for the 136-garden campaign (with its unused candidates); `poems/pass2_replacements.py` and `poems/pass3_astra_review.py` are
  the second and third curation passes (25 swaps and 4 translation repairs, logged in the JSON's `about.revisions`); `poems/originals-brainstorm.md` holds original
  candidates for three gardens, not used. `src/poems.py` attaches the poems to the gardens.
- `src/daily.py`: a year of daily gardens (`out/daily_gardens.json`, 366; stats in `out/daily_stats.json`): the campaign's
  unused candidates plus fresh ones from its generators (cached in `out/daily_cache/`), none sharing a layout with the
  campaign, exported like the campaign's gardens; the page picks the day's by the local date.
- `sound/`: the recordings carried in the page (`tracks.json`: credits and sources; `make_sound_bundle.py`: loudness
  measurement, static gain, mono MP3, writes `out/sound_pieces.json`; `README.md`: how the sounds work and how a
  SingleFile/gwtar step can move the recordings out of the page). `src/sound_pieces.py` writes them into the page as
  data: URLs for both builds. The drips, pipe, wind, birds, crickets, footsteps, rake scrape, bell and seal are
  synthesized in the prototype itself.
- `proto/samon.template.html`, `proto/build.py`: the prototype (one page; canvas; no dependencies, no requests).
  `build.py` embeds the gardens, their poems, the daily year and the recordings, and writes `samon.html` (a page body, for publishing as an artifact) and
  `samon.standalone.html` (a complete HTML document, with the social-media `<meta>` tags, for self-hosting). Sharing lives in
  the template too: a raking is a code in the URL's hash (`#r=`: the walk as a number in the mixed radix of its legal moves,
  8–17 base64url characters; `walkEncode`/`walkDecode`, `rakingCode`/`readRaking`), shown but not credited to whoever opens
  it; all progress is another (`#p=`: `progressCode`/`readProgress`/`importProgress`, offered and merged); gardens are
  named in both by a 32-bit hash of their layout (`layoutHash`/`byLayout`), never by index, so links survive a reordering
  and a changed garden is refused rather than mistaken; `#daily` opens today's garden; the share text draws the raking in
  box-drawing characters (`gridText`).
- `proto/test/wrap.py`, `proto/test/run.py`: Playwright test; the first visit (the monk's demonstration, which plays once,
  resets the garden, and is replayed by Watch or skipped by a tap; then the waiting monk and the first-drag hint);
  draws every garden's intended raking with a real mouse, lets go, and checks that it is raked at par and sealed; walking back
  after raking, the keyboard's beat, and a complete walk that may not end where it does; refusals (wide rake, pebbles); Restore; the abbot's verdicts and rewinds;
  the grey seal (and that Step back and Restore keep it; that inconclusive answers and the reason a walk stopped are free);
  unlocking by the main path, extras (folded behind "+N more", unfolding while one is open; locked worlds as one line), and Next; for every garden, that the bends drawn in the raked stroke = the turns
  counted = par; that finished sand is never a pivot; that a dead end is explained by the rule that actually fails;
  assists; the practice garden; the poems (every garden has one; it follows any raking, helped or off par; the next
  garden fades in after it; a tap stays; 'original' shows the Japanese; none during a demonstration); the scroll of poems earned;
  the daily garden (picked by date, Rake commits, the record and streak survive a reload, the next day is fresh); drag
  intent (a hand drifting off the row still draws the row; the grip survives leaving the field; arcs, rounded corners,
  reversals and wobbles); sound (no context before a gesture; each synthesized sound renders audibly; Sound off persists;
  the recording is found, credited with its source, put down to follow a world's bell, heard, and let go; the scrape,
  footsteps, birds, crickets and wind render; the scrape runs with the raking; day and night by the clock; volume and
  haptics settings remembered, in the panel under the speaker's chevron; the speaker in the header and M mute; a browser that will not play is shown; a tap on the
  poem goes on and a tap beside it stays); sharing (every intended raking round-trips through its code; an illegal walk
  has none; the share text's heading, drawing and link; a `#r=` link shows the raking with its seal, credited to nobody,
  and a tap or Rake it yourself gives the viewer the garden fresh with nothing to Restore; the grey seal travels; a daily
  garden's raking from another day sends the viewer on to today's; Share this raking copies text and link, and the link
  shows it; `#p=` is offered, then merged with better-state-wins, carrying the day's record and an unfinished walk with
  its help; the daily share carries shape and score but no stroke; `#daily`; bad codes are refused with a word; every
  garden's layout hash is distinct; a tampered hash is refused; links made before a reordering of the campaign still
  open, and credit, the right garden; every control has a tooltip); screenshots in light and dark. Every garden's intended raking is stored in `out/proto_gardens.json`, so nothing is solved at test time; the
  minutes go on real mouse drags and real-time reveals. `SAMON_QUICK=1` (about 3 minutes) drags three gardens instead of
  148 and samples the abbot's explanations on 30 random routes per garden instead of 400.
- `proto/test/fuzz.py`: a gesture fuzzer: random drags, taps, keys, buttons, pips, pills and reloads against the page's
  invariants (a legal walk, consistent mode and controls, bracketed status, bounded sound timers, progress that parses and
  a draft that survives a reload; no errors). `python3 proto/test/fuzz.py [seconds] [seed]`. It found two bugs (a draft lost
  to a page closed within the save's debounce; the page returning to the first unraked garden instead of the one open), so it stays.
- `proto/test/thumbnail.py`: renders the social-media thumbnail (`deliver/2026-10-05-gwern-samon-thumbnail.png`, 1200×630:
  a garden raked at par beside the title); `thumbnail.py pick` renders candidates.
- `proto/test/gt_check.py`: checks the page's hint search (can still reach par / can still be finished / nearest
  completable footprint) against the C solver on random routes; writes `out/hint_check.json` (`gt_check.py 15`: 30 routes per
  garden, the 4,440 routes quoted in the document).
- `standalone/build_standalone.py`: builds the single-file edition, `standalone/2026-10-03-gwern-samon.html`: the design
  document converted from `samon.md` by Pandoc (figures from `fig/final/` embedded as SVG data URIs; Wikipedia shorthand
  resolved) with the playable prototype in the same page, an appendix holding this tree (code shown; every file in an
  embedded .zip, also written to `standalone/2026-10-03-gwern-samon-code.zip`), and no external resources.
  `standalone/test_standalone.py` checks that the page makes no requests, that figures, contents and collapsed sections
  work, that the appendix's files and its .zip match this tree and the download delivers the .zip, and runs the
  prototype tests (it is generated from `proto/test/run.py` by `standalone/make_test_standalone.py`).
- `samon.md`: the design document (Gwerndown). `fig/final/`: its figures (PNG and SVG).
- `out/`: saved levels, picks, and experiment outputs.
samon.md · 613 lines · 100.0 kB 
---
title: "_Samon_: Designing a Zen Garden Raking Puzzle"
author: Gwern, Claude-5.5-opus
description: "Full design for a one-stroke sand-raking puzzle game: a novice monk walks into a dry garden and must rake his way back out to the gate. Solver experiments show the bare rule is loose (1.5 million rakings of a 7×7 garden), but scoring by turns recovers the traditional ripple and wave patterns, and a 'wide rake' rule forbidding U-turns in open sand yields unique, garden-like solutions. Covers rules, mechanics, progression, level generation & grading, interface (including hints that are proofs about the player's own route), presentation, a production plan, and prior art, with a playable prototype."
thumbnail: /doc/design/2026-10-05-gwern-samon-thumbnail.png
thumbnail-text: "The prototype's garden 'Ripples' raked at par: the stroke's grooves ring a stone in square waves, with the abbot's vermilion seal in the corner; beside it the title Samon 砂紋 and the rule in one sentence."
created: 2023-02-13
modified: 2026-10-05
status: in progress
confidence: likely
importance: 3
css-extension: dropcaps-kanzlei
...

<!-- LLM META (handoff note; ≤20 lines)
Provenance: expanded by Claude-5.5-opus (configured model id claude-opus-5-5) on 2026-10-03 from Gwern's 2023-02-13 note (the first paragraph and the margin-note voice are his).
All counts, pars, grades & yields come from the attached solver code (zgsolve.c exact DFS; zcp.py CP-SAT; grader.py) and are reproducible from final_figs.py / final_tables.py / exp_*.py.
Solver validated against OEIS A145157 & A145156, an independent brute-force enumerator (0 mismatches in ≈340,000 random small gardens, ≈67,000 solvable, every rule), and the CP-SAT model.
Playtesting: only Gwern's sessions and one outside player's note so far. All difficulty claims are from a mechanical grader, not humans. Treat 'tier' as a proxy.
Prototype: proto/samon.html (artifact page body) & proto/samon.standalone.html, built by proto/build.py from out/proto_gardens.json (src/export_proto.py); proto/test/run.py draws each garden's intended raking with a real mouse and checks par & seal; proto/test/gt_check.py checks the in-page hint search against the C solver.
Revision (same day): merged the best of a parallel design by GPT-6 Astra ("The Last Footprint", supplied by Gwern): proof-backed hints on the player's route, restorable rewinds, explicit Rake commit, practice garden, 'any turn' clue (here pebbles, plus 'straight'), world template, slice plan with pass/fail criteria, symmetry proof. Credited in Methods; pebble numbers from exp_marks.py / fig_pebbles.py.
After Gwern's first play session: helped rakings earn a grey seal; the reveal is now a continuous rake over a lit heightfield (it was cell-by-cell line art).
After the second play session: no Rake button (letting go of a complete walk rakes it; a tap on the raked garden walks back); the monk waits in the gateway, with a beckoning ring and, until the first walk, a ghost fingertip showing the drag (the gate was a dashed circle).
Decisions after the open-questions list: scoring rules kept (gate turn counts; braced U-turns not free); worlds open at two-thirds raked, any turns; the seal unchanged (redundant but harmless in strict gardens); title Samon; worlds ordered by the template (roles computed in src/campaign.py); hints chosen as the plan-changing fact (src/hints.py), with the abbot pointing at its cells.
Campaign: all 148 gardens are in the prototype (src/campaign.py): 34 teaching gardens (13 of them contrasting pairs, src/pairs.py) + 114 generated with garden-like stones (src/garden_sampler.py; yields vs single-cell stones from src/exp_sampler.py); 72 on the main path, 76 extras; first hints for generated gardens are solver facts (src/hints.py). Not curated, not playtested. The in-page hint search memoizes dead positions and finds rewinds by binary search (proof of monotonicity in Methods).
After Astra's review of the prototype (2026-10-04): fixed side-gate turn counting and finished sand as a wide-rake pivot in all solvers (src/test_side_pivot.py), the grey-seal hole, the free inconclusive hint, the dead-end explanation order; short main path + extras; contrasting pairs; continuity claim weakened with numbers (src/exp_continuity.py). At Gwern's request: the monk demonstrates each new rule once, then resets the garden (Watch replays). After an outside player's note (phone; no sense of the potential until garden 6, then the first real garden): it is now third, and the 2×3 garden is gone.
Rendering fixes after Gwern's look (2026-10-04): the stroke starts flush with the back of its first cell (was a quarter-cell in); the poem sits on a plate (frosted then; a white card with a hairline since the gwern.net restyle of 2026-10-05, which made the whole page monochrome: proto/samon.template.html's stylesheet and the single-file edition's PAGE_CSS); stones of several cells fill their cells (lumpy superellipses) instead of floating in bare sand; sand grain is procedural (two height octaves + albedo fleck and mottle), seeded afresh per layout, field up to 144 samples/cell.
Then (at Gwern's request): the scroll of poems earned (enterPoems in the template) and the daily garden (src/daily.py → out/daily_gardens.json, 366 gardens embedded; the page appends today's as a slot after the campaign, keyed daily:YYYY-MM-DD in progress; Rake commits; streak from consecutive keys; tests in run.py section 11). Drag intent (after Gwern's note on clunky mouse control): the walk follows the axis of hand motion with 30° hysteresis (dragTo/dragAxis/settle in the template), not the cell under the pointer, and pointer capture plus clamping keep the grip beyond the walls; tests in run.py section 12. Sound (at Gwern's request; the rake's scrape dialled down 15% at his ear, SCRAPE_LEVEL): drip, pipe, bonshō and tok synthesized in Web Audio (SND in the template; renderSound renders them offline for tests); the shakuhachi recording (sound/tracks.json → sound/make_sound_bundle.py → out/sound_pieces.json) is carried in the page as a data: URL in an inert script.piece, which Gwern's SingleFile/gwtar step can move into an archive (samonAsset in the template resolves data: URLs, plain paths, or gwtar members); tests in run.py section 13. Then the rest of the sound: the rake's scrape driven by revealTick's speed (scrapeStart/Set/Stop), footsteps (crunch) on every step, wind bed and gusts, birds by day and crickets by night (dayPart, by the local clock), a volume slider and a haptics switch (navigator.vibrate, 8 ms per cell); then a mute button in the header (M), navigator.audioSession = ambient, Media Session pause/stop, and a blocked state read from the context (soundBlocked), after Gwern asked for mute controls and whatever browser silence preferences exist (there is no CSS one). Poem taps remapped after dbohdan's note. Astra's second review (2026-10-05): focus returns to the canvas after an abbot action and rewinds restate what holds (addAction/rewindTo); stones repainted as sumi-e (contour/contourPath/paintStone in the template: lumpy contours, shadowBlur contact shadows, gradient wash, one variable-width ink band with dry breaks); the monk is an ensō (makeEnso), at Gwern's suggestion, then rubricated (vermilion) in place of the gate triangle, which he found redundant; both smoothed after his look (no per-segment strokes, no stepped fills). Editorial brackets (gwern.net's convention) on everything the game says to the player: setStatus/appendStatus in the template, the help, lock notes, the scroll's notes and credits, the single-file edition's own messages. Sharing (at Gwern's request, 2026-10-05, to make the one page travel): a raking is a code in the URL's hash (#r=), the walk written as a number in the mixed radix of its legal moves Poem attributions are set in by an indent, with no dash and no kanji (the 原文 button is now "original"; the poet's kanji name is gone from the source line), and the beckoning ring and the ghost finger stop after three pulses (Gwern, 2026-10-05). The page below the garden slimmed at his request: locked worlds as one line, extras folded (renderNav: extrasOpen, .fold), the help a <details>, sound/volume/haptics in a panel under the speaker's chevron (#soundMore/#soundPanel), the rule box only for rule-bearing gardens with the raking counts in Par's tooltip, no Sand counter; assists stay in view. (walkEncode/walkDecode in the template: 8–17 base64url characters for a whole raking, a mean of 22 bits against 86 at two bits a step), shown to whoever opens it but credited to nobody; all progress as another code (#p=, progressCode/readProgress/importProgress, offered and merged with whichever is better); gardens are named in both codes by a 32-bit layout hash (layoutHash/byLayout), never by index, after Astra's review of the first version (8-bit check, index-keyed progress); #daily opens today's garden; the share text draws the raking in box-drawing characters (gridText) and goes through the share sheet or to the clipboard; the daily garden shares its shape and score, never its stroke; tests in run.py section 14. Tooltips on every control, count and mark (gwern.net's habit), including the footnotes and the collapsed sections of the single-file edition.
Poems (at Gwern's request): every garden's reward is a real waka/haiku laid over the raked sand, then a fade to the next garden after ~10 s (a tap on the poem goes on, a tap beside it stays; after dbohdan's note, 2026-10-05). Selection by GPT-6 Astra for the 136-garden campaign; carried over by layout-stable names (out/garden_names.json), 13 new pairings, two Shiki poems whose texts could not be confirmed replaced; all 149 Japanese texts re-checked against linked pages; a second pass swapped 18 weak pairings for better classical poems (poems/pass2_replacements.py), and Astra's review of it 7 more plus 4 translation repairs (poems/pass3_astra_review.py; its two originals declined); log in the JSON's about.revisions; 25 translations revised, 17 new. Data: poems/samon-poems.json (src/poems.py).
Image paths under /doc/design/ are suggestions; files are named per the MoS. Figures are programmatic renders (no generative-image model).
Open decisions flagged inline as TODO.
-->

<div class="abstract">
Raking a Zen dry garden is a chore with the shape of a puzzle: a novice who walks into the sand leaves footprints, and must rake his way back out to the gate without crossing his own lines or raking anything twice.
We design _Samon_ ("sand patterns"), a puzzle game built on that rule: the player draws one stroke from the gate covering every cell of sand; the monk walks it in, sets down the rake, and walks backwards out, revealing the pattern.

We formalize gardens as grid graphs and rakings as Hamiltonian paths ending at the gate, show that "walk in, rake out" and the simpler "mirror raking" are exactly equivalent (so the interface can be "draw from the gate"), and test candidate mechanics on thousands of random gardens with an exact solver, a constraint-programming optimizer, and a deductive difficulty grader.

The bare rule is loose: an empty 7×7 garden with a corner gate has 1,510,446 valid rakings, and random stone layouts are mostly either unsolvable (often by checkerboard parity) or wide open.
Scoring by turns fixes the slack without new rules: in that 7×7 garden only 66 rakings reach the minimum ("par") of 12 turns, and par rakings in lightly-stoned gardens tend to be the traditional motifs of parallel lines and rings around stones.
A single physical rule, the **wide rake** (no U-turn in open sand; set the rake down against a stone or wall), collapses gardens to a handful of rakings, often exactly one, and in about half of the calm-looking unique gardens that one is also a par raking of the unrestricted garden: in one 8×8 garden it picks out the only U-turn-free raking among 12 par rakings of 6,310,226.
Ripple stones and wave bands, the obvious "pattern" constraints, add texture but not tightness; given grooves pin down unique answers with ~5--8 clues, and pebbles that say only "straight here" or "turn here" need more clues (a median of 11 against 7) but make nearly every garden a lookahead puzzle.

The resulting design has 7 worlds, each adding one rule (shown by a before-and-after pair of small gardens, the second raked once by the monk as a demonstration); free gardens scored against a proven par with [_Opus Magnum_](!W)-style histograms; strict gardens with unique, graceful answers; hints that are proofs about the player's own route (it can still reach par; it can still be finished; or it cannot, and here is the nearest footprint to step back to); a daily garden that can be raked only once; a classical poem as the reward for every raked garden, after which the next garden fades in; a heightfield sand renderer and a free-raking practice garden; a level pipeline whose yields and difficulty distribution we measure; and a production plan with pass/fail criteria.
A playable prototype implements most of it, with the whole campaign of 148 gardens and their poems.
</div>

Zen sand gardens are often intended for meditation, whether to look at or while maintaining it by raking the sand into the prescribed patterns which are formed by the tines of the rake yielding parallel lines (like `|||`).
Such a chore might be tedious, but there is also a great deal of satisfaction to restoring an esthetic order from chaos, and raking sand seems like one of those things, like popping bubblewrap, which can be oddly satisfying---I recall the lifeguards at my summer camp seeming to rather enjoy the task of raking the beach in front of their office every day and not allowing any campers to do so.
Puzzle-solving is also often described as being 'meditative' and 'oddly satisfying', even when it looks a lot like a tedious chore.
So for those of us who do not live at a Zen temple, can we make a Zen sand garden *puzzle*?

This page turns the original 2023 sketch into a full design.
Where the sketch speculated (would "mirror raking" be too restrictive? could a neural net rank solutions by beauty? how should difficulty be raised?), we answer with a solver and measurements; where it left gaps (what exactly makes a garden hard, what the worlds are, how levels are generated, what the player touches), we fill them.
The title is _Samon_ (砂紋, "sand crest"), the term for the patterns themselves.[^samon]

[^samon]: [JAANUS](https://projects.mcah.columbia.edu/jaanus/node/1584 "'samon 砂紋', JAANUS"): "The technique of making a pattern or design in sand or gravel... Typically in such gardens, the gravel is raked into a series of ridges or ripples which suggest waves in an ocean, but may well be pure abstraction." Garden associations list ~7 standard patterns, including _sazanami_ (continuous ripples), _mizumon_ (concentric rings), _ryūsui_ (flowing water) and _ichimatsu_ (checkerboard) ([NAJGA 2023](https://najga.org/7-common-raking-patterns-in-japanese-dry-landscape-gardens/ "'7 Common Raking Patterns In Japanese Dry Landscape Gardens', NAJGA 2023")).

# The Garden

[Pieces.]{.margin-note} A garden is a walled rectangle of white gravel---in Kyoto, typically _Shirakawa-suna_, weathered granite from the Kitashirakawa area ([Nabunken](https://www.nabunken.go.jp/org/bunka/jgd/pages/ShirakawaSand.html "'Shirakawa sand', Nara National Research Institute for Cultural Properties"))---divided into square cells, with stones as obstacles.
One cell on the veranda side is the **gate**: the gap in front of the _kutsunugi-ishi_, the flat stone below the veranda where one removes one's shoes ([JAANUS](https://projects.mcah.columbia.edu/jaanus/node/2714 "'kutsunugi-ishi 沓脱石', JAANUS")).
The player is an _[unsui](!W)_, a novice monk, and the garden is viewed, as at [Ryōan-ji](!W), from the abbot's veranda: the bottom edge of the screen.

[The rule.]{.margin-note} The novice steps off the shoe-stone, walks into the sand, puts down his rake, and walks *backwards* to the gate, dragging the rake over his own footprints.
The garden is finished when every cell of sand has been raked exactly once, the lines are unbroken, and he is back on the shoe-stone.
That is the whole rule: one stroke, every cell once, no crossing, ending at the gate.

![*Anatomy of a raking.*<br>(*1*) A 6×6 garden with 2 stones; the gate is the gap in the bottom wall.<br>(*2*) The player draws the walk in from the gate; the monk leaves footprints.<br>(*3*) At the end of the walk he sets down the rake (red dot) and walks backwards, the rake erasing his footprints.<br>(*4*) The finished garden: 10 turns, which is par; 2 of this garden's 1,023 valid rakings reach it.](/doc/design/2026-10-03-gwern-samon-anatomy.png){.width-full}

[Formally: Hamiltonian paths.]{.margin-note} Formally, the sand cells form a grid graph, and a valid raking is a [Hamiltonian path](!W) of that graph which ends at the gate cell.
The rake lines are the path's shape: a cell traversed straight gets parallel grooves; a cell where the path turns gets concentric quarter-circles; the cell where the rake was set down gets the groove's start.
Every claim below about counts, pars, and difficulty comes from an exact solver for this formulation (see [Methods](#methods)).

[Walking in is free.]{.margin-note} The original sketch distinguished the forward walk (leaving footprints) from the backward raking, and proposed "mirror raking"---the monk retraces his walk exactly---as a possibly-too-restrictive simplification.
It is not a restriction at all.
Any walk in, followed by a raking that covers every cell exactly once and ends at the gate, erases every footprint, because every cell gets raked; so the walk in is irrelevant except for where it ends, and the set of finished gardens is exactly the set of Hamiltonian paths ending at the gate.
Conversely, every such path is achieved by walking it in and raking it out.[^mirror-proof]
So mirror raking loses nothing, and the interface can be the simplest possible: *draw a path from the gate*; that path is the walk in; the raking is its reverse.

[^mirror-proof]: Proof sketch: let the walk in be any walk _W_ from the gate to cell _P_, and the raking _R_ any path from _P_ to the gate that visits every sand cell once. Every cell _W_ touched is on _R_, so all footprints are raked over; the finished pattern is _R_ alone. Hence {finished gardens} = {Hamiltonian paths ending at the gate} regardless of _W_, and choosing _W_ = reverse(_R_) shows every such path is reachable by mirror raking. The one physical constraint---never step on raked sand---holds automatically, since walking backwards the monk only steps onto cells the rake has not yet reached.

This matters for design because the story stays intact (the reveal is the monk raking himself out of the garden) while the player's task is a single, legible gesture.

# The Bare Rule Is Loose

[Too many rakings.]{.margin-note} The one-stroke rule alone is a weak constraint in open sand.
For an empty garden with a corner gate, the number of valid rakings is the number of Hamiltonian paths from a corner, [OEIS A145157](https://oeis.org/A145157 "'Number of Greek-key tours on an n X n board; i.e., self-avoiding walks on n X n grid starting in top left corner', OEIS"), which grows explosively: 824 for 5×5, 1,510,446 for 7×7, 54,986,690,944 for 9×9.
(Our solver reproduces A145157 and the 5×_n_ sequence [A145156](https://oeis.org/A145156) exactly, which is a useful correctness check.)
Anyone can rake an open garden: walk up and down in rows.

<div class="table-small">
| Garden | Valid rakings | Par (min. turns) | Rakings at par | Median turns | Max turns | Wide-rake rakings |
|:------:|--------------:|:----------------:|---------------:|:------------:|:---------:|:-----------------:|
| 3×3    | 8             | 4                | 2              | 5            | 6         | 2                 |
| 4×4    | 52            | 6                | 4              | 8            | 13        | 4                 |
| 5×5    | 824           | 8                | 10             | 13           | 20        | 4                 |
| 6×6    | 22,144        | 10               | 24             | 20           | 31        | 6                 |
| 7×7    | 1,510,446     | 12               | 66             | 27           | 42        | 6                 |
| 6×8    | 899,432       | 11               | 66             | 26           | 42        | 8                 |

Table: Empty gardens, gate in the bottom-left corner of the south (veranda) wall. "Turns" counts every cell where the stroke changes direction, including leaving through the gate. The last column anticipates the [wide rake](#the-wide-rake).
</div>

[Parity.]{.margin-note} Stones make the rule bite, but mostly by making gardens *impossible*.
Color the cells like a checkerboard: every step of the stroke changes color, so a stroke over _n_ cells alternates colors, and if one color has more cells, the stroke must begin and end on it.
A 5×5 garden has 13 dark and 12 light cells, so its gate must be on a dark cell; move the gate one cell along the wall and there is no raking at all.
Of random 7×7 gardens with a gate somewhere on the veranda wall, 57% pass this parity test with no stones and only 31% with 6 stones.

![*Checkerboard parity decides solvability before anything else.*<br>A 5×5 garden has 13 dark and 12 light cells, so the stroke must start and end on dark cells.<br>(*Left*) Gate on a dark corner: 824 valid rakings (one at par, 8 turns, shown).<br>(*Right*) Gate one cell over, on a light cell: no raking exists.](/doc/design/2026-10-03-gwern-samon-parity.png)

[Random stones: unsolvable or open.]{.margin-note} Past parity, stones create dead ends (a cell with one free neighbor must be where the rake is set down---at most one such cell may exist) and corridors (a cell with two free neighbors must use both).
These are the classic deductions of one-stroke puzzles, and they are real, but random stone placement produces them in the wrong proportion: gardens are either unsolvable or still have thousands of rakings.
In 300 random 7×7 gardens per row, *none* with 0--6 random stones had a unique raking:

<div class="table-small">
| Stones | Parity OK | One stroke: solvable | One stroke: unique | One stroke: median rakings | No U-turns at all: solvable | Wide rake: solvable | Wide rake: unique | Wide rake: median rakings |
|:------:|:---------:|:--------------------:|:------------------:|:--------------------------:|:---------------------------:|:-------------------:|:-----------------:|:-------------------------:|
| 0      | 57%       | 57%                  | 0%                 | ≥100,000                   | 0%                          | 57%                 | 0%                | 6                         |
| 2      | 43%       | 42%                  | 0%                 | 28,326                     | 0%                          | 18%                 | 2.7%              | 2                         |
| 4      | 34%       | 20%                  | 0%                 | 610                        | 0%                          | 2.7%                | 0.3%              | 3                         |
| 6      | 31%       | 8%                   | 0%                 | 96                         | 0%                          | 0.3%                | 0%                | 4                         |

Table: 300 random 7×7 gardens per row, random 1×1 stones, gate at a random cell of the south wall. Medians are over solvable gardens (counts capped at 100,000). The wide-rake columns are explained [below](#the-wide-rake).
</div>

The bare rule therefore cannot carry a game by itself: completion is trivial in open sand, and the obstacles that make it non-trivial mostly make it impossible.
Two things fix this: a *score* that makes the slack matter, and *rules* that remove it.
Pleasingly, both come from what makes a raked garden beautiful.

# Calm: Turns As Score

[The abbot counts turns.]{.margin-note} The commonest _samon_ are long parallel lines, interrupted where they ring a stone.
A raking that zigzags looks agitated.
So the game's single esthetic metric is the number of **turns**: every cell where the stroke changes direction.
Fewer turns is calmer.
Each garden has a **par**, the minimum number of turns over all valid rakings, *proven* by a constraint solver (in a few seconds at most, for gardens up to 9×9).
Any valid raking finishes the garden; a raking at par earns the abbot's vermilion seal.

![*The same 7×8 garden raked three ways.*<br>(*Left*) Par, 17 turns: 6 of its 103,812 valid rakings achieve it.<br>(*Middle*) A median raking, 31 turns.<br>(*Right*) The most agitated possible raking, 48 turns (5 rakings).](/doc/design/2026-10-03-gwern-samon-calm-agitated.png){.width-full}

[Par is a needle.]{.margin-note} Turns make the looseness useful.
The distribution of turns over all valid rakings is a broad hump, and par sits in the far tail: in the empty 7×7 garden, 66 of 1,510,446 rakings (0.004%) reach par, and the median raking has more than twice as many turns.
Finishing a garden is easy; finishing it *calmly* is a real optimization puzzle, of the kind where intuition ("make long lines; circle the stones; use the walls") gets close and the last two turns take thought.

![*Turn counts of all 103,812 valid rakings of the garden above.* Par (17 turns) is achieved by 6 rakings; the median raking takes 31.](/doc/design/2026-10-03-gwern-samon-histogram.png)

[Histograms.]{.margin-note} This is the structure of [Zachtronics](!W) optimization games, where after solving, the player's solution is ranked against every other player's on a few metrics ([_Opus Magnum_](!W) compares cycles, cost, and area).
_Samon_ shows one histogram per garden after completion: the player's turn count against everyone's, with par marked---a picture very like the one above, filled in by real players.
Because par is *proven*, the seal is never a trap (no "par" that nobody can reach) and never a lie (no raking can beat it).

[Par tends to look traditional.]{.margin-note} The surprise is that minimizing a single number recovers the traditional motifs.
In the open and lightly-stoned gardens we inspected, par rakings were nested U-bends and spirals: long parallel lines across the garden (_sazanami_), and concentric rings around stones (_mizumon_), without either being specified.
The rings appear because circling a stone is the cheapest way to use the cells next to it; the parallel lines appear because every turn costs.
This is not a law, though: with crowded or awkwardly-placed stones, par rakings degenerate into meanders full of tight U-turns in open sand.
Among 60 random solvable gardens (6×7--7×8, 1--3 stones), only 40% admitted *any* raking free of open-sand U-turns, and only 20% had a par raking that was free of them; so "par is beautiful" is a property for the level curator to select for (and we do: see [Generating Gardens](#generating-gardens)), not something the rule guarantees.

[Why not a neural esthetic score?]{.margin-note} The sketch proposed ranking solutions with a learned esthetic model, such as the CLIP-based predictor behind [LAION-Aesthetics](https://laion.ai/blog/laion-aesthetics/).
For *scoring the player*, a hand-countable metric is better: the player can see it, count it, and deliberately optimize it, and a proven optimum exists; an opaque neural score can be none of these, and would invite adversarial raking.
The learned model is better spent upstream, ranking *layouts* (which stone arrangements look like real gardens, which par rakings are prettiest when several tie), where opacity costs nothing.
<!-- TODO: test whether a CLIP-aesthetic score of renders agrees with turn count among rakings of one garden; untested here (no model weights available offline). -->

# Mechanics

[One rule per world.]{.margin-note} The standing criticism of [_Zen Puzzle Garden_](https://www.lexaloffle.com/zen.php "'Zen Puzzle Garden', Lexaloffle 2003"), the closest ancestor, was monotony: "there's still a persisting feeling that, even by the final few stages, _Zen Puzzle Garden_ hasn't developed one step beyond what the tutorial already demonstrated" ([Wilson 2011](https://www.pocketgamer.com/zen-puzzle-garden/review/ "'Zen Puzzle Garden Review', Wilson 2011")).
So each world adds one rule, and each rule must earn its place by changing *how the player thinks*, not just the scenery.
Every rule below is diegetic---a thing in the garden or a property of the rake---and every one was tested with the solver on random gardens before being kept.
Here are the candidates that survived, in the order the game introduces them, and then the ones that did not.

## Ripple Stones

[Rings.]{.margin-note} A **ripple stone** (marked with faint concentric rings in the abbot's sketch) must be encircled: its 8 neighboring cells must be raked as one unbroken arc, entering once and leaving once.
This is _mizumon_, the rings-around-an-island motif, as a hard constraint instead of an esthetic tendency.

[Texture, not tightness.]{.margin-note} The surprise of testing was how little it constrains.
In the garden below, the ripple rule cuts the number of valid rakings from 183,986 (with an ordinary stone) to 35,570, and leaves par unchanged at 12; 4 of the 5 par rakings already ringed the stone.
Across random 7×7 gardens with one ripple stone (and 0--2 plain stones), the median number of rakings was still 5,573.
Ripple stones teach "rings are entered from outside and left from the cell next to the entrance" and make the reveal prettier, but they cannot make a garden tight on their own.

![*A ripple stone.*<br>(*Left*) The abbot's sketch marks the stone with rings: its 8 neighbors must be raked as one unbroken arc.<br>(*Right*) The par raking (12 turns; 4 of 35,570 rakings) starts at the stone and spirals outward, the ripple becoming a whirlpool.](/doc/design/2026-10-03-gwern-samon-ripple.png){.width-full}

## Wave Bands

[Lines that run through.]{.margin-note} A **wave band** is a hatched region of the sketch: inside it, every row (or column, for a vertical band) must be raked as one straight, unbroken run, edge to edge.
This is _sazanami_: the long parallel ripples that most gardens are mostly made of.
Mechanically, each row of the band becomes a corridor that must be crossed in one go, so the player must plan where the turnarounds go outside the band.

[Somewhat tighter.]{.margin-note} Bands constrain more than rings: in the garden below, the band cuts 1,492,008 rakings to 2,984, and raises par from 16 to 17 (the calmest raking of the bare garden does not run its lines where the abbot wants them).
But across random 7×7 gardens with a band (and 0--2 stones), the median count was still 3,799.

![*A wave band.*<br>(*Left*) The hatched band must be crossed by straight runs, edge to edge.<br>(*Right*) A par raking: 17 turns, one of 34 at par among 2,984 valid rakings.](/doc/design/2026-10-03-gwern-samon-waves.png){.width-full}

[Ichimatsu.]{.margin-note} Alternate horizontal and vertical bands in a checkerboard and you get _ichimatsu_.
Here the band rule finally becomes tight, because every cell now belongs to a run and the stroke can turn only at run ends, handing off from one run to the next: the 9×9 garden of nine 3×3 blocks below has exactly 2 valid rakings, versus 54,986,690,944 for the empty 9×9 garden, and the 6×6 version (four blocks) has exactly 1 for most gate positions.
These make good end-of-world showpieces: a real checkerboard _samon_, drawn in one stroke.

![*Ichimatsu.*<br>(*Left*) Nine 3×3 blocks, raked alternately across and down.<br>(*Right*) One of its 2 valid rakings (both 45 turns).](/doc/design/2026-10-03-gwern-samon-ichimatsu.png){.width-full}

## The Master's Grooves

[Continue his lines.]{.margin-note} The abbot has begun the raking and been called away: a few cells already carry grooves, and the finished garden must be one stroke that runs *along* them.
(Re-raking a groove in its own direction leaves no trace, so the novice may pass through these cells, but only following them.)
These are the "partial raking" of the original sketch, and are the game's general-purpose clue: unlike abstract numbers or symbols, a given groove is self-explanatory, since it *is* a piece of the answer.

[Uniqueness for ~5--8 grooves.]{.margin-note} Grooves are how free gardens become strict ones.
The level generator picks a beautiful raking as the intended answer---a par raking with no U-turn in open sand---and adds grooves from it until it is the only valid raking left.
In the garden below, 5 grooves cut 12,375 rakings to 1, and the answer is a par raking (21 turns; 1 of 11 at par).
Over 69 generated 7×7--7×8 gardens, the median was 7 grooves.

![*The master's grooves.*<br>(*Left*) Five cells are pre-raked; the stroke must run along them.<br>(*Right*) The unique raking (21 turns, which is also par).](/doc/design/2026-10-03-gwern-samon-grooves.png){.width-full}

[Two kinds of groove.]{.margin-note} The abbot can also leave *finished* work: a closed ring raked around a stone, as real gardens have.
A single stroke can never close a ring, so these are effectively obstacles---and a way to put the iconic closed ripple into gardens without violating the one-stroke rule.
They are also the "partial raking which makes an insoluble problem soluble" of the sketch: pre-raked cells are removed from the stroke's job, which changes the garden's topology (and, if the finished shape covers unequal numbers of dark and light cells, its parity).

## The Abbot's Pebbles

[Straight or turn, nothing more.]{.margin-note} A weaker clue than a groove: the abbot leaves a pebble on a cell.
A pale pebble means the stroke runs straight through that cell, across or down; a dark pebble means it turns there, either way.
A pebble's cell is never where the rake is set down, since that cell has only one connection.
(The "any turn" clue comes from a parallel design by GPT-6 Astra; Nikoli's [Masyu](!W) uses white and black pearls the same way, with extra conditions on the neighboring cells that pebbles do not have.)

[More clues, harder deduction.]{.margin-note} We made 88 generated 7×7--7×8 gardens unique both ways, from the same intended raking (a graceful par raking) by the same greedy procedure.
Grooves needed a median of 7 clues and pebbles 11 (more in 92% of gardens), but the pebble gardens were much harder: 92% needed lookahead by our grader (median 7 steps) and 12.5% nested lookahead, against 47% (median 2 steps; 58% in the pipeline's separate sample, [below](#generating-gardens)) and none for grooves.
A groove gives away a piece of the answer; a pebble is a disjunction to be resolved against its neighbors, which is the kind of deduction the strict worlds want.
So pebbles come after grooves in world 4, and supply the hardest strict gardens of world 7.
Below, 8 pebbles leave exactly one of a garden's 59,498 rakings; the same answer needs only 6 grooves, which make the garden tier 1, where the pebbles make it tier 3.

![*The abbot's pebbles.*<br>(*Left*) A 7×7 garden made unique by 6 grooves: tier 1, every step forced.<br>(*Middle*) The same garden and answer made unique by 8 pebbles (pale: straight through; dark: a turn): tier 3, 14 lookahead steps.<br>(*Right*) The only raking: 13 turns, which is par (6 of the bare garden's 59,498 rakings reach it).](/doc/design/2026-10-03-gwern-samon-pebbles.png){.width-full}

## The Wide Rake

[No U-turns in open sand.]{.margin-note} The novice is promoted and given the senior's rake, which is too wide to pivot.
With it, the stroke may turn 90° anywhere, but it may make a tight U-turn (two consecutive turns the same way) only when the U is braced against a wall or a stone; and the rake must be set down in a cell touching a wall or a stone, never in the open.
The abbot's finished sand (his closed rings, above) is neither: turning against it would rake it.
A reversal in open sand therefore needs *something inside it*: a stone to wrap around, or a nest of wider U's around a stone.
The esthetic motivation is the same as for counting turns: a groove that doubles back on itself in open sand is the ugliest thing a raking can contain, and in real gardens lines end at walls and stones, not in open gravel.

![*The wide rake.*<br>(*Left*) A tight U-turn in open sand is forbidden.<br>(*Middle*) Against a wall, it is allowed.<br>(*Right*) A wider U-turn around a stone is allowed.](/doc/design/2026-10-03-gwern-samon-widerake-rule.png){.width-full}

[It tightens everything.]{.margin-note} This one physical rule does what the pattern constraints could not.
In empty gardens it leaves 2--8 rakings out of up to millions (see the first table); in random 7×7 gardens with 2 stones, 18% are solvable and the median solvable garden has 2 rakings; uniqueness, which the bare rule did not produce in any of the 1,200 random 7×7 layouts, appears by itself (2.7% of 2-stone layouts).
Forbidding every U-turn, braced or not, is too much: no empty garden of any size we tried, and none of 1,200 random 7×7 gardens, could be raked at all.

[Often the calm answer.]{.margin-note} The striking result is *which* raking survives.
The 8×8 garden below has 6,310,226 valid rakings under the bare rule; 12 of them reach par (17 turns); exactly one of those 12 has no U-turn in open sand; and that one is the garden's *only* raking under the wide rake.
This is not a fluke of one garden, though it is not a law either: over 964 random gardens with a unique wide-rake raking, that raking was a par raking of the unrestricted garden in 32% of cases, and in 47% of the calm-looking ones (turn density ≤0.40) that the curator would keep.
So the novice's tool often selects the very raking the abbot would have chosen; but it is a stricter rule than his taste, not the same one (see [Free, then strict](#progression)).

![*A wide-rake garden.*<br>(*Left*) Two stones.<br>(*Right*) Its only valid raking under the wide rake (17 turns): a spiral around one stone, a horseshoe around the other, parallel lines everywhere else. It is the only one of the garden's 12 par rakings without a U-turn in open sand.](/doc/design/2026-10-03-gwern-samon-widerake.png){.width-full}

[Hard.]{.margin-note} Wide-rake gardens are the hardest in the game by our grader: every one we generated required lookahead (assume a move, propagate, find a contradiction), and about a quarter required *nested* lookahead.
The player-facing heuristics---"every reversal needs a pivot", "count the cells inside each U", "lines run wall to wall"---are learnable, but this needs playtesting more than anything else in the design, and is why it comes fifth rather than second.

## The Side Gate

[Both ends fixed.]{.margin-note} Some gardens have a second, side gate, and the novice must rake his way *in* from it (walking backwards from the start, so leaving no footprints at all) and out by the veranda.
Fixing both ends of the stroke doubles the force of parity (both endpoint colors are now given) and removes the player's freedom of where to begin.
The stroke comes in through the wall, so a bend in its first cell counts as a turn, as leaving by the veranda gate does.
Under the bare rule this is still loose (the garden below has 311,080 such rakings); combined with the wide rake it gives a unique, graceful answer that is also par.

![*The side gate.*<br>(*Left*) The stroke must begin at the side gate (top, outlined) and end at the veranda gate.<br>(*Right*) The only valid raking under the wide rake: 20 turns, counting the bend just inside the side gate, which is also par for the bare rule (32 of the 311,080 rakings reach it).](/doc/design/2026-10-03-gwern-samon-twogates.png){.width-full}

## Rejected Mechanics {.collapse}

<div class="abstract-collapse">
Mechanics we considered and dropped, with the reason for each.
</div>

- **Ice-sliding**: _Zen Puzzle Garden_'s rule (the monk slides until blocked, and may only turn when blocked) is the obvious alternative core. It is taken, and it makes the puzzle about stopping places rather than shapes.
- **Braced turns only** (90° turns allowed only next to walls or stones): a softer cousin of ice-sliding. Too strict: only 31% of empty 7×7 gardens remain solvable, and almost no gardens with stones.
- **No U-turns at all**: infeasible; see above.
- **Pushable lanterns, ordered leaves**: _Zen Puzzle Garden_'s later additions ([JayIsGames](https://jayisgames.com/review/zen-puzzle-garden.php "'Zen Puzzle Garden', JayIsGames")). They import Sokoban into a game whose identity is the line.
- **Repairing a disturbed garden** (a deer walked through overnight; re-rake only what you must): it sounds rich but is trivial. For the repaired garden to be one stroke, the grooves the novice leaves untouched must be an initial segment of the old stroke, ending where his new raking begins; so the cheapest repair is always to re-rake everything from the first damaged cell (in the old stroke's order) back to the gate, and the route he takes doesn't matter.
- **Multiple monks / multiple strokes / stepping stones that lift the rake**: each adds slack to a rule whose problem is already slack.
- **Bridges over a dry stream** (the only cells the stroke may cross): workable (_Flow Free: Bridges_ does it), but untested here, and it weakens "never cross your own lines", the rule that makes the fiction work.
- **Symmetric gardens** (mirror-image halves raked in mirror image): against the grain of the esthetic, which prizes _fukinsei_, asymmetry ([Japanese aesthetics](!W)), and impossible anyway. A symmetry of the garden that preserves the stroke maps the path onto itself, and a path's only symmetries are the identity and reversal; reversal would swap the gate's cell with the rake's start, so it must be the identity, which fixes every cell of the stroke: all the sand would have to lie on the mirror line (Astra's argument). *Local* symmetry, two marked patches raked in mirror image, is possible; Astra proposes it as an optional rule, untested here.

# Progression

[Seven worlds, one day.]{.margin-note} The worlds follow one day at the temple, dawn to moonrise, which gives each a lighting (and so a look: low sun makes grooves cast long shadows; noon flattens them; moonlight silvers them) without any text.
Each introduces exactly one rule; gardens grow from one-cell corridors to 9×9.

<div class="table-small">
| World | Time | New rule | What it teaches | Gardens (main path + extras) |
|-------|------|----------|-----------------|--------:|
| 1. First Stroke | Dawn | One stroke from the gate; stones | A dead end is where the walk in must end; corridors force the route; parity | 9 + 10 |
| 2. Ripples | Morning | Ripple stones | Rings: entered once, left beside the entrance | 10 + 10 |
| 3. Waves | Noon | Wave bands; _ichimatsu_ | Runs are atomic; plan turnarounds outside them | 11 + 9 |
| 4. The Master's Grooves | Afternoon | Given grooves (open & closed); then pebbles | Continuation; reading a unique answer; resolving "straight or turn" against the neighbors | 16 + 9 |
| 5. The Wide Rake | Evening | No U-turn in open sand; set rake down at an edge | Every reversal needs a pivot | 9 + 17 |
| 6. The Side Gate | Dusk | Two gates | Both ends fixed; double parity | 9 + 9 |
| 7. Silver Sand | Night | All of the above, larger gardens | Synthesis | 8 + 12 |

Table: World structure (148 gardens, all in the prototype: 72 on the main path, 76 optional extras). Worlds 1--3 are mostly *free* gardens (many valid rakings, scored against par); worlds 4--7 mostly *strict* (one valid raking).
</div>

[Free, then strict.]{.margin-note} The arc runs from optimization to deduction.
In worlds 1--3 any valid raking finishes a garden and the seal rewards calm; the player learns, by seeing their own rakings next to par, that calm rakings are long lines and rings.
From world 4 on, gardens have one valid raking, and the rules (grooves, then the wide rake) *enforce* a calm that the first half only rewarded.
It is not quite the same calm.
The generator keeps a free garden only if *some* par raking has no U-turn in open sand, but par is usually reached by others too: in 42 of the 54 free gardens of worlds 1--3 the par rakings include both kinds, and 327 of all 407 par rakings there turn back in open sand somewhere (_Waves_, for instance, has 34 par rakings, and 32 of them do).
So the wide rake is a stricter rule than the abbot's taste, to be learned as a new rule, not the taste made physical (Astra's correction: an earlier version of this page claimed the latter).
Curating the free worlds so that every par raking is graceful would leave too few gardens (11 of 54 qualify), so instead the wide rake's introduction makes the difference visible: its first garden is one whose par raking turns back in open sand (below).
The seal works the same way throughout; in a strict garden every valid raking is at par, so there it marks a garden raked without the abbot's help.
The final world is named for the "sea of silver sand" at [Ginkaku-ji](!W), raked "so that in moonlight its ridges appear as ripples in water" ([Cartwright 2019](https://www.worldhistory.org/Ginkakuji/ "'Ginkakuji', Cartwright 2019")).

[A short main path.]{.margin-note} Each world has a short main path, which is the template below and nothing more (8--16 gardens; 72 in all), followed by optional extras (76), listed after it as "more": the rest of the generated gardens, for practice and for seals.
In the first version of the campaign every garden was on the path, so grooves came after 38 gardens and the wide rake after 52 (Astra's count, from the unlocking rule); now they come after 21 and 32.

[Opening worlds.]{.margin-note} A world opens once two-thirds of the main path of the world before it has been raked, at any number of turns; extras raked instead count too.
Par is never required, and any garden can be finished, by persistence or with the abbot's help, so no player is ever locked out; within an open world every garden is open, so a player stuck on one rakes others.

[The first five gardens.]{.margin-note} Teaching is wordless, by construction:
(1) a one-cell-wide corridor (drawing, and the reveal);
(2) a 3×3 garden with a corner gate (more than one answer, and U-turns at the walls; the turn counter and par appear);
(3) the first real garden, the easiest of world 1's generated ones (5×5, one stone), where long lines turning at a stone first appear;
(4) a garden with a one-cell pocket behind a stone (the pocket can only be where the rake is set down);
(5) a garden of two-tone gravel in which the obvious place to end the walk is the wrong color, so every attempt ending there strands a cell (parity, made visible; the two tones fade to uniform white over the next two gardens).
The real garden comes before the two techniques, which only matter at its size: an early player, on a phone, did not "sense the potential until about six levels in", which is where the first real garden then came.
A 2×3 garden that taught the U-turn was dropped to make room; the 3×3 garden's par rakings turn back at the walls anyway.

[Inside a world.]{.margin-note} Each world after the first follows one template (Astra's): an introduction (a contrasting pair, below); two applications; a garden whose obvious move, learned from the gardens before it, is wrong; two combinations with earlier rules; a calm, easy garden; and a capstone.
Every garden is stored with its intended raking, the one observation it teaches, and the abbot's first hint, which states that observation; the test for keeping a garden is *what fact changes the player's plan?*
A garden is cut if the only way to solve it is trial and error, or if its clues trace the answer outright.

[Show the change.]{.margin-note} Each later rule is introduced by a contrasting pair (also Astra's suggestion): a small garden raked as usual, then the same garden with one change, both found by search.
A ripple stone, a band, or the wide rake is added to a garden whose every par raking breaks the new rule, so the plan that just worked visibly fails; one groove cuts a garden's 8 rakings to one; pebbles, one pale and one dark, replace the two grooves that pinned another garden down; a side gate leaves one of a garden's two wide-rake rakings, and not the par one.
The pairs have 17--23 cells of sand; the first campaign introduced bands in a 55-cell garden, and grooves and pebbles in 47-cell ones, which now come after two smaller applications.
The changed garden of each pair is demonstrated before the player rakes it (see [Interface](#interface)).

[The campaign.]{.margin-note} The prototype has all 148 gardens.
Thirty-four are teaching gardens: 13 of the prototype's original 14 (the 2×3 garden was dropped), the corridor, three _ichimatsu_ variants, four small introductions found by search (among them the two-tone parity garden), and the 13 gardens of the contrasting pairs.
The other 114 come from the pipeline below, with garden-like stones: for each size and mechanic, it generates up to twice as many candidates as needed (235 from 16,437 sampled layouts, in 33 minutes of computing) and keeps a subset spread over their difficulty range, no two alike up to rotation and reflection (apart from the pairs, which are alike on purpose).
Half of that time went on failing to find a single 9×9 pebble garden (none of 227 layouts passed the filters), so world 7's pebble gardens are 8×8.

[Ordered by the template.]{.margin-note} Each world is put in the template's order, with every role assigned by measurement.
Applications run from easy to middling (the hardest are extras): free gardens by size, then by how rare par is among their rakings; strict gardens by grade.
The trap is the garden whose obvious move is most clearly wrong: in a free garden, the one whose key observation (below) the near-misses, rakings one or two turns over par, most often get wrong (between 74% and 94% of them, in worlds 1--3); in a strict garden, one whose only raking turns aside at once, where walking straight in from the gate can be refuted only by lookahead.
The combinations add a rule from an earlier world (a second ripple stone; a ripple stone among bands; grooves or pebbles with a ripple or a band; the wide rake with either; pebbles at a side gate), generated for the purpose.
The calm garden is the one with the fewest turns per cell among the easier half of the rest, and the capstone is the hardest, or a hand-made showpiece (_Calm_, _Spiral_, _Side gate_, _Silver sand_).
A rule introduced mid-world (_ichimatsu_; the abbot's finished rings; his pebbles) gets its own short run of gardens after the trap.

[Hints that change the plan.]{.margin-note} Each generated garden's first hint is the one provable fact that most changes the player's plan, and the abbot points at the cells it names.
In a free garden, it is the fact true of every par raking (where it ends; how it begins; a row, column, or stretch of one raked in one straight line; whether the line turns or runs straight in some cell) that the most near-misses get wrong: a median of 50% of them.
In a strict garden, it is the fact about the only raking (where the rake is set down, or the line's shape in one cell) that most lowers the grade; when the rake is set down in a cell with one way in, the hint says so, giving the reason before the place.
Of the 80 strict gardens with such a hint, 62 need lookahead without it; with it, 16 need none, the 10 that needed nested lookahead need only one level, and the other 36 need fewer steps.
Of the 18 that need no lookahead anyway, it lets 4 be solved by local rules alone (the other 14 already are).
In a strict trap, the observation is the trap itself: walking straight in goes wrong.
None of this is curated or playtested: it is the raw material for step 3 of the [production plan](#production-plan), not its result.

[Daily garden: _ichi-go ichi-e_.]{.margin-note} One generated garden per day, the same for everyone, alternating free and strict.
The player may draw and undo freely, and here nothing is raked until they press **Rake** (the only garden with the button); that first raking is the day's record---"one time, one meeting" ([_ichi-go ichi-e_](!W "Ichi-go ichi-e")).
The result shared is the seal (or not) and turns versus par, with the garden's shape but not the stroke, which spoils nothing (everyone else is still raking it that day).
The prototype has it: a year of 366 gardens made by the campaign's generators and never used in it (even days free, odd days strict; 5×5 to 8×8; ripple stones, bands, grooves, pebbles, the wide rake and side gates all appear, with the rule stated in a line, since the daily garden does not wait for the player to reach the world that teaches it), picked by the local date, so everyone gets the same garden on the same day and the year repeats.
Drawing and undoing are free; **Rake** commits; the raking is then final for the day, with the record, the poem (the campaign's, by day of the year) and a streak of consecutive days shown, a share to pass on ("Samon · daily garden 2026-10-04 · 14 turns, par 12 · 静 · streak 3", the garden's shape drawn in box-drawing characters without the stroke, and a link that opens today's garden: see [Sharing](#sharing) below), and a new garden tomorrow.
[LinkedIn's _Zip_](https://www.gmpuzzles.com/blog/2025/03/introducing-zip "'Introducing Zip', Snyder 2025"), a daily Hamiltonian-path puzzle, suggests the appetite is there.

# Generating Gardens

[Pipeline.]{.margin-note} Handmade levels set the curriculum; generated ones fill it, and supply the daily garden.
The pipeline:

#. Sample a layout: garden size, stones, gate on the veranda wall, and the world's markings. Stones are placed the way a garden designer would: an odd number of pieces (1, 3 or 5), each of 1--3 cells (single, straight or L-shaped), none within a cell of another, with no mirror or half-turn symmetry, and covering at most ~15% of the garden. (The experiments elsewhere on this page used single-cell stones; the two samplers are compared below.)
#. Reject on parity (instant).
#. Count valid rakings with the exact solver (milliseconds for ≤8×8; the empty 7×7 garden's 1.5 million rakings take 2 seconds).
#. Prove par with the constraint solver, and also the best turn count achievable with no U-turn in open sand; reject free gardens where the two differ (their par raking is a meander, and the seal would reward an ugly answer).
#. Strict gardens: keep wide-rake layouts that are unique by themselves, or add grooves or pebbles from the graceful par raking until it is unique.
#. Grade with the deductive solver (below), and reject or reorder.
#. Curate the survivors for layout esthetics (a learned model could help here; see above), and write each one's observation and first hint.

[Yields.]{.margin-note} Strictness is cheap to generate.
Of 1,151 random 7×7--7×8 layouts, 69 (6.0%) passed all filters for groove gardens, needing a median 7 grooves.
Of 95,673 random 7×7--8×8 layouts with 1--3 stones, 2,444 (2.6%) were unique under the wide rake with no grooves at all; 58% of those had a turn density (turns per cell) of at most 0.40, which in practice separates calm answers from meandering ones.
At these rates, a year of daily gardens takes on the order of a CPU-hour; the prototype's year took 8 minutes of generation on two cores (`src/daily.py`), plus the campaign's unused candidates.

[Garden-like stones cost nothing.]{.margin-note} Realistic stone groups do not make good gardens rarer.
On 1,000 random 7×7 layouts from each sampler (garden-like layouts averaged 4.7 stone cells, single-cell ones 2.0), fewer garden-like layouts were solvable at all (28% against 39%), but the filters discard those anyway: usable free gardens came slightly more often (4.7% of layouts against 3.8%), gardens unique under the wide rake about as often (2.2% against 2.7%), and unique side-gate gardens slightly more often (1.8% against 1.3%).
Groove gardens needed the same median of 6 grooves; on small samples (25 each), the garden-like ones needed lookahead less often (28% against 84%), plausibly because larger stones make more corridors, which are local deductions.

[Grading.]{.margin-note} Difficulty is estimated by a deductive solver that works like a human, on the state of each boundary between cells (raked through, or not):

- *Tier 1*, local rules: a cell with two raked boundaries closes its others; a cell with exactly two open boundaries uses both; a cell with one is where the rake is set down; no closed loops; grooves, pebbles, bands, rings, and the wide rake's ban on open U-turns.
- *Tier 2*, global rules: checkerboard parity of the starting cell, connectivity of the unraked sand, and no premature joining of the stroke's two ends.
- *Tier 3*, lookahead: assume a boundary is raked through, propagate tiers 1--2, and reject it on contradiction.
- *Tier 4*, nested lookahead.

Generated groove gardens were 38% tier 1, 4% tier 2, and 58% tier 3 (median 3 lookahead steps); wide-rake gardens were 74% tier 3 (median 16 steps) and 26% tier 4.
Within a world, gardens follow the template ([Progression](#progression)), and its strict applications are ordered by tier, then lookahead count, then size.
These tiers are a proxy; only playtesting can say whether, for instance, the wide rake's lookahead is the kind players enjoy or the kind they resent.

# Interface

[Show, then do.]{.margin-note} A garden that brings in something new is first raked once by the monk himself: a ghostly fingertip leads him in from the gate along the intended raking, he rakes his way back out, the result rests for a moment ("That is a calm way to rake it: 3 turns, which is par. Now you."), and the garden is smoothed for the player.
This happens in the first garden (what raking *is*: in play, the first garden alone left it unclear what to do), in the first one scored by turns, and at the introduction of each later rule; it plays once by itself, **Watch** replays it, a tap skips it, and it costs nothing.
Watching the answer to an introduction spoils little, since an introduction is not meant to be hard; what it teaches is what the rule *looks like* when obeyed.

[Draw from the gate.]{.margin-note} Before the walk, the monk waits in the gateway, with a ring that pulses around him three times and then rests (it pulsed without end at first, which nags once the idea has been had); the player drags him in, and the path snaps cell to cell and leaves footprints.
Until a player's first walk, and always in the first garden, a ghostly fingertip then demonstrates the drag itself, three times; a press anywhere else says where to begin, and demonstrates again.
(The prototype first marked the gate's cell with a dashed circle, which says nothing about where to start, or that one drags.)
The walk follows the *direction* the hand is moving, not the cell the pointer happens to be in: arms are not mounted on tracks, and a hand sweeping along a row drifts off it, so a drag from one end of a row to the other that strays into the next row still draws the row (the first prototype snapped to whichever cell the pointer entered, and a long run would hop rows partway).
The axis of motion holds until the motion itself turns to within 30° of the other axis, or until the pointer is two cells off the run; a turn first finishes the run as far as the pointer went, so a rounded corner comes out as a corner; the direction is read over the last 0.4 cell of motion, so a wobble in place reads as nothing; and the pointer is captured, so the grip survives leaving the field or the window, a pointer beyond the walls being read as the nearest cell.
(This is the submenu problem of hierarchical menus, solved the same way: the intended movement is what matters, and a cursor that leaves the menu item on a slight diagonal toward the submenu has not left it.)
Dragging back over the last footprint retracts it; tapping any footprint walks the monk back to it; tapping the gate starts over.
Since either can throw away a long route by accident, the last rewind or restart can always be undone (**Restore**).
A rewind says what it leaves ("Stepped back 3 footprints: par is still possible from here"), replacing whatever verdict stood before, and the garden keeps the keyboard afterwards (after Astra's note that arrow keys scrolled the page after "Step back 3 for par" and that the old impossibility message lingered).
Moves into stones, walls, or footprints are ignored, and so is any move that breaks a rule on the spot (an open-sand U-turn under the wide rake, a turn at a pale pebble), with a soft scrape and a brief judder of the rake; a route that is doomed but not yet broken stays drawable, since noticing the trap is the puzzle.
A turn counter and the par sit at the top of the screen; nothing else does.
Whatever the game says to the player, as opposed to shows ("Drag the monk in from the gate", "Morning is open: 20 new gardens", a locked world's note, the help, the scroll's counts) is set in square brackets, as gwern.net sets the editor's words to the reader; rules, poems, counts and controls are not.
Every control, count and mark carries a tooltip with the word more that a hover might want (what Par is and who proved it, what Ask the abbot costs, what a world's extras are, what the seal says), as gwern.net puts one on everything that would bear it; the garden itself has none, since a tooltip over the sand would follow the hand through every walk.

[Letting go rakes.]{.margin-note} When the walk covers every cell and the finger lifts, the monk rakes; from the keyboard, the last step is followed by a beat in which it can still be taken back.
Raking is not final: tapping any cell of the raked garden smooths it back to the footprints, as far as that cell, and the walk goes on from there (and **Restore** rakes it again as it was), so a player who sees 19 turns against a par of 17 backs up with one tap, having seen the pattern.
An earlier design asked for an explicit **Rake** press after every garden (Astra's); in play it was a confirmation for an action that costs nothing, and it is kept only in the daily garden, where the first raking is the record.

[Sharing.]{#sharing .margin-note} A raking is worth passing on, and the page has no server, so the raking itself travels in the link: **Share this raking** gives the share sheet (or the clipboard) the raking drawn in box-drawing characters---light lines for the stroke, heavy for the walls with a tee where the gate breaks them, blocks for the stones---under a line with the garden, the turns, the par and the seal, and a link whose hash holds the raking as a code.
The code is small because the rules are strict about what a walk can do next: at each step there are at most three legal moves (and often one), so the walk is written as a number in the *mixed radix* of its legal moves, step by step in the order north, east, south, west, and a whole raking of a 7×7 garden takes some 20--50 bits (a mean of 22 bits over the campaign's intended rakings, against 86 at two bits a step; with the garden's identity and whether the abbot helped, 8--17 characters of base64url).
The garden is named in the code by 32 bits of a hash of its layout and rules, never by its place in the campaign, so a link survives a reordering, and a garden changed since is recognized as another rather than mistaken for the one meant (Astra's point: an 8-bit check, the first version, would have collided once in 256, and a seal must never land on a different puzzle).
No code decodes to a walk that breaks a rule, since the decoder re-derives the legal moves as it goes, and a code whose garden is gone or changed, or whose walk does not fit, is refused with a word.
Opening such a link shows the garden raked, with its seal, and says whose it is: it is credited to nobody, the viewer's own progress is untouched, and any tap smooths it away and leaves the garden theirs to rake from the gate (a raking so received cannot be walked back into, or restored).
The same idea carries a player's whole progress between devices without an account: **Carry my progress** makes a link whose hash lists every raked garden, by the same layout hash, with its state (raked and by how many turns over par, sealed in grey, sealed), the daily records by date with today's raking, and the unfinished walk of the garden last open, with the abbot's mark on it: about 7 characters a garden and 4 a day, so some 140 characters for 20 gardens and 2,500 for everything.
Opened on another device, it is offered rather than taken ("This link carries a player's progress: 23 of 148 gardens raked, 19 sealed, 4 daily gardens... Take it over here?"), and taking it joins the two: whichever side has the better state of a garden keeps it, the first raking of a day stays its record, and an entry whose garden is not in this edition, or has changed, is left out and said so, never assigned to the garden now at its place.
A link ending in `#daily` opens today's garden.
(The alternative, a shared raking as its index in a canonical enumeration of the garden's rakings, would be smaller still for free gardens, but a million-raking garden cannot be enumerated in the browser; the mixed radix is the same ordering, counted from one end, without the enumeration.)

[Phone geometry.]{.margin-note} Portrait phones are ≈390 points wide, so a 9-cell garden gives ≈42-point cells, about the 44-point minimum touch target; that sets 9×9 as the largest garden that needs no zoom, and the design never exceeds it.
The same interface works with a mouse.

[The reveal.]{.margin-note} When the walk is complete, the monk turns, sets down the rake, and walks backwards to the gate, pulling it; then the camera tilts from top-down to the abbot's view from the veranda, low sun on the ridges, and the seal is stamped or not.
The rake must glide: it follows the stroke's centerline continuously, through every quarter-circle, at a walking pace that eases in where it is set down and out at the gate, carving lit grooves and erasing the footprints as it passes, with a few grains flicked aside by the tines.
(The prototype's first reveal drew finished cells one at a time as line art; it read as chunky and unsatisfying, and the reveal is the game's payoff, so it is worth the most polish.)
It lasts a few seconds, scaled to the garden's size, and a tap skips it.

[The abbot judges your route, not his.]{.margin-note} Asking the abbot never reveals *his* answer; it decides, by search, what the player's own route can still become, and says only what it can prove.
Gardens this small are decided quickly (the slowest of the prototype's 4,440 test routes took 0.7 seconds in a desktop browser).
There are three verdicts.
The route can still be raked at par: he sketches its next groove, taken from a par raking that continues the player's route.
It can still be finished, but not at par: he marks the last footprint from which par was possible and says so with the action first ("step back 3 footprints to the marked footprint for par"), and offers the step.
It cannot be finished: he says why when there is a short reason (sand cut off, two dead ends, the wrong count of dark and light cells), marks the cells involved, and offers to step back to the nearest footprint from which the garden can still be finished, proven to be the nearest (if, in a large garden, that proof runs out of budget, he offers a footprint from which it can be finished, without calling it the nearest).
In a fresh garden the first request gives instead the garden's observation ("the top-right corner has only one way in").
If a search ever runs out of budget, he says he cannot tell yet, rather than guess.

[Help costs the vermilion.]{.margin-note} The observation is free, and so is the reason a walk has *stopped* (no step on is allowed, or every cell is covered but the walk can't end there), since a player can see that much already; an answer the abbot cannot give in time is free too.
Anything more---a verdict on a live route, a sketched groove, where to step back to---marks the garden as helped, and a helped raking at par earns the abbot's seal in grey ink instead of vermilion.
Astra argues for making every diagnosis free and charging only for a revealed move; but a verdict on a live route ("this can still reach par") is an oracle, which a player can consult after every step to search the garden mechanically, and the seal would then certify nothing.
The mark lasts until the garden is next raked, so starting over does not wash it out, and it travels with the route it helped: walking back into a helped raking and restoring it rakes it grey again (an early version let Step back and Restore turn a grey seal vermilion; Astra found it).
Raking the garden again unaided can still earn the vermilion seal, and the garden list keeps the better of the two (seals are drawn square, rakings off par as round dots).
The redemption is deliberate: a player can memorize a helped answer and rake it again, but making one peek cost a garden's seal forever would punish curiosity more than it rewards honesty.
Assists (below) do not affect the seal.
This replaces an earlier design, in which a hint revealed one groove of *the* intended raking; in free gardens that hint contradicted valid routes the player was already on.
(Hints that judge the player's own route, with proven rewinds, are Astra's idea.)

[Assists.]{.margin-note} Off by default after world 1, on toggles: a faint preview of the grooves under the footprints; the checkerboard; and hatching on sand the route has cut off from the gate.
There are no timers, move limits, or failure states; an impossible position is simply one the player has not yet noticed.

[Below the garden.]{.margin-note} Under the garden sit the turn count against par and three buttons (Step back, Start over, Ask the abbot; Restore, Watch, Rake and Next garden appear when they apply), then the status line, the assists (left in view, since a new player cannot know what an assist is until shown), a garden's rule where it brings one in, the list of gardens, and the help, folded into one line.
The list shows the worlds that are open, each as its main path, with the extras folded behind "+10 more" (unfolded by themselves while one of them is the garden open); a world not yet open is one muted line with its count and what opens it, so a new player sees a short list that lengthens as the day goes on.
Sound, volume and haptics live under a chevron beside the speaker in the header rather than in the list; the counts of a garden's rakings live in the tooltip of its par.
(The first version showed every world's pips, greyed, the three sound controls in the list, a "Sand" counter, the rule for every garden, and 290 words of help; on a phone the page ran to 2,400 pixels, and now to 1,400, with everything that was cut a hover or a tap away.)

[Accessibility.]{.margin-note} Every rule-bearing mark differs in shape, not only in color: hatching for bands, dashed rings for ripple stones, pale against dark pebbles, a dashed outline for cells the abbot points at, hatching for cut-off sand.
The game is fully playable from a keyboard (arrows walk, Backspace steps back, Enter skips the reveal and goes on) and respects reduced-motion settings, as the prototype already does.
A text description of each cell (terrain, mark, footprint, monk) with a coordinate-grid view would support non-visual play; that needs testing with screen-reader users rather than assuming.

[Prototype.]{.margin-note} A one-page HTML prototype, included with the code, implements this interface without the camera tilt: worlds that open in turn (with a switch that opens them all, for testing), main paths and extras, the monk's demonstrations and the first-drag hint, the monk waiting at the gate, drawing, rewinds and Restore, the turn counter against par, raking on letting go and stepping back after it, the refusals, the abbot's three verdicts, the three assists, the gliding reveal, the vermilion and grey seals, the poem after each raking and the fade to the next garden, the scroll of poems earned, the daily garden, the practice garden, the sound (the synthesized pipe, drips, wind, birds, crickets, footsteps, scrape, bell and _tok_; the shakuhachi recording; haptics), and the sharing (rakings and progress as codes in the URL's hash) (all in [Presentation](#presentation) and [Progression](#progression)).
It has the whole campaign of 148 gardens (see [Progression](#progression)), each with its poem; every par and raking count shown in it comes from the solvers, and its hint search agreed with the C solver on every one of 4,440 test routes (none ran out of budget).
Where the game would teach wordlessly, the prototype states each new rule in a line of text.

# Presentation

[Sand as a heightfield.]{.margin-note} The original sketch suggested a falling-sand particle simulation.
Raking does not make sand flow; it displaces it, so the right representation is a *heightfield* computed per pixel in a fragment shader: each cell's groove type (straight, quarter-circle around the corner, or the rake's start) gives the distance to each tine's centerline, a smooth groove profile turns distance into height, and the height's gradient is lit by a low sun whose angle follows the world's time of day.
Footprints are depressions stamped by decal; the reveal is a mask advancing along the stroke.
This is cheap on any phone, resolution-independent, and gives the one thing that matters visually: ridges catching light.
The prototype does a simpler version on the CPU, with about one height sample per screen pixel (32 to 144 per cell, so the sand is never visibly upscaled): footprints are pressed into the field as the player walks in, the abbot's grooves are raked into it in advance, and the reveal writes the rake's cross-section along the stroke frame by frame, the same engine as the practice garden (below).
Gravel grain is procedural and never the same twice: two octaves of value noise in the height (so the ridges catch light unevenly), a per-pixel fleck and a slow mottle in the color, a few specks on top, and a fresh seed every time a garden is laid out.
The rake is set down flush with whatever is behind its first cell (wall, stone, or the lane raked before), so no patch of sand is left between the start of the stroke and its neighbor.
The grid is an abstraction, and the rendering admits it: a turn is drawn as an ideal quarter-circle stamp, which no rigid rake sweeps exactly; the picture is faithful to the stroke, not to rake physics.

[Practice garden.]{.margin-note} A garden with no rules is always open: drag anywhere to rake, curves and all; place or lift stones (the sand under a lifted stone stays unraked); smooth it flat and start again.
Here the heightfield is written directly: the rake head, held perpendicular to its motion, sets the sand's cross-section (a ridge between every pair of tines) along its path, and the same low sun lights it.
It costs almost nothing, and it is the honest home for the sketch's falling-sand toy: a sandbox for the hands, kept apart from the puzzles so it never weakens their rules (Astra's suggestion).
The prototype has one.

[Palette.]{.margin-note} White-grey granite gravel, _sumi-e_ ink for stones and walls, moss green around the larger stones, and one accent: the vermilion of the novice's _ensō_ and the abbot's seal (the two marks of a brush in the garden; the first prototype had a vermilion triangle in the gateway as well, redundant with the gap in the wall and with the monk waiting in it, so it is gone, and only a side gate's exit keeps an outline triangle).
No sepia or yellow cast; the gravel at Kyoto temples is white.
The stones are ink-wash paintings, not 3D models; [_Ōkami_](!W) shows how far _sumi-e_ can go in motion.
In the prototype each stone is drawn as one: a contour that is lumpy and a little rough, flatter at its base where it sits in the sand; a soft contact shadow cast down and to the right, where the sand's light is from, as the blurred shadow of one path (so the two parts of a bent stone never darken each other), with a tighter, darker one at the base; an ink wash lighter toward the light; and a brushed edge, one band of varying weight, heavy where the ink pools on the shadow side and thinning to nothing where the brush ran dry on the lit side (in dark mode the edge is the pale rim light, so it is heavy on the lit side instead).
This replaced the first prototype's smooth capsules with a highlight, after GPT-6 Astra's second review of it; a first attempt with per-segment strokes and stepped washes looked jagged and banded, which Gwern caught, so everything is now one smooth band or gradient.
The novice is an [_ensō_](!W): one brushed circle, coming down rounded and heavy, thinning as it goes round, lifting before it closes, with two dry strands trailing past the lift, in vermilion, rubricated like an initial (the first prototype drew a filled black disc); he is drawn at the head of the walk, and, walking backwards, behind the rake; while he waits in the gateway he is its marker.
Everything around the garden follows [gwern.net's design](https://gwern.net/design): black text on white, greys for whatever is secondary, serif text with sans-serif controls, square corners, hairline rules, no shadows, and dark mode as gwern.net has it (#161616 under #f1f1f1); the vermilion stays the one colour, on the novice's ensō and the seal, and the abbot's marks, the hint highlights and the faint route through the footprints, which the first prototype tinted blue and red, are ink.

[Sound.]{.margin-note} The garden has its own sounds and almost no music.
A bamboo pipe drips into a stone basin now and then (every 7--23 s), and every minute or two the pipe tips onto its stone with a hollow knock and spills a few drops; a _bonshō_ bell sounds once at the close of each world, its low partials beating and lasting; the seal lands with a dull wooden _tok_.
Under them a wind moves, a bed of low filtered noise that wanders in level and colour every few seconds and swells into a gust every half-minute or so; by the local clock, small birds sing by day (phrases of two to six gliding notes, every 12--45 s, from 6 to 19) and crickets trill after dark (a high sine pulsed at about 28 Hz, from 20 to 5), so a page left open follows the evening.
Each footprint is a step on gravel (a burst of grains over a soft thump), and the raking is heard: the rake's scrape is grains of gravel under the tines, a looping crackle through a bandpass whose loudness, brightness and density follow the stroke's speed frame by frame, with the rasp of tines dragged sideways in every bend and the monk's backward step at every cell, lifting as the rake goes out through the gate.
All of these are synthesized in the page with Web Audio (decaying sines, filtered noise and synthesized crackle; a short feedback delay for the basin's echo), so they cost nothing to carry and never repeat exactly.
After a world's bell, and now and then otherwise (never within a quarter-hour of the last), a shakuhachi recording follows, faded in over 3 s and out over 4: the recordings are real and public domain, carried in the page as data: URLs with their credits and sources shown on the scroll of poems.
The first is _Shika no Tōne_ (鹿の遠音, "The Distant Cry of Deer"), the Kinko-school _honkyoku_ for two shakuhachi answering each other, played by Araki Kodō III (1879--1935) on Victor 13029 between 1925 and 1935, [from Wikimedia Commons](https://commons.wikimedia.org/wiki/File:Shikanotoone_new.ogg) (a Japanese phonogram whose neighbouring rights have expired; the de-noised derivative released as public domain by its maker), re-encoded as mono MP3 at 32 kbps (1.5 MB).
Nothing sounds until the player has touched the page, as browsers require, and the first time it does a word beside the speaker says so.
Silence is one tap away: a speaker button in the header (or M) mutes everything, **Sound off** in the list of gardens is the same switch, a slider beside it sets the volume, and all of it is remembered.
The page also defers to what the browser and the device say.
There is no CSS media query for a sound preference (Media Queries Level 5 has `prefers-reduced-motion`, `-transparency`, `-contrast`, `-data` and `forced-colors`, nothing for audio), so the signals that exist are used instead: where the browser has the [Audio Session API](https://www.w3.org/TR/audio-session/) (Safari 17), the page's session is _ambient_, so a phone's silent switch mutes it and it mixes with whatever the player was already listening to rather than stopping it; the system's media keys (pause, stop) mute it while a recording plays, through the Media Session API, which also names the piece on the lock screen; and when the browser keeps the page's audio from running after a gesture (a site set to block sound, or Firefox's autoplay policy answering _disallowed_), the speaker shows it in red instead of pretending to play.
Haptics tick once per cell (8 ms, where the device can; a switch beside the sound's, shown only on such devices).

[A poem for each garden.]{.margin-note} The reward for raking a garden, at par or not, helped or not, is a poem: a moment after the reveal, a classical waka or haiku is laid over the raked sand in English on a white card with a hairline border (the first prototype blurred the grooves through a frosted plate), with the poet's name set in by an indent (no dash, and no kanji: the first version put 原文 beside the name as the button for the original, and the poet's name in kanji over the source, which Gwern found clutter), and a tap on "original" adds the Japanese and its source.
About ten seconds later the garden and its poem fade together and the next garden fades in, so the campaign can be played as one slow, unbroken sequence; a tap on the poem goes on at once, a tap beside it (on the sand around the card, or the walls) puts the poem away and stays with the garden (a player's note: interacting with the poem should go forth, dismissing it should go back; the first prototype had any tap stay), **Next garden** or Enter goes on at once, and nothing moves on while the page is hidden or scrolled away.
The poem rewards finishing, not par, so it never punishes asking for help; the seal still records how the garden was raked.
Every poem earned is kept on a scroll (**Poems**, beside the practice garden), in the order of the gardens with their Japanese and sources, with a count of what remains in each world, so the 76 extras have something to give besides seals; a garden's name on the scroll leads back to it.

The poems are real, and all written before 1900 (all but one by poets who died before it): 149 (one for each garden, and one for the practice garden) by 25 poets, from Bashō (34), Teika (23), Buson (20) and Rengetsu (13) down to single poems by Tsurayuki, Izumi Shikibu and Shiki.
Each is paired by image or by structure.
The first garden's poem is Bashō sweeping a temple garden before he goes on ("Let me sweep the garden / before I go— / willow leaves fall in the temple.").
The garden before the first ripple stone gets Ryōkan's new pond, where the frog jumps in and makes no sound; the stone arrives with Bashō's old pond.
The pair of gardens that introduces wave bands gets the two drafts of his Kiyotaki poem: waves "without a speck of dust" under the summer moon, and then, as he rewrote it on his deathbed, waves with green pine needles scattering into them.
The last garden of the main path gets the one classical poem about temple sand hauled and tended by monks, Bashō's at Kehi shrine, where the Yugyō abbots carried the sand in themselves: "The moon is clear / on the sand / the pilgrim monk carried."
The English versions are working translations, not quotations of published ones (see [Credits](#methods)).

[No koans.]{.margin-note} There is no narrative text, and no invented wisdom: apart from the rules, the only words are the poems, which are real.
The novice's progress is shown by his rake (bamboo, then the wide rake) and his robes (in the prototype he is an _ensō_, see [Palette](#presentation)); the abbot appears only as a seated figure on the veranda and as his seal.
Fake Zen aphorisms (or fake haiku) would cheapen exactly the thing the game is trying to be.

# Prior Art

- [**_Zen Puzzle Garden_**](https://www.lexaloffle.com/zen.php) (Lexaloffle 2003): raking as a puzzle, with the ice-sliding rule ("rake all of the sand without turning corners or breaking raked lines"); 64 gardens across 4 seasons, an editor, and later pushable lanterns and leaves to collect in order. Its solvability is NP-complete ([Houston et al 2012](https://arxiv.org/abs/1106.2104 "'Zen Puzzle Garden is NP-complete', Houston et al 2012")).
- [**_Zen Rock Garden_**](https://apps.apple.com/us/app/zen-rock-garden/id1553652093 "'Zen Rock Garden', ZenRock 2021") (ZenRock 2021): "Just trace over all the sand panels with a single stroke!", 300+ stages, with famous temple gardens as late motifs. This is the bare one-stroke rule, so _Samon_ does not claim it; what is new here is the gate-anchored walk-in/rake-out reveal, proven-par scoring, the _samon_ constraints, and the wide rake.
- **One-stroke and fill-the-board path puzzles**: _hitofude-gaki_ (一筆書き) apps; [_Flow Free_](!W) (2012: "Pair all colors, and cover the entire board"); [_Zip_](https://www.gmpuzzles.com/blog/2025/03/introducing-zip) (2025; Hamiltonian path through numbered waypoints, with walls); Nikoli genres like Numbrix/[Hidato](!W), whose given path fragments resemble the master's grooves, and [Masyu](!W), whose pearls resemble the pebbles.
- **Line puzzles with deep progression**: [_The Witness_](!W "The Witness (2016 video game)") (constraints on a drawn line, taught wordlessly); [_LYNE_](https://www.appspy.com/review/8095/lyne "'LYNE Review', AppSpy 2014") (2014); [_Cosmic Express_](https://apps.apple.com/us/app/cosmic-express/id1152474226 "'Cosmic Express', Draknek 2017") (2017; routing a single non-crossing track).
- **Optimization metrics**: [_Opus Magnum_](!W) and other Zachtronics games' histograms.
- **Exits and pulling**: [_Stephen's Sausage Roll_](!W) (the puzzle includes getting out), and [Sokoban](!W), of which raking is a kind of inverse: one pulls a rake rather than pushing blocks.

The general problem is NP-complete, as Hamiltonian path is NP-complete even on grid graphs ([Itai et al 1982](https://doi.org/10.1137/0211056 "'Hamilton Paths in Grid Graphs', Itai et al 1982")), so no simple trick solves every garden; but garden-sized instances are trivial for a computer, which is what makes proven pars, proof-backed hints, and generated levels possible.

# Production Plan

[Rules before art.]{.margin-note} The order of work (Astra's plan, adapted):
(1) prove the rules in a plain prototype---drawing, undo, the rules, the hint search, a few gardens---before any art (done: the prototype attached to this page);
(2) build a vertical slice of 12 gardens on real phones: the first five gardens, a ripple garden, a band garden, a groove garden, a pebble garden, a wide-rake garden, a side-gate garden, and a daily garden; one polished reveal; the histogram screen;
(3) curate the campaign from generated candidates, several times more than needed, ordered by grade and revised by the mistakes players actually make (a first, uncurated pass is in the prototype);
(4) only then the sensory layer: the shader, sound, haptics, low-power rendering, and accessibility passes.

[Pass or fail.]{.margin-note} The slice passes if players can say why a route failed after asking the abbot, carry an observation from one garden to the next, and choose to rake a finished garden again for the seal.
If they brute-force, shrink the gardens; if they only trace forced moves, remove clues; if the reveal feels like a wait, shorten it; if they spend their time in the practice garden instead of the gardens, the puzzles are the problem, not the sand.

# Risks and Open Questions

- **The free worlds may feel trivial.** Finishing an open garden is easy, and par is optional. Mitigation: keep world 1 short (9 gardens on its main path), give the seal real status, and introduce grooves (world 4) by the second hour (after 21 gardens). One early outside player raked at least 26 gardens on a phone, in the longer first campaign, and meant to play on.
- **The wide rake may be too hard.** The grader says every wide-rake garden needs lookahead. Mitigation: start it at 5×5, use the hint, and order by grade; if playtesting is bad, move it to an optional post-game.
- **Genre saturation.** One-stroke grids are a crowded genre. The differentiators are esthetic: the reveal, the score that *is* the beauty, the diegetic constraint vocabulary.
- **Scoring details.** Decided: leaving through the gate counts as a turn, and U-turns braced against walls are not free (as they are in real raking). Both stay simple, and the turn count already rewards long lines; either would be a one-line change, but would change every par.
- **Parity frustration.** Players who never discover parity will hit unwinnable endgames repeatedly. The two-tone tutorial gravel and the checkerboard assist are the mitigation; whether they suffice is an empirical question.
- **Esthetic curation.** Turn density and "no open U-turns" are crude proxies for beauty. A learned layout scorer, validated against human ratings of rendered gardens, is the obvious next experiment.
- **Hints as a crutch.** Proof-backed hints are strong: pressed repeatedly in a par-reachable route, the abbot sketches the whole answer. Decided after a first play session: helped rakings earn a grey seal (see [Interface](#interface)); Astra's design kept hints free and unpenalized, and its review argued for free diagnosis; explanations of a stopped walk are now free, verdicts on a live route are not. Open: whether players will memorize helped answers to re-rake them for vermilion, and whether that matters.
- **Demonstrations may teach copying.** Showing the calm raking of each introduction makes the rule's look concrete, but the player then retraces it. If playtests show players learning the shape rather than the rule, demonstrate each rule on a separate, smaller garden, and leave the pair's changed garden to the player.

# Methods {.collapse}

<div class="abstract-collapse">
How the numbers on this page were computed, and how they were checked. Everything is reproducible from the attached code.
</div>

[Exact solver.]{.margin-note} A depth-first search in C draws strokes from the gate inward (the reverse of raking; turns and U-turns are symmetric under reversal).
Sand cells are bits in a 128-bit mask (gardens ≤128 cells).
Each node prunes on: connectivity of the remaining sand through the head (a bitwise flood fill); degree (every unvisited cell needs two usable neighbors, except at most one, which must be the rake's start); checkerboard parity of the remaining cells; required and forbidden directions per cell (encoding bands and grooves); pebbles (a pale pebble's cell needs both neighbors on one axis usable, a dark pebble's one on each axis); ring contiguity; and the U-turn rules.
It enumerates the 1,510,446 rakings of the empty 7×7 garden in 1.9 seconds (33 million nodes).

[Validation.]{.margin-note} Its counts for empty gardens match [OEIS A145157](https://oeis.org/A145157) (_n_ = 3--7) and [A145156](https://oeis.org/A145156) (5×4, 5×6).
Against an independent brute-force enumerator (no pruning; rules checked only on complete paths) on ≈340,000 random small gardens (≈67,000 of them solvable) exercising every rule, it agreed on every count and every turn histogram; and on medium gardens it agreed with a separately-written [CP-SAT](https://developers.google.com/optimization/cp/cp_solver) model on counts, histograms, and minimum turns.
Pebbles were checked the same way, on 6,000 further random small gardens against brute force and 108 medium ones against CP-SAT, with no disagreements; the grader's pebble rules never derived a contradiction in a solvable garden, and every garden it solved, it solved to the right answer (711 random gardens, 79 of them with a unique raking).
Astra's review found two rules that the solvers and the prototype shared, and so could not catch by agreeing with each other: the bend just inside a side gate was drawn but not counted (in 13 of the 18 side-gate gardens then in the campaign), and the abbot's finished sand counted as a pivot for the wide rake.
Both were fixed in all three solvers and in the prototype, and checked again on 3,192 random small gardens against brute force (411 solvable, 132 of those with a side gate and 149 with finished sand) and on 120 medium ones against CP-SAT, with no disagreements.
The recheck also found a bug in the CP-SAT model: a sand cell walled in by finished sand has no arcs, and a circuit constraint silently leaves such nodes out, so the model "solved" gardens that cannot be raked; every cell now needs exactly one incoming arc.

[Par.]{.margin-note} The CP-SAT model is a single circuit through all sand cells plus a dummy node (arcs from the dummy to any start cell, and from the gate cell back to the dummy), with straightness indicators per cell and direction, so turns = (cells − 1) − Σ straight; rings constrain the number of arcs entering the ring set to exactly 1; bands and grooves fix edges; a pebble fixes its cell's straightness indicator and excludes it as the start; U-turns are three-literal clauses.
It proves par for random 8×8 gardens in 0.2--1.8 seconds, and for the empty 9×9 garden in 4.4 seconds (2 CPU cores).

[Grader and generator.]{.margin-note} The grader is the tiered propagation solver described in [Generating Gardens](#generating-gardens), in Python.
Groove clues and pebbles are chosen greedily (each round, of ~24 random candidate cells from the intended raking, add the one leaving the fewest valid rakings), then pruned of redundant ones.
Yields count every sampled layout, including those rejected on parity.

[Hint search.]{.margin-note} The prototype's hints run a JavaScript depth-first search that uses the same rule code as the player's moves and the C solver's pruning (connectivity, dead ends, parity, pebbles), with a node budget.
"Can this route still reach par?" needs no search: each garden ships with every one of its par rakings, and a route can reach par exactly when it is a prefix of one of them.
Two things keep the search small.
It remembers every position it has proved hopeless, where a position is the footprints, the head, and the last two moves (all the rules ever look at), so a dead end reached again by another route costs nothing.
And because a completion of a route also completes every shorter part of it, whether a route can be finished switches from yes to no only once along it, so the nearest footprint to step back to is found by binary search, and is proven nearest by a single proof of impossibility at the footprint after it.
On 4,440 routes over all 148 gardens (random legal walks, and prefixes of real rakings), every verdict it gave matched the C solver's, and no search ran out of budget (an earlier campaign had one such route, in a 9×9 wide-rake garden, where the abbot said he could not tell yet); of the 2,011 rewinds offered from stuck routes, 1,989 were proven nearest and matched the C solver's nearest footprint, and the other 22 (in three of the largest gardens) were footprints from which the garden could be finished, offered without the claim.

[Credits.]{.margin-note} Several ideas on this page come from a parallel design for the same sketch by GPT-6 Astra, "The Last Footprint" (unpublished): hints that judge the player's own route, with proven rewinds; restorable rewinds and an explicit commit (now kept only for the daily garden); the practice garden; the "any turn" clue (the dark pebble; we added the pale one); the world template and per-garden records; the production plan and its pass/fail criteria; and the symmetry argument.
Its five example gardens all check out under our solver.
Astra later reviewed the prototype: it found four bugs (the grey seal washed out by Step back and Restore; the uncounted side-gate bends; finished sand as a pivot; an inconclusive hint that cost the seal), a dead end explained by the wrong rule (a side gate's cell blamed on the wide rake), and the overclaim that the abbot's taste *is* the wide rake's physics, and it proposed the short main path and the contrasting pairs.
Astra also chose the poems: a sourced waka or haiku, with a working translation, for each garden of the earlier 136-garden campaign.
We carried them over to the 148 gardens (garden names follow layouts, so most pairings survived the reordering), paired the 13 new gardens (with 3 poems new to the selection), replaced two poems (Shiki's) whose texts we could not confirm, checked every Japanese text and attribution against a page that prints it, and revised 22 of the translations.
Two further passes (ours, then Astra's review of it) replaced 25 pairings judged weak (a felicitation poem, filler, poems fastened to their gardens by a pun or by a shared noun) with better classical poems, among them Ryōkan's new pond before Bashō's old one, Bashō's master carpenter for the abbot's grooves, the Man'yōshū moon-boat for the star garden, Tamemasa's unseen carp under trembling lotus leaves, and Saigyō's snipe at the threshold; Astra also offered two original poems of its own, which we left out, so that every word the player reads is still a real poem.
The texts, sources, pairings, credits and the log of replacements are in `poems/samon-poems.json`.

# License

[CC0.]{.margin-note} Everything here that is ours---Gwern's contributions and all the LLM-written work: this document, the design, the solvers, experiments, tests and build scripts, the level generator and the 148 gardens and 366 daily gardens it produced, the prototype and its sounds (every one synthesized in the page), the figures, the English working translations of the poems, and the ideas merged from GPT-6 Astra's parallel design---is released into the public domain under [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/).
No attribution is required; none of it may be used to claim that Gwern or a contributor endorses a derived work.

The poems are classical Japanese texts, all first published before 1900, and in the public domain everywhere; the pages each text was checked against are linked beside it in the game and in `poems/samon-poems.json`.

The one recording, _Shika no Tōne_ played by Araki Kodō III (Victor 13029, recorded between 1925 and 1935), is [File:Shikanotoone_new.ogg](https://commons.wikimedia.org/wiki/File:Shikanotoone_new.ogg) on Wikimedia Commons, re-encoded as mono MP3.
Commons records it as public domain: a Japanese phonogram whose neighbouring rights have expired under Article 101 of the Japanese Copyright Act, with the performer dead for over seventy years, and the de-noised derivative (Torsodog, 2008) released into the public domain by its maker.
In the United States, sound recordings fixed before 1946 came under federal copyright by the Music Modernization Act for 100 years from publication, so a pressing from 1930 is not public domain there until 2031; Commons hosts the file as public domain in both Japan and the United States, and we rely on that.

The page's look follows gwern.net's own stylesheets (colours and font stacks), which are [CC0](https://gwern.net/about#license); no fonts are bundled, and no third-party code or images are included.
The design document's figures are our own renders; the puzzle counts quoted from the [OEIS](https://oeis.org/) (A145157, A145156) are facts, not copies.

# See Also

- [Research ideas](/idea)

Solvers, experiments, the level generator, and the document's checks ( src/ )

src/campaign.py · 700 lines · 40.8 kB — Generate the full campaign (7 worlds, 148 gardens) with the garden-like stone sampler. 
"""campaign.py: generate the full campaign (7 worlds, 148 gardens) with the garden-like stone sampler.

Each world mixes hand-made teaching gardens (the prototype's original 14, plus a few searched for) with generated
ones. Generated gardens pass the pipeline's filters (parity; solvable; free gardens: par reachable with no U-turn in
open sand, a calm par raking, a manageable number of par rakings; strict gardens: exactly one raking, solved by the
deductive grader) and are ordered by difficulty (free: by size, then how rare par is; strict: by grader tier,
lookahead steps, size). Names and intros are assigned here; first hints marked "auto" are derived and verified by
export_proto.py. Writes out/campaign.json and out/campaign_stats.json.
"""
from __future__ import annotations

import json
import math
import random
import sys
import time
from copy import deepcopy
from multiprocessing import Pool

from zg import Garden, turns_of
from zcp import optimize
from gen import parity_ok
from garden_sampler import garden_layout, add_side_gate, ichimatsu
from clues import make_unique, with_clues, make_unique_marks, with_marks
from grader import grade
from levels import from_dict, to_dict

OUT = "/home/claude/zen/out/campaign.json"
STATS = "/home/claude/zen/out/campaign_stats.json"
L = json.load(open("/home/claude/zen/out/levels.json"))
NAMES_REG = "/home/claude/zen/out/garden_names.json"


def key_of(g):
    return (tuple(g.rows), tuple(g.gate), tuple(g.end) if g.end else None)


SWAP = str.maketrans("hv=|", "vh|=")   # band and block orientations swap when the axes do


def canon(g):
    """The layout (terrain, gate cell, side-gate cell) up to rotation and reflection: two gardens with the same canon
    look the same to a player, even if their gates face different ways or their clues differ."""
    H, W = g.H, g.W
    pts = [(g.gate[0], g.gate[1])] + ([tuple(g.end[:2])] if g.end else [])
    forms = []
    for tr in (False, True):
        for fr in (False, True):
            for fc in (False, True):
                def f(r, c):
                    r, c = (H - 1 - r if fr else r), (W - 1 - c if fc else c)
                    return (c, r) if tr else (r, c)
                h2, w2 = (W, H) if tr else (H, W)
                grid = [[None] * w2 for _ in range(h2)]
                for r in range(H):
                    for c in range(W):
                        a, b = f(r, c)
                        grid[a][b] = g.rows[r][c].translate(SWAP) if tr else g.rows[r][c]
                forms.append((tuple("".join(x) for x in grid), tuple(f(*q) for q in pts)))
    return min(forms)


# ---------------------------------------------------------------- generators (each returns a candidate dict or None)

def cand_free(g, max_at_par=40, min_rakings=20):
    if not parity_ok(g):
        return None
    r = g.solve(mode=0, cap=3 * 10**6, nodelimit=2 * 10**9)
    if r["nsol"] < min_rakings or not r["complete"] or r["capped"]:
        return None
    free = optimize(g, time_limit=20)
    grace = optimize(g, nohairpin=2, time_limit=20)
    if free[1] != "OPTIMAL" or grace[1] != "OPTIMAL" or grace[0] != free[0]:
        return None
    at_par = r["hist"].get(free[0], 0)
    cells = len(g.sand_cells())
    if not (1 <= at_par <= max_at_par) or free[0] / cells > 0.45:
        return None
    return {"kind": "free", "garden": g, "cells": cells, "par": free[0], "rakings": r["nsol"], "at_par": at_par,
            "rarity": math.log(r["nsol"] / at_par), "target": list(reversed(grace[2]))}


def cand_clued(g, rng, marks=False, max_clues=None):
    if not parity_ok(g) or g.solve(mode=0, cap=2)["nsol"] < 2:
        return None
    free = optimize(g, time_limit=30)
    grace = optimize(g, nohairpin=2, time_limit=30)
    if free[1] != "OPTIMAL" or grace[1] != "OPTIMAL" or grace[0] != free[0]:
        return None
    c = {"cells": len(g.sand_cells()), "target": list(reversed(grace[2]))}
    if free[0] / c["cells"] > 0.45:
        return None
    cl = (make_unique_marks if marks else make_unique)(g, c["target"], rng)
    if cl is None or len(cl) > (max_clues or (13 if marks else 9)):
        return None
    gu = (with_marks if marks else with_clues)(g, cl)
    gr = grade(gu)
    if not gr["solved"]:
        return None
    return {"kind": "strict", "garden": gu, "cells": c["cells"], "tier": gr["tier"], "trials": gr["trials"] or 0,
            "clues": len(cl), "target": c["target"]}


def cand_wide(g, max_density=0.45):
    if not parity_ok(g) and g.end is None:
        return None
    g.nohairpin = 2
    g.braced_start = True
    r = g.solve(mode=0, cap=2, stream=True)
    if r["nsol"] != 1:
        return None
    d = r["sols"][0]["draw"]
    cells = len(g.sand_cells())
    if turns_of(g, d) / cells > max_density:
        return None
    gr = grade(g)
    if not gr["solved"]:
        return None
    return {"kind": "strict", "garden": g, "cells": cells, "tier": gr["tier"], "trials": gr["trials"] or 0, "target": d}


def cand_side_marks(g, rng):
    """Side gate without the wide rake: pebbles from a graceful par raking until unique."""
    r = g.solve(mode=0, cap=10**6)
    if r["nsol"] == 0 or not r["complete"]:
        return None
    free = optimize(g, time_limit=20)
    grace = optimize(g, nohairpin=2, time_limit=20)
    if free[1] != "OPTIMAL" or grace[1] != "OPTIMAL" or grace[0] != free[0]:
        return None
    target = list(reversed(grace[2]))
    cl = make_unique_marks(g, target, rng)
    if cl is None or len(cl) > 13:
        return None
    gu = with_marks(g, cl)
    gr = grade(gu)
    if not gr["solved"]:
        return None
    return {"kind": "strict", "garden": gu, "cells": len(g.sand_cells()), "tier": gr["tier"], "trials": gr["trials"] or 0,
            "clues": len(cl), "target": target}


def make(spec, rng):
    kind, p = spec["gen"], spec
    H, W = p["H"], p["W"]
    feats = dict(ripple=p.get("ripple", False), band=p.get("band"), closed_ring=p.get("closed_ring", False))
    if kind == "free":
        g = garden_layout(H, W, rng, **feats)
        if g is not None and p.get("ripple2"):
            from garden_sampler import add_ripple
            rows = [list(r) for r in g.rows]
            if not add_ripple(rows, rng, H, W):
                return None
            g = Garden(["".join(r) for r in rows], g.gate)
        return cand_free(g) if g is not None else None
    if kind in ("grooves", "pebbles"):
        g = garden_layout(H, W, rng, **feats)
        return cand_clued(g, rng, marks=(kind == "pebbles")) if g is not None else None
    if kind == "wide":
        g = garden_layout(H, W, rng, **feats)
        return cand_wide(g) if g is not None else None
    if kind == "side_wide":
        g = garden_layout(H, W, rng, **feats)
        if g is None or not add_side_gate(g, rng):
            return None
        return cand_wide(g)
    if kind == "side_pebbles":
        g = garden_layout(H, W, rng, **feats)
        if g is None or not add_side_gate(g, rng):
            return None
        return cand_side_marks(g, rng)
    raise ValueError(kind)


CACHE = "/home/claude/zen/out/campaign_cache"


def run_spec(args):
    spec, seed = args
    import os, pickle
    os.makedirs(CACHE, exist_ok=True)
    path = f"{CACHE}/{spec['id']}.pkl"
    if os.path.exists(path):
        return pickle.load(open(path, "rb"))
    res = _run_spec(spec, seed)
    pickle.dump(res, open(path, "wb"))
    return res


def _run_spec(spec, seed):
    rng = random.Random(seed)
    t0, tried, found, seen = time.time(), 0, [], set()
    want = math.ceil(spec["n"] * spec.get("over", 2))   # generate extra, keep a subset spread over difficulty
    while len(found) < want and time.time() - t0 < spec.get("budget", 300):
        tried += 1
        c = make(spec, rng)
        if c is None or key_of(c["garden"]) in seen:
            continue
        seen.add(key_of(c["garden"]))
        c["spec"] = spec["id"]
        found.append(c)
    return spec["id"], found, tried, time.time() - t0


# ---------------------------------------------------------------- the campaign

SPECS = [
    # world 1: free gardens with stones
    dict(id="w1-55", world=1, gen="free", H=5, W=5, n=2), dict(id="w1-56", world=1, gen="free", H=5, W=6, n=2),
    dict(id="w1-66", world=1, gen="free", H=6, W=6, n=3), dict(id="w1-67", world=1, gen="free", H=6, W=7, n=2),
    dict(id="w1-77", world=1, gen="free", H=7, W=7, n=2), dict(id="w1-78", world=1, gen="free", H=7, W=8, n=2),
    # world 2: ripple stones
    dict(id="w2-66", world=2, gen="free", H=6, W=6, ripple=True, n=3), dict(id="w2-67", world=2, gen="free", H=6, W=7, ripple=True, n=4),
    dict(id="w2-77", world=2, gen="free", H=7, W=7, ripple=True, n=4), dict(id="w2-78", world=2, gen="free", H=7, W=8, ripple=True, n=2),
    dict(id="w2-88", world=2, gen="free", H=8, W=8, ripple=True, ripple2=True, n=1, budget=600, role="combo", names="two_ripple"),
    dict(id="w2-77x2", world=2, gen="free", H=7, W=7, ripple=True, ripple2=True, n=2, budget=600, role="combo", names="two_ripple"),
    # world 3: wave bands (ichimatsu gardens are added by hand below)
    dict(id="w3-67", world=3, gen="free", H=6, W=7, band=True, n=4), dict(id="w3-77", world=3, gen="free", H=7, W=7, band=True, n=4),
    dict(id="w3-78", world=3, gen="free", H=7, W=8, band=True, n=3),
    dict(id="w3-88", world=3, gen="free", H=8, W=8, band=True, ripple=True, n=1, budget=600, role="combo", names="band_ripple"),
    dict(id="w3-77br", world=3, gen="free", H=7, W=7, band=True, ripple=True, n=1, budget=600, role="combo", names="band_ripple"),
    # world 4: grooves, closed rings, pebbles
    dict(id="w4-g66", world=4, gen="grooves", H=6, W=6, n=3), dict(id="w4-g77", world=4, gen="grooves", H=7, W=7, n=3),
    dict(id="w4-g78", world=4, gen="grooves", H=7, W=8, n=3), dict(id="w4-ring", world=4, gen="grooves", H=7, W=8, closed_ring=True, n=2, budget=600, over=1.5),
    dict(id="w4-p66", world=4, gen="pebbles", H=6, W=6, n=2), dict(id="w4-p77", world=4, gen="pebbles", H=7, W=7, n=1),
    dict(id="w4-p78", world=4, gen="pebbles", H=7, W=8, n=2),
    dict(id="w4-gr", world=4, gen="grooves", H=7, W=7, ripple=True, n=1, over=4, budget=600, role="combo", names="groove_ripple"),
    dict(id="w4-pb", world=4, gen="pebbles", H=7, W=7, band=True, n=1, budget=600, role="combo", names="pebble_band"),
    # world 5: the wide rake
    dict(id="w5-55", world=5, gen="wide", H=5, W=5, n=3), dict(id="w5-56", world=5, gen="wide", H=5, W=6, n=2),
    dict(id="w5-66", world=5, gen="wide", H=6, W=6, n=3), dict(id="w5-67", world=5, gen="wide", H=6, W=7, n=3),
    dict(id="w5-77", world=5, gen="wide", H=7, W=7, n=4), dict(id="w5-78", world=5, gen="wide", H=7, W=8, n=2),
    dict(id="w5-88", world=5, gen="wide", H=8, W=8, n=2),
    dict(id="w5-wr", world=5, gen="wide", H=7, W=7, ripple=True, n=1, budget=600, role="combo", names="wide_ripple"),
    dict(id="w5-wb", world=5, gen="wide", H=7, W=7, band=True, n=1, budget=600, role="combo", names="wide_band"),
    # world 6: the side gate
    dict(id="w6-67", world=6, gen="side_wide", H=6, W=7, n=3),
    dict(id="w6-77", world=6, gen="side_wide", H=7, W=7, n=4), dict(id="w6-78", world=6, gen="side_wide", H=7, W=8, n=3),
    dict(id="w6-88", world=6, gen="side_wide", H=8, W=8, n=1), dict(id="w6-p77", world=6, gen="side_pebbles", H=7, W=7, n=3, budget=600, role="combo"),
    # world 7: everything, larger
    dict(id="w7-wr", over=1.5, world=7, gen="wide", H=8, W=8, ripple=True, n=4, budget=900), dict(id="w7-wb", over=1.5, world=7, gen="wide", H=8, W=8, band=True, n=3, budget=900),
    dict(id="w7-prb", over=1.5, world=7, gen="pebbles", H=8, W=8, ripple=True, band=True, n=3, budget=900, role="combo"),
    dict(id="w7-swr", over=1.5, world=7, gen="side_wide", H=8, W=8, ripple=True, n=2, budget=900, role="combo"),
    dict(id="w7-ring", over=1.5, world=7, gen="wide", H=8, W=8, closed_ring=True, n=2, budget=900),
    dict(id="w7-w99", over=1.5, world=7, gen="wide", H=9, W=9, n=3, budget=900),
    dict(id="w7-p99", over=1.5, world=7, gen="pebbles", H=9, W=9, n=2, budget=900,
         fallback=dict(id="w7-p88", over=1.5, gen="pebbles", H=8, W=8, budget=900)),
]

NAMES = {   # names for gardens with no special feature, in order of play
    1: ["Morning dew", "Mist", "First bell", "Sparrow", "Cold sand", "Lantern", "Bamboo", "The long wall", "Wind chime",
        "Pine shadow", "Heron's step", "Quiet", "Sunrise", "Kettle", "Swept path"],
    2: ["Koi", "Bell's echo", "Turtle", "Stepping stone", "Pond edge", "Water lily", "Drip", "Carp", "Lotus", "Still water",
        "Well", "Spring", "Frog", "Mossy bank", "Moon in a bowl", "Rain on the roof"],
    3: ["High tide", "Breakers", "Current", "Shallows", "Sandbar", "Reef", "Low tide", "Crosscurrent", "Undertow", "Swell",
        "Estuary", "Spring tide", "Noon bell"],
    4: ["Afternoon", "Tea house"],
    5: ["Pivot", "Hairpin", "Brace", "Long rake", "Turning stone", "Horseshoe", "Coil", "Shell", "Crescent", "Evening bell",
        "Swallow", "Bat", "Owl", "Ember", "Firefly", "Long shadow", "Last light", "Cicada", "Dragonfly", "Lanternlight", "Smoke"],
    6: ["Two doors", "Visitor", "Garden path", "Through the wall", "Gatekeeper", "Postern", "Threshold", "Late arrival",
        "Crossing", "Wicket", "Hinge", "Latch", "Bolt", "Dusk", "Errand"],
    7: ["Moonrise", "Starfield", "Lantern festival", "Night heron", "Owl's hour", "Milky Way", "Frost", "Moonlit sea", "Midnight",
        "Deep night", "Night rain", "Dark water", "Stargazer", "The abbot's garden", "Moonbeam", "Moon pool", "Last bell",
        "Small hours", "Before dawn", "Sleeping temple"],
}
POOLS = {   # names that describe a feature, used only for gardens that have it
    "two_ripple": ["Twin rings", "Two moons", "Paired wells"],
    "band_ripple": ["Noon sun", "Island in the current"],
    "ichimatsu": ["Tatami", "Weave", "Lattice", "Checkered shade"],
    "groove": ["His first lines", "Unfinished", "Interrupted", "Called away", "Old grooves", "Continuation", "Halfway",
               "Left for you", "Plain speaking", "Margin notes"],
    "ring": ["The abbot's ring", "Closed circle"],
    "pebble": ["Pale and dark", "White stones", "Black stones", "Go board", "Counting stones", "Afternoon tea"],
    "groove_ripple": ["Ripple and groove"],
    "pebble_band": ["Pebbles in the current"],
    "wide_ripple": ["Whorl"],
    "wide_band": ["Wake"],
}


def search_two_tones(rng, budget=120):
    """World 1's parity lesson: a small garden where the obvious place to end (the cell farthest from the gate)
    is the wrong colour, so every walk that ends there strands a cell."""
    t0 = time.time()
    best = None
    while time.time() - t0 < budget:
        H, W = rng.choice([(4, 5), (5, 5), (5, 4)])
        g = garden_layout(H, W, rng, n_pieces=1, max_frac=0.1)
        if g is None or not parity_ok(g):
            continue
        r = g.solve(mode=0, cap=10**6, stream=True)
        if not (2 <= r["nsol"] <= 40):
            continue
        # BFS distance from the gate
        from collections import deque
        start = (g.gate[0], g.gate[1]); dist = {start: 0}; q = deque([start])
        while q:
            a = q.popleft()
            for d in ((1, 0), (-1, 0), (0, 1), (0, -1)):
                b = (a[0] + d[0], a[1] + d[1])
                if g.is_sand(*b) and b not in dist:
                    dist[b] = dist[a] + 1; q.append(b)
        far = max(dist, key=lambda c: (dist[c], c in ((0, 0), (0, W - 1), (H - 1, 0), (H - 1, W - 1))))
        ends = {tuple(s["draw"][-1]) for s in r["sols"]}
        if far in ends or far[0] not in (0, H - 1) or far[1] not in (0, W - 1):
            continue
        c = cand_free(g, max_at_par=10**9, min_rakings=2)
        if c is None:
            continue
        score = (len(ends) <= 3, -r["nsol"])
        if best is None or score > best[0]:
            best = (score, c, far, sorted(ends))
    return best


def search_small(gen, H, W, rng, budget=120, side=False, ripple=False, max_tier=3):
    t0 = time.time()
    best = None
    while time.time() - t0 < budget:
        if ripple:
            g = garden_layout(H, W, rng, ripple=True, n_pieces=1)
        else:
            g = garden_layout(H, W, rng, n_pieces=1, max_frac=0.1)
        if g is None:
            continue
        if side and not add_side_gate(g, rng):
            continue
        c = cand_free(g) if gen == "free" else cand_wide(g)
        if c is None or (gen != "free" and c["tier"] > max_tier):
            continue
        score = (-c.get("tier", 0), c.get("rarity", 0))
        if best is None or score > best[0]:
            best = (score, c)
    return best[1] if best else None


def hand(name, world, g, intro=None, hint="auto", tone=0.0, kind=None, cells=None):
    return {"name": name, "world": world, "garden": to_dict(g), "intro": intro, "hint": hint, "tone": tone, "kind": kind,
            "hint_cells": cells}


def main():
    t_start = time.time()
    rng = random.Random(2026)
    stats = {}
    with Pool(2) as pool:
        jobs = [(s, 1000 + i) for i, s in enumerate(SPECS)]
        results = []
        for res in pool.imap_unordered(run_spec, jobs, chunksize=1):
            results.append(res)
            print(f"  done {res[0]} ({len(results)}/{len(jobs)})", flush=True)
    by_spec = {}
    for sid, found, tried, secs in results:
        by_spec[sid] = found
        stats[sid] = {"tried": tried, "found": len(found), "seconds": round(secs)}
        print(f"{sid:8s} tried {tried:6d} found {len(found):3d} in {secs:5.0f}s", flush=True)
    # a spec that found too few falls back to an easier one in the same slot (recorded in the stats)
    for i, s in enumerate(SPECS):
        if len(by_spec[s["id"]]) < s["n"] and "fallback" in s:
            fb = dict(s["fallback"], world=s["world"], n=s["n"])
            sid, found, tried, secs = run_spec((fb, 5000 + i))
            print(f"{s['id']} found {len(by_spec[s['id']])} < {s['n']}: fallback {sid} found {len(found)} in {secs:.0f}s", flush=True)
            stats[sid] = {"tried": tried, "found": len(found), "seconds": round(secs), "fallback_for": s["id"]}
            by_spec[s["id"]] = found + by_spec[s["id"]]

    # hand-made and searched teaching gardens (the searches are cached too)
    import os, pickle
    tpath = f"{CACHE}/teaching.pkl"
    if os.path.exists(tpath):
        two, wide_intro, side_intro, ripple_intro = pickle.load(open(tpath, "rb"))
    else:
        two = search_two_tones(rng)
        wide_intro = search_small("wide", 5, 5, rng)
        side_intro = search_small("wide", 6, 6, rng, side=True)
        ripple_intro = search_small("free", 5, 6, rng, ripple=True)
        pickle.dump((two, wide_intro, side_intro, ripple_intro), open(tpath, "wb"))
    if not (two and wide_intro and side_intro and ripple_intro):
        sys.exit(f"teaching search failed: {bool(two)}, {bool(wide_intro)}, {bool(side_intro)}, {bool(ripple_intro)}")
    stats["two_tones"] = {"far_cell": two[2], "ends": two[3]}
    # contrasting pairs that introduce each rule: the same small garden, before and after one change (pairs.py)
    ppath = f"{CACHE}/pairs.pkl"
    if os.path.exists(ppath):
        PAIRS = pickle.load(open(ppath, "rb"))
    else:
        import pairs as pairmod
        with Pool(2) as pool:
            PAIRS = dict(pool.imap_unordered(pairmod.run, [(nm, 7000 + i, None) for i, nm in enumerate(pairmod.MAKERS)]))
        pickle.dump(PAIRS, open(ppath, "wb"))
    if any(v is None for v in PAIRS.values()):
        sys.exit(f"pair search failed: {[k for k, v in PAIRS.items() if v is None]}")
    stats["pairs"] = {k: dict(v["facts"], tried=v["tried"], size=[v["B"].H, v["B"].W]) for k, v in PAIRS.items()}
    # a searched teaching garden can coincide with a generated candidate of the same size: keep it for teaching only
    reserved = {key_of(c["garden"]) for c in (two[1], wide_intro, side_intro, ripple_intro)}
    for sid in by_spec:
        dropped = [c for c in by_spec[sid] if key_of(c["garden"]) in reserved]
        if dropped:
            print(f"{sid}: {len(dropped)} candidate(s) kept back as teaching gardens", flush=True)
            by_spec[sid] = [c for c in by_spec[sid] if key_of(c["garden"]) not in reserved]

    w = json.load(open("/home/claude/zen/out/wide_pick.json"))
    t = json.load(open("/home/claude/zen/out/two_pick.json"))
    wide_list = json.load(open("/home/claude/zen/out/search3_wide.json"))
    gw = next(from_dict(d) for dens, trials, k, d, draw in wide_list
              if d["rows"] == ['........', '........', '..#.....', '....#...', '........', '........', '......#.'] and d["gate"] == [6, 5, "S"])
    mp = json.load(open("/home/claude/zen/out/marks_pick.json"))

    # layouts already in the campaign, up to symmetry: a generated garden may be a mirror image of a teaching garden,
    # and the same stones and gate can pass the filters of two worlds (free, and wide)
    teaching = [Garden(["....", "###.", "...."], (2, 0, "S")), Garden(["...", "..."], (1, 0, "S")), Garden(["...", "...", "..."], (2, 0, "S")),
                Garden(["#....", "....#", ".....", "....."], (3, 0, "S")), two[1]["garden"], wide_intro["garden"], side_intro["garden"],
                ripple_intro["garden"], gw, from_dict(w["garden"]), from_dict(t["garden"]), from_dict(mp["garden"])] + \
               [from_dict(L[k]["garden"]) for k in L] + [v[x] for v in PAIRS.values() for x in ("A", "B") if x in v]
    used = {canon(x) for x in teaching}

    def pick(sid, n, order):
        cs, ks = [], set()
        for c in sorted(by_spec[sid], key=order):      # also drop mirror images among the spec's own candidates
            k = canon(c["garden"])
            if k not in used and k not in ks:
                cs.append(c); ks.add(k)
        if len(cs) < len(by_spec[sid]):
            print(f"{sid}: {len(by_spec[sid]) - len(cs)} candidate(s) skipped, their layout (up to symmetry) already used", flush=True)
        if len(cs) < n:
            sys.exit(f"spec {sid}: only {len(cs)} of {n}")
        # spread over the difficulty range rather than taking the easiest n
        idx = [round(i * (len(cs) - 1) / max(1, n - 1)) for i in range(n)] if n > 1 else [len(cs) // 2]
        out = [cs[i] for i in sorted(set(idx))][:n] if len(set(idx)) == n else cs[:n]
        used.update(canon(c["garden"]) for c in out)
        return out

    free_order = lambda c: (c["cells"], c["rarity"])
    strict_order = lambda c: (c["tier"], c["trials"], c["cells"])

    def gen_entries(world, sids, order):
        out = []
        for s in SPECS:
            if s["world"] == world and s["id"] in sids:
                out += pick(s["id"], s["n"], order)
        return sorted(out, key=order)

    # ---------------------------------------------------------------- order each world by the template
    # intro; applications (easy to hard); a trap (the garden that most punishes the obvious first move, walking straight
    # in from the gate); combinations with earlier rules; a calm, easy garden; a capstone. Worlds that introduce a second
    # mechanic (ichimatsu; closed rings and pebbles) give it its own short section, after the trap.
    from hints import trap_score, near_misses

    def profile(c):
        """Ordering data: difficulty, turn density (calmness) and trap score, from the solvers."""
        g = c["garden"] if not isinstance(c["garden"], dict) else from_dict(c["garden"])
        if c["kind"] == "free":
            P = [s_["draw"] for s_ in g.solve(mode=2, bound=c["par"], cap=10**6, stream=True)["sols"]]
            c["trap"] = trap_score(g, c["par"], P, near_misses(g, c["par"]))
            c["density"] = c["par"] / c["cells"]
            c["diff"] = (c["cells"], c["rarity"])
        else:
            c["trap"] = trap_score(g, None, draw=c["target"])
            c["density"] = turns_of(g, c["target"]) / c["cells"]
            c["diff"] = (c["tier"], c["trials"], c["cells"])
        return c

    def hand_profile(e, kind):
        g = from_dict(e["garden"])
        if kind == "free":
            r = g.solve(mode=0, cap=10**8)
            free = optimize(g, time_limit=60)
            e.update(kind="free", par=free[0], cells=len(g.sand_cells()), rarity=math.log(r["nsol"] / r["hist"][free[0]]))
        else:
            gr = grade(g)
            d = g.solve(mode=0, cap=2, stream=True)["sols"][0]["draw"]
            e.update(kind="strict", tier=gr["tier"], trials=gr["trials"] or 0, cells=len(g.sand_cells()), target=d)
        return profile(e)

    def spread(n, k):
        """k indices spread over the easier part of range(n): the easiest, then evenly spaced short of the hardest (the
        capstone is the world's hard garden; the hardest applications are extras)."""
        if k >= n:
            return list(range(n))
        return [round(i * (n - 1) / k) for i in range(k)]

    def arrange(apps, combos=(), capstone=None, sections=(), n_apps=2, n_combos=2, showpiece=None):
        """The template after a world's introduction: two applications (easy to hard, then a hand-made showpiece if the
        world has one), the trap, any mid-world sections, two combinations, the calm garden and the capstone form the
        main path; the other applications and combinations are optional extras. Returns (main, extras), with roles
        recorded."""
        apps, combos = list(apps), list(combos)
        if capstone is None:                                  # the hardest garden closes the world
            capstone = max(apps + combos, key=lambda c: c["diff"])
            (apps if capstone in apps else combos).remove(capstone)
        apps.sort(key=lambda c: c["diff"])
        trap = None
        cands = [c for c in apps if c["trap"] > 0]
        if cands:                                             # the strongest trap; ties: nearest the middle difficulty
            trap = max(cands, key=lambda c: (round(c["trap"], 2), -abs(apps.index(c) - len(apps) // 2)))
            apps.remove(trap)
        easy = apps[: max(1, len(apps) // 2)]                 # a calm garden: easy, and with the fewest turns per cell
        calm = min(easy, key=lambda c: c["density"])
        apps.remove(calm)
        main_apps = [apps[i] for i in spread(len(apps), n_apps)]
        combos.sort(key=lambda c: c["diff"])
        main = main_apps + ([showpiece] if showpiece else []) + ([trap] if trap else [])
        extras = [c for c in apps if c not in main_apps]
        for sec_main, sec_extra in sections:                  # (main part, extra part) of a rule introduced mid-world
            main += sec_main; extras += sec_extra
        main += combos[:n_combos] + [calm, capstone]
        extras += combos[n_combos:]
        for c in main + extras:
            c.setdefault("role", "application")
        if trap: trap["role"] = "trap"
        calm["role"], capstone["role"] = "calm", "capstone"
        for c in combos: c["role"] = "combination"
        for c in extras: c["extra"] = True
        return main, extras

    def gen(world, combo=False, only=None):
        out = []
        for s_ in SPECS:
            if s_["world"] == world and (s_.get("role") == "combo") == combo and (only is None or s_["id"] in only):
                for c in pick(s_["id"], s_["n"], lambda c: (c["cells"], c["rarity"]) if c["kind"] == "free" else (c["tier"], c["trials"], c["cells"])):
                    out.append(profile(dict(c, world=world, name=None, pool=s_.get("names"))))
        return out

    def pair_entry(kind, which, world, name, intro, role):
        return dict(hand(name, world, PAIRS[kind][which], intro), role=role, pair=kind)

    SAME = "The next garden is this one again, with one change."
    W, X = {}, {}       # each world's main path, and its optional extras
    # ---- world 1: the teaching gardens, then the template's tail (no combinations: there is nothing earlier). The first
    # real garden comes third, before the two technique lessons (dead ends, parity), which only matter at its size: an
    # early player did not "sense the potential until about six levels in", when it came sixth. (A 2x3 garden that taught
    # the U-turn was dropped: the 3x3 garden's par rakings make U-turns anyway.)
    fs = hand_profile(hand("First stroke", 1, from_dict(L["first"]["garden"]), None,
                           "Both par rakings begin by walking straight up from the gate, and end near the top-left corner."), "free")
    calm1 = hand("Calm", 1, from_dict(L["calm"]["garden"]), None, "At par, the walk ends right beside the gate.")
    m, X[1] = arrange(gen(1) + [fs], capstone=calm1)
    teach = [hand("The corridor", 1, Garden(["....", "###.", "...."], (2, 0, "S")),
                  "Drag the monk in from the gate, covering every cell of sand in one stroke; when you let go, he rakes his way back out.",
                  "There is only one way through: follow the corridor."),
             hand("Nine squares", 1, Garden(["...", "...", "..."], (2, 0, "S")),
                  "Fewer turns is calmer. Match par for the abbot's seal.",
                  "Fewer turns mean longer lines: run each one from wall to wall."),
             hand("The pocket", 1, Garden(["#....", "....#", ".....", "....."], (3, 0, "S")),
                  "A cell with only one way in is where the walk must end: the rake is set down there.",
                  "The top-right corner has only one way in, so the walk has to end there.", cells=[[0, 4]]),
             hand("Two tones", 1, two[1]["garden"],
                  "Dark and light: every step changes colour, so a stroke ends on the colour there is more of.", "auto-parity", tone=1.0)]
    for e in teach:
        e["role"] = "intro"
    W[1] = teach[:2] + m[:1] + teach[2:] + m[1:]
    W[1][5]["tone"], W[1][6]["tone"] = 0.6, 0.3          # the two tones fade over the next gardens
    # ---- world 2: ripple stones, introduced by a pair; combinations have two ripple stones
    W[2] = [pair_entry("ripple", "A", 2, "Plain stone", f"Rake this one as usual. {SAME}", "pair"),
            pair_entry("ripple", "B", 2, "First ripple", "The same garden, with a ripple stone: its eight neighbours must be raked as one unbroken ring.", "intro")]
    ripples = hand("Ripples", 2, from_dict(L["ripple"]["garden"]), None,
                   "Rake the ring in one run; at par, the rest of the garden wraps around it in long lines.")
    m, X[2] = arrange(gen(2), gen(2, combo=True), showpiece=ripples)
    W[2] += m
    X[2] = [dict(ripple_intro, name=None, world=2)] + X[2]
    # ---- world 3: wave bands, introduced by a pair; then ichimatsu, its own short section; combinations add a ripple stone
    ich = []
    # ichimatsu variants: only 3x3 blocks can be raked at all (2x2 blocks have no valid raking at any gate), and a 6x6
    # garden with its blocks flipped is a mirror image of the original, as a 9x6 one is of a 6x9; so the variants move
    # the gate off the corner and widen the garden (each has one raking; all are distinct up to symmetry)
    for nm, (nbr, nbc, b, gc, flip) in (("ichimatsu-gate2", (2, 2, 3, 2, False)), ("ichimatsu-6x9", (2, 3, 3, 0, False)),
                                        ("ichimatsu-6x9-gate2", (2, 3, 3, 2, False))):
        gi = ichimatsu(nbr, nbc, b, gc, flip)
        r = gi.solve(mode=0, cap=50)
        if not (1 <= r["nsol"] <= 6 and parity_ok(gi)):
            sys.exit(f"ichimatsu variant {nm}: {r['nsol']} rakings")
        ich.append((r["nsol"], nm, gi))
    ich.sort(key=lambda x: (len(x[2].sand_cells()), x[0]))
    stats["ichimatsu_variants"] = [(n, nm, x.rows, list(x.gate)) for n, nm, x in ich]
    ich_entries = [dict(hand(None, 3, x, None, "auto"), role="application", pool="ichimatsu") for n, nm, x in ich]
    ich_section = ([dict(hand("Ichimatsu", 3, Garden(["===|||"] * 3 + ["|||==="] * 3, (5, 0, "S")),
                              "Checkerboard blocks: rake each block across or down, as hatched.",
                              "Rake each block as three parallel lines, then step into the next block."), role="intro"), ich_entries[0]],
                   ich_entries[1:])
    W[3] = [pair_entry("band", "A", 3, "Calm water", f"Rake this one as usual. {SAME}", "pair"),
            pair_entry("band", "B", 3, "First wave", "The same garden, with a wave band: each row of the hatched band is crossed in one straight run, edge to edge.", "intro")]
    waves = hand("Waves", 3, from_dict(L["waves"]["garden"]), None,
                 "Each row of the band is one straight run, so every turn happens in the plain sand at its ends.")
    m, X[3] = arrange(gen(3), gen(3, combo=True), sections=[ich_section], showpiece=waves)
    W[3] += m
    # ---- world 4: grooves, introduced by a pair; then the finished ring and the pebbles, each its own short section
    rings = sorted(pick("w4-ring", 2, lambda c: (c["tier"], c["trials"], c["cells"])), key=lambda c: (c["tier"], c["trials"], c["cells"]))
    for c_ in rings:
        c_.update(name=None, world=4, pool="ring", role="application")
    ring_section = ([dict(hand("The finished ring", 4, PAIRS["ring"]["B"],
                               "A ring the abbot has already raked around a stone: leave it as it is, and rake around it."), role="intro", pair="ring"),
                     rings[0]], [rings[1]])
    pebbles = sorted(gen(4, only={"w4-p66", "w4-p77", "w4-p78"}), key=lambda c: c["diff"])
    for c_ in pebbles:
        c_.update(pool="pebble", role="application")
    pebble_section = ([pair_entry("pebbles", "A", 4, "His grooves", f"Another garden with the abbot's grooves. {SAME}", "pair"),
                       pair_entry("pebbles", "B", 4, "His pebbles",
                                  "The same garden, with pebbles where the grooves were. Pale pebble: the line runs straight through it, either way. "
                                  "Dark pebble: the line turns at it, either way.", "intro"),
                       pebbles[0], hand("The abbot's pebbles", 4, from_dict(mp["garden"]), None,
                                        "A pebble beside a wall or a stone often leaves only one way through; settle those first.")],
                      pebbles[1:])
    W[4] = [pair_entry("grooves", "A", 4, "Open ground", f"Many rakings fit this garden. {SAME}", "pair"),
            pair_entry("grooves", "B", 4, "First grooves", "The same garden: the abbot began the raking. Your stroke must run along his grooves, and now only one raking fits.", "intro")]
    masters = hand("The master's grooves", 4, from_dict(L["grooves"]["garden"]), None,
                   "A groove fixes how the stroke enters and leaves its cell, and often its neighbours too: extend each one first.")
    grooves = gen(4, only={"w4-g66", "w4-g77", "w4-g78"})
    for c_ in grooves:
        c_["pool"] = "groove"
    m, X[4] = arrange(grooves, gen(4, combo=True), sections=[ring_section, pebble_section], showpiece=masters)
    W[4] += m
    # ---- world 5: the wide rake, introduced by a pair whose every par raking turns back in open sand; Spiral closes it
    wide_rake = hand_profile(hand("Wide rake", 5, gw, None, "Every reversal needs a pivot: each U-turn must hug a wall or wrap around a stone."), "strict")
    spiral = hand("Spiral", 5, from_dict(w["garden"]), None,
                  "One stone stands in open sand: the only way to turn back around it is to spiral in toward it.")
    W[5] = [pair_entry("wide", "A", 5, "Narrow rake", f"Rake this one with the ordinary rake. {SAME}", "pair"),
            pair_entry("wide", "B", 5, "Wide blade", "The same garden, with the wide rake: it cannot turn back in open sand, since a U-turn needs a wall "
                       "or a stone to pivot on; and it is set down against one.", "intro")]
    m, X[5] = arrange(gen(5) + [wide_rake], gen(5, combo=True), capstone=spiral)
    W[5] += m
    X[5] = [dict(wide_intro, name=None, world=5)] + X[5]
    # ---- world 6: the side gate, introduced by a pair; combinations add pebbles; Side gate closes it
    sg = hand("Side gate", 6, from_dict(t["garden"]), None,
              "Both ends are fixed: plan backwards from the side gate as well as forwards from the veranda.")
    W[6] = [pair_entry("side", "A", 6, "One gate", f"A garden for the wide rake. {SAME}", "pair"),
            pair_entry("side", "B", 6, "Two gates", "The same garden, with a side gate: the stroke must begin there, and end at the veranda. "
                       "Of the first garden's rakings, only one begins there.", "intro")]
    m, X[6] = arrange(gen(6), gen(6, combo=True), capstone=sg)
    W[6] += m
    X[6] = [dict(side_intro, name=None, world=6)] + X[6]
    # ---- world 7: everything, larger; three-rule gardens are the combinations; Silver sand closes the game
    silver = hand("Silver sand", 7, Garden(["".join("=" if ((r // 3) + (c // 3)) % 2 == 0 else "|" for c in range(9)) for r in range(9)], (8, 0, "S")),
                  None, "Nine blocks, each raked as three parallel lines: find the order in which each block hands the stroke to the next.")
    apps7 = sorted(gen(7), key=lambda c: c["diff"])
    first7 = dict(apps7.pop(0), role="intro", intro="Night: everything the day taught, in larger gardens.")
    m, X[7] = arrange(apps7, gen(7, combo=True), capstone=silver)
    W[7] = [first7] + m
    for wld in X:
        for e in X[wld]:
            e["extra"] = True
            e.setdefault("role", "application")
    W = {wld: W[wld] + X[wld] for wld in W}
    for wld in W:
        for e in W[wld]:
            e.setdefault("role", "application")

    # names, and the final list: a garden keeps the name its layout had before (out/garden_names.json, seeded by
    # seed_garden_names.py), so that names, poems and saved progress survive rearranging the campaign; a garden new to the
    # campaign takes the next name of its pool that no other layout owns
    def full_key(d):
        return json.dumps([d["rows"], d["gate"], d["end"], d.get("end_side"), d["nohairpin"], bool(d["braced_start"]),
                           sorted(d["clues"].items()), sorted((d.get("marks") or {}).items())])
    REG = {r["key"]: r["name"] for r in json.load(open(NAMES_REG))} if os.path.exists(NAMES_REG) else {}
    owned = set(REG.values())
    final = []
    pools = {k: iter(v) for k, v in POOLS.items()}
    for world in range(1, 8):
        names = iter(NAMES[world])
        taken = {e.get("name") for e in W[world] if e.get("name")}
        for e in W[world]:
            if "garden" in e and not isinstance(e["garden"], dict):
                e["garden"] = to_dict(e["garden"])
            if not e.get("name"):
                nm = REG.get(full_key(e["garden"]))
                if nm is None or nm in taken:
                    src = pools[e["pool"]] if e.get("pool") else names
                    nm = next(src)
                    while nm in taken or nm in owned:
                        nm = next(src)
                e["name"] = nm
                taken.add(nm)
            final.append({"name": e["name"], "world": world, "garden": e["garden"], "intro": e.get("intro") if not e.get("extra") else None,
                          "hint": e.get("hint", "auto"), "tone": e.get("tone", 0.0), "role": e["role"],
                          "hint_cells": e.get("hint_cells"), "trap": round(e.get("trap", 0.0), 3),
                          "extra": bool(e.get("extra")), "pair": e.get("pair")})
        stats[f"world{world}"] = len(W[world])
        print(world, " | ".join(f"{e['name']} ({e['role'][:4]}{', x' if e.get('extra') else ''})" for e in W[world]), flush=True)
    counts = [stats[f"world{w}"] for w in range(1, 8)]
    print("gardens per world:", counts, "total", sum(counts))
    stats["main"] = [sum(not e.get("extra") for e in W[w]) for w in range(1, 8)]
    stats["extras"] = [sum(bool(e.get("extra")) for e in W[w]) for w in range(1, 8)]
    print("main path per world:", stats["main"], "extras:", stats["extras"])
    # the whole definition: a contrasting pair is one layout twice, differing only in its clues, pebbles or rake
    def full_key(d):
        return (tuple(d["rows"]), tuple(d["gate"]), tuple(d["end"] or ()), d.get("end_side"), d["nohairpin"], bool(d["braced_start"]),
                tuple(sorted(d["clues"].items())), tuple(sorted((d.get("marks") or {}).items())))
    seen_k = {}
    for e in final:
        k = full_key(e["garden"])
        if k in seen_k:
            print("duplicate:", seen_k[k], "=", e["name"], flush=True)
        seen_k[k] = e["name"]
    assert len({full_key(e["garden"]) for e in final}) == len(final), "duplicate gardens"
    same = [e["name"] for e in final if e.get("pair") and e["role"] == "pair"]
    stats["layouts"] = len({canon(from_dict(e["garden"])) for e in final})
    json.dump(final, open(OUT, "w"))
    stats["seconds"] = round(time.time() - t_start)
    json.dump(stats, open(STATS, "w"), indent=1)
    print("saved", OUT, f"{stats['seconds']}s")


if __name__ == "__main__":
    main()
src/clues.py · 78 lines · 2.7 kB — Greedy 'master's grooves' clue selection: add target-solution glyphs until the garden has a unique raking. 
"""Greedy 'master's grooves' clue selection: add target-solution glyphs until the garden has a unique raking."""
import random
from copy import deepcopy
from zg import Garden, glyphs

def with_clues(g, clues):
    g2 = deepcopy(g)
    g2.clues = dict(g.clues); g2.clues.update(clues)
    return g2

def count(g, cap):
    r = g.solve(mode=0, cap=cap, nodelimit=2 * 10**8)
    return r["nsol"], r

def make_unique(g, target_draw, rng, cap=3000, max_clues=40, sample=24):
    gl = glyphs(g, target_draw)
    start = target_draw[-1]
    cand_all = [c for c in gl if c != start and len(gl[c]) == 2 and c not in g.clues]
    clues = {}
    n, _ = count(g, cap)
    while n > 1 and len(clues) < max_clues:
        cand = [c for c in cand_all if c not in clues]
        rng.shuffle(cand)
        best, bestn = None, None
        for c in cand[:sample]:
            m, _ = count(with_clues(g, {**clues, c: gl[c]}), cap)
            if bestn is None or m < bestn:
                best, bestn = c, m
                if m == 1: break
        clues[best] = gl[best]
        n = bestn
    if n != 1:
        return None
    # prune redundant clues
    for c in list(clues):
        trial = {k: v for k, v in clues.items() if k != c}
        m, _ = count(with_clues(g, trial), 2)
        if m == 1:
            clues = trial
    return clues


def with_marks(g, marks):
    g2 = deepcopy(g)
    g2.marks = dict(g.marks); g2.marks.update(marks)
    return g2


def make_unique_marks(g, target_draw, rng, cap=3000, max_marks=60, sample=24):
    """Like make_unique, but with the abbot's *marks*: each says only 'straight here' ('S') or 'turn here' ('B'),
    taken from the target raking; returns None if marks alone cannot make the target unique."""
    gl = glyphs(g, target_draw)
    start = target_draw[-1]
    kind = {c: ("S" if set(ds) in ({"E", "W"}, {"N", "S"}) else "B") for c, ds in gl.items() if len(ds) == 2}
    cand_all = [c for c in kind if c != start and c not in g.clues and c not in g.marks]
    marks = {}
    n, _ = count(g, cap)
    while n > 1 and len(marks) < max_marks:
        cand = [c for c in cand_all if c not in marks]
        if not cand:
            break
        rng.shuffle(cand)
        best, bestn = None, None
        for c in cand[:sample]:
            m, _ = count(with_marks(g, {**marks, c: kind[c]}), cap)
            if bestn is None or m < bestn:
                best, bestn = c, m
                if m == 1: break
        marks[best] = kind[best]
        n = bestn
    if n != 1:
        return None
    for c in list(marks):
        trial = {k: v for k, v in marks.items() if k != c}
        m, _ = count(with_marks(g, trial), 2)
        if m == 1:
            marks = trial
    return marks
src/daily.py · 207 lines · 10.8 kB — A year of daily gardens (366), the same for everyone on a given date, none of them in the campaign. 
"""daily.py: a year of daily gardens (366), the same for everyone on a given date, none of them in the campaign.

Free and strict gardens alternate (free: scored against par; strict: exactly one raking, under grooves, pebbles, the
wide rake or a side gate), sizes 5x5 to 8x8, made by the campaign's own generators (campaign.make) and filters. The
candidates the campaign generated and did not use are taken first; the rest are generated here (cached in
out/daily_cache/). Each garden is exported as the prototype expects (export_proto.py's fields: par, raking counts,
every par raking as move strings for the in-page hints, an intended raking, a first observation). Writes
out/daily_gardens.json (day 0 = 2026-01-01; the page picks day-of-cycle from the local date) and out/daily_stats.json.
"""
from __future__ import annotations

import glob
import json
import math
import os
import pickle
import random
import sys
import time
from multiprocessing import Pool

from zg import turns_of, check_solution
from zcp import optimize
from levels import from_dict, to_dict
from campaign import canon, key_of, _run_spec, CACHE
from hints import best_free_hint, best_strict_hint, near_misses

OUT = "/home/claude/zen/out/daily_gardens.json"
STATS = "/home/claude/zen/out/daily_stats.json"
DCACHE = "/home/claude/zen/out/daily_cache"
YEAR = 366
MAX_AT_PAR = 24           # free gardens with more par rakings than this are left out (the page carries every par raking)
MOVE = {(-1, 0): "N", (1, 0): "S", (0, 1): "E", (0, -1): "W"}


def moves_of(d):
    return "".join(MOVE[(b[0] - a[0], b[1] - a[1])] for a, b in zip(d, d[1:]))


FREE = [
    dict(id="d-f66", gen="free", H=6, W=6, n=18), dict(id="d-f67", gen="free", H=6, W=7, n=20),
    dict(id="d-f77", gen="free", H=7, W=7, n=22), dict(id="d-f78", gen="free", H=7, W=8, n=12),
    dict(id="d-r66", gen="free", H=6, W=6, ripple=True, n=14), dict(id="d-r67", gen="free", H=6, W=7, ripple=True, n=18),
    dict(id="d-r77", gen="free", H=7, W=7, ripple=True, n=20), dict(id="d-r78", gen="free", H=7, W=8, ripple=True, n=10),
    dict(id="d-r77x2", gen="free", H=7, W=7, ripple=True, ripple2=True, n=8, budget=400),
    dict(id="d-b67", gen="free", H=6, W=7, band=True, n=14), dict(id="d-b77", gen="free", H=7, W=7, band=True, n=18),
    dict(id="d-b78", gen="free", H=7, W=8, band=True, n=10), dict(id="d-br77", gen="free", H=7, W=7, band=True, ripple=True, n=8, budget=400),
    dict(id="d-f66b", gen="free", H=6, W=6, n=8), dict(id="d-r67b", gen="free", H=6, W=7, ripple=True, n=8),   # (the first round fell 4 short after the par-raking cap)
]
STRICT = [
    dict(id="d-g66", gen="grooves", H=6, W=6, n=16), dict(id="d-g77", gen="grooves", H=7, W=7, n=20),
    dict(id="d-g78", gen="grooves", H=7, W=8, n=8, budget=400), dict(id="d-gr77", gen="grooves", H=7, W=7, ripple=True, n=6, budget=400),
    dict(id="d-ring78", gen="grooves", H=7, W=8, closed_ring=True, n=6, budget=400),
    dict(id="d-p66", gen="pebbles", H=6, W=6, n=12), dict(id="d-p77", gen="pebbles", H=7, W=7, n=12),
    dict(id="d-p78", gen="pebbles", H=7, W=8, n=4, budget=300), dict(id="d-pb77", gen="pebbles", H=7, W=7, band=True, n=4, budget=300),
    dict(id="d-w55", gen="wide", H=5, W=5, n=8), dict(id="d-w56", gen="wide", H=5, W=6, n=8),
    dict(id="d-w66", gen="wide", H=6, W=6, n=14), dict(id="d-w67", gen="wide", H=6, W=7, n=14),
    dict(id="d-w77", gen="wide", H=7, W=7, n=16), dict(id="d-w78", gen="wide", H=7, W=8, n=10), dict(id="d-w88", gen="wide", H=8, W=8, n=8),
    dict(id="d-wr77", gen="wide", H=7, W=7, ripple=True, n=8), dict(id="d-wb77", gen="wide", H=7, W=7, band=True, n=8),
    dict(id="d-s67", gen="side_wide", H=6, W=7, n=12), dict(id="d-s77", gen="side_wide", H=7, W=7, n=14),
    dict(id="d-s78", gen="side_wide", H=7, W=8, n=10), dict(id="d-s88", gen="side_wide", H=8, W=8, n=6),
    dict(id="d-sp77", gen="side_pebbles", H=7, W=7, n=6, budget=300),
]
for s in FREE + STRICT:
    s.setdefault("over", 1)
    s.setdefault("budget", 240)


def run_cached(args):
    spec, seed = args
    os.makedirs(DCACHE, exist_ok=True)
    path = f"{DCACHE}/{spec['id']}.pkl"
    if os.path.exists(path):
        return pickle.load(open(path, "rb"))
    res = _run_spec(spec, seed)
    pickle.dump(res, open(path, "wb"))
    return res


def leftovers(used):
    """Candidates the campaign generated and did not use (by layout up to symmetry)."""
    out = []
    for f in sorted(glob.glob(f"{CACHE}/w*.pkl")):
        sid, found, tried, secs = pickle.load(open(f, "rb"))
        for c in found:
            if canon(c["garden"]) not in used and c["garden"].H <= 8 and c["garden"].W <= 8:   # (the daily garden stays at 8x8 or smaller)
                c = dict(c)
                c["spec"] = "campaign:" + sid
                out.append(c)
    return out


EXPORTS = f"{DCACHE}/exports.pkl"


def export(c, k):
    """The prototype's garden record for candidate c (see export_proto.add), as day k; cached by layout."""
    cache = pickle.load(open(EXPORTS, "rb")) if os.path.exists(EXPORTS) else {}
    ck = (key_of(c["garden"]), tuple(sorted(to_dict(c["garden"])["clues"].items())), tuple(sorted(to_dict(c["garden"])["marks"].items())), c["garden"].nohairpin)
    if ck in cache:
        return cache[ck]
    e = _export(c, k)
    cache[ck] = e
    pickle.dump(cache, open(EXPORTS, "wb"))
    return e


def _export(c, k):
    g = c["garden"]
    intended = [tuple(x) for x in c["target"]]
    assert check_solution(g, intended), c["spec"]
    r = g.solve(mode=0, cap=10**9, nodelimit=4 * 10**9)
    assert r["nsol"] > 0 and r["complete"] and not r["capped"], c["spec"]
    if c["kind"] == "free":
        free = optimize(g, time_limit=120)
        assert free[1] == "OPTIMAL", c["spec"]
        par = free[0]
    else:
        assert r["nsol"] == 1, c["spec"]
        par = turns_of(g, intended)
    assert turns_of(g, intended) == par, c["spec"]
    at_par = r["hist"].get(par, 0)
    assert at_par > 0, c["spec"]
    if at_par > MAX_AT_PAR:
        return None
    rp = g.solve(mode=2, bound=par, cap=10**6, stream=True)
    par_rakings = [s_["draw"] for s_ in rp["sols"]]
    assert rp["complete"] and not rp["capped"] and len(par_rakings) == rp["nsol"] == at_par, c["spec"]
    assert all(check_solution(g, d_) and turns_of(g, d_) == par for d_ in par_rakings), c["spec"]
    par_rakings.sort(key=lambda d_: [tuple(x) for x in d_] != intended)
    assert [tuple(x) for x in par_rakings[0]] == intended, c["spec"]
    if r["nsol"] == 1:
        hint, hint_cells, _ = best_strict_hint(g, intended)
    else:
        hint, hint_cells, _ = best_free_hint(g, par, par_rakings, near_misses(g, par))
    d = to_dict(g)
    return {"name": f"daily-{k:03d}", "world": 0, "rows": g.rows, "gate": list(g.gate), "end": list(g.end) if g.end else None,
            "end_side": g.end_side, "clues": d["clues"], "wide": g.nohairpin == 2, "par": par, "rakings": r["nsol"], "at_par": at_par,
            "intended": [list(x) for x in intended], "intro": None, "marks": d["marks"], "par_moves": [moves_of(d_) for d_ in par_rakings],
            "hint": hint, "tone": 0.0, "hint_cells": hint_cells or [], "role": "daily", "trap": 0.0, "extra": False, "pair": None,
            "demo": False, "kind": c["kind"], "spec": c["spec"]}


def main():
    t0 = time.time()
    C = json.load(open("/home/claude/zen/out/campaign.json"))
    used = {canon(from_dict(e["garden"])) for e in C}
    cands = leftovers(used)
    print(f"{len(cands)} candidates left over from the campaign", flush=True)
    specs = FREE + STRICT
    with Pool(2) as pool:
        results = pool.map(run_cached, [(s, 9000 + i) for i, s in enumerate(specs)])
    stats = {}
    for sid, found, tried, secs in results:
        stats[sid] = {"tried": tried, "found": len(found), "seconds": round(secs)}
        print(f"{sid:9s} tried {tried:6d} found {len(found):3d} in {secs:5.0f}s", flush=True)
        cands += found
    # one of each layout (up to symmetry), none from the campaign
    seen, uniq = set(used), []
    for c in cands:
        cn = canon(c["garden"])
        if cn in seen:
            continue
        seen.add(cn)
        uniq.append(c)
    free = [c for c in uniq if c["kind"] == "free"]
    strict = [c for c in uniq if c["kind"] == "strict"]
    print(f"{len(uniq)} distinct layouts: {len(free)} free, {len(strict)} strict", flush=True)
    rng = random.Random(2026)
    rng.shuffle(free)
    rng.shuffle(strict)
    half = YEAR // 2
    days, picked = [], {"free": 0, "strict": 0}
    for kind, pool_ in (("free", free), ("strict", strict)):
        out = []
        for c in pool_:
            if len(out) == half:
                break
            e = export(c, 0)
            if e is not None:
                out.append(e)
                print(f"{kind} {len(out):3d}/{half} {c['spec']:14s} {c['garden'].H}x{c['garden'].W} rakings={e['rakings']:>8,} par={e['par']:2d} at_par={e['at_par']:2d}  {e['hint']}", flush=True)
        assert len(out) == half, f"only {len(out)} {kind} gardens"
        picked[kind] = out
    for k in range(YEAR):                   # even days free, odd days strict
        e = (picked["free"] if k % 2 == 0 else picked["strict"])[k // 2]
        e["name"] = f"daily-{k:03d}"
        days.append(e)
    assert len({(tuple(e["rows"]), tuple(e["gate"]), tuple(e["end"] or []), e["wide"], json.dumps(e["clues"], sort_keys=True), json.dumps(e["marks"], sort_keys=True)) for e in days}) == YEAR
    json.dump(days, open(OUT, "w"))
    sizes = {}
    for e in days:
        sizes[f"{len(e['rows'])}x{len(e['rows'][0])}"] = sizes.get(f"{len(e['rows'])}x{len(e['rows'][0])}", 0) + 1
    kinds = {}
    for e in days:
        kinds[e["spec"].split(":")[-1][:5] if e["spec"].startswith("campaign") else e["spec"]] = kinds.get(e["spec"].split(":")[-1][:5] if e["spec"].startswith("campaign") else e["spec"], 0) + 1
    stats_out = {"days": YEAR, "free": half, "strict": YEAR - half, "sizes": sizes, "from_campaign_leftovers": sum(e["spec"].startswith("campaign") for e in days),
                 "with_ripple": sum(any("O" in r for r in e["rows"]) for e in days), "with_band": sum(any(ch in r for r in e["rows"] for ch in "hv") for e in days),
                 "with_grooves": sum(bool(e["clues"]) for e in days), "with_pebbles": sum(bool(e["marks"]) for e in days),
                 "wide": sum(e["wide"] for e in days), "side_gate": sum(bool(e["end"]) for e in days), "closed_ring": sum(any("R" in r for r in e["rows"]) for e in days),
                 "max_at_par": MAX_AT_PAR, "bytes": os.path.getsize(OUT), "specs": stats, "seconds": round(time.time() - t0)}
    json.dump(stats_out, open(STATS, "w"), indent=1)
    print(json.dumps({k: v for k, v in stats_out.items() if k != "specs"}, indent=1), flush=True)


if __name__ == "__main__":
    main()
src/exp_clues.py · 31 lines · 1.4 kB 
import random, statistics, sys, time
from multiprocessing import Pool
from gen import random_layout
from clues import make_unique

def job(args):
    seed, H, W, k, variant = args
    rng = random.Random(seed * 31 + k)
    g = random_layout(H, W, k, rng, gate_side="S")
    if g is None: return None
    if variant == "wide":
        g.nohairpin = 2; g.braced_start = True
    r = g.solve(mode=0, cap=1, nodelimit=10**8)
    if r["nsol"] == 0: return None
    target = r["sols"][0]["draw"]
    clues = make_unique(g, target, rng)
    if clues is None: return ("fail",)
    return (len(clues), len(g.sand_cells()))

if __name__ == "__main__":
    H, W = int(sys.argv[1]), int(sys.argv[2])
    for variant in ["plain", "wide"]:
        for k in [0, 2, 4]:
            t0 = time.time()
            with Pool(2) as p:
                res = [x for x in p.map(job, [(s, H, W, k, variant) for s in range(120)]) if x]
            ok = [x for x in res if x[0] != "fail"]
            nc = [x[0] for x in ok]
            print(f"{variant:5s} {H}x{W} k={k}: n={len(ok)} fails={len(res)-len(ok)} clues median={statistics.median(nc) if nc else 0} "
                  f"mean={statistics.mean(nc) if nc else 0:.1f} max={max(nc) if nc else 0} frac0={sum(c==0 for c in nc)/max(1,len(nc)):.2f} "
                  f"clues/cell={statistics.mean(c/x[1] for c, x in zip(nc, ok)) if nc else 0:.3f} ({time.time()-t0:.0f}s)", flush=True)
src/exp_constraints.py · 64 lines · 2.8 kB 
import random, statistics, time
from multiprocessing import Pool
from zg import Garden

def place_ripples(H, W, n_ripple, n_plain, rng):
    rows = [["."] * W for _ in range(H)]
    tries = 0; placed = 0
    while placed < n_ripple and tries < 500:
        tries += 1
        r, c = rng.randint(1, H - 2), rng.randint(1, W - 2)
        if all(rows[a][b] == "." for a in range(r - 1, r + 2) for b in range(c - 1, c + 2)):
            # keep other ripple rings disjoint-ish: require 3x3 free
            rows[r][c] = "O"; placed += 1
    placed = 0; tries = 0
    while placed < n_plain and tries < 500:
        tries += 1
        r, c = rng.randrange(H), rng.randrange(W)
        if rows[r][c] != ".": continue
        # don't break any ripple ring
        if any(0 <= r+dr < H and 0 <= c+dc < W and rows[r+dr][c+dc] == "O" for dr in (-1,0,1) for dc in (-1,0,1)): continue
        rows[r][c] = "#"; placed += 1
    gc = rng.choice([c for c in range(W) if rows[H-1][c] == "."])
    return Garden(["".join(x) for x in rows], (H - 1, gc, "S"))

def place_band(H, W, rng, orient):
    rows = [["."] * W for _ in range(H)]
    if orient == "h":
        h = rng.randint(2, 3); w = rng.randint(3, W - 2)
        r0 = rng.randint(0, H - h - 1); c0 = rng.randint(1, W - w - 1)
    else:
        w = rng.randint(2, 3); h = rng.randint(3, H - 2)
        r0 = rng.randint(1, H - h - 1); c0 = rng.randint(0, W - w)
    for r in range(r0, r0 + h):
        for c in range(c0, c0 + w):
            rows[r][c] = orient
    # one or two plain stones away from band
    for _ in range(rng.randint(0, 2)):
        r, c = rng.randrange(H), rng.randrange(W)
        if rows[r][c] == ".": rows[r][c] = "#"
    gcs = [c for c in range(W) if rows[H-1][c] == "."]
    if not gcs: return None
    return Garden(["".join(x) for x in rows], (H - 1, rng.choice(gcs), "S"))

def job(args):
    kind, seed, H, W = args
    rng = random.Random(seed)
    if kind.startswith("ripple"):
        n = int(kind[-1])
        g = place_ripples(H, W, n, rng.randint(0, 2), rng)
    else:
        g = place_band(H, W, rng, kind[-1])
    if g is None: return None
    r = g.solve(mode=0, cap=200000, nodelimit=3*10**8)
    return r["nsol"]

if __name__ == "__main__":
    for kind in ["ripple1", "ripple2", "band_h", "band_v"]:
        for (H, W) in [(7, 7), (8, 8)]:
            t0 = time.time()
            with Pool(2) as p:
                res = [x for x in p.map(job, [(kind, s, H, W) for s in range(300)]) if x is not None]
            solv = [x for x in res if x > 0]
            print(f"{kind:8s} {H}x{W}: solvable={len(solv)/len(res):.3f} unique={sum(x==1 for x in res)/len(res):.3f} "
                  f"1-20={sum(1<=x<=20 for x in res)/len(res):.3f} median={statistics.median(solv) if solv else 0:g} ({time.time()-t0:.0f}s)", flush=True)
src/exp_continuity.py · 46 lines · 2.3 kB — Does the abbot's taste in worlds 1-3 match the wide rake's physics (worlds 5-7)? 
"""Does the abbot's taste in worlds 1-3 match the wide rake's physics (worlds 5-7)?

The generator keeps a free garden only if *some* par raking has no U-turn in open sand. Astra's review pointed out that
this does not make *every* rewarded raking graceful. For each free garden of worlds 1-3 in the exported campaign, count
the par rakings (all of them are stored) with and without an open-sand U-turn. Writes out/continuity.json.
"""
import json
from zg import Garden, hairpin_list

P = json.load(open('/home/claude/zen/out/proto_gardens.json'))
STEP = {"N": (-1, 0), "S": (1, 0), "E": (0, 1), "W": (0, -1)}


def garden_of(p):
    marks = {tuple(map(int, k.split(','))): v for k, v in (p.get('marks') or {}).items()}
    clues = {tuple(map(int, k.split(','))): v for k, v in (p['clues'] or {}).items()}
    return Garden(p['rows'], tuple(p['gate']), clues=clues, marks=marks, end=tuple(p['end']) if p['end'] else None,
                  nohairpin=2 if p['wide'] else 0, braced_start=p['wide'], end_side=p['end_side'])


def cells(p, ms):
    r, c = p['gate'][0], p['gate'][1]
    out = [(r, c)]
    for ch in ms:
        r, c = r + STEP[ch][0], c + STEP[ch][1]
        out.append((r, c))
    return out


rows = []
for p in P:
    if p['world'] > 3 or p['rakings'] == 1:
        continue
    g = garden_of(p)
    opened = [any(not braced for *_, braced in hairpin_list(g, cells(p, ms))) for ms in p['par_moves']]
    rows.append({"name": p["name"], "world": p["world"], "extra": p["extra"], "role": p["role"], "at_par": len(opened),
                 "with_open_u": sum(opened)})
mixed = [r for r in rows if 0 < r["with_open_u"] < r["at_par"]]
clean = [r for r in rows if r["with_open_u"] == 0]
most = [r for r in rows if r["with_open_u"] * 2 > r["at_par"]]
out = {"free_gardens": len(rows), "all_graceful": len(clean), "mixed": len(mixed), "majority_open_u": len(most),
       "none_graceful": sum(r["with_open_u"] == r["at_par"] for r in rows),
       "par_rakings": sum(r["at_par"] for r in rows), "par_rakings_with_open_u": sum(r["with_open_u"] for r in rows),
       "worst": max(rows, key=lambda r: (r["with_open_u"] / r["at_par"], r["at_par"])), "gardens": rows}
json.dump(out, open('/home/claude/zen/out/continuity.json', 'w'), indent=1)
print({k: v for k, v in out.items() if k != "gardens"})
src/exp_graceful.py · 28 lines · 1.5 kB 
import random, statistics, time, sys
from gen import random_layout, parity_ok
from zcp import optimize
rng = random.Random(int(sys.argv[1]))
rows = []
t0 = time.time()
while len(rows) < int(sys.argv[2]):
    H, W = rng.choice([(6,7),(7,7),(7,8)])
    g = random_layout(H, W, rng.choice([1,2,3]), rng, gate_side="S")
    if g is None or not parity_ok(g): continue
    if g.solve(mode=0, cap=1)['nsol'] == 0: continue
    t_free = optimize(g, time_limit=60)
    if t_free[1] != 'OPTIMAL': continue
    t_grace = optimize(g, nohairpin=2, time_limit=60)
    g.braced_start = True
    t_wide = optimize(g, nohairpin=2, time_limit=60)
    rows.append((t_free[0], t_free[4], t_grace[0], t_grace[1], t_wide[0], t_wide[1]))
free = [r[0] for r in rows]
same = sum(1 for r in rows if r[2] == r[0])
feas_grace = sum(1 for r in rows if r[2] is not None)
feas_wide = sum(1 for r in rows if r[4] is not None)
gaps = [r[2] - r[0] for r in rows if r[2] is not None]
gapw = [r[4] - r[0] for r in rows if r[4] is not None]
hp_at_par = [r[1] for r in rows]
print(f"n={len(rows)} ({time.time()-t0:.0f}s)")
print(f"graceful (no open hairpin) feasible: {feas_grace/len(rows):.2f}; graceful optimum == free par: {same/len(rows):.2f}; gap when feasible: mean {statistics.mean(gaps):.2f} max {max(gaps)}")
print(f"wide rake feasible: {feas_wide/len(rows):.2f}; gap mean {statistics.mean(gapw) if gapw else float('nan'):.2f}")
print("hairpins in (one) free-par solution: mean", statistics.mean(hp_at_par))
src/exp_marks.py · 48 lines · 2.6 kB — The abbot's marks vs the master's grooves: on the same gardens and the same intended raking (a graceful par 
"""The abbot's marks vs the master's grooves: on the same gardens and the same intended raking (a graceful par
raking), how many clues does each need for uniqueness, and how hard does the grader find the result?"""
import json, random, statistics, sys, time
from multiprocessing import Pool
from gen import random_layout, parity_ok
from zg import turns_of
from zcp import optimize
from clues import make_unique, with_clues, make_unique_marks, with_marks
from grader import grade

BUDGET = float(sys.argv[1]) if len(sys.argv) > 1 else 600

def job(seed):
    rng = random.Random(seed); out = []; t0 = time.time()
    while time.time() - t0 < BUDGET:
        H, W = rng.choice([(7, 7), (7, 8)])
        g = random_layout(H, W, rng.choice([1, 2, 3]), rng, gate_side="S", min_gap=1)
        if g is None or not parity_ok(g) or g.solve(mode=0, cap=1)["nsol"] == 0:
            continue
        free = optimize(g, time_limit=30); grace = optimize(g, nohairpin=2, time_limit=30)
        if free[1] != "OPTIMAL" or grace[1] != "OPTIMAL" or grace[0] != free[0]:
            continue
        target = list(reversed(grace[2]))
        s = rng.randrange(10**9)
        grooves = make_unique(g, target, random.Random(s))
        marks = make_unique_marks(g, target, random.Random(s))
        row = {"rows": g.rows, "gate": list(g.gate), "cells": len(target), "turns": turns_of(g, target),
               "target": [list(c) for c in target]}
        if grooves is not None:
            gr = grade(with_clues(g, grooves))
            row.update(grooves={f"{r},{c}": v for (r, c), v in grooves.items()}, g_tier=gr["tier"], g_trials=gr["trials"])
        if marks is not None:
            gr = grade(with_marks(g, marks))
            row.update(marks={f"{r},{c}": v for (r, c), v in marks.items()}, m_tier=gr["tier"], m_trials=gr["trials"])
        out.append(row)
    return out

if __name__ == "__main__":
    with Pool(2) as p:
        rows = [x for r in p.map(job, [2000, 2001]) for x in r]
    json.dump(rows, open("/home/claude/zen/out/marks_vs_grooves.json", "w"))
    both = [r for r in rows if "grooves" in r and "marks" in r]
    print(f"gardens {len(rows)}; grooves ok {sum('grooves' in r for r in rows)}; marks ok {sum('marks' in r for r in rows)}; both {len(both)}")
    if both:
        print("clues median: grooves", statistics.median(len(r["grooves"]) for r in both), "marks", statistics.median(len(r["marks"]) for r in both))
        for k in ("g", "m"):
            tiers = [r[f"{k}_tier"] for r in both]
            print(k, "tiers", {t: tiers.count(t) for t in sorted(set(tiers))})
src/exp_open.py · 13 lines · 690 bytes 
import time
from zg import Garden
for (H, W) in [(3,3),(3,4),(4,4),(4,5),(5,5),(5,6),(6,6)]:
    for gate in [((H-1, 0, "W"), "corner"), ((H-1, W//2, "S"), "mid-edge")]:
        g = Garden(["."*W]*H, gate[0])
        t0 = time.time()
        r = g.solve(mode=0, cap=10**12, timeout=3600)
        dt = time.time()-t0
        h = r['hist']; n = r['nsol']
        tmin = min(h) if h else None; tmax = max(h) if h else None
        frac_par = h[tmin]/n if n else 0
        hp = r['hphist']
        print(f"{H}x{W} {gate[1]:8s} n={n:>10} complete={r['complete']} minT={tmin} maxT={tmax} atPar={h.get(tmin,0)} ({frac_par:.4f}) zeroHairpin={hp.get(0,0)} nodes={r['nodes']} {dt:.1f}s", flush=True)
src/exp_open2.py · 12 lines · 639 bytes 
import time
from zg import Garden
for (H, W) in [(6,7),(7,7),(6,8)]:
    for gate in [((H-1, 0, "W"), "corner")]:
        g = Garden(["."*W]*H, gate[0])
        t0 = time.time()
        r = g.solve(mode=0, cap=10**12, timeout=3600)
        dt = time.time()-t0
        h = r['hist']; n = r['nsol']
        tmin = min(h) if h else None; tmax = max(h) if h else None
        print(f"{H}x{W} {gate[1]:8s} n={n:>10} complete={r['complete']} minT={tmin} maxT={tmax} atPar={h.get(tmin,0)} ({h.get(tmin,0)/n:.5f}) zeroHairpin={r['hphist'].get(0,0)} nodes={r['nodes']} {dt:.1f}s", flush=True)
        print('  hist', sorted(h.items()), flush=True)
src/exp_pipeline.py · 59 lines · 3.0 kB — Run the generation pipeline in bulk and record yield / difficulty statistics per mechanic. 
"""Run the generation pipeline in bulk and record yield / difficulty statistics per mechanic."""
import json, random, statistics, sys, time
from multiprocessing import Pool
from gen import random_layout, parity_ok
from zg import turns_of, hairpin_list, Garden
from zcp import optimize
from clues import make_unique, with_clues
from grader import grade

def grooves_job(seed):
    rng = random.Random(seed); out = []; t0 = time.time(); tried = 0
    while time.time() - t0 < float(sys.argv[2]):
        H, W = rng.choice([(7, 7), (7, 8)])
        g = random_layout(H, W, rng.choice([1, 2, 3]), rng, gate_side="S", min_gap=1)
        if g is None: continue
        tried += 1
        if not parity_ok(g) or g.solve(mode=0, cap=1)["nsol"] == 0: continue
        free = optimize(g, time_limit=30); grace = optimize(g, nohairpin=2, time_limit=30)
        if free[1] != "OPTIMAL" or grace[1] != "OPTIMAL" or grace[0] != free[0]: continue
        target = list(reversed(grace[2]))
        clues = make_unique(g, target, rng)
        if clues is None: continue
        gr = grade(with_clues(g, clues))
        out.append({"cells": len(target), "clues": len(clues), "tier": gr["tier"], "trials": gr["trials"],
                    "dens": turns_of(g, target) / len(target)})
    return out, tried

def wide_job(seed):
    rng = random.Random(seed); out = []; t0 = time.time(); tried = 0
    while time.time() - t0 < float(sys.argv[2]):
        H, W = rng.choice([(7, 7), (7, 8), (8, 8)])
        g = random_layout(H, W, rng.choice([1, 2, 3]), rng, gate_side="S", min_gap=1)
        if g is None: continue
        tried += 1
        if not parity_ok(g): continue
        g.nohairpin = 2; g.braced_start = True
        r = g.solve(mode=0, cap=2)
        if r["nsol"] != 1: continue
        d = r["sols"][0]["draw"]
        dens = turns_of(g, d) / len(d)
        gr = grade(g)
        out.append({"cells": len(d), "clues": 0, "tier": gr["tier"], "trials": gr["trials"], "dens": dens})
    return out, tried

if __name__ == "__main__":
    kind = sys.argv[1]
    fn = grooves_job if kind == "grooves" else wide_job
    with Pool(2) as p:
        res = p.map(fn, [1000 + i for i in range(2)])
    rows = [x for r, _ in res for x in r]; tried = sum(t for _, t in res)
    print(f"{kind}: layouts tried {tried}, levels {len(rows)} (yield {len(rows)/tried:.3f})")
    if not rows: sys.exit()
    for t in sorted(set(r['tier'] for r in rows)):
        sub = [r for r in rows if r['tier'] == t]
        print(f"  tier {t}: {len(sub)/len(rows):.2f}  median trials {statistics.median([r['trials'] or 0 for r in sub])}")
    print(f"  clues median {statistics.median(r['clues'] for r in rows)} mean {statistics.mean(r['clues'] for r in rows):.1f}")
    pretty = [r for r in rows if r['dens'] <= 0.40]
    print(f"  turn density median {statistics.median(r['dens'] for r in rows):.3f}; share with density<=0.40: {len(pretty)/len(rows):.2f}")
    json.dump(rows, open(f"/home/claude/zen/out/pipeline_{kind}.json", "w"))
src/exp_sampler.py · 84 lines · 4.0 kB — Yields of the garden-like stone sampler (garden_sampler.py) against the single-cell sampler used in all earlier 
"""Yields of the garden-like stone sampler (garden_sampler.py) against the single-cell sampler used in all earlier
experiments (gen.random_layout, 1-3 1x1 stones kept apart), on 7x7 gardens with the gate on the south wall:
  free:  parity, solvable, and par reachable with no U-turn in open sand ("graceful"); plus rakings at par;
  grooves: grooves added from the graceful par raking until unique (count, grade);
  wide:  unique under the wide rake (yield, grade);
  side:  unique under the wide rake with a random side gate (yield).
Writes out/sampler_yields.json."""
import json, random, statistics, sys, time
from multiprocessing import Pool
from gen import random_layout, parity_ok
from garden_sampler import garden_layout, add_side_gate
from zcp import optimize
from clues import make_unique, with_clues
from grader import grade

N = int(sys.argv[1]) if len(sys.argv) > 1 else 300


def layout(kind, rng):
    if kind == "single":
        return random_layout(7, 7, rng.choice([1, 2, 3]), rng, gate_side="S", min_gap=1)
    return garden_layout(7, 7, rng)


def job(args):
    kind, seed = args
    rng = random.Random(seed)
    out = {"kind": kind, "layouts": 0, "parity": 0, "solvable": 0, "graceful": 0, "at_par": [], "grooves": [], "g_tiers": [],
           "wide_unique": 0, "wide_tiers": [], "side_unique": 0, "stone_cells": []}
    while out["layouts"] < N:
        g = layout(kind, rng)
        if g is None:
            continue
        out["layouts"] += 1
        out["stone_cells"].append(sum(ch == "#" for row in g.rows for ch in row))
        if not parity_ok(g):
            continue
        out["parity"] += 1
        r = g.solve(mode=0, cap=10**6, nodelimit=5 * 10**8)
        if r["nsol"] == 0:
            continue
        out["solvable"] += 1
        free = optimize(g, time_limit=20); grace = optimize(g, nohairpin=2, time_limit=20)
        if free[1] == "OPTIMAL" and grace[1] == "OPTIMAL" and grace[0] == free[0]:
            out["graceful"] += 1
            if r["complete"] and not r["capped"]:
                out["at_par"].append(r["hist"].get(free[0], 0))
            if len(out["grooves"]) < 25:
                cl = make_unique(g, list(reversed(grace[2])), rng)
                if cl is not None:
                    out["grooves"].append(len(cl)); out["g_tiers"].append(grade(with_clues(g, cl))["tier"])
        gw = Garden_copy(g); gw.nohairpin = 2; gw.braced_start = True
        rw = gw.solve(mode=0, cap=2)
        if rw["nsol"] == 1:
            out["wide_unique"] += 1
            out["wide_tiers"].append(grade(gw)["tier"])
        gs = Garden_copy(gw)
        if add_side_gate(gs, rng) and gs.solve(mode=0, cap=2)["nsol"] == 1:
            out["side_unique"] += 1
    return out


def Garden_copy(g):
    from copy import deepcopy
    return deepcopy(g)


if __name__ == "__main__":
    t0 = time.time()
    with Pool(2) as p:
        res = p.map(job, [("single", 11), ("garden", 12)])
    summary = {}
    for o in res:
        k = o["kind"]; n = o["layouts"]
        summary[k] = {"layouts": n, "parity_ok": round(o["parity"] / n, 3), "solvable": round(o["solvable"] / n, 3),
                      "graceful_free": round(o["graceful"] / n, 3), "median_at_par": statistics.median(o["at_par"]) if o["at_par"] else None,
                      "median_grooves": statistics.median(o["grooves"]) if o["grooves"] else None,
                      "grooves_lookahead": round(sum(t >= 3 for t in o["g_tiers"]) / max(1, len(o["g_tiers"])), 2),
                      "wide_unique": round(o["wide_unique"] / n, 3),
                      "wide_nested": round(sum(t >= 4 for t in o["wide_tiers"]) / max(1, len(o["wide_tiers"])), 2),
                      "side_unique": round(o["side_unique"] / n, 3), "mean_stone_cells": round(statistics.mean(o["stone_cells"]), 2)}
        print(k, summary[k], flush=True)
    json.dump({"summary": summary, "raw": res}, open("/home/claude/zen/out/sampler_yields.json", "w"))
    print(f"{time.time() - t0:.0f}s")
src/exp_variants.py · 40 lines · 1.7 kB 
import random, sys, time, statistics, json
from multiprocessing import Pool
from gen import random_layout, parity_ok
from zg import Garden

VARIANTS = {
    "plain": dict(nohairpin=0, braced_turns=False),
    "nohairpin": dict(nohairpin=1, braced_turns=False),
    "braced_hairpin": dict(nohairpin=2, braced_turns=False),
    "braced_turns": dict(nohairpin=0, braced_turns=True),
}
CAP = 200000

def job(args):
    seed, H, W, k = args
    rng = random.Random(seed)
    g = random_layout(H, W, k, rng, gate_side="S")
    if g is None: return None
    out = {"parity": parity_ok(g)}
    for name, kw in VARIANTS.items():
        g.nohairpin = kw["nohairpin"]; g.braced_turns = kw["braced_turns"]
        r = g.solve(mode=0, cap=CAP, nodelimit=3 * 10**8)
        out[name] = (r["nsol"], r["complete"], r["capped"])
    return out

if __name__ == "__main__":
    H, W, k, n = int(sys.argv[1]), int(sys.argv[2]), int(sys.argv[3]), int(sys.argv[4])
    t0 = time.time()
    with Pool(2) as p:
        res = [r for r in p.map(job, [(s, H, W, k) for s in range(n)]) if r]
    print(f"== {H}x{W}, {k} stones, gate on S edge, {len(res)} layouts ({time.time()-t0:.0f}s)")
    print(f"   parity-feasible: {sum(r['parity'] for r in res)/len(res):.2f}")
    for name in VARIANTS:
        counts = [r[name][0] for r in res]
        capped = sum(r[name][2] for r in res)
        solv = [c for c in counts if c > 0]
        uniq = sum(c == 1 for c in counts)
        few = sum(1 <= c <= 10 for c in counts)
        med = statistics.median(solv) if solv else 0
        print(f"   {name:15s} solvable={len(solv)/len(res):.3f} unique={uniq/len(res):.3f} 1-10 sols={few/len(res):.3f} median#(solvable)={med:g} capped@{CAP}={capped}")
src/exp_variants2.py · 38 lines · 1.5 kB 
import random, sys, time, statistics
from multiprocessing import Pool
from gen import random_layout, parity_ok

VARIANTS = {
    "plain": dict(nohairpin=0, braced_start=False),
    "braced_hp": dict(nohairpin=2, braced_start=False),
    "wide_rake": dict(nohairpin=2, braced_start=True),
}
CAP = 200000

def job(args):
    seed, H, W, k = args
    rng = random.Random(seed * 7919 + k)
    g = random_layout(H, W, k, rng, gate_side="S")
    if g is None: return None
    out = {}
    for name, kw in VARIANTS.items():
        g.nohairpin = kw["nohairpin"]; g.braced_start = kw["braced_start"]
        r = g.solve(mode=0, cap=CAP, nodelimit=3 * 10**8)
        out[name] = (r["nsol"], r["capped"])
    return out

if __name__ == "__main__":
    H, W, n = int(sys.argv[1]), int(sys.argv[2]), int(sys.argv[3])
    for k in [0, 1, 2, 3, 4, 6]:
        t0 = time.time()
        with Pool(2) as p:
            res = [r for r in p.map(job, [(s, H, W, k) for s in range(n)]) if r]
        line = f"{H}x{W} k={k} ({time.time()-t0:.0f}s):"
        for name in VARIANTS:
            counts = [r[name][0] for r in res]
            solv = [c for c in counts if c > 0]
            uniq = sum(c == 1 for c in counts)
            few = sum(1 <= c <= 20 for c in counts)
            med = statistics.median(solv) if solv else 0
            line += f"  {name}: solv={len(solv)/len(res):.3f} uniq={uniq/len(res):.3f} 1-20={few/len(res):.3f} med={med:g}"
        print(line, flush=True)
src/exp_wide_par.py · 23 lines · 1.3 kB — How often is a wide-rake-unique garden's only raking also a par raking of the bare garden? 
"""How often is a wide-rake-unique garden's only raking also a par raking of the bare garden?"""
import random, time, statistics, sys
from gen import random_layout, parity_ok
from zg import Garden, turns_of
from zcp import optimize
rng = random.Random(int(sys.argv[1]))
rows = []; t0 = time.time()
while time.time() - t0 < float(sys.argv[2]):
    H, W = rng.choice([(7, 7), (7, 8), (8, 8)])
    g = random_layout(H, W, rng.choice([1, 2, 3]), rng, gate_side="S", min_gap=1)
    if g is None or not parity_ok(g): continue
    g.nohairpin = 2; g.braced_start = True
    r = g.solve(mode=0, cap=2)
    if r["nsol"] != 1: continue
    t_wide = r["sols"][0]["turns"]
    p = optimize(Garden(g.rows, g.gate), time_limit=60, workers=2)
    if p[1] != "OPTIMAL": continue
    rows.append((t_wide, p[0], t_wide / len(r["sols"][0]["draw"])))
gaps = [a - b for a, b, _ in rows]
print(f"n={len(rows)}; wide answer == bare par: {sum(g == 0 for g in gaps)/len(rows):.2f}; within 1: {sum(g <= 1 for g in gaps)/len(rows):.2f}; within 2: {sum(g <= 2 for g in gaps)/len(rows):.2f}; median gap {statistics.median(gaps)}")
calm = [r for r in rows if r[2] <= 0.40]
cg = [a - b for a, b, _ in calm]
print(f"among turn density <= 0.40 (n={len(calm)}): == par {sum(g == 0 for g in cg)/max(1,len(calm)):.2f}; within 1: {sum(g <= 1 for g in cg)/max(1,len(calm)):.2f}")
src/export_proto.py · 99 lines · 6.2 kB — Export the campaign (out/campaign.json, from campaign.py) for the prototype, re-deriving and checking everything: 
"""Export the campaign (out/campaign.json, from campaign.py) for the prototype, re-deriving and checking everything:
each garden's raking count (exact solver), par (CP-SAT), rakings at par, every par raking (as move strings, for the
in-page hints), an intended raking (asserted valid and at par), and the first hints marked "auto" (hints.py)."""
import json
from zg import Garden, turns_of, check_solution
from zcp import optimize
from levels import from_dict, to_dict
import copy
from hints import hint_parity, far_cell, best_free_hint, best_strict_hint, near_misses, with_fact, first_dir, STRAIGHT_IN, gate_word
from grader import grade
from clues import with_marks

CAMPAIGN = json.load(open('/home/claude/zen/out/campaign.json'))
MOVE = {(-1, 0): "N", (1, 0): "S", (0, 1): "E", (0, -1): "W"}


def moves_of(d):
    return "".join(MOVE[(b[0] - a[0], b[1] - a[1])] for a, b in zip(d, d[1:]))


gardens = []


HINT_STATS = []

# the gardens that bring in something new are first raked once by the monk, the calm way, and then reset for the player:
# the first garden (what raking is), the first one scored by turns, and the introduction of each later rule (the second
# garden of each contrasting pair, and the two rules introduced mid-world without a pair)
DEMO = {"The corridor", "Nine squares", "Ichimatsu", "The finished ring"}


def demo_of(name, role, pair, extra):
    return not extra and (name in DEMO or (role == "intro" and pair is not None))


def add(name, world, g, intro=None, hint=None, tone=0.0, hint_cells=None, role=None, trap=0.0, extra=False, pair=None):
    r = g.solve(mode=0, cap=10**9, nodelimit=4 * 10**9)
    assert r['nsol'] > 0 and r['complete'] and not r['capped'], name
    free = optimize(g, time_limit=120)
    assert free[1] == 'OPTIMAL', name
    par = free[0]
    at_par = r['hist'].get(par, 0)  # how many of the valid rakings reach par (exact: the count is complete)
    assert at_par > 0, name
    # intended raking: graceful par if available, else par
    grace = optimize(g, nohairpin=2, time_limit=120) if not g.nohairpin else free
    intended = list(reversed(grace[2])) if (grace[1] == 'OPTIMAL' and grace[0] == par) else list(reversed(free[2]))
    assert check_solution(g, intended) and turns_of(g, intended) == par, name
    # every par raking (the in-page hint decides "can this route still reach par?" exactly, by prefix)
    rp = g.solve(mode=2, bound=par, cap=10**6, stream=True)
    par_rakings = [s_["draw"] for s_ in rp["sols"]]
    assert rp["complete"] and not rp["capped"] and len(par_rakings) == rp["nsol"] == at_par, name
    assert all(check_solution(g, d_) and turns_of(g, d_) == par for d_ in par_rakings), name
    par_rakings.sort(key=lambda d_: d_ != intended)          # the intended raking first
    assert par_rakings[0] == intended, name
    if role == "trap" and r['nsol'] == 1:
        # a strict trap teaches its trap: the only raking turns aside at once, though walking straight in looks fine
        gate = (g.gate[0], g.gate[1])
        assert first_dir(intended) != STRAIGHT_IN[g.gate[2]], name
        hint, hint_cells = f"Walking straight in from the {gate_word(g)} goes wrong here: the line turns at once.", [list(gate)]
        before, after = grade(g), grade(with_fact(g, hint_cells, hint))
        assert after["solved"], name
        HINT_STATS.append({"name": name, "kind": "strict", "trap": True, "tier": before["tier"], "tier_after": after["tier"],
                           "trials": before["trials"] or 0, "trials_after": after["trials"] or 0})
    elif hint in (None, "auto", "auto-parity"):
        if hint == "auto-parity":
            hint = hint_parity(g, [s_["draw"] for s_ in g.solve(mode=0, cap=10**6, stream=True)["sols"]])
            hint_cells = [list(far_cell(g))]
        elif r['nsol'] == 1:
            # the fact about the unique raking that lets the grader's tier 1-2 rules decide the most; with it as a
            # given, how hard is the garden still?
            hint, hint_cells, (d0, d1) = best_strict_hint(g, intended)
            before = grade(g)
            after = grade(with_fact(g, hint_cells, hint)) if hint_cells else before
            assert after["solved"], name
            HINT_STATS.append({"name": name, "kind": "strict", "tier": before["tier"], "tier_after": after["tier"],
                               "trials": before["trials"] or 0, "trials_after": after["trials"] or 0, "decided": [d0, d1]})
        else:
            near = near_misses(g, par)
            hint, hint_cells, miss = best_free_hint(g, par, par_rakings, near)
            HINT_STATS.append({"name": name, "kind": "free", "near": len(near), "miss": miss})
    d = to_dict(g)
    gardens.append({"name": name, "world": world, "rows": g.rows, "gate": list(g.gate), "end": list(g.end) if g.end else None,
                    "end_side": g.end_side, "clues": d["clues"], "wide": g.nohairpin == 2, "par": par,
                    "rakings": r['nsol'], "at_par": at_par, "intended": [list(c) for c in intended], "intro": intro,
                    "marks": d["marks"], "par_moves": [moves_of(d_) for d_ in par_rakings], "hint": hint, "tone": tone,
                    "hint_cells": hint_cells or [], "role": role, "trap": trap, "extra": bool(extra), "pair": pair,
                    "demo": demo_of(name, role, pair, extra)})
    print(f"{world} {name:22s} {g.H}x{g.W} rakings={r['nsol']:>9,} par={par:2d} at_par={at_par:3d}  {hint}", flush=True)


for e in CAMPAIGN:
    add(e["name"], e["world"], from_dict(e["garden"]), e.get("intro"), e.get("hint"), e.get("tone", 0.0),
        e.get("hint_cells"), e.get("role"), e.get("trap", 0.0), e.get("extra", False), e.get("pair"))
assert len({g["name"] for g in gardens}) == len(gardens), "duplicate names"
json.dump(gardens, open('/home/claude/zen/out/proto_gardens.json', 'w'))
json.dump(HINT_STATS, open('/home/claude/zen/out/hint_stats.json', 'w'), indent=1)
assert all(g["name"] in {x["name"] for x in gardens} for g in [{"name": n} for n in DEMO]), "a demonstration garden is missing"
print(len(gardens), "gardens exported;", sum(not g["extra"] for g in gardens), "on the main path;",
      sum(g["demo"] for g in gardens), "with a demonstration:", ", ".join(g["name"] for g in gardens if g["demo"]))
src/fig_pebbles.py · 36 lines · 2.9 kB — The abbot's pebbles figure (same garden and answer: grooves vs pebbles), plus the marks-vs-grooves 
"""fig_pebbles.py: the abbot's pebbles figure (same garden and answer: grooves vs pebbles), plus the marks-vs-grooves
statistics quoted in the document (from out/marks_vs_grooves.json, written by exp_marks.py)."""
import json, os, statistics
from zg import Garden, turns_of, check_solution
from levels import from_dict
from grader import grade
from render import garden_svg, hstack, save

FIG = "/home/claude/zen/fig/final"; PREFIX = "2026-10-03-gwern-samon-"
mp = json.load(open("/home/claude/zen/out/marks_pick.json"))
gm = from_dict(mp["garden"])
draw = [tuple(c) for c in mp["draw"]]
grooves = {tuple(map(int, k.split(","))): v for k, v in mp["grooves"].items()}
gg = Garden(gm.rows, gm.gate, clues=grooves)
plain = Garden(gm.rows, gm.gate)
rm, rg, rp = gm.solve(mode=0, cap=5), gg.solve(mode=0, cap=5), plain.solve(mode=0, cap=10**8)
assert rm["nsol"] == 1 and rg["nsol"] == 1 and check_solution(gm, draw) and check_solution(gg, draw)
grm, grg = grade(gm), grade(gg)
facts = {"pebbles": len(gm.marks), "pale": sum(v == "S" for v in gm.marks.values()), "dark": sum(v == "B" for v in gm.marks.values()),
         "grooves": len(grooves), "plain_rakings": rp["nsol"], "par": min(rp["hist"]), "at_par": rp["hist"][min(rp["hist"])],
         "answer_turns": turns_of(gm, draw), "tier_pebbles": grm["tier"], "trials_pebbles": grm["trials"], "tier_grooves": grg["tier"]}
panels = [garden_svg(gg, label=f"{len(grooves)} grooves", sublabel=f"tier {grg['tier']}", seed=5),
          garden_svg(gm, label=f"{len(gm.marks)} pebbles", sublabel=f"tier {grm['tier']} ({grm['trials']} lookahead steps)", seed=5),
          garden_svg(gm, draw=draw, label="The only raking", sublabel=f"{facts['answer_turns']} turns = par", seed=5)]
save(hstack(panels), os.path.join(FIG, PREFIX + "pebbles.png"), os.path.join(FIG, PREFIX + "pebbles.svg"))
rows = json.load(open("/home/claude/zen/out/marks_vs_grooves.json"))
ng, nm = [len(r["grooves"]) for r in rows], [len(r["marks"]) for r in rows]
gt, mt = [r["g_tier"] for r in rows], [r["m_tier"] for r in rows]
facts["experiment"] = {"gardens": len(rows), "grooves_median": statistics.median(ng), "pebbles_median": statistics.median(nm),
    "share_more_pebbles": round(sum(b > a for a, b in zip(ng, nm)) / len(rows), 3),
    "grooves_lookahead_share": round(sum(t >= 3 for t in gt) / len(rows), 3), "pebbles_lookahead_share": round(sum(t >= 3 for t in mt) / len(rows), 3),
    "grooves_nested_share": round(sum(t >= 4 for t in gt) / len(rows), 3), "pebbles_nested_share": round(sum(t >= 4 for t in mt) / len(rows), 3),
    "grooves_median_trials": statistics.median([r["g_trials"] for r in rows if r["g_tier"] >= 3]),
    "pebbles_median_trials": statistics.median([r["m_trials"] for r in rows if r["m_tier"] >= 3])}
json.dump(facts, open("/home/claude/zen/out/facts_pebbles.json", "w"), indent=1)
print(json.dumps(facts, indent=1))
src/final_figs.py · 233 lines · 12.0 kB — Render all figures for the design document and record every number they show. 
"""final_figs.py: render all figures for the design document and record every number they show."""
import json
import os
import statistics

from zg import Garden, glyphs, DIRS, DR, DC, OPP, turns_of, check_solution
from levels import from_dict, to_dict
from render import garden_svg, hstack, save, SEAL, INK, S
from zcp import optimize

FIG = "/home/claude/zen/fig/final"
os.makedirs(FIG, exist_ok=True)
PREFIX = "2026-10-03-gwern-samon-"
L = json.load(open("/home/claude/zen/out/levels.json"))
facts = {}


def out(name):
    return os.path.join(FIG, PREFIX + name + ".png")


def svgout(name):
    return os.path.join(FIG, PREFIX + name + ".svg")


def draw_of(lv):
    return [tuple(c) for c in lv["intended"]]


# ------------------------------------------------------------------ 1. anatomy
lv = L["first"]
g = from_dict(lv["garden"])
draw = draw_of(lv)
rake = list(reversed(draw))
gl = glyphs(g, draw)
k = int(len(rake) * 0.45)
raked_cells = rake[:k]
partial = {c: gl[c] for c in raked_cells}
foot_prefix = draw[: len(draw) - k]  # cells still showing footprints (gate ... monk position)
panels = [
    garden_svg(g, label="1. the garden", seed=2),
    garden_svg(g, footprints=draw, monk=draw[-1], label="2. walk in", seed=2),
    garden_svg(g, glyph_override=partial, footprints=foot_prefix, monk=rake[k - 1], label="3. rake back out", seed=2),
    garden_svg(g, draw=draw, label=f"4. raked: {turns_of(g, draw)} turns", seed=2),
]
sv = hstack(panels, gap=14)
save(sv, out("anatomy"), svgout("anatomy"))
facts["anatomy"] = {"rows": g.rows, "gate": g.gate, "turns": turns_of(g, draw), "par": lv["par"],
                    "n_rakings": lv["n_rakings"], "at_par": lv["at_par"]}

# ------------------------------------------------------------------ 2. calm vs agitated
lv = L["calm"]
g = from_dict(lv["garden"])
par_draw = draw_of(lv)
r = g.solve(mode=0, cap=10**9)
hist = {int(t): n for t, n in r["hist"].items()}
total = sum(hist.values())
# median turn count over all valid rakings
acc = 0
median_t = None
for t in sorted(hist):
    acc += hist[t]
    if acc >= total / 2:
        median_t = t
        break
mid = optimize(g, objective=None, fix={"turns_eq": median_t}, time_limit=60)
mid_draw = list(reversed(mid[2]))
worst_draw = r["worst"]["draw"]
assert check_solution(g, mid_draw) and check_solution(g, worst_draw) and check_solution(g, par_draw)
panels = [
    garden_svg(g, draw=par_draw, label=f"par: {turns_of(g, par_draw)} turns", seed=5),
    garden_svg(g, draw=mid_draw, label=f"median: {turns_of(g, mid_draw)} turns", seed=5),
    garden_svg(g, draw=worst_draw, label=f"most agitated: {turns_of(g, worst_draw)} turns", seed=5),
]
save(hstack(panels, gap=14), out("calm-agitated"), svgout("calm-agitated"))
facts["calm"] = {"rows": g.rows, "gate": g.gate, "n_rakings": total, "par": min(hist), "at_par": hist[min(hist)],
                 "median_turns": median_t, "max_turns": max(hist), "at_max": hist[max(hist)],
                 "mode_turns": max(hist, key=hist.get), "hist": hist}

# ------------------------------------------------------------------ 3. histogram (static figure; single series + one emphasised bar)
W_, H_ = 760, 330
ml, mr, mt, mb = 70, 24, 24, 58
pw, ph = W_ - ml - mr, H_ - mt - mb
ts = list(range(min(hist), max(hist) + 1))
ymax = 10000
slot = pw / len(ts)
bw = min(24, slot - 2)
el = [f'<svg xmlns="http://www.w3.org/2000/svg" width="{W_}" height="{H_}" viewBox="0 0 {W_} {H_}">',
      f'<rect width="{W_}" height="{H_}" fill="#ffffff"/>']
for yv in range(0, ymax + 1, 2000):
    y = mt + ph - ph * yv / ymax
    el.append(f'<line x1="{ml}" y1="{y:.1f}" x2="{ml + pw}" y2="{y:.1f}" stroke="#e4e4e1" stroke-width="1"/>')
    el.append(f'<text x="{ml - 8}" y="{y + 4:.1f}" font-family="Helvetica, Arial, sans-serif" font-size="12" text-anchor="end" fill="#52514e">{yv:,}</text>')
for i, t in enumerate(ts):
    n = hist.get(t, 0)
    x = ml + i * slot + (slot - bw) / 2
    h = ph * n / ymax
    y = mt + ph - h
    color = SEAL if t == min(hist) else "#8e9aa6"
    if h > 0:
        rr = min(4, h, bw / 2)
        # rounded data end, square at the baseline
        el.append(f'<path d="M{x:.1f},{mt + ph:.1f} V{y + rr:.1f} Q{x:.1f},{y:.1f} {x + rr:.1f},{y:.1f} H{x + bw - rr:.1f} '
                  f'Q{x + bw:.1f},{y:.1f} {x + bw:.1f},{y + rr:.1f} V{mt + ph:.1f} Z" fill="{color}"/>')
    if t % 5 == 0:
        el.append(f'<text x="{x + bw / 2:.1f}" y="{mt + ph + 18}" font-family="Helvetica, Arial, sans-serif" font-size="12" text-anchor="middle" fill="#52514e">{t}</text>')
el.append(f'<line x1="{ml}" y1="{mt + ph}" x2="{ml + pw}" y2="{mt + ph}" stroke="#9a9a96" stroke-width="1"/>')
el.append(f'<text x="{ml + pw / 2}" y="{H_ - 12}" font-family="Helvetica, Arial, sans-serif" font-size="13" text-anchor="middle" fill="#0b0b0b">turns in the raking</text>')
el.append(f'<text x="18" y="{mt + ph / 2}" font-family="Helvetica, Arial, sans-serif" font-size="13" text-anchor="middle" fill="#0b0b0b" transform="rotate(-90 18 {mt + ph / 2})">number of valid rakings</text>')
# par annotation with leader
i0 = ts.index(min(hist))
px = ml + i0 * slot + slot / 2
el.append(f'<circle cx="{px:.1f}" cy="{mt + ph - 3:.1f}" r="4" fill="{SEAL}" stroke="#ffffff" stroke-width="2"/>')
el.append(f'<line x1="{px:.1f}" y1="{mt + ph - 9:.1f}" x2="{px:.1f}" y2="{mt + 108:.1f}" stroke="#52514e" stroke-width="1"/>')
el.append(f'<text x="{px - 4:.1f}" y="{mt + 82:.1f}" font-family="Helvetica, Arial, sans-serif" font-size="12.5" fill="#0b0b0b">par: {min(hist)} turns</text>')
el.append(f'<text x="{px - 4:.1f}" y="{mt + 98:.1f}" font-family="Helvetica, Arial, sans-serif" font-size="12.5" fill="#52514e">{hist[min(hist)]} of {total:,} rakings</text>')
im = ts.index(median_t)
mx = ml + im * slot + slot / 2
my = mt + ph - ph * hist[median_t] / ymax
el.append(f'<text x="{mx + 16:.1f}" y="{my + 4:.1f}" font-family="Helvetica, Arial, sans-serif" font-size="12.5" fill="#52514e">median: {median_t} turns</text>')
el.append("</svg>")
save("\n".join(el), out("histogram"), svgout("histogram"))

# ------------------------------------------------------------------ 4. parity
g_ok = Garden(["....."] * 5, (4, 0, "S"))
g_bad = Garden(["....."] * 5, (4, 1, "S"))
r_ok = g_ok.solve(mode=0, cap=10**9)
r_bad = g_bad.solve(mode=0, cap=10**9)
best_ok = optimize(g_ok)
panels = [garden_svg(g_ok, draw=list(reversed(best_ok[2])), checker=True, label=f"gate on a dark square: {r_ok['nsol']} rakings", seed=1),
          garden_svg(g_bad, checker=True, label=f"gate on a light square: {r_bad['nsol']} rakings", seed=1)]
save(hstack(panels, gap=14), out("parity"), svgout("parity"))
facts["parity"] = {"ok_rakings": r_ok["nsol"], "bad_rakings": r_bad["nsol"]}


# ------------------------------------------------------------------ 5-8, 10-11: puzzle | solution pairs
def pair(key_or_garden, draw, name, left="the garden", right=None, seed=3, g=None):
    g = g or from_dict(L[key_or_garden]["garden"])
    right = right or f"its raking: {turns_of(g, draw)} turns"
    sv = hstack([garden_svg(g, label=left, seed=seed), garden_svg(g, draw=draw, label=right, seed=seed)], gap=14)
    save(sv, out(name), svgout(name))
    return g


for key, name in [("ripple", "ripple"), ("waves", "waves"), ("grooves", "grooves")]:
    lv = L[key]
    g = pair(key, draw_of(lv), name,
             right=(f"unique raking: {lv['intended_turns']} turns" if lv["n_rakings"] == 1 else
                    f"par raking: {lv['intended_turns']} turns"))
    facts[key] = {k2: lv[k2] for k2 in ("n_rakings", "par", "at_par", "graceful_par", "intended_turns", "grade") if k2 in lv}
    facts[key]["rows"] = g.rows
    facts[key]["gate"] = g.gate
    facts[key]["clues"] = len(g.clues)

# ichimatsu 9x9
gi = Garden(["".join("=" if ((r // 3) + (c // 3)) % 2 == 0 else "|" for c in range(9)) for r in range(9)], (8, 0, "S"))
ri = gi.solve(mode=0, cap=100)
pair(None, ri["sols"][0]["draw"], "ichimatsu", g=gi, right=f"one of its {ri['nsol']} rakings: {ri['sols'][0]['turns']} turns")
facts["ichimatsu"] = {"n_rakings": ri["nsol"], "turns": [s["turns"] for s in ri["sols"]], "rows": gi.rows}

# wide rake: chosen candidate
wide = json.load(open("/home/claude/zen/out/wide_pick.json"))
gw = from_dict(wide["garden"])
dw = [tuple(c) for c in wide["draw"]]
assert check_solution(gw, dw)
assert gw.solve(mode=0, cap=2)["nsol"] == 1
gw_free = Garden(gw.rows, gw.gate)
rf = gw_free.solve(mode=0, cap=10**9, nodelimit=3 * 10**9)
pair(None, dw, "widerake", g=gw, right=f"its only raking: {turns_of(gw, dw)} turns")
facts["widerake"] = {"rows": gw.rows, "gate": gw.gate, "turns": turns_of(gw, dw), "grade": wide.get("grade"),
                     "plain_rakings": rf["nsol"], "plain_complete": rf["complete"] and not rf["capped"],
                     "plain_par": min(rf["hist"]) if rf["hist"] else None}

# two gates
two = json.load(open("/home/claude/zen/out/two_pick.json"))
gt = from_dict(two["garden"])
dt = [tuple(c) for c in two["draw"]]
assert check_solution(gt, dt) and gt.solve(mode=0, cap=2)["nsol"] == 1
pair(None, dt, "twogates", g=gt, right=f"its only raking: {turns_of(gt, dt)} turns")
facts["twogates"] = {"rows": gt.rows, "gate": gt.gate, "end": gt.end, "end_side": gt.end_side,
                     "turns": turns_of(gt, dt), "grade": two.get("grade")}


# ------------------------------------------------------------------ 9. the wide-rake rule
def frag_glyphs(cells, ext_start, ext_end):
    """Glyphs along a fragment; ends extended straight in the given directions."""
    gl = {}
    for i, c in enumerate(cells):
        ds = set()
        if i > 0:
            p = cells[i - 1]
            ds.add([d for d in DIRS if (c[0] + DR[d], c[1] + DC[d]) == p][0])
        else:
            ds.add(ext_start)
        if i + 1 < len(cells):
            n = cells[i + 1]
            ds.add([d for d in DIRS if (c[0] + DR[d], c[1] + DC[d]) == n][0])
        else:
            ds.add(ext_end)
        gl[c] = "".join(ds)
    return gl


def cross(ox, oy, r, c, ok):
    x, y = ox + c * S + S / 2, oy + r * S + S / 2
    if ok:
        return f'<path d="M{x - 9:.1f},{y:.1f} L{x - 2:.1f},{y + 7:.1f} L{x + 10:.1f},{y - 8:.1f}" fill="none" stroke="{INK}" stroke-width="3" stroke-linecap="round"/>'
    return (f'<path d="M{x - 8:.1f},{y - 8:.1f} L{x + 8:.1f},{y + 8:.1f} M{x + 8:.1f},{y - 8:.1f} L{x - 8:.1f},{y + 8:.1f}" '
            f'stroke="{SEAL}" stroke-width="3.2" stroke-linecap="round"/>')


ga = Garden(["......"] * 5, (4, 0, "S"))
fa = [(1, 0), (1, 1), (1, 2), (1, 3), (2, 3), (2, 2), (2, 1), (2, 0)]
gb = Garden(["...."] * 5, (4, 0, "S"))
fb = [(1, 0), (1, 1), (1, 2), (1, 3), (2, 3), (2, 2), (2, 1), (2, 0)]
gc = Garden(["......", "......", "...#..", "......", "......"], (4, 0, "S"))
fc = [(1, 0), (1, 1), (1, 2), (1, 3), (1, 4), (2, 4), (3, 4), (3, 3), (3, 2), (3, 1), (3, 0)]
panels = [
    garden_svg(ga, glyph_override=frag_glyphs(fa, "W", "W"), show_gate_arrow=False,
               marks_svg=lambda ox, oy: cross(ox, oy, 1.5, 4.3, False), label="open sand: forbidden", seed=7, show_gate=False),
    garden_svg(gb, glyph_override=frag_glyphs(fb, "W", "W"), show_gate_arrow=False,
               marks_svg=lambda ox, oy: cross(ox, oy, 3.5, 2.0, True), label="against a wall: allowed", seed=7, show_gate=False),
    garden_svg(gc, glyph_override=frag_glyphs(fc, "W", "W"), show_gate_arrow=False,
               marks_svg=lambda ox, oy: cross(ox, oy, 2, 5, True), label="around a stone: allowed", seed=7, show_gate=False),
]
save(hstack(panels, gap=14), out("widerake-rule"), svgout("widerake-rule"))

json.dump(facts, open("/home/claude/zen/out/facts.json", "w"), indent=1, default=str)
print(json.dumps({k: {kk: vv for kk, vv in v.items() if kk not in ("hist", "rows")} for k, v in facts.items()}, indent=1, default=str))

# ------------------------------------------------------------------ thumbnail: the wide-rake garden, raked
sv = garden_svg(gw, draw=dw, seed=3, highlight_start=False)
save(sv, out("thumbnail"), svgout("thumbnail"), scale=2.5)
src/final_tables.py · 63 lines · 2.7 kB — Final numbers for the document's tables (run in the background; prints Markdown-ready rows). 
"""Final numbers for the document's tables (run in the background; prints Markdown-ready rows)."""
import random, statistics, sys, time, json
from multiprocessing import Pool
from zg import Garden
from gen import random_layout, parity_ok

def median_from_hist(h):
    tot = sum(h.values()); acc = 0
    for t in sorted(h):
        acc += h[t]
        if acc >= tot / 2: return t

def table1():
    rows = []
    for (H, W) in [(3, 3), (4, 4), (5, 5), (6, 6), (7, 7), (6, 8)]:
        g = Garden(["." * W] * H, (H - 1, 0, "S"))
        r = g.solve(mode=0, cap=10**12)
        h = r["hist"]
        gw = Garden(["." * W] * H, (H - 1, 0, "S"), nohairpin=2, braced_start=True)
        rw = g.solve(mode=0, cap=1)  # placeholder to keep API symmetric
        rw = gw.solve(mode=0, cap=10**9)
        rows.append(dict(size=f"{H}x{W}", rakings=r["nsol"], complete=r["complete"], par=min(h), at_par=h[min(h)],
                         median=median_from_hist(h), max=max(h), wide=rw["nsol"],
                         wide_turns=sorted(rw["hist"].items())))
        print(rows[-1], flush=True)
    json.dump(rows, open("/home/claude/zen/out/table1.json", "w"))

RULES = {"one stroke": dict(nohairpin=0, braced_start=False),
         "no U-turns at all": dict(nohairpin=1, braced_start=False),
         "wide rake": dict(nohairpin=2, braced_start=True)}
CAP = 100000

def job(args):
    seed, k = args
    rng = random.Random(seed * 1009 + k)
    g = random_layout(7, 7, k, rng, gate_side="S")
    if g is None: return None
    out = {"parity": parity_ok(g)}
    for name, kw in RULES.items():
        g.nohairpin = kw["nohairpin"]; g.braced_start = kw["braced_start"]
        r = g.solve(mode=0, cap=CAP, nodelimit=5 * 10**8)
        out[name] = (r["nsol"], r["capped"])
    return out

def table2(n):
    res_all = {}
    for k in [0, 2, 4, 6]:
        with Pool(2) as p:
            res = [r for r in p.map(job, [(s, k) for s in range(n)]) if r]
        res_all[k] = res
        line = f"k={k} n={len(res)} parity-ok={sum(r['parity'] for r in res)/len(res):.2f}"
        for name in RULES:
            counts = [r[name][0] for r in res]
            solv = [c for c in counts if c > 0]
            capped = sum(1 for r in res if r[name][1])
            med = statistics.median(solv) if solv else 0
            line += f" | {name}: solvable {len(solv)/len(res):.3f} unique {sum(c == 1 for c in counts)/len(res):.3f} median {med:g} capped {capped}"
        print(line, flush=True)
    json.dump({k: v for k, v in res_all.items()}, open("/home/claude/zen/out/table2.json", "w"))

if __name__ == "__main__":
    if sys.argv[1] == "1": table1()
    else: table2(int(sys.argv[2]))
src/garden_sampler.py · 160 lines · 6.2 kB — Garden-like layouts for the campaign. 
"""garden_sampler.py: garden-like layouts for the campaign.

Stones come in an odd number of pieces (1, 3 or 5), each piece 1-3 cells (single, I2, I3 or L3), kept apart
(no two pieces within one cell of each other, diagonals included), placed so the layout has no left-right,
top-bottom or 180-degree symmetry (karesansui prize asymmetry, fukinsei), and covering at most ~15% of the garden.
Optional features: a ripple stone (O, with its 8-cell ring of sand), a wave band (h/v rectangle), a closed ring
(a stone whose 8 neighbours are already raked: R cells, which are obstacles to the stroke), and a side gate.
"""
from __future__ import annotations

import random

from zg import Garden

PIECES = [  # (weight, cells)
    (9, [(0, 0)]),
    (4, [(0, 0), (0, 1)]), (4, [(0, 0), (1, 0)]),
    (1, [(0, 0), (0, 1), (0, 2)]), (1, [(0, 0), (1, 0), (2, 0)]),
    (1, [(0, 0), (1, 0), (1, 1)]), (1, [(0, 0), (0, 1), (1, 0)]), (1, [(0, 0), (0, 1), (1, 1)]), (1, [(0, 1), (1, 0), (1, 1)]),
]


def _free(rows, cells, H, W, gap):
    for r, c in cells:
        if not (0 <= r < H and 0 <= c < W) or rows[r][c] != ".":
            return False
        for dr in range(-gap, gap + 1):
            for dc in range(-gap, gap + 1):
                a, b = r + dr, c + dc
                if 0 <= a < H and 0 <= b < W and rows[a][b] in "#OR":
                    return False
    return True


def symmetric(rows):
    """Stones (any of #, O, R) symmetric under a left-right or top-bottom mirror, or a half turn."""
    H, W = len(rows), len(rows[0])
    S = {(r, c) for r in range(H) for c in range(W) if rows[r][c] in "#OR"}
    if not S:
        return True
    return any(S == {f(r, c) for r, c in S} for f in (lambda r, c: (r, W - 1 - c), lambda r, c: (H - 1 - r, c),
                                                       lambda r, c: (H - 1 - r, W - 1 - c)))


def add_ripple(rows, rng, H, W):
    for _ in range(200):
        r, c = rng.randint(1, H - 2), rng.randint(1, W - 2)
        block = [(r + a, c + b) for a in (-1, 0, 1) for b in (-1, 0, 1)]
        if all(rows[a][b] == "." for a, b in block) and _free(rows, block, H, W, 1):
            rows[r][c] = "O"
            return True
    return False


def add_closed_ring(rows, rng, H, W):
    for _ in range(200):
        r, c = rng.randint(1, H - 2), rng.randint(1, W - 2)
        block = [(r + a, c + b) for a in (-1, 0, 1) for b in (-1, 0, 1)]
        if all(rows[a][b] == "." for a, b in block) and _free(rows, block, H, W, 1):
            for a, b in block:
                rows[a][b] = "R"
            rows[r][c] = "#"
            return True
    return False


def add_band(rows, rng, H, W, sym=None):
    sym = sym or rng.choice("hv")
    for _ in range(200):
        if sym == "h":
            h, w = rng.randint(1, 2), rng.randint(3, W - 2)
        else:
            h, w = rng.randint(3, H - 2), rng.randint(1, 2)
        r, c = rng.randrange(0, H - h + 1), rng.randrange(0, W - w + 1)
        cells = [(r + a, c + b) for a in range(h) for b in range(w)]
        if all(rows[a][b] == "." for a, b in cells):
            for a, b in cells:
                rows[a][b] = sym
            return True
    return False


def add_stones(rows, rng, H, W, n_pieces, max_frac=0.15, gap=1):
    placed, tries = 0, 0
    budget = max(1, int(H * W * max_frac)) - sum(ch in "#" for row in rows for ch in row)
    weights = [w for w, _ in PIECES]
    while placed < n_pieces and tries < 500:
        tries += 1
        cells0 = rng.choices([p for _, p in PIECES], weights)[0]
        if len(cells0) > budget - (n_pieces - placed - 1):
            continue
        r, c = rng.randrange(H), rng.randrange(W)
        cells = [(r + a, c + b) for a, b in cells0]
        if not _free(rows, cells, H, W, gap):
            continue
        for a, b in cells:
            rows[a][b] = "#"
        budget -= len(cells)
        placed += 1
    return placed == n_pieces


def garden_layout(H, W, rng, n_pieces=None, ripple=False, band=None, closed_ring=False, gate_side="S", max_frac=0.15):
    """A garden-like layout, or None if the constraints could not be met."""
    rows = [["."] * W for _ in range(H)]
    if ripple and not add_ripple(rows, rng, H, W):
        return None
    if closed_ring and not add_closed_ring(rows, rng, H, W):
        return None
    if band and not add_band(rows, rng, H, W, None if band is True else band):
        return None
    if n_pieces is None:
        n_pieces = rng.choice([1, 3, 3]) if H * W < 49 else rng.choice([1, 3, 3, 5])
    extra = sum(1 for row in rows for ch in row if ch in "O#")   # a ripple stone or ringed stone counts as a piece
    n_pieces = max(0, n_pieces - extra)
    if n_pieces and not add_stones(rows, rng, H, W, n_pieces, max_frac=max_frac):
        return None
    if (n_pieces + extra) % 2 == 0 or symmetric(rows):
        return None
    if gate_side == "S":
        cand = [(H - 1, c) for c in range(W)]
    elif gate_side == "N":
        cand = [(0, c) for c in range(W)]
    elif gate_side == "W":
        cand = [(r, 0) for r in range(H)]
    else:
        cand = [(r, W - 1) for r in range(H)]
    cand = [p for p in cand if rows[p[0]][p[1]] in ".hv"]
    if not cand:
        return None
    gr, gc = rng.choice(cand)
    return Garden(["".join(r) for r in rows], (gr, gc, gate_side))


def add_side_gate(g: Garden, rng):
    """Pick a second gate on the north, east or west wall (the stroke must begin there)."""
    opts = []
    for c in range(g.W):
        if g.rows[0][c] == ".":
            opts.append(((0, c), "N"))
    for r in range(g.H - 1):
        if g.rows[r][0] == ".":
            opts.append(((r, 0), "W"))
        if g.rows[r][g.W - 1] == ".":
            opts.append(((r, g.W - 1), "E"))
    if not opts:
        return False
    (cell, side) = rng.choice(opts)
    if cell == (g.gate[0], g.gate[1]):
        return False
    g.end, g.end_side = cell, side
    return True


def ichimatsu(nbr, nbc, b, gate_col, flip=False):
    """A whole garden of b x b blocks raked alternately across (=) and down (|)."""
    rows = []
    for r in range(nbr * b):
        rows.append("".join(("=" if ((r // b) + (c // b) + flip) % 2 == 0 else "|") for c in range(nbc * b)))
    return Garden(rows, (nbr * b - 1, gate_col, "S"))
src/gen.py · 45 lines · 1.7 kB — Random garden layouts for experiments and generation. 
"""Random garden layouts for experiments and generation."""
import random
from zg import Garden

def random_layout(H, W, n_stones, rng, shapes=((1,1),), gate_side=None, min_gap=0):
    rows = [["."] * W for _ in range(H)]
    placed = 0; tries = 0
    while placed < n_stones and tries < 1000:
        tries += 1
        h, w = rng.choice(shapes)
        r, c = rng.randrange(0, H - h + 1), rng.randrange(0, W - w + 1)
        cells = [(r + i, c + j) for i in range(h) for j in range(w)]
        if any(rows[a][b] != "." for a, b in cells):
            continue
        if min_gap:
            near = [(a + da, b + db) for a, b in cells for da in range(-min_gap, min_gap + 1) for db in range(-min_gap, min_gap + 1)]
            if any(0 <= a < H and 0 <= b < W and rows[a][b] != "." for a, b in near):
                continue
        for a, b in cells:
            rows[a][b] = "#"
        placed += 1
    side = gate_side or rng.choice("NESW")
    if side == "S":
        cand = [(H - 1, c) for c in range(W)]
    elif side == "N":
        cand = [(0, c) for c in range(W)]
    elif side == "W":
        cand = [(r, 0) for r in range(H)]
    else:
        cand = [(r, W - 1) for r in range(H)]
    cand = [p for p in cand if rows[p[0]][p[1]] == "."]
    if not cand:
        return None
    gr, gc = rng.choice(cand)
    return Garden(["".join(r) for r in rows], (gr, gc, side))

def parity_ok(g):
    cells = g.sand_cells()
    b = sum((r + c) % 2 == 0 for r, c in cells); w = len(cells) - b
    gr, gc, _ = g.gate
    gb = (gr + gc) % 2 == 0
    if abs(b - w) > 1: return False
    if b == w + 1 and not gb: return False
    if w == b + 1 and gb: return False
    return True
src/grader.py · 419 lines · 17.0 kB — Human-style deductive solver for one-stroke raking puzzles, used to grade difficulty. 
"""grader.py: human-style deductive solver for one-stroke raking puzzles, used to grade difficulty.

Edge states: 0 unknown, 1 raked (ON), -1 not raked (OFF). The gate's exit half-edge is always ON.
Technique tiers:
  tier 1 (local):   saturation, forcing, dead-end => rake-start, no loops, given grooves/bands,
                    ripple-ring counting, wide-rake hairpin exclusion, shape marks (straight / turn).
  tier 2 (global):  checkerboard parity of the rake-start, connectivity of unraked sand,
                    no premature closure of the stroke.
  tier 3 (trial):   assume an edge, propagate tiers 1-2, refute by contradiction (depth 1).
  tier 4:           nested trials (depth 2).
grade() returns the highest tier needed plus counts.
"""
from __future__ import annotations

from zg import Garden, DIRS, DR, DC, OPP

PERP = {"N": "EW", "S": "EW", "E": "NS", "W": "NS"}


class Contradiction(Exception):
    pass


class Board:
    def __init__(self, g: Garden):
        self.g = g
        self.cells = g.sand_cells()
        self.idx = {c: i for i, c in enumerate(self.cells)}
        self.n = len(self.cells)
        self.gate = self.idx[(g.gate[0], g.gate[1])]
        self.edges = []          # (u, v) with u < v
        self.eid = {}            # (u, v) and (v, u) -> edge id
        self.inc = [[] for _ in range(self.n)]   # incident edge ids
        self.dir_edge = [dict() for _ in range(self.n)]  # direction -> edge id
        for c in self.cells:
            u = self.idx[c]
            for d in "ES":
                nb = (c[0] + DR[d], c[1] + DC[d])
                if nb in self.idx:
                    v = self.idx[nb]
                    e = len(self.edges)
                    self.edges.append((u, v))
                    self.eid[(u, v)] = self.eid[(v, u)] = e
                    self.inc[u].append(e)
                    self.inc[v].append(e)
                    self.dir_edge[u][d] = e
                    self.dir_edge[v][OPP[d]] = e
        # start candidates
        cand = set(range(self.n))
        if self.n > 1:
            cand.discard(self.gate)
        if g.end is not None:
            cand = {self.idx[tuple(g.end)]}
        if g.braced_start:
            cand = {u for u in cand if any(g.pivot(self.cells[u][0] + DR[d], self.cells[u][1] + DC[d]) for d in DIRS)}
        self.marks = {self.idx[c]: t for c, t in g.marks.items()}
        cand -= set(self.marks)  # a marked cell has two connections, so it is never the rake start
        self.cand0 = cand
        # parity-derived candidates (tier 2)
        black = [((r + c) % 2 == 0) for r, c in self.cells]
        nb_, nw_ = sum(black), self.n - sum(black)
        gate_black = black[self.gate]
        if nb_ == nw_:
            want = not gate_black
        elif nb_ == nw_ + 1:
            want = True
        elif nw_ == nb_ + 1:
            want = False
        else:
            want = None  # impossible garden
        self.parity_cand = {u for u in range(self.n) if want is not None and black[u] == want}
        self.parity_impossible = want is None or (nb_ == nw_ + 1 and not gate_black) or (nw_ == nb_ + 1 and gate_black)
        # fixed edges from bands and grooves
        self.fixed = {}
        for c, need in g.zone_requirements().items():
            for d in need:
                self._fix(c, d, 1)
        for c, want_dirs in g.clues.items():
            for d in DIRS:
                self._fix(c, d, 1 if d in want_dirs else -1)
        # ripple rings
        self.rings = []
        for ring in g.ring_sets():
            S = {self.idx[c] for c in ring}
            cross = [e for u in S for e in self.inc[u] if (self.edges[e][0] in S) != (self.edges[e][1] in S)]
            internal = [e for e in range(len(self.edges)) if self.edges[e][0] in S and self.edges[e][1] in S]
            self.rings.append((S, cross, internal))
        # hairpin triples (e_arm_u, e_mid, e_arm_v) that are forbidden
        self.hairpins = []
        if g.nohairpin:
            for c in self.cells:
                u = self.idx[c]
                for b in "ES":  # mid edge u -> v in direction b
                    if b not in self.dir_edge[u]:
                        continue
                    em = self.dir_edge[u][b]
                    v = self.edges[em][1] if self.edges[em][0] == u else self.edges[em][0]
                    for a in PERP[b]:
                        # arms go in direction -a from u and v (the U's bottom faces a)
                        back = OPP[a]
                        eu = self.dir_edge[u].get(back)
                        ev = self.dir_edge[v].get(back)
                        cu, cv = self.cells[u], self.cells[v]
                        braced = g.pivot(cu[0] + DR[a], cu[1] + DC[a]) and g.pivot(cv[0] + DR[a], cv[1] + DC[a])
                        if g.nohairpin == 2 and braced:
                            continue
                        # arms may also be the gate's exit half-edge
                        arm_u = eu if eu is not None else ("X" if (u == self.gate and g.gate[2] == back) else None)
                        arm_v = ev if ev is not None else ("X" if (v == self.gate and g.gate[2] == back) else None)
                        if arm_u is None or arm_v is None:
                            continue
                        self.hairpins.append((arm_u, em, arm_v))

    def _fix(self, c, d, val):
        u = self.idx[c]
        if c == (self.g.gate[0], self.g.gate[1]) and d == self.g.gate[2]:
            if val == -1:
                self.fixed["contradiction"] = True
            return
        if d not in self.dir_edge[u]:
            if val == 1:
                self.fixed["contradiction"] = True
            return
        e = self.dir_edge[u][d]
        if self.fixed.get(e, val) != val:
            self.fixed["contradiction"] = True
        self.fixed[e] = val


class State:
    def __init__(self, B: Board, use_global=True):
        self.B = B
        self.s = [0] * len(B.edges)
        self.cand = set(B.cand0)
        if use_global:
            self.cand &= B.parity_cand
        self.use_global = use_global
        self.start = None

    def copy(self):
        o = State.__new__(State)
        o.B, o.s, o.cand, o.use_global, o.start = self.B, list(self.s), set(self.cand), self.use_global, self.start
        return o

    def set(self, e, val):
        if self.s[e] == val:
            return False
        if self.s[e] != 0:
            raise Contradiction
        self.s[e] = val
        return True

    def ondeg(self, u):
        d = sum(1 for e in self.B.inc[u] if self.s[e] == 1)
        return d + (1 if u == self.B.gate else 0)

    def unk(self, u):
        return [e for e in self.B.inc[u] if self.s[e] == 0]

    def hp_val(self, x):
        return 1 if x == "X" else self.s[x]

    def dstate(self, u, d):
        """State of cell u's side d: the gate's exit is ON; a side with no edge (wall, stone) is OFF."""
        if u == self.B.gate and d == self.B.g.gate[2]:
            return 1
        e = self.B.dir_edge[u].get(d)
        return -1 if e is None else self.s[e]

    def dset(self, u, d, val):
        if u == self.B.gate and d == self.B.g.gate[2]:
            if val != 1:
                raise Contradiction
            return False
        e = self.B.dir_edge[u].get(d)
        if e is None:
            if val == 1:
                raise Contradiction
            return False
        return self.set(e, val)

    def propagate(self):
        B = self.B
        if B.fixed.get("contradiction"):
            raise Contradiction
        if self.use_global and B.parity_impossible:
            raise Contradiction
        for e, v in B.fixed.items():
            if e != "contradiction":
                self.set(e, v)
        changed = True
        while changed:
            changed = False
            if not self.cand:
                raise Contradiction
            if len(self.cand) == 1:
                self.start = next(iter(self.cand))
            # cell rules
            for u in range(B.n):
                on, un = self.ondeg(u), self.unk(u)
                is_start = self.start == u
                maxdeg = 1 if is_start else 2
                if on > maxdeg:
                    raise Contradiction
                if on == maxdeg and un:
                    for e in un:
                        changed |= self.set(e, -1)
                    continue
                if on + len(un) == 0:
                    raise Contradiction
                if on + len(un) == 1:
                    # dead end: must be the rake start
                    if u not in self.cand:
                        raise Contradiction
                    if self.start != u:
                        self.cand = {u}
                        self.start = u
                        changed = True
                    for e in un:
                        changed |= self.set(e, 1)
                    continue
                if u not in self.cand and on + len(un) == 2:
                    for e in un:
                        changed |= self.set(e, 1)
                if is_start and on + len(un) >= 1 and on == 0 and len(un) == 1:
                    changed |= self.set(un[0], 1)
            # shape marks: 'S' = both sides of one axis ON (the other OFF); 'B' = exactly one side of each axis ON
            for u, t in B.marks.items():
                st = {d: self.dstate(u, d) for d in DIRS}
                if t == "S":
                    for a, b, oa, ob in (("N", "S", "E", "W"), ("E", "W", "N", "S")):
                        if st[a] == 1 or st[b] == 1:
                            for d, v in ((a, 1), (b, 1), (oa, -1), (ob, -1)):
                                changed |= self.dset(u, d, v)
                            break
                        if st[a] == -1 or st[b] == -1:
                            for d, v in ((a, -1), (b, -1), (oa, 1), (ob, 1)):
                                changed |= self.dset(u, d, v)
                            break
                else:
                    for a, b in (("N", "S"), ("E", "W")):
                        if st[a] == st[b] != 0:
                            raise Contradiction
                        if st[a] != 0:
                            changed |= self.dset(u, b, -st[a])
                        elif st[b] != 0:
                            changed |= self.dset(u, a, -st[b])
            # fragments (union-find over ON edges): forbid loops, premature closure
            parent = list(range(B.n))

            def find(x):
                while parent[x] != x:
                    parent[x] = parent[parent[x]]
                    x = parent[x]
                return x
            for e, (u, v) in enumerate(B.edges):
                if self.s[e] == 1:
                    ru, rv = find(u), find(v)
                    if ru == rv:
                        raise Contradiction
                    parent[ru] = rv
            size = {}
            for u in range(B.n):
                size[find(u)] = size.get(find(u), 0) + 1
            for e, (u, v) in enumerate(B.edges):
                if self.s[e] != 0:
                    continue
                ru, rv = find(u), find(v)
                if ru == rv:
                    changed |= self.set(e, -1)
                    continue
                if self.use_global and self.start is not None:
                    terminals = {find(B.gate), find(self.start)}
                    if {ru, rv} == terminals and size[ru] + size[rv] < B.n:
                        changed |= self.set(e, -1)
            # hairpins
            for (a, m, b) in B.hairpins:
                vals = [self.hp_val(a), self.hp_val(m), self.hp_val(b)]
                if vals.count(1) == 3:
                    raise Contradiction
                if vals.count(1) == 2 and 0 in vals:
                    x = (a, m, b)[vals.index(0)]
                    if x != "X":
                        changed |= self.set(x, -1)
            # rings
            for (S, cross, internal) in B.rings:
                gate_in = B.gate in S
                p_true = self.start is not None and self.start in S
                p_poss = bool(self.cand & S)
                maxT = 2 - gate_in - (1 if p_true else 0)
                minT = 2 - gate_in - (1 if p_poss else 0)
                on = sum(1 for e in cross if self.s[e] == 1)
                un = [e for e in cross if self.s[e] == 0]
                if on > maxT or on + len(un) < minT:
                    raise Contradiction
                if on == maxT:
                    for e in un:
                        changed |= self.set(e, -1)
                elif minT == maxT and on + len(un) == minT:
                    for e in un:
                        changed |= self.set(e, 1)
                ion = sum(1 for e in internal if self.s[e] == 1)
                ioff = [e for e in internal if self.s[e] == -1]
                iun = [e for e in internal if self.s[e] == 0]
                need = len(S) - 1
                if ion > need or ion + len(iun) < need:
                    raise Contradiction
                if ion == need:
                    for e in iun:
                        changed |= self.set(e, -1)
                elif ion + len(iun) == need:
                    for e in iun:
                        changed |= self.set(e, 1)
            # connectivity of the sand through non-OFF edges
            if self.use_global:
                seen = {B.gate}
                stack = [B.gate]
                while stack:
                    u = stack.pop()
                    for e in B.inc[u]:
                        if self.s[e] != -1:
                            a, b = B.edges[e]
                            w = b if a == u else a
                            if w not in seen:
                                seen.add(w)
                                stack.append(w)
                if len(seen) != B.n:
                    raise Contradiction
        return self

    def solved(self):
        return all(x != 0 for x in self.s)

    def unknown_edges(self):
        return [e for e, x in enumerate(self.s) if x == 0]

    def valid_final(self):
        B = self.B
        degs = [self.ondeg(u) for u in range(B.n)]
        ones = [u for u in range(B.n) if degs[u] == 1]
        if len(ones) != 1 or any(d not in (1, 2) for d in degs):
            return False
        # the path from the gate must cover everything
        seen = {B.gate}
        stack = [B.gate]
        while stack:
            u = stack.pop()
            for e in B.inc[u]:
                if self.s[e] == 1:
                    a, b = B.edges[e]
                    w = b if a == u else a
                    if w not in seen:
                        seen.add(w)
                        stack.append(w)
        return len(seen) == B.n


def try_assume(st: State, e, val, depth):
    t = st.copy()
    try:
        t.set(e, val)
        t.propagate()
        if depth > 0:
            trial_loop(t, depth - 1, stats=None)
        if t.solved() and not t.valid_final():
            raise Contradiction
    except Contradiction:
        return False
    return True


def trial_loop(st: State, depth, stats):
    """Repeatedly refute edge assumptions at the given depth; mutates st."""
    progress = True
    while progress and not st.solved():
        progress = False
        for e in st.unknown_edges():
            if st.s[e] != 0:
                continue
            for val in (1, -1):
                if not try_assume(st, e, val, depth):
                    st.set(e, -val)
                    st.propagate()
                    if stats is not None:
                        stats["trials"] += 1
                    progress = True
                    break
    return st


def grade(g: Garden, max_depth=2):
    """Return dict(tier=1..5, trials, solved). tier 5 = not solved by depth<=max_depth trials."""
    B = Board(g)
    # tier 1: local rules only
    st = State(B, use_global=False)
    try:
        st.propagate()
    except Contradiction:
        return {"tier": 0, "trials": 0, "solved": False, "note": "contradiction (unsolvable)"}
    if st.solved() and st.valid_final():
        return {"tier": 1, "trials": 0, "solved": True}
    st = State(B, use_global=True)
    try:
        st.propagate()
    except Contradiction:
        return {"tier": 0, "trials": 0, "solved": False, "note": "contradiction (unsolvable)"}
    if st.solved() and st.valid_final():
        return {"tier": 2, "trials": 0, "solved": True}
    for depth in range(max_depth):
        stats = {"trials": 0}
        t = st.copy()
        try:
            trial_loop(t, depth, stats)
        except Contradiction:
            return {"tier": 0, "trials": stats["trials"], "solved": False, "note": "contradiction in trials"}
        if t.solved() and t.valid_final():
            return {"tier": 3 + depth, "trials": stats["trials"], "solved": True}
        st = t
    return {"tier": 3 + max_depth, "trials": None, "solved": False}
src/hints.py · 417 lines · 19.0 kB — The abbot's first observation for generated gardens, derived from solver facts (so true by construction). 
"""hints.py: the abbot's first observation for generated gardens, derived from solver facts (so true by construction).

The observation is chosen as the one fact that most changes the player's plan:
  free gardens:   among facts true of every par raking (where it ends, how it begins, a whole row or column raked in one
                  line, whether the line turns or runs straight in some cell), the one that the most near-misses (rakings
                  one or two turns over par) get wrong;
  strict gardens: among facts about the unique raking (where the rake is set down; whether the line turns or runs straight
                  in some cell), the one that lets the grader's local and global rules (tiers 1-2) decide the most of
                  the garden, i.e. the step a player would otherwise need lookahead for.
Each observation comes with the cells the abbot points at. Parity lessons: the colour count, and why the far corner fails.
trap_score() measures how strongly a garden punishes the obvious move (see there).
"""
from __future__ import annotations

import copy
from collections import deque

ORD = ["first", "second", "third", "fourth", "fifth", "sixth", "seventh", "eighth", "ninth"]
WORD = {"N": "straight up", "E": "right", "W": "left", "S": "down"}


def _dir(a, b):
    return {(-1, 0): "N", (1, 0): "S", (0, 1): "E", (0, -1): "W"}[(b[0] - a[0], b[1] - a[1])]


def row_phrase(g, r):
    if r == 0:
        return "in the top row"
    if r == g.H - 1:
        return "in the bottom row"
    return f"in the {ORD[r]} row from the top" if r < g.H / 2 else f"in the {ORD[g.H - 1 - r]} row from the bottom"


def col_phrase(g, c):
    if c == 0:
        return "against the left wall"
    if c == g.W - 1:
        return "against the right wall"
    return f"{ORD[c]} from the left" if c < g.W / 2 else f"{ORD[g.W - 1 - c]} from the right"


def gate_word(g):
    return "veranda gate" if g.end is not None else "gate"      # with a side gate, "the gate" is ambiguous


def describe(g, cell):
    r, c = cell
    gr, gc = g.gate[0], g.gate[1]
    if (r, c) == (gr, gc):
        return f"in the cell just inside the {gate_word(g)}"
    if r == gr and abs(c - gc) == 1:
        return f"right beside the {gate_word(g)}"
    corners = {(0, 0): "in the top-left corner", (0, g.W - 1): "in the top-right corner",
               (g.H - 1, 0): "in the bottom-left corner", (g.H - 1, g.W - 1): "in the bottom-right corner"}
    if (r, c) in corners:
        return corners[(r, c)]
    return f"{row_phrase(g, r)}, {col_phrase(g, c)}"


def cell_noun(g, cell):
    """The cell as a noun phrase: 'the top-left corner', 'the cell just inside the gate', 'the cell in the top row, ...'."""
    d = describe(g, cell)
    if d.startswith("in the cell"):
        return d[len("in "):]
    if d.startswith("in the ") and d.endswith(" corner"):
        return d[len("in "):]
    return "the cell " + d


def one_way_in(g, cell):
    r, c = cell
    return (r, c) != (g.gate[0], g.gate[1]) and sum(g.is_sand(r + a, c + b) for a, b in ((-1, 0), (1, 0), (0, -1), (0, 1))) == 1


def features(g):
    s = "".join(g.rows)
    return {"wide": g.nohairpin == 2, "side": g.end is not None, "pebbles": bool(g.marks), "grooves": bool(g.clues),
            "ripple": "O" in s, "band": ("h" in s or "v" in s), "blocks": ("=" in s or "|" in s), "ring": "R" in s}


def line_name(g, kind, i):
    if kind == "row":
        return "top row" if i == 0 else "bottom row" if i == g.H - 1 else (
            f"{ORD[i]} row from the top" if i < g.H / 2 else f"{ORD[g.H - 1 - i]} row from the bottom")
    return "left-hand column" if i == 0 else "right-hand column" if i == g.W - 1 else (
        f"{ORD[i]} column from the left" if i < g.W / 2 else f"{ORD[g.W - 1 - i]} column from the right")


def common_line(g, par_rakings):
    """A whole row or column of plain sand (wall to wall) that every par raking rakes as one straight line."""
    lines = [("row", r, [(r, c) for c in range(g.W)]) for r in range(g.H)] + \
            [("column", c, [(r, c) for r in range(g.H)]) for c in range(g.W)]
    best = None
    for kind, i, cells in lines:
        if not all(g.rows[x[0]][x[1]] == "." for x in cells) or len(cells) < 4:   # plain sand: a band row is straight anyway
            continue
        fwd, back = [tuple(x) for x in cells], [tuple(x) for x in reversed(cells)]
        n = len(cells)

        def has(d):
            d = [tuple(x) for x in d]
            return any(d[k:k + n] in (fwd, back) for k in range(len(d) - n + 1))
        if all(has(d) for d in par_rakings) and (best is None or n > best[0]):
            best = (n, kind, i)
    if best is None:
        return None
    n, kind, i = best
    return f"Every par raking rakes the whole {line_name(g, kind, i)} in one straight line."


def hint_free(g, par, par_rakings):
    ends = {tuple(d[-1]) for d in par_rakings}
    firsts = {_dir(d[0], d[1]) for d in par_rakings if len(d) > 1}
    gate = (g.gate[0], g.gate[1])
    f = features(g)
    rings = "".join(g.rows).count("O")
    ring_txt = "Rake the ring in one run" if rings == 1 else "Rake each ring in one run"
    lead = (f"{ring_txt}, and cross each band in one straight run. " if f["ripple"] and f["band"] else
            f"{ring_txt}. " if f["ripple"] else "Cross each band in one straight run. " if f["band"] else "")
    if len(ends) == 1:
        return lead + f"At par, the walk ends {describe(g, next(iter(ends)))}."
    if all(abs(e[0] - gate[0]) + abs(e[1] - gate[1]) == 1 for e in ends):
        return lead + "At par, the walk ends right beside the gate."
    rows, cols = {e[0] for e in ends}, {e[1] for e in ends}
    if len(rows) == 1:
        return lead + f"At par, the walk ends {row_phrase(g, next(iter(rows)))}."
    if len(cols) == 1 and next(iter(cols)) in (0, g.W - 1):
        return lead + f"At par, the walk ends {col_phrase(g, next(iter(cols)))}."
    if len(firsts) == 1:
        return lead + f"Every par raking begins by walking {WORD[next(iter(firsts))]} from the gate."
    line = common_line(g, par_rakings)
    if line:
        return lead + line
    return lead + f"Par is {par} turns: long lines from wall to wall, and as few U-turns as the stones allow."


def hint_strict(g, draw, all_draws=None):
    f = features(g)
    draws = all_draws or [draw]
    ends = {tuple(d[-1]) for d in draws}
    end_txt = f"the rake is set down {describe(g, next(iter(ends)))}" if len(ends) == 1 else None
    if f["side"]:
        heads = {_dir(d[0], d[1]) for d in draws if len(d) > 1}
        if len(heads) == 1:
            first = f"From the veranda gate, the walk first heads {WORD[next(iter(heads))]}."
        else:
            first = "Plan from both gates at once."
        tail = " Every reversal needs a pivot." if f["wide"] else " Settle the pebbles beside walls and stones first." if f["pebbles"] else ""
        return first + tail
    if f["blocks"]:
        lead = "Rake each block as parallel lines, then step into the next."
    elif f["pebbles"]:
        lead = "Settle the pebbles beside walls and stones first: there, often only one way through fits."
    elif f["grooves"]:
        lead = "Extend each of the abbot's grooves into its neighbours first."
    elif f["wide"]:
        lead = "Every reversal needs a pivot: a wall or a stone."
    else:
        lead = "Long lines first."
    if f["ring"] and not f["pebbles"]:
        lead += " Leave the finished ring as it is."
    return lead + (f" Here {end_txt}." if end_txt else "")


def far_cell(g):
    start = (g.gate[0], g.gate[1])
    dist, q = {start: 0}, deque([start])
    while q:
        a = q.popleft()
        for d in ((1, 0), (-1, 0), (0, 1), (0, -1)):
            b = (a[0] + d[0], a[1] + d[1])
            if g.is_sand(*b) and b not in dist:
                dist[b] = dist[a] + 1
                q.append(b)
    H, W = g.H, g.W
    return max(dist, key=lambda c: (dist[c], c in ((0, 0), (0, W - 1), (H - 1, 0), (H - 1, W - 1))))


def hint_parity(g, all_draws):
    cells = g.sand_cells()
    dark = sum((r + c) % 2 == 0 for r, c in cells)
    light = len(cells) - dark
    gate_dark = (g.gate[0] + g.gate[1]) % 2 == 0
    if dark == light + 1:
        rule, end_dark = "There is one more dark cell than light, so the walk ends on a dark cell", True
    elif light == dark + 1:
        rule, end_dark = "There is one more light cell than dark, so the walk ends on a light cell", False
    else:
        end_dark = not gate_dark
        rule = f"Dark and light cells are equal in number and the gate is on a {'dark' if gate_dark else 'light'} cell, so the walk ends on a {'dark' if end_dark else 'light'} one"
    far = far_cell(g)
    far_dark = (far[0] + far[1]) % 2 == 0
    assert all(((d[-1][0] + d[-1][1]) % 2 == 0) == end_dark for d in all_draws), "parity rule contradicted by a raking"
    if far_dark != end_dark:
        assert all(tuple(d[-1]) != far for d in all_draws)
        return f"{rule}: not {describe(g, far)}, which is {'dark' if far_dark else 'light'}."
    return rule + "."


# ---------------------------------------------------------------- the observation that changes the plan

STRAIGHT_IN = {"S": "N", "N": "S", "E": "W", "W": "E"}   # the obvious first move: straight in through the gate
ZONE = set("hv=|")


def first_dir(d):
    return _dir(d[0], d[1]) if len(d) > 1 else None


def shapes(g, d):
    """Cell -> 'S' (the line runs straight through it) or 'B' (it turns there), along a walk from the gate; the last cell
    (where the rake is set down) has no shape."""
    d = [tuple(c) for c in d]
    out = {}
    for i, c in enumerate(d[:-1]):
        din = STRAIGHT_IN[g.gate[2]] if i == 0 else _dir(d[i - 1], c)
        out[c] = "S" if _dir(c, d[i + 1]) == din else "B"
    return out


def near_misses(g, par, slack=2, cap=300000):
    """Rakings within `slack` turns over par (the plans a player is most likely to find instead of par)."""
    from zg import turns_of
    r = g.solve(mode=2, bound=par + slack, cap=cap, stream=True)
    return [s_["draw"] for s_ in r["sols"] if turns_of(g, s_["draw"]) > par]


def _beside_gate(g):
    gr, gc, side = g.gate
    return {(gr, gc - 1), (gr, gc + 1)} if side in "NS" else {(gr - 1, gc), (gr + 1, gc)}


def free_facts(g, par_rakings):
    """Facts true of every par raking: (kind rank, text, cells pointed at, test on a raking)."""
    P = [[tuple(c) for c in d] for d in par_rakings]
    gate = (g.gate[0], g.gate[1])
    F = []
    ends = {d[-1] for d in P}
    if len(ends) == 1:
        e = next(iter(ends))
        F.append((0, f"At par, the walk ends {describe(g, e)}.", [e], lambda d, e=e: tuple(d[-1]) == e))
    else:
        bg = _beside_gate(g)
        if ends <= bg:
            F.append((0, "At par, the walk ends right beside the gate.", sorted(ends), lambda d, bg=bg: tuple(d[-1]) in bg))
        rows, cols = {e[0] for e in ends}, {e[1] for e in ends}
        if len(rows) == 1:
            r = next(iter(rows))
            F.append((1, f"At par, the walk ends in the {line_name(g, 'row', r)}.", sorted(ends), lambda d, r=r: d[-1][0] == r))
        if len(cols) == 1:
            c = next(iter(cols))
            F.append((1, f"At par, the walk ends in the {line_name(g, 'column', c)}.", sorted(ends), lambda d, c=c: d[-1][1] == c))
    firsts = {first_dir(d) for d in P}
    if len(firsts) == 1:
        f = next(iter(firsts))
        nxt = P[0][1]
        F.append((0, f"Every par raking begins by walking {WORD[f]} from the gate.", [gate, nxt], lambda d, f=f: first_dir(d) == f))
    for kind, i, cells in [("row", r, [(r, c) for c in range(g.W)]) for r in range(g.H)] + \
                          [("column", c, [(r, c) for r in range(g.H)]) for c in range(g.W)]:
        if len(cells) < 4 or not all(g.rows[x[0]][x[1]] == "." for x in cells):
            continue
        fwd, back, n = list(cells), list(reversed(cells)), len(cells)

        def has(d, fwd=fwd, back=back, n=n):
            d = [tuple(x) for x in d]
            return any(d[k:k + n] in (fwd, back) for k in range(len(d) - n + 1))
        if all(has(d) for d in P):
            F.append((1, f"Every par raking rakes the whole {line_name(g, kind, i)} in one straight line.", cells, has))
    # a stretch of a row or column between stones or walls (not the whole line), raked in one straight run at par
    for kind, i, cells in [("row", r, [(r, c) for c in range(g.W)]) for r in range(g.H)] + \
                          [("column", c, [(r, c) for r in range(g.H)]) for c in range(g.W)]:
        runs, cur = [], []
        for x in cells:
            if g.rows[x[0]][x[1]] == ".":
                cur.append(x)
            else:
                runs.append(cur); cur = []
        runs.append(cur)
        for seg in runs:
            if len(seg) < 3 or len(seg) == len(cells):
                continue
            fwd, back, n = list(seg), list(reversed(seg)), len(seg)

            def has(d, fwd=fwd, back=back, n=n):
                d = [tuple(x) for x in d]
                return any(d[k:k + n] in (fwd, back) for k in range(len(d) - n + 1))
            if all(has(d) for d in P):
                F.append((1, f"Every par raking rakes the marked stretch of the {line_name(g, kind, i)} in one straight line.", seg, has))
    S = [shapes(g, d) for d in P]
    for c in sorted(set.intersection(*[set(x) for x in S])):
        if g.rows[c[0]][c[1]] in ZONE or c in g.marks or c in g.clues:
            continue
        t = {x[c] for x in S}
        if len(t) == 1:
            t = next(iter(t))
            text = (f"At par, the line turns at {cell_noun(g, c)}." if t == "B"
                    else f"At par, the line runs straight through {cell_noun(g, c)}.")
            F.append((2, text, [c], lambda d, c=c, t=t: shapes(g, d).get(c) == t))
    return F


def best_free_hint(g, par, par_rakings, near=None):
    """The fact true of every par raking that the most near-misses get wrong (ties: the more natural kind of fact)."""
    F = free_facts(g, par_rakings)
    if not F:
        return f"Par is {par} turns: long lines from wall to wall, and as few U-turns as the stones allow.", [], None
    near = near_misses(g, par) if near is None else near
    scored = []
    for rank, text, cells, test in F:
        assert all(test(d) for d in par_rakings), text          # true of every par raking, by construction
        miss = sum(not test(d) for d in near) / len(near) if near else 0.0
        scored.append((miss - 0.04 * rank, -rank, text, cells, miss))
    scored.sort(key=lambda x: (x[0], x[1]), reverse=True)
    _, _, text, cells, miss = scored[0]
    return text, [list(c) for c in cells], round(miss, 3)


def _decided(B, end=None, mark=None):
    """How much of the garden the grader's tier 1-2 rules decide, given one more fact; and whether that solves it."""
    from grader import State, Contradiction
    B2 = copy.copy(B)
    if end is not None:
        B2.cand0 = {B.idx[end]}
    if mark is not None:
        c, t = mark
        B2.marks = dict(B.marks)
        B2.marks[B.idx[c]] = t
        B2.cand0 = set(B.cand0) - {B.idx[c]}
    st = State(B2, use_global=True)
    try:
        st.propagate()
    except Contradiction:
        raise AssertionError("a true fact led to a contradiction")
    return sum(1 for x in st.s if x != 0), st.solved() and st.valid_final()


def with_fact(g, cells, text):
    """The garden with a strict hint's fact added as a given: the end cell fixed, or a pebble at the cell."""
    g2 = copy.deepcopy(g)
    c = tuple(cells[0])
    if "rake is set down" in text:
        g2.end = c
    else:
        g2.marks = dict(g.marks); g2.marks[c] = "B" if "turns" in text else "S"
    return g2


def best_strict_hint(g, draw, regrade=8):
    """The fact about the unique raking that most lowers the garden's grade (tier, then lookahead steps), among the
    end cell and the `regrade` facts that let the tier 1-2 rules decide the most; ties: more decided, then where the
    rake is set down, then the cell earliest in the walk."""
    from grader import Board, grade
    B = Board(g)
    d = [tuple(c) for c in draw]
    base, _ = _decided(B)
    cands = []
    if g.end is None:
        e = d[-1]
        n, solved = _decided(B, end=e)
        # say why, when the reason is short: a cell with one way in is where the walk must end
        noun = cell_noun(g, e)
        noun = noun[0].upper() + noun[1:] + ("," if "," in noun else "")
        text = (f"{noun} has only one way in, so the rake is set down there." if one_way_in(g, e)
                else f"The rake is set down {describe(g, e)}.")
        cands.append([solved, n, 1, 0, text, [e]])
    sh = shapes(g, d)
    for i, c in enumerate(d[:-1]):
        if g.rows[c[0]][c[1]] in ZONE or c in g.marks or c in g.clues:
            continue
        n, solved = _decided(B, mark=(c, sh[c]))
        text = (f"The line turns at {cell_noun(g, c)}." if sh[c] == "B"
                else f"The line runs straight through {cell_noun(g, c)}.")
        cands.append([solved, n, 0, -i, text, [c]])
    if not cands:
        return "Follow the hatching: every block is raked in parallel lines.", [], (base, base)
    cands.sort(key=lambda x: x[:4], reverse=True)
    short = cands[:regrade] + [c for c in cands[regrade:] if c[2] == 1]
    best = None
    for c in short:
        gr = grade(with_fact(g, c[5], c[4]))
        assert gr["solved"], c[4]
        key = (gr["tier"], gr["trials"] or 0, -c[1], -c[2], -c[3])
        if best is None or key < best[0]:
            best = (key, c)
    _, (solved, n, _, _, text, cells) = best
    return text, [list(c) for c in cells], (base, n)


def trap_score(g, par, par_rakings=None, near=None, draw=None):
    """How strongly the garden punishes the obvious move (0 = not at all).
    Free: the share of near-misses (rakings one or two turns over par) that get the garden's key observation wrong (the
    best_free_hint fact), counted only when it is most of them: the plan that comes naturally is wrong there.
    (The first move is no use here: in almost every free garden, the near-misses' favourite first move is also a par move.)
    Strict: 1 when the only raking turns aside at once, but walking straight in through the gate is not refuted by the
    tier 1-2 rules (it takes lookahead to see why it fails)."""
    if draw is None:
        if not near:
            return 0.0
        _, _, miss = best_free_hint(g, par, par_rakings, near)
        return miss if miss >= 0.5 else 0.0
    obvious = STRAIGHT_IN[g.gate[2]]
    gr, gc = g.gate[0], g.gate[1]
    nxt = {"N": (gr - 1, gc), "S": (gr + 1, gc), "E": (gr, gc + 1), "W": (gr, gc - 1)}[obvious]
    if not g.is_sand(*nxt) or first_dir(draw) == obvious:
        return 0.0
    from grader import Board, State, Contradiction
    B = Board(g)
    st = State(B, use_global=True)
    st.propagate()
    e = B.dir_edge[B.idx[(gr, gc)]].get(obvious)
    if e is None:
        return 0.0
    try:
        t = st.copy(); t.set(e, 1); t.propagate()
    except Contradiction:
        return 0.0
    return 1.0
src/levels.py · 262 lines · 10.1 kB — Search for example gardens per mechanic, verify them, and save to JSON. 
"""levels.py: search for example gardens per mechanic, verify them, and save to JSON.

Every saved level records: layout, rules, the number of valid rakings (exact, or a lower bound if capped),
par (proven by CP-SAT), whether par is achievable 'gracefully' (no hairpin turn in open sand),
the intended raking, and the deductive grade."""
from __future__ import annotations

import json
import random
import sys
import time
from copy import deepcopy

from zg import Garden, glyphs, hairpin_list, turns_of, check_solution
from zcp import optimize
from gen import random_layout, parity_ok
from clues import make_unique, with_clues
from grader import grade

OUT = "/home/claude/zen/out/levels.json"


def to_dict(g: Garden):
    return {"rows": g.rows, "gate": list(g.gate), "clues": {f"{r},{c}": v for (r, c), v in g.clues.items()},
            "end": list(g.end) if g.end else None, "nohairpin": g.nohairpin, "braced_start": g.braced_start,
            "name": g.name, "end_side": g.end_side, "marks": {f"{r},{c}": t for (r, c), t in g.marks.items()}}


def from_dict(d) -> Garden:
    g = Garden(d["rows"], tuple(d["gate"]), clues={tuple(map(int, k.split(","))): v for k, v in d["clues"].items()},
               end=tuple(d["end"]) if d["end"] else None, nohairpin=d["nohairpin"], braced_start=d["braced_start"],
               name=d.get("name", ""), end_side=d.get("end_side"),
               marks={tuple(map(int, k.split(","))): v for k, v in d.get("marks", {}).items()})
    return g


def par_info(g: Garden, time_limit=120):
    free = optimize(g, time_limit=time_limit)
    if free[1] != "OPTIMAL":
        return None
    grace = optimize(g, nohairpin=2, time_limit=time_limit)
    return {"par": free[0], "par_draw": list(reversed(free[2])),
            "graceful_par": grace[0] if grace[1] == "OPTIMAL" else None,
            "graceful_draw": list(reversed(grace[2])) if grace[2] else None}


def summarize(g: Garden, intended_draw=None, cap=10**7):
    r = g.solve(mode=0, cap=cap, nodelimit=2 * 10**9)
    info = {"n_rakings": r["nsol"], "count_exact": bool(r["complete"] and not r["capped"]),
            "hist": r["hist"]}
    p = par_info(g)
    if p:
        info.update({k: v for k, v in p.items() if not k.endswith("draw")})
        info["at_par"] = r["hist"].get(p["par"], 0) if info["count_exact"] else None
    if intended_draw is None and p:
        intended_draw = p["graceful_draw"] if p["graceful_par"] == p["par"] else p["par_draw"]
    if intended_draw is not None:
        assert check_solution(g, intended_draw), "intended raking fails the rules"
        info["intended"] = [list(c) for c in intended_draw]
        info["intended_turns"] = turns_of(g, intended_draw)
        info["intended_open_hairpins"] = sum(1 for *_, br in hairpin_list(g, intended_draw) if not br)
    info["grade"] = grade(g) if r["nsol"] == 1 else None
    return info


def graceful(g, info):
    return info.get("graceful_par") is not None and info["graceful_par"] == info["par"]


# ---------------------------------------------------------------- searches

def search_first_stroke(rng, budget=60):
    t0 = time.time()
    best = None
    while time.time() - t0 < budget:
        g = random_layout(6, 6, 2, rng, gate_side="S", min_gap=1)
        if g is None or not parity_ok(g) or g.solve(mode=0, cap=1)["nsol"] == 0:
            continue
        p = par_info(g, 30)
        if not p or p["graceful_par"] != p["par"]:
            continue
        g.name = "First stroke"
        return g
    return best


def search_ripple(rng, budget=120):
    t0 = time.time()
    while time.time() - t0 < budget:
        H, W = 7, 7
        rows = [["."] * W for _ in range(H)]
        r, c = rng.randint(2, H - 3), rng.randint(2, W - 3)
        rows[r][c] = "O"
        for _ in range(rng.randint(1, 2)):
            a, b = rng.randrange(H), rng.randrange(W)
            if rows[a][b] == "." and max(abs(a - r), abs(b - c)) > 2:
                rows[a][b] = "#"
        gc = rng.choice([x for x in range(W) if rows[H - 1][x] == "."])
        g = Garden(["".join(x) for x in rows], (H - 1, gc, "S"), name="Ripples")
        if not parity_ok(g) or g.solve(mode=0, cap=1)["nsol"] == 0:
            continue
        p = par_info(g, 30)
        if p and p["graceful_par"] == p["par"]:
            return g
    return None


def search_waves(rng, budget=120):
    t0 = time.time()
    while time.time() - t0 < budget:
        H, W = 7, 8
        rows = [["."] * W for _ in range(H)]
        r0 = rng.randint(1, 2)
        for r in range(r0, r0 + 2):
            for c in range(1, W - 1):
                rows[r][c] = "h"
        for _ in range(rng.randint(1, 2)):
            a, b = rng.randint(r0 + 3, H - 2), rng.randrange(1, W - 1)
            rows[a][b] = "#"
        gc = rng.choice([x for x in range(W) if rows[H - 1][x] == "."])
        g = Garden(["".join(x) for x in rows], (H - 1, gc, "S"), name="Waves")
        if not parity_ok(g) or g.solve(mode=0, cap=1)["nsol"] == 0:
            continue
        p = par_info(g, 30)
        if p and p["graceful_par"] == p["par"]:
            return g
    return None


def ichimatsu_garden():
    rows = ["===|||", "===|||", "===|||", "|||===", "|||===", "|||==="]
    return Garden(rows, (5, 0, "S"), name="Ichimatsu")


def search_grooves(rng, budget=300, size=(7, 7), stones=(2, 3), min_tier=3, max_clues=6, name="Master's grooves", extra=None):
    t0 = time.time()
    cands = []
    while time.time() - t0 < budget:
        g = random_layout(size[0], size[1], rng.choice(stones), rng, gate_side="S", min_gap=1)
        if g is None or not parity_ok(g):
            continue
        if extra:
            extra(g, rng)
        if g.solve(mode=0, cap=1)["nsol"] == 0:
            continue
        p = par_info(g, 30)
        if not p or p["graceful_par"] != p["par"]:
            continue
        target = p["graceful_draw"]
        clues = make_unique(g, target, rng)
        if clues is None or len(clues) > max_clues:
            continue
        gu = with_clues(g, clues)
        gr = grade(gu)
        if gr["solved"] and gr["tier"] >= min_tier:
            gu.name = name
            return gu, target, gr
        cands.append((gr["tier"], gu, target, gr))
    if cands:
        cands.sort(key=lambda x: -x[0])
        _, gu, target, gr = cands[0]
        gu.name = name
        return gu, target, gr
    return None


def search_wide(rng, budget=300, size=(7, 7)):
    t0 = time.time()
    best = None
    while time.time() - t0 < budget:
        g = random_layout(size[0], size[1], rng.choice([2, 3]), rng, gate_side="S", min_gap=1)
        if g is None or not parity_ok(g):
            continue
        g.nohairpin = 2
        g.braced_start = True
        r = g.solve(mode=0, cap=2)
        if r["nsol"] != 1:
            continue
        gr = grade(g)
        if not gr["solved"]:
            continue
        key = (gr["tier"], gr["trials"] or 0)
        if best is None or key > best[0]:
            g.name = "Wide rake"
            best = (key, g, r["sols"][0]["draw"], gr)
        if gr["tier"] >= 3 and (gr["trials"] or 0) >= 8:
            break
    return best


def search_two_gates(rng, budget=300):
    t0 = time.time()
    while time.time() - t0 < budget:
        g = random_layout(7, 7, rng.choice([2, 3]), rng, gate_side="S", min_gap=1)
        if g is None:
            continue
        # side gate on the east wall: the rake must start there
        er = rng.randint(0, 4)
        if g.rows[er][6] != ".":
            continue
        g.end = (er, 6)
        r = g.solve(mode=0, cap=3)
        if r["nsol"] != 1:
            continue
        gr = grade(g)
        if gr["solved"] and gr["tier"] >= 2:
            g.name = "Two gates"
            return g, r["sols"][0]["draw"], gr
    return None


def main():
    seed = int(sys.argv[1]) if len(sys.argv) > 1 else 1
    rng = random.Random(seed)
    levels = {}

    g1 = Garden(["........", "........", "..#.....", "........", ".....#..", "........", "........"], (6, 3, "S"),
                name="Calm and agitated")
    levels["calm"] = {"garden": to_dict(g1), **summarize(g1)}
    print("calm", {k: v for k, v in levels["calm"].items() if k not in ("garden", "intended", "hist")}, flush=True)

    g = search_first_stroke(rng)
    levels["first"] = {"garden": to_dict(g), **summarize(g)}
    print("first", {k: v for k, v in levels["first"].items() if k not in ("garden", "intended", "hist")}, flush=True)

    g = search_ripple(rng)
    levels["ripple"] = {"garden": to_dict(g), **summarize(g)}
    print("ripple", {k: v for k, v in levels["ripple"].items() if k not in ("garden", "intended", "hist")}, flush=True)

    g = search_waves(rng)
    levels["waves"] = {"garden": to_dict(g), **summarize(g)}
    print("waves", {k: v for k, v in levels["waves"].items() if k not in ("garden", "intended", "hist")}, flush=True)

    g = ichimatsu_garden()
    levels["ichimatsu"] = {"garden": to_dict(g), **summarize(g)}
    print("ichimatsu", {k: v for k, v in levels["ichimatsu"].items() if k not in ("garden", "intended", "hist")}, flush=True)

    res = search_grooves(rng)
    if res:
        gu, target, gr = res
        levels["grooves"] = {"garden": to_dict(gu), **summarize(gu, intended_draw=target)}
        print("grooves", {k: v for k, v in levels["grooves"].items() if k not in ("garden", "intended", "hist")}, flush=True)

    res = search_wide(rng)
    if res:
        _, g, draw, gr = res
        levels["wide"] = {"garden": to_dict(g), **summarize(g, intended_draw=draw)}
        print("wide", {k: v for k, v in levels["wide"].items() if k not in ("garden", "intended", "hist")}, flush=True)

    res = search_two_gates(rng)
    if res:
        g, draw, gr = res
        levels["twogates"] = {"garden": to_dict(g), **summarize(g, intended_draw=draw)}
        print("twogates", {k: v for k, v in levels["twogates"].items() if k not in ("garden", "intended", "hist")}, flush=True)

    with open(OUT, "w") as f:
        json.dump(levels, f, indent=1)
    print("saved", OUT)


if __name__ == "__main__":
    main()
src/pairs.py · 268 lines · 10.4 kB — Contrasting pairs that introduce each rule: the same small garden twice, before and after one change. 
"""pairs.py: contrasting pairs that introduce each rule: the same small garden twice, before and after one change.

Each search samples small layouts and keeps the best pair it finds in a time budget. A pair is kept only if the change
matters: the old way of raking the first garden must fail in the second (ripple stone, band, wide rake), or the
second must be pinned down to one raking by a few marks (grooves; pebbles in place of grooves), or the side gate must
change the answer. Smaller and easier pairs are preferred.
"""
from __future__ import annotations

import copy
import math
import random
import time

from zg import Garden, check_solution, hairpin_list, turns_of, glyphs
from gen import parity_ok
from garden_sampler import add_ripple, add_band, add_stones, add_closed_ring, add_side_gate, garden_layout
from clues import make_unique, with_clues, make_unique_marks, with_marks
from grader import grade
from zcp import optimize

SIZES = [(4, 5), (5, 4), (5, 5), (4, 6), (5, 6)]


def counted(g, cap=10**6):
    r = g.solve(mode=0, cap=cap)
    return r if (r["complete"] and not r["capped"]) else None


def par_rakings(g, par):
    return [s["draw"] for s in g.solve(mode=2, bound=par, cap=10**5, stream=True)["sols"]]


def open_u_turn(g, d):
    return any(not braced for *_, braced in hairpin_list(g, d))


def bottom_gate(rows, rng):
    H = len(rows)
    cand = [c for c in range(len(rows[0])) if rows[H - 1][c] == "."]
    return (H - 1, rng.choice(cand), "S") if cand else None


def as_rows(rows):
    return ["".join(r) for r in rows]


def habit_fails(A, B):
    """Every par raking of A breaks B's rule: the plan that worked before no longer does."""
    ra, rb = counted(A), counted(B)
    if ra is None or rb is None or ra["nsol"] < 3 or rb["nsol"] < 1:
        return None
    pa = min(ra["hist"])
    PA = par_rakings(A, pa)
    if any(check_solution(B, d) for d in PA):
        return None
    return ra, rb, pa


def search(make, budget, rng):
    best, t0, tried = None, time.time(), 0
    while time.time() - t0 < budget:
        tried += 1
        c = make(rng)
        if c is not None and (best is None or c["score"] < best["score"]):
            best = c
    if best:
        best["tried"] = tried
    return best


# ---------------------------------------------------------------- the pairs

def graceful_par(g):
    """Some par raking has no U-turn in open sand (the generator's filter for free gardens: the seal should not reward
    only meanders)."""
    free, grace = optimize(g, time_limit=20), optimize(g, nohairpin=2, time_limit=20)
    return free[1] == "OPTIMAL" and grace[1] == "OPTIMAL" and grace[0] == free[0]


def make_ripple(rng):
    H, W = rng.choice(SIZES)
    rows = [["."] * W for _ in range(H)]
    if not add_ripple(rows, rng, H, W):
        return None
    if rng.random() < 0.5 and not add_stones(rows, rng, H, W, 1, max_frac=0.12):
        return None
    gate = bottom_gate(rows, rng)
    if gate is None:
        return None
    B = Garden(as_rows(rows), gate)
    A = Garden([r.replace("O", "#") for r in B.rows], gate)
    if not parity_ok(B):
        return None
    h = habit_fails(A, B)
    if h is None or h[1]["nsol"] < 2 or not (graceful_par(A) and graceful_par(B)):
        return None
    return {"A": A, "B": B, "score": (len(B.sand_cells()), h[1]["nsol"]), "facts": {"A_rakings": h[0]["nsol"], "B_rakings": h[1]["nsol"]}}


def make_band(rng):
    H, W = rng.choice([(5, 5), (5, 6), (4, 6), (6, 5)])
    rows = [["."] * W for _ in range(H)]
    if rng.random() < 0.6 and not add_stones(rows, rng, H, W, 1, max_frac=0.12):
        return None
    plain = [r[:] for r in rows]
    sym = "h" if H < 5 else "v" if W < 5 else rng.choice("hv")   # (a band needs room for at least three cells)
    if not add_band(rows, rng, H, W, sym):
        return None
    gate = bottom_gate(rows, rng)
    if gate is None:
        return None
    A, B = Garden(as_rows(plain), gate), Garden(as_rows(rows), gate)
    if not parity_ok(A):
        return None
    h = habit_fails(A, B)
    if h is None or h[1]["nsol"] < 2 or not (graceful_par(A) and graceful_par(B)):
        return None
    return {"A": A, "B": B, "score": (len(B.sand_cells()), h[1]["nsol"]), "facts": {"A_rakings": h[0]["nsol"], "B_rakings": h[1]["nsol"]}}


def graceful_target(g):
    free, grace = optimize(g, time_limit=20), optimize(g, nohairpin=2, time_limit=20)
    if free[1] != "OPTIMAL" or grace[1] != "OPTIMAL" or grace[0] != free[0]:
        return None
    return list(reversed(grace[2]))


def make_grooves(rng, max_clues=3):
    H, W = rng.choice([(5, 5), (5, 6), (6, 5), (6, 6)])
    A = garden_layout(H, W, rng, n_pieces=1, max_frac=0.12)
    if A is None or not parity_ok(A):
        return None
    ra = counted(A)
    if ra is None or ra["nsol"] < 8:
        return None
    target = graceful_target(A)
    if target is None:
        return None
    cl = make_unique(A, target, rng)
    if cl is None or len(cl) > max_clues:
        return None
    B = with_clues(A, cl)
    gr = grade(B)
    if not gr["solved"] or gr["tier"] > 2:
        return None
    return {"A": A, "B": B, "target": target, "score": (len(cl), gr["tier"], len(B.sand_cells())),
            "facts": {"A_rakings": ra["nsol"], "grooves": len(cl), "tier": gr["tier"]}}


def make_pebbles(rng):
    """A: a garden pinned down by a few grooves. B: the same garden, with a pebble in each groove's cell (it only says
    straight or turn), plus whatever pebbles are needed for one raking again."""
    c = make_grooves(rng, max_clues=3)
    if c is None:
        return None
    A, target = c["B"], c["target"]
    base = copy.deepcopy(A); base.clues = {}
    gl = glyphs(base, target)
    marks0 = {cell: ("S" if set(gl[cell]) in ({"E", "W"}, {"N", "S"}) else "B") for cell in A.clues}
    B0 = with_marks(base, marks0)
    extra = make_unique_marks(B0, target, rng)
    if extra is None or len(marks0) + len(extra) > 6:
        return None
    B = with_marks(B0, extra)
    gr = grade(B)
    if not gr["solved"] or gr["tier"] > 2:
        return None
    kinds = set(B.marks.values())          # an introduction should show both kinds of pebble
    return {"A": A, "B": B, "score": (len(kinds) < 2, len(extra), gr["tier"], len(B.sand_cells())),
            "facts": {"grooves": len(A.clues), "pebbles": len(B.marks), "kinds": len(kinds), "tier": gr["tier"]}}


def make_wide(rng):
    """A: an ordinary garden whose every par raking turns back in open sand. B: the same garden with the wide rake."""
    H, W = rng.choice([(4, 5), (5, 4), (5, 5), (5, 6), (4, 6)])
    A = garden_layout(H, W, rng, n_pieces=1, max_frac=0.12)
    if A is None or not parity_ok(A):
        return None
    ra = counted(A)
    if ra is None or ra["nsol"] < 3:
        return None
    pa = min(ra["hist"])
    if not all(open_u_turn(A, d) for d in par_rakings(A, pa)):
        return None
    B = copy.deepcopy(A); B.nohairpin = 2; B.braced_start = True
    rb = B.solve(mode=0, cap=2)
    if rb["nsol"] != 1:
        return None
    gr = grade(B)
    if not gr["solved"] or gr["tier"] > 3:
        return None
    return {"A": A, "B": B, "score": (gr["tier"], gr["trials"] or 0, len(B.sand_cells())),
            "facts": {"A_rakings": ra["nsol"], "A_par": pa, "tier": gr["tier"]}}


def side_options(g):
    """Every cell of the north, east or west wall where a side gate could go (not the veranda gate's cell)."""
    out = [((0, c), "N") for c in range(g.W) if g.rows[0][c] == "."]
    out += [((r, 0), "W") for r in range(g.H - 1) if g.rows[r][0] == "."]
    out += [((r, g.W - 1), "E") for r in range(g.H - 1) if g.rows[r][g.W - 1] == "."]
    return [o for o in out if o[0] != (g.gate[0], g.gate[1])]


def make_side(rng):
    """A: a small garden for the wide rake with a few rakings. B: the same garden with a side gate, which leaves exactly one
    of them, the one that begins there. (B's rakings are always among A's: the side gate's cell touches a wall, so a
    raking that begins there is a valid raking of A. So A must have more than one.) Kept only if B's raking is not a par
    raking of A: the plan that was best before has to change."""
    H, W = rng.choice([(5, 5), (5, 6), (6, 5), (6, 6)])
    A = garden_layout(H, W, rng, n_pieces=1, max_frac=0.12)
    if A is None:
        return None
    A.nohairpin = 2; A.braced_start = True
    ra = A.solve(mode=0, cap=10**4)
    if not ra["complete"] or ra["capped"] or not (2 <= ra["nsol"] <= 8):
        return None
    pa = min(ra["hist"])
    best = None
    for cell, side in side_options(A):
        B = copy.deepcopy(A); B.end, B.end_side = cell, side
        rb = B.solve(mode=0, cap=2, stream=True)
        if rb["nsol"] != 1:
            continue
        d = rb["sols"][0]["draw"]
        if turns_of(A, d) == pa:
            continue
        gb = grade(B)
        if not gb["solved"] or gb["tier"] > 4:
            continue
        c = {"A": A, "B": B, "score": (gb["tier"], gb["trials"] or 0, len(B.sand_cells()), ra["nsol"]),
             "facts": {"A_rakings": ra["nsol"], "A_par": pa, "B_turns": turns_of(B, d), "B_tier": gb["tier"]}}
        if best is None or c["score"] < best["score"]:
            best = c
    return best


def make_ring(rng):
    """A small first garden with a ring the abbot has finished, pinned down by at most two grooves."""
    H, W = rng.choice([(5, 6), (6, 5), (6, 6)])
    g = garden_layout(H, W, rng, closed_ring=True, n_pieces=1, max_frac=0.1)
    if g is None or not parity_ok(g):
        return None
    if counted(g) is None:
        return None
    target = graceful_target(g)
    if target is None:
        return None
    cl = make_unique(g, target, rng)
    if cl is None or len(cl) > 2:
        return None
    B = with_clues(g, cl)
    gr = grade(B)
    if not gr["solved"] or gr["tier"] > 2:
        return None
    return {"B": B, "score": (len(cl), gr["tier"], len(B.sand_cells())), "facts": {"grooves": len(cl), "tier": gr["tier"]}}


MAKERS = {"ripple": make_ripple, "band": make_band, "grooves": make_grooves, "pebbles": make_pebbles,
          "wide": make_wide, "side": make_side, "ring": make_ring}


BUDGET = {"ripple": 120, "band": 120, "grooves": 120, "pebbles": 180, "wide": 240, "side": 420, "ring": 120}


def run(args):
    name, seed, budget = args
    return name, search(MAKERS[name], budget or BUDGET[name], random.Random(seed))
src/poems.py · 29 lines · 1.2 kB — The poems: one classical waka or haiku for each garden, and one for the practice garden, shown when the garden is 
"""The poems: one classical waka or haiku for each garden, and one for the practice garden, shown when the garden is
raked. poems/samon-poems.json is the curated selection (see its notes); attach() gives each garden the compact form the
page embeds, and returns the practice garden's."""
import json

PATH = '/home/claude/zen/poems/samon-poems.json'


def load():
    return json.load(open(PATH))


def compact(p):
    src = p.get("display_source")
    if src is None:
        src = " ".join(str(x) for x in (p.get("collection"), p.get("poem_number")) if x)
    return {"en": p["translation"], "ja": p["japanese"], "poet": p["poet"], "poet_ja": p.get("author_japanese", ""),
            "dates": p.get("author_dates", ""), "src": src, "url": p.get("source_url") or ""}


def attach(gardens):
    D = load()
    missing = [g["name"] for g in gardens if g["name"] not in D["gardens"]]
    assert not missing, f"gardens without a poem: {missing}"
    for g in gardens:
        p = D["poems"][D["gardens"][g["name"]]]
        assert p["translation"].strip() and p["japanese"].strip(), g["name"]
        g["poem"] = compact(p)
    return compact(D["poems"][D["practice"]])
src/preview.py · 10 lines · 511 bytes 
import json, sys
from levels import from_dict
from render import garden_svg, hstack, save
L = json.load(open('/home/claude/zen/out/levels.json'))
for key, lv in L.items():
    g = from_dict(lv['garden'])
    draw = [tuple(c) for c in lv['intended']]
    sv = hstack([garden_svg(g, label=f"{key}: puzzle", seed=3), garden_svg(g, draw=draw, label=f"{lv['intended_turns']} turns (par {lv['par']}), {lv['n_rakings']} rakings", seed=3)])
    save(sv, f"/home/claude/zen/fig/prev_{key}.png", scale=1.0)
print("done")
src/render.py · 337 lines · 15.8 kB — SVG/PNG figures of gardens, puzzles and rakings (no external drawing libraries). 
"""render.py: SVG/PNG figures of gardens, puzzles and rakings (no external drawing libraries)."""
from __future__ import annotations

import math
import random

from zg import Garden, glyphs, DIRS, DR, DC, OPP

S = 44            # cell size (px)
WALL = 14         # wall thickness
TINES = (0.2, 0.5, 0.8)
SAND = "#eeefec"
SAND_DOT = "#dcdedb"
GROOVE = "#8e928d"
GROOVE_HI = "#ffffff"
STONE = "#4a4d4b"
STONE_HI = "#6b6f6c"
WALLC = "#2f3130"
INK = "#1d1f1e"
SEAL = "#c23b22"
MARK = "#5b7a99"     # puzzle markings (bands, rings) in a cool slate blue


def _xy(r, c, ox, oy):
    return ox + c * S, oy + r * S


def _stone_path(cx, cy, rad, rng, sides=9):
    pts = []
    phase = rng.random() * math.tau
    for i in range(sides):
        a = phase + i * math.tau / sides
        rr = rad * (0.78 + 0.3 * rng.random())
        pts.append((cx + rr * math.cos(a), cy + rr * math.sin(a) * 0.86))
    # smooth closed path through midpoints (quadratic)
    d = ""
    n = len(pts)
    for i in range(n):
        p0, p1 = pts[i], pts[(i + 1) % n]
        mid = ((p0[0] + p1[0]) / 2, (p0[1] + p1[1]) / 2)
        if i == 0:
            pm = ((pts[-1][0] + p0[0]) / 2, (pts[-1][1] + p0[1]) / 2)
            d += f"M{pm[0]:.1f},{pm[1]:.1f} "
        d += f"Q{p0[0]:.1f},{p0[1]:.1f} {mid[0]:.1f},{mid[1]:.1f} "
    return d + "Z"


def _components(g: Garden, sym_set):
    """Connected components (4-neighbour) of blocked cells, for drawing multi-cell stones."""
    seen = set()
    comps = []
    for r in range(g.H):
        for c in range(g.W):
            if g.rows[r][c] in sym_set and (r, c) not in seen:
                comp = []
                stack = [(r, c)]
                seen.add((r, c))
                while stack:
                    a, b = stack.pop()
                    comp.append((a, b))
                    for d in DIRS:
                        nb = (a + DR[d], b + DC[d])
                        if 0 <= nb[0] < g.H and 0 <= nb[1] < g.W and g.rows[nb[0]][nb[1]] == g.rows[r][c] and nb not in seen:
                            seen.add(nb)
                            stack.append(nb)
                comps.append(comp)
    return comps


def _groove_cell(x, y, dirs, start=False, cls_color=GROOVE):
    """SVG path data for the tines through one cell; dirs is a set of 1-2 directions."""
    out = []
    dirs = set(dirs)
    if len(dirs) == 2 and dirs in ({"E", "W"}, {"N", "S"}):
        for o in TINES:
            if dirs == {"E", "W"}:
                out.append(f"M{x:.1f},{y + o * S:.1f} H{x + S:.1f}")
            else:
                out.append(f"M{x + o * S:.1f},{y:.1f} V{y + S:.1f}")
    elif len(dirs) == 2:
        # corner: arc centred on the cell corner shared by the two sides
        cx = x + (S if "E" in dirs else 0)
        cy = y + (S if "S" in dirs else 0)
        for o in TINES:
            rad = o * S if True else 0
            # point on first side and second side
            pts = []
            for d in sorted(dirs):
                if d in "NS":
                    yy = y + (0 if d == "N" else S)
                    xx = cx + (-rad if cx > x else rad)
                    pts.append((xx, yy))
                else:
                    xx = x + (0 if d == "W" else S)
                    yy = cy + (-rad if cy > y else rad)
                    pts.append((xx, yy))
            (x1, y1), (x2, y2) = pts
            out.append(f"M{x1:.1f},{y1:.1f} A{rad:.1f},{rad:.1f} 0 0 {_sweep(x1, y1, x2, y2, cx, cy)} {x2:.1f},{y2:.1f}")
    else:
        (d,) = tuple(dirs)
        # rake start: tines run from the edge in direction d back to just past the centre
        for o in TINES:
            if d in "EW":
                x_edge = x + (S if d == "E" else 0)
                x_end = x + (0.22 * S if d == "E" else 0.78 * S)
                out.append(f"M{x_edge:.1f},{y + o * S:.1f} H{x_end:.1f}")
            else:
                y_edge = y + (S if d == "S" else 0)
                y_end = y + (0.22 * S if d == "S" else 0.78 * S)
                out.append(f"M{x + o * S:.1f},{y_edge:.1f} V{y_end:.1f}")
    return out


def _sweep(x1, y1, x2, y2, cx, cy):
    # choose the short arc: sign of cross product of (p1-c) x (p2-c)
    cr = (x1 - cx) * (y2 - cy) - (y1 - cy) * (x2 - cx)
    return 1 if cr > 0 else 0


def garden_svg(g: Garden, draw=None, show_marks=True, show_clue_grooves=True, title=None, footprints=None,
               seed=1, label=None, sublabel=None, highlight_start=True, show_gate_arrow=True,
               ghost_path=None, glyph_override=None, marks_svg=None, checker=False, monk=None, show_gate=True):
    """Return an SVG string for one garden panel.
    draw: solution in drawing order (gate first) -> rendered as raked grooves.
    footprints: partial path (drawing order) rendered as footprints."""
    rng = random.Random(seed)
    top = 30 if title else 8
    bottom = 46 if (label or sublabel) else 16
    ox, oy = WALL + 10, top + WALL
    width = g.W * S + 2 * WALL + 20
    height = g.H * S + 2 * WALL + top + bottom
    el = []
    el.append(f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" viewBox="0 0 {width} {height}">')
    el.append(f'<rect x="0" y="0" width="{width}" height="{height}" fill="#ffffff"/>')
    if title:
        el.append(f'<text x="{width/2:.1f}" y="20" font-family="Georgia, serif" font-size="15" text-anchor="middle" fill="{INK}">{title}</text>')
    # wall
    el.append(f'<rect x="{ox - WALL}" y="{oy - WALL}" width="{g.W * S + 2 * WALL}" height="{g.H * S + 2 * WALL}" fill="{WALLC}" rx="2"/>')
    # sand (one rectangle: per-cell rects leave hairline seams when rasterised at fractional scales)
    el.append(f'<rect x="{ox}" y="{oy}" width="{g.W * S}" height="{g.H * S}" fill="{SAND}"/>')
    # unraked texture: sparse dots on cells without grooves
    raked = set()
    gl = {}
    if draw:
        gl = glyphs(g, draw)
        side = getattr(g, "end_side", None)
        if side and g.end is not None and tuple(draw[-1]) == tuple(g.end):
            gl[tuple(g.end)] = gl[tuple(g.end)] + side  # the stroke enters through the side gate
        raked = set(gl)
    if glyph_override is not None:
        gl = dict(glyph_override)
        raked = set(gl)
    if checker:
        for r in range(g.H):
            for c in range(g.W):
                if (r + c) % 2 == 0 and g.is_sand(r, c):
                    x, y = _xy(r, c, ox, oy)
                    el.append(f'<rect x="{x}" y="{y}" width="{S}" height="{S}" fill="#d6d9d4"/>')
    if show_clue_grooves:
        for cell, dirs in g.clues.items():
            raked.add(cell)
    for r in range(g.H):
        for c in range(g.W):
            if g.is_sand(r, c) and (r, c) not in raked:
                x, y = _xy(r, c, ox, oy)
                for _ in range(5):
                    el.append(f'<circle cx="{x + rng.random() * S:.1f}" cy="{y + rng.random() * S:.1f}" r="0.9" fill="{SAND_DOT}"/>')
    # puzzle marks: bands / ichimatsu hatching
    if show_marks and not draw:
        for r in range(g.H):
            for c in range(g.W):
                s = g.rows[r][c]
                x, y = _xy(r, c, ox, oy)
                if s in "h=":
                    for o in (0.25, 0.5, 0.75):
                        el.append(f'<line x1="{x}" y1="{y + o * S:.1f}" x2="{x + S}" y2="{y + o * S:.1f}" stroke="{MARK}" stroke-width="1" stroke-dasharray="3 4" opacity="0.8"/>')
                elif s in "v|":
                    for o in (0.25, 0.5, 0.75):
                        el.append(f'<line x1="{x + o * S:.1f}" y1="{y}" x2="{x + o * S:.1f}" y2="{y + S}" stroke="{MARK}" stroke-width="1" stroke-dasharray="3 4" opacity="0.8"/>')
        # band outlines
        for comp in _components(g, set("hv=|")):
            cs = set(comp)
            for (r, c) in comp:
                x, y = _xy(r, c, ox, oy)
                for d, (x1, y1, x2, y2) in {"N": (x, y, x + S, y), "S": (x, y + S, x + S, y + S),
                                             "W": (x, y, x, y + S), "E": (x + S, y, x + S, y + S)}.items():
                    if (r + DR[d], c + DC[d]) not in cs:
                        el.append(f'<line x1="{x1}" y1="{y1}" x2="{x2}" y2="{y2}" stroke="{MARK}" stroke-width="1.6"/>')
    # grooves
    paths = []
    if gl:
        for cell, dirs in gl.items():
            x, y = _xy(cell[0], cell[1], ox, oy)
            paths += _groove_cell(x, y, dirs)
    elif show_clue_grooves:
        for cell, dirs in g.clues.items():
            x, y = _xy(cell[0], cell[1], ox, oy)
            paths += _groove_cell(x, y, set(dirs))
    if paths:
        d = " ".join(paths)
        el.append(f'<path d="{d}" fill="none" stroke="{GROOVE_HI}" stroke-width="2.4" stroke-linecap="butt" transform="translate(-0.9,-0.9)"/>')
        el.append(f'<path d="{d}" fill="none" stroke="{GROOVE}" stroke-width="1.9" stroke-linecap="butt"/>')
    if ghost_path:
        pts = " ".join(f"{ox + c * S + S / 2:.1f},{oy + r * S + S / 2:.1f}" for r, c in ghost_path)
        el.append(f'<polyline points="{pts}" fill="none" stroke="{SEAL}" stroke-width="2" stroke-dasharray="5 4" opacity="0.8"/>')
    if footprints:
        for i, (r, c) in enumerate(footprints):
            x, y = _xy(r, c, ox, oy)
            cx, cy = x + S / 2, y + S / 2
            if i + 1 < len(footprints):
                nr, nc = footprints[i + 1]
            else:
                pr, pc_ = footprints[i - 1]
                nr, nc = 2 * r - pr, 2 * c - pc_
            ang = math.degrees(math.atan2(nr - r, nc - c))
            for side in (-1, 1):
                el.append(f'<ellipse cx="0" cy="{side * 5}" rx="6" ry="3" fill="{STONE_HI}" opacity="0.55" '
                          f'transform="translate({cx:.1f},{cy:.1f}) rotate({ang:.1f}) translate({side * 4},0)"/>')
    # ripple-stone marks: dotted circle through the ring
    if show_marks and not draw:
        for r in range(g.H):
            for c in range(g.W):
                if g.rows[r][c] == "O":
                    x, y = _xy(r, c, ox, oy)
                    cx, cy = x + S / 2, y + S / 2
                    for rad in (S * 0.95, S * 1.25):
                        el.append(f'<circle cx="{cx:.1f}" cy="{cy:.1f}" r="{rad:.1f}" fill="none" stroke="{MARK}" stroke-width="1.3" stroke-dasharray="2 4"/>')
    # stones
    for comp in _components(g, set("#O")):
        rs = [p[0] for p in comp]
        cs_ = [p[1] for p in comp]
        cx = ox + (min(cs_) + max(cs_) + 1) / 2 * S
        cy = oy + (min(rs) + max(rs) + 1) / 2 * S
        rad = 0.5 * S * max(max(rs) - min(rs) + 1, max(cs_) - min(cs_) + 1) * 0.92
        if len(comp) > 1 and len(comp) != (max(rs) - min(rs) + 1) * (max(cs_) - min(cs_) + 1):
            # irregular shape: draw each cell as a rounded blob
            for (r, c) in comp:
                x, y = _xy(r, c, ox, oy)
                el.append(f'<path d="{_stone_path(x + S / 2, y + S / 2, S * 0.5, rng)}" fill="{STONE}"/>')
            continue
        el.append(f'<path d="{_stone_path(cx, cy + 2, rad * 1.02, rng)}" fill="#c9cbc8" opacity="0.6"/>')
        el.append(f'<path d="{_stone_path(cx, cy, rad, rng)}" fill="{STONE}"/>')
        el.append(f'<path d="{_stone_path(cx - rad * 0.18, cy - rad * 0.22, rad * 0.45, rng, 7)}" fill="{STONE_HI}" opacity="0.7"/>')
    # the abbot's pebbles (shape marks): pale = straight through, dark = turn; small stones off-centre in the cell
    if show_marks and not draw:
        for (r, c), t in sorted(getattr(g, "marks", {}).items()):
            x, y = _xy(r, c, ox, oy)
            prng = random.Random(r * 1009 + c * 17 + 5)
            px, py, pr = x + S / 2 + 0.17 * S, y + S / 2 - 0.17 * S, 0.14 * S
            el.append(f'<path d="{_stone_path(px + 1, py + 1.3, pr * 1.08, random.Random(r * 1009 + c * 17 + 5))}" fill="#c9cbc8" opacity="0.8"/>')
            if t == "S":
                el.append(f'<path d="{_stone_path(px, py, pr, prng)}" fill="#ffffff" stroke="{STONE}" stroke-width="1.3"/>')
            else:
                el.append(f'<path d="{_stone_path(px, py, pr, prng)}" fill="{STONE}"/>')
    # gate: gap in the wall + shoe-stone outside
    gr, gc, gd = g.gate
    x, y = _xy(gr, gc, ox, oy)
    if gd == "S":
        gx, gy, gw, gh = x + 4, y + S, S - 8, WALL
        sx, sy = x + S / 2, y + S + WALL + 0
    elif gd == "N":
        gx, gy, gw, gh = x + 4, y - WALL, S - 8, WALL
        sx, sy = x + S / 2, y - WALL
    elif gd == "W":
        gx, gy, gw, gh = x - WALL, y + 4, WALL, S - 8
        sx, sy = x - WALL, y + S / 2
    else:
        gx, gy, gw, gh = x + S, y + 4, WALL, S - 8
        sx, sy = x + S + WALL, y + S / 2
    if show_gate:
        el.append(f'<rect x="{gx}" y="{gy}" width="{gw}" height="{gh}" fill="{SAND}"/>')
    if show_gate and show_gate_arrow:
        ax, ay = gx + gw / 2, gy + gh / 2
        rot = {"S": 0, "N": 180, "W": 90, "E": -90}[gd]
        el.append(f'<path d="M-5,-3 L5,-3 L0,4 Z" fill="{SEAL}" transform="translate({ax:.1f},{ay:.1f}) rotate({rot})"/>')
    # two-gate gardens: the side gate through which the stroke must begin
    side = getattr(g, "end_side", None)
    if g.end is not None and side:
        x, y = _xy(g.end[0], g.end[1], ox, oy)
        if side == "N":
            bx, by, bw, bh, rot = x + 4, y - WALL, S - 8, WALL, 0
        elif side == "S":
            bx, by, bw, bh, rot = x + 4, y + S, S - 8, WALL, 180
        elif side == "W":
            bx, by, bw, bh, rot = x - WALL, y + 4, WALL, S - 8, -90
        else:
            bx, by, bw, bh, rot = x + S, y + 4, WALL, S - 8, 90
        el.append(f'<rect x="{bx}" y="{by}" width="{bw}" height="{bh}" fill="{SAND}"/>')
        el.append(f'<path d="M-5,-3 L5,-3 L0,4 Z" fill="none" stroke="{SEAL}" stroke-width="1.5" transform="translate({bx + bw / 2:.1f},{by + bh / 2:.1f}) rotate({rot})"/>')
    if draw and highlight_start:
        r, c = draw[-1]
        x, y = _xy(r, c, ox, oy)
        el.append(f'<circle cx="{x + S / 2:.1f}" cy="{y + S / 2:.1f}" r="3.2" fill="{SEAL}"/>')
    if monk is not None:
        r, c = monk
        x, y = _xy(r, c, ox, oy)
        el.append(f'<circle cx="{x + S / 2:.1f}" cy="{y + S / 2:.1f}" r="9" fill="{INK}" stroke="#ffffff" stroke-width="2"/>')
        el.append(f'<circle cx="{x + S / 2:.1f}" cy="{y + S / 2 - 1:.1f}" r="4.2" fill="#e8e3da"/>')
    if marks_svg:
        el.append(marks_svg(ox, oy))
    if label:
        el.append(f'<text x="{width/2:.1f}" y="{oy + g.H * S + WALL + 20:.1f}" font-family="Georgia, serif" font-size="14" text-anchor="middle" fill="{INK}">{label}</text>')
    if sublabel:
        el.append(f'<text x="{width/2:.1f}" y="{oy + g.H * S + WALL + 37:.1f}" font-family="Georgia, serif" font-size="12" font-style="italic" text-anchor="middle" fill="#555">{sublabel}</text>')
    el.append("</svg>")
    return "\n".join(el)


def hstack(svgs, gap=10):
    """Place several SVG panels side by side in one SVG."""
    import re
    dims = []
    bodies = []
    for s in svgs:
        m = re.search(r'width="([\d.]+)" height="([\d.]+)"', s)
        w, h = float(m.group(1)), float(m.group(2))
        dims.append((w, h))
        body = s.split(">", 1)[1].rsplit("</svg>", 1)[0]
        bodies.append(body)
    W = sum(w for w, _ in dims) + gap * (len(svgs) - 1)
    H = max(h for _, h in dims)
    out = [f'<svg xmlns="http://www.w3.org/2000/svg" width="{W:.0f}" height="{H:.0f}" viewBox="0 0 {W:.0f} {H:.0f}">',
           f'<rect x="0" y="0" width="{W:.0f}" height="{H:.0f}" fill="#ffffff"/>']
    x = 0
    for (w, h), body in zip(dims, bodies):
        out.append(f'<g transform="translate({x:.1f},{(H - h) / 2:.1f})">{body}</g>')
        x += w + gap
    out.append("</svg>")
    return "\n".join(out)


def save(svg, path_png, path_svg=None, scale=2.0):
    import cairosvg
    if path_svg:
        with open(path_svg, "w") as f:
            f.write(svg)
    cairosvg.svg2png(bytestring=svg.encode(), write_to=path_png, scale=scale)
src/search2.py · 64 lines · 2.5 kB — Second-pass searches: prettier unique gardens for the wide rake and two-gate mechanics. 
"""Second-pass searches: prettier unique gardens for the wide rake and two-gate mechanics."""
import json, random, sys, time
from multiprocessing import Pool
from gen import random_layout, parity_ok
from zg import Garden, turns_of, hairpin_list
from grader import grade
from levels import to_dict

def wide_job(seed):
    rng = random.Random(seed)
    out = []
    t0 = time.time()
    while time.time() - t0 < 240:
        H, W = rng.choice([(7, 7), (7, 8), (8, 8)])
        g = random_layout(H, W, rng.choice([1, 2, 3]), rng, gate_side="S", min_gap=1)
        if g is None or not parity_ok(g): continue
        g.nohairpin = 2; g.braced_start = True
        r = g.solve(mode=0, cap=2)
        if r["nsol"] != 1: continue
        d = r["sols"][0]["draw"]
        dens = turns_of(g, d) / len(d)
        if dens > 0.45: continue
        gr = grade(g)
        if gr["solved"] and gr["tier"] >= 3:
            out.append((dens, gr["trials"], to_dict(g), d))
    return out

def two_job(seed):
    rng = random.Random(seed)
    out = []
    t0 = time.time()
    while time.time() - t0 < 240:
        H, W = rng.choice([(7, 7), (6, 8), (7, 8)])
        g = random_layout(H, W, rng.choice([1, 2, 3]), rng, gate_side="S", min_gap=1)
        if g is None: continue
        side = rng.choice("EWN")
        if side == "E": cells = [(r, W - 1) for r in range(H - 2)]
        elif side == "W": cells = [(r, 0) for r in range(H - 2)]
        else: cells = [(0, c) for c in range(W)]
        cells = [c for c in cells if g.rows[c[0]][c[1]] == "."]
        if not cells: continue
        g.end = rng.choice(cells)
        r = g.solve(mode=0, cap=2)
        if r["nsol"] != 1: continue
        d = r["sols"][0]["draw"]
        dens = turns_of(g, d) / len(d)
        ub = sum(1 for *_, br in hairpin_list(g, d) if not br)
        if dens > 0.42 or ub > 0: continue
        gr = grade(g)
        if gr["solved"]:
            dd = to_dict(g); dd["end_side"] = side
            out.append((dens, gr["tier"], gr["trials"], dd, d))
    return out

if __name__ == "__main__":
    kind = sys.argv[1]
    with Pool(2) as p:
        res = p.map(wide_job if kind == "wide" else two_job, [11, 12])
    res = [x for r in res for x in r]
    res.sort(key=lambda x: x[0])
    print(kind, "found", len(res))
    for x in res[:8]:
        print(x[:3] if kind == "wide" else x[:3], x[-2]["rows"] if kind == "wide" else x[3]["rows"])
    json.dump(res[:20], open(f"/home/claude/zen/out/search2_{kind}.json", "w"))
src/search3.py · 38 lines · 1.4 kB 
import json, random, sys, time
from multiprocessing import Pool
from gen import random_layout, parity_ok
from zg import turns_of, hairpin_list
from grader import grade
from levels import to_dict

def wide_job(seed):
    rng = random.Random(seed)
    out = {}
    t0 = time.time()
    while time.time() - t0 < float(sys.argv[2]):
        H, W = rng.choice([(7, 7), (7, 8), (8, 8)])
        k = rng.choice([2, 3, 4])
        g = random_layout(H, W, k, rng, gate_side="S", min_gap=1)
        if g is None or not parity_ok(g): continue
        g.nohairpin = 2; g.braced_start = True
        key = (tuple(g.rows), tuple(g.gate))
        if key in out: continue
        r = g.solve(mode=0, cap=2)
        if r["nsol"] != 1: continue
        d = r["sols"][0]["draw"]
        dens = turns_of(g, d) / len(d)
        if dens > 0.40: continue
        gr = grade(g)
        if gr["solved"] and gr["tier"] >= 3:
            out[key] = (dens, gr["trials"], k, to_dict(g), d)
    return list(out.values())

if __name__ == "__main__":
    with Pool(2) as p:
        res = p.map(wide_job, [101, 202])
    res = [x for r in res for x in r]
    res.sort(key=lambda x: (x[0], -x[2]))
    print("found", len(res))
    for x in res[:12]:
        print(round(x[0],3), x[1], x[2], x[3]["rows"], x[3]["gate"])
    json.dump(res, open("/home/claude/zen/out/search3_wide.json", "w"))
src/search4.py · 48 lines · 1.9 kB 
import json, random, sys, time
from multiprocessing import Pool
from gen import random_layout, parity_ok
from zg import turns_of, hairpin_list
from grader import grade
from levels import to_dict

def job(seed):
    rng = random.Random(seed)
    out = {}
    t0 = time.time()
    while time.time() - t0 < float(sys.argv[1]):
        H, W = rng.choice([(7, 7), (7, 8), (8, 8)])
        k = rng.choice([1, 2, 3])
        g = random_layout(H, W, k, rng, gate_side="S", min_gap=1)
        if g is None: continue
        side = rng.choice("EWN")
        if side == "E": cells = [(r, W - 1) for r in range(H - 2)]
        elif side == "W": cells = [(r, 0) for r in range(H - 2)]
        else: cells = [(0, c) for c in range(W)]
        cells = [c for c in cells if g.rows[c[0]][c[1]] == "."]
        if not cells: continue
        g.end = rng.choice(cells)
        wide = rng.random() < 0.5
        if wide: g.nohairpin = 2; g.braced_start = True
        key = (tuple(g.rows), tuple(g.gate), g.end, wide)
        if key in out: continue
        r = g.solve(mode=0, cap=2)
        if r["nsol"] != 1: continue
        d = r["sols"][0]["draw"]
        dens = turns_of(g, d) / len(d)
        ub = sum(1 for *_, br in hairpin_list(g, d) if not br)
        if dens > 0.40 or ub > 0: continue
        gr = grade(g)
        if gr["solved"]:
            dd = to_dict(g); dd["end_side"] = side
            out[key] = (dens, gr["tier"], gr["trials"], wide, dd, d)
    return list(out.values())

if __name__ == "__main__":
    with Pool(2) as p:
        res = p.map(job, [7, 8])
    res = [x for r in res for x in r]
    res.sort(key=lambda x: x[0])
    print("found", len(res), "wide:", sum(x[3] for x in res))
    for x in res[:12]:
        print(round(x[0],3), x[1], x[2], x[3], x[4]["rows"], x[4]["gate"], x[4]["end"], x[4]["end_side"])
    json.dump(res, open("/home/claude/zen/out/search4_two.json", "w"))
src/search_finale.py · 61 lines · 2.5 kB 
import json, random, sys, time
from multiprocessing import Pool
from zg import Garden, turns_of, hairpin_list
from gen import parity_ok
from levels import to_dict

def polyomino(k, rng):
    cells = {(0, 0)}
    while len(cells) < k:
        r, c = rng.choice(sorted(cells))
        d = rng.choice([(0,1),(1,0),(0,-1),(-1,0)])
        cells.add((r + d[0], c + d[1]))
    mr = min(r for r, c in cells); mc = min(c for r, c in cells)
    return {(r - mr, c - mc) for r, c in cells}

def layout(H, W, sizes, rng):
    rows = [["."] * W for _ in range(H)]
    occupied = set()
    for k in sizes:
        for _ in range(200):
            p = polyomino(k, rng)
            ph = max(r for r, c in p) + 1; pw = max(c for r, c in p) + 1
            if ph > min(3, H - 2) or pw > W - 2: continue
            r0 = rng.randint(1, H - ph - 1); c0 = rng.randint(1, W - pw - 1)
            cells = {(r0 + r, c0 + c) for r, c in p}
            halo = {(r + dr, c + dc) for r, c in cells for dr in (-2, -1, 0, 1, 2) for dc in (-2, -1, 0, 1, 2)}
            if halo & occupied: continue
            for r, c in cells: rows[r][c] = "#"
            occupied |= cells
            break
        else:
            return None
    return rows

def job(seed):
    rng = random.Random(seed); out = []; t0 = time.time(); tried = 0
    H, W = int(sys.argv[2]), int(sys.argv[3])
    while time.time() - t0 < float(sys.argv[1]):
        sizes = [5, 3, 3, 2, 2]; rng.shuffle(sizes)
        rows = layout(H, W, sizes, rng)
        if rows is None: continue
        gc = rng.choice([c for c in range(W) if rows[H-1][c] == "."])
        g = Garden(["".join(r) for r in rows], (H - 1, gc, "S"), nohairpin=2, braced_start=True)
        tried += 1
        if not parity_ok(g): continue
        r = g.solve(mode=0, cap=2, nodelimit=5 * 10**7)
        if r["nsol"] != 1 or not r["complete"]: continue
        d = r["sols"][0]["draw"]
        dens = turns_of(g, d) / len(d)
        out.append((dens, to_dict(g), d, sizes))
    return out, tried

if __name__ == "__main__":
    with Pool(2) as p:
        res = p.map(job, [int(sys.argv[4]), int(sys.argv[4]) + 1])
    rows = [x for r, _ in res for x in r]; tried = sum(t for _, t in res)
    rows.sort(key=lambda x: x[0])
    print("tried", tried, "unique", len(rows))
    for x in rows[:6]:
        print(round(x[0], 3), x[3], x[1]["rows"], x[1]["gate"])
    json.dump(rows[:30], open(f"/home/claude/zen/out/finale_{sys.argv[2]}x{sys.argv[3]}.json", "w"))
src/search_finale2.py · 27 lines · 1.2 kB 
import json, random, sys, time
from zg import Garden, turns_of, hairpin_list
from gen import parity_ok
from levels import to_dict
from zcp import optimize
from search_finale import layout
rng = random.Random(int(sys.argv[3]))
H, W = int(sys.argv[1]), int(sys.argv[2])
t0 = time.time(); out = []; tried = 0
while time.time() - t0 < float(sys.argv[4]):
    sizes = [5, 3, 3, 2, 2]; rng.shuffle(sizes)
    rows = layout(H, W, sizes, rng)
    if rows is None: continue
    gc = rng.choice([c for c in range(W) if rows[H-1][c] == "."])
    g = Garden(["".join(r) for r in rows], (H - 1, gc, "S"))
    tried += 1
    if not parity_ok(g): continue
    if g.solve(mode=0, cap=1, nodelimit=10**7)['nsol'] == 0: continue
    free = optimize(g, time_limit=60, workers=2)
    if free[1] != 'OPTIMAL': continue
    grace = optimize(g, nohairpin=2, time_limit=60, workers=2)
    ok = grace[1] == 'OPTIMAL' and grace[0] == free[0]
    print(f"tried {tried} par {free[0]} graceful {grace[0]} {grace[1]} ok={ok} ({time.time()-t0:.0f}s)", flush=True)
    if ok:
        out.append((free[0], to_dict(g), list(reversed(grace[2])), sizes))
json.dump(out, open(f"/home/claude/zen/out/finale2_{H}x{W}_{sys.argv[3]}.json", "w"))
print("found", len(out))
src/seed_garden_names.py · 61 lines · 3.9 kB — One-time seeding of out/garden_names.json: the names the gardens had in the 136-garden campaign (the one Astra chose 
"""One-time seeding of out/garden_names.json: the names the gardens had in the 136-garden campaign (the one Astra chose
poems for), recovered by matching each current garden's layout to that campaign's per-level record (size, gate, pars and
counts, stones, marks, and side gate; side-gate pars are matched without par, since the bend inside a side gate now
counts as a turn). From now on campaign.py names a garden by its layout, so a garden keeps its name (and its poem, and a
player's saved progress) when the campaign is rearranged. The two world-7 ring gardens generated since then take the
names of the two ring gardens they replaced."""
import json, collections

A = json.load(open('/home/claude/zen/poems/astra-samon-poem-selection.json'))
C = json.load(open('/home/claude/zen/out/campaign.json'))


def full_key(d):
    return json.dumps([d["rows"], d["gate"], d["end"], d.get("end_side"), d["nohairpin"], bool(d["braced_start"]),
                       sorted(d["clues"].items()), sorted((d.get("marks") or {}).items())])


def sig_new(e, par=True):
    d = e["garden"]; rows = d["rows"]; s = "".join(rows); marks = d.get("marks") or {}
    return (len(rows), len(rows[0]), tuple(d["gate"]), e.get("par") if par else None, s.count('#'), s.count('O'), s.count('R'),
            s.count('h'), s.count('v'), s.count('='), s.count('|'), len(d["clues"]), sum(v == 'S' for v in marks.values()),
            sum(v == 'B' for v in marks.values()), d["nohairpin"] == 2, tuple(d["end"]) if d["end"] else None)


def sig_old(L, par=True):
    m = L["mechanics"]
    return (m["rows"], m["columns"], tuple(m["gate_zero_based"]), m["par_turns"] if par else None, m["ordinary_stone_cells"],
            m["ripple_stones"], m["protected_cells"], m["horizontal_band_cells"], m["vertical_band_cells"], m["horizontal_block_cells"],
            m["vertical_block_cells"], m["given_grooves"], m["pale_pebbles"], m["dark_pebbles"], m["wide_rake"],
            tuple(m["second_endpoint_zero_based"]) if m["second_endpoint_zero_based"] else None)


P = {p["name"]: p for p in json.load(open('/home/claude/zen/out/proto_gardens.json'))}
for e in C:
    e["par"] = P[e["name"]]["par"]
old = collections.defaultdict(list)
for L in A["levels"]:
    old[sig_old(L, False)].append(L)
reg, used = {}, set()
for e in C:
    cands = [L for L in old.get(sig_new(e, False), []) if L["name"] not in used]
    if not cands:
        continue
    exact = [L for L in cands if L["mechanics"]["par_turns"] == e["par"]]
    side = e["garden"]["end"] is not None
    cands = exact if exact else ([L for L in cands if L["mechanics"]["par_turns"] + 1 == e["par"]] if side else [])
    if len(cands) > 1:   # two 5x5 wide-rake gardens with a two-cell stone on the right wall: told apart by Astra's description
        rows = e["garden"]["rows"]
        above = rows[1][-1] == "#"
        cands = [L for L in cands if ("above the midpoint" in L["description"]) == above]
    assert len(cands) == 1, (e["name"], [L["name"] for L in cands])
    reg[full_key(e["garden"])] = cands[0]["name"]; used.add(cands[0]["name"])
# the two world-7 ring gardens that replaced "Lantern festival" and "Dark water" (the ring spec was regenerated)
rings7 = [e for e in C if e["world"] == 7 and "R" in "".join(e["garden"]["rows"]) and full_key(e["garden"]) not in reg]
assert len(rings7) == 2 and {"Lantern festival", "Dark water"}.isdisjoint(used), [e["name"] for e in rings7]
rings7.sort(key=lambda e: (e["role"] != "intro", e["name"]))
for e, nm in zip(rings7, ("Lantern festival", "Dark water")):
    reg[full_key(e["garden"])] = nm; used.add(nm)
missing = [L["name"] for L in A["levels"] if L["name"] not in used]
print(len(reg), "layouts named;", "old names with no layout now:", missing)
json.dump([{"name": v, "key": k} for k, v in sorted(reg.items(), key=lambda kv: kv[1])], open('/home/claude/zen/out/garden_names.json', 'w'), indent=0)
src/sound_pieces.py · 26 lines · 1.3 kB — The recordings carried in the page, as HTML for the builders (proto/build.py, standalone/build_standalone.py). 
"""The recordings carried in the page, as HTML for the builders (proto/build.py, standalone/build_standalone.py).

out/sound_pieces.json (sound/make_sound_bundle.py) holds each encoded recording with its credits. Each becomes an inert
<script type="text/plain" class="piece"> whose text is a data: URL; the page reads them (loadPieces in the template), and
Gwern's gwtar tool moves them into its archive, leaving a path the page resolves through the gwtar loader (samonAsset).
"""
import html, json, os

ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
PIECES = os.path.join(ROOT, "out", "sound_pieces.json")


def load():
    return json.load(open(PIECES)) if os.path.exists(PIECES) else []


def pieces_html(pieces=None):
    pieces = load() if pieces is None else pieces
    if not pieces:
        print("warning: no recordings (out/sound_pieces.json missing): the page will have no shakuhachi")
        return "<!-- no recordings -->"
    out = []
    for i, p in enumerate(pieces):
        attrs = " ".join(f'data-{k}="{html.escape(str(p[k]), quote=True)}"' for k in ("title", "player", "license", "source", "mime", "gain", "dur"))
        out.append(f'<script type="text/plain" class="piece" id="piece-{i}" {attrs}>data:{p["mime"]};base64,{p["b64"]}</script>')
    return "\n".join(out)
src/test_crosscheck.py · 70 lines · 2.7 kB — Cross-check the C solver against the brute-force Python enumerator on random small gardens. 
"""Cross-check the C solver against the brute-force Python enumerator on random small gardens."""
import random
import sys
from zg import Garden, brute_force, check_solution

random.seed(int(sys.argv[1]) if len(sys.argv) > 1 else 1)


def random_garden():
    H = random.randint(3, 5)
    W = random.randint(3, 5)
    while H * W > 22:
        W -= 1
    rows = [["." for _ in range(W)] for _ in range(H)]
    for _ in range(random.randint(0, 3)):
        r, c = random.randrange(H), random.randrange(W)
        rows[r][c] = "#"
    # maybe a ripple stone
    if random.random() < 0.3 and H >= 3 and W >= 3:
        r, c = random.randint(1, H - 2), random.randint(1, W - 2)
        for rr in range(r - 1, r + 2):
            for cc in range(c - 1, c + 2):
                rows[rr][cc] = "."
        rows[r][c] = "O"
    # maybe a wave band
    if random.random() < 0.3:
        sym = random.choice("hv")
        r0, c0 = random.randrange(H), random.randrange(W)
        for rr in range(r0, min(H, r0 + random.randint(1, 2))):
            for cc in range(c0, min(W, c0 + random.randint(1, 3))):
                if rows[rr][cc] == ".":
                    rows[rr][cc] = sym
    # gate on the border
    border = [(r, c) for r in range(H) for c in range(W) if (r in (0, H - 1) or c in (0, W - 1)) and rows[r][c] in ".hv"]
    if not border:
        return None
    gr, gc = random.choice(border)
    sides = [s for s, ok in (("N", gr == 0), ("S", gr == H - 1), ("W", gc == 0), ("E", gc == W - 1)) if ok]
    clues = {}
    if random.random() < 0.3:
        sand = [(r, c) for r in range(H) for c in range(W) if rows[r][c] == "."]
        if sand:
            cell = random.choice(sand)
            clues[cell] = random.choice(["EW", "NS", "NE", "ES", "SW", "NW"])
    g = Garden(["".join(r) for r in rows], (gr, gc, random.choice(sides)), clues=clues,
               nohairpin=random.choice([0, 0, 1, 2]), braced_turns=random.random() < 0.25, braced_start=random.random() < 0.3)
    if random.random() < 0.15:
        sand = g.sand_cells()
        g.end = random.choice(sand)
    return g


ok = bad = 0
for trial in range(int(sys.argv[2]) if len(sys.argv) > 2 else 150):
    g = random_garden()
    if g is None:
        continue
    try:
        g.ring_sets()
    except ValueError:
        continue
    res = g.solve(mode=0, cap=10**9)
    n_bf, hist_bf = brute_force(g)
    sols_ok = all(check_solution(g, s["draw"]) for s in res["sols"])
    if res["nsol"] != n_bf or res["hist"] != hist_bf or not sols_ok:
        bad += 1
        print("MISMATCH", g, res["nsol"], n_bf, res["hist"], hist_bf, sols_ok)
    else:
        ok += 1
print("ok", ok, "bad", bad)
src/test_grader.py · 27 lines · 902 bytes 
import random, time
from gen import random_layout
from clues import make_unique, with_clues
from grader import grade, Board, State
from zg import glyphs
rng = random.Random(3)
tiers = {}
t0 = time.time()
done = 0
while done < 40:
    variant = rng.choice(["plain", "wide"])
    g = random_layout(6, 6, rng.choice([0, 1, 2, 3]), rng, gate_side="S")
    if g is None: continue
    if variant == "wide": g.nohairpin = 2; g.braced_start = True
    r = g.solve(mode=0, cap=1)
    if r["nsol"] == 0: continue
    clues = make_unique(g, r["sols"][0]["draw"], rng)
    if clues is None: continue
    gu = with_clues(g, clues)
    ru = gu.solve(mode=0, cap=2)
    assert ru["nsol"] == 1
    res = grade(gu)
    tiers.setdefault((variant, res["tier"]), 0)
    tiers[(variant, res["tier"])] += 1
    done += 1
    print(variant, len(clues), res, flush=True)
print(sorted(tiers.items()), f"{time.time()-t0:.0f}s")
src/test_grader_marks.py · 58 lines · 2.7 kB — Soundness of the grader's deductions (incl. shape marks): on random gardens, it must never derive a 
"""Soundness of the grader's deductions (incl. shape marks): on random gardens, it must never derive a
contradiction when a raking exists, and when it claims a solution, that must be the unique raking."""
import random, sys
from zg import Garden, glyphs
from grader import grade, Board, State, trial_loop, Contradiction

rng = random.Random(int(sys.argv[1]) if len(sys.argv) > 1 else 7)
n_ok = n_bad = n_uniq = n_solved = 0
for t in range(int(sys.argv[2]) if len(sys.argv) > 2 else 300):
    H, W = rng.randint(4, 6), rng.randint(4, 6)
    rows = [["."] * W for _ in range(H)]
    for _ in range(rng.randint(0, 3)):
        rows[rng.randrange(H)][rng.randrange(W)] = "#"
    if rows[H - 1][0] != ".":
        continue
    g = Garden(["".join(r) for r in rows], (H - 1, 0, "S"))
    sand = g.sand_cells()
    if rng.random() < 0.3:
        g.nohairpin = 2; g.braced_start = True
    if rng.random() < 0.5:                       # random marks (often unsolvable)
        for _ in range(rng.randint(1, 6)):
            g.marks[rng.choice(sand)] = rng.choice("SB")
    else:                                        # marks taken from one raking (solvable, often unique)
        r0 = g.solve(mode=0, cap=200, stream=True)
        if not r0["sols"]:
            continue
        draw = rng.choice(r0["sols"])["draw"]
        gl = glyphs(g, draw)
        two = [c for c in draw[:-1] if len(gl[c]) == 2]
        for c in rng.sample(two, min(len(two), rng.randint(2, 10))):
            g.marks[c] = "S" if set(gl[c]) in ({"E", "W"}, {"N", "S"}) else "B"
    r = g.solve(mode=0, cap=3, stream=True)
    res = grade(g)
    bad = False
    if r["nsol"] >= 1 and res["tier"] == 0:
        bad = True                                      # contradiction although a raking exists
    if r["nsol"] == 0 and res["solved"]:
        bad = True
    if r["nsol"] >= 2 and res["solved"]:
        bad = True
    if r["nsol"] == 1:
        n_uniq += 1
        if res["solved"]:
            n_solved += 1
            # re-derive the grader's final edge set and compare with the solver's raking
            B = Board(g); st = State(B, use_global=True); st.propagate()
            for depth in range(2):
                if st.solved(): break
                st = trial_loop(st.copy(), depth, stats=None)
            draw = r["sols"][0]["draw"]
            want = {B.eid[(B.idx[a], B.idx[b])] for a, b in zip(draw, draw[1:])}
            got = {e for e, x in enumerate(st.s) if x == 1}
            bad |= want != got
    if bad:
        n_bad += 1; print("BAD", g, r["nsol"], res)
    else:
        n_ok += 1
print(f"grader soundness with marks: ok {n_ok} bad {n_bad} (unique {n_uniq}, solved by grader {n_solved})")
src/test_marks.py · 100 lines · 4.0 kB — Cross-check shape marks ('S' straight / 'B' turn): C solver vs brute force (small gardens, all rules), 
"""Cross-check shape marks ('S' straight / 'B' turn): C solver vs brute force (small gardens, all rules),
streamed solutions vs counts, and C vs CP-SAT (medium gardens: counts, histograms, par)."""
import random
import sys
from zg import Garden, brute_force, check_solution
from zcp import count_solutions, optimize

seed = int(sys.argv[1]) if len(sys.argv) > 1 else 1
trials = int(sys.argv[2]) if len(sys.argv) > 2 else 2000
random.seed(seed)


def random_garden(Hr=(3, 5), Wr=(3, 5), maxcells=22, rules=True):
    H, W = random.randint(*Hr), random.randint(*Wr)
    while H * W > maxcells:
        W -= 1
    rows = [["." for _ in range(W)] for _ in range(H)]
    for _ in range(random.randint(0, 3)):
        rows[random.randrange(H)][random.randrange(W)] = "#"
    if rules and random.random() < 0.25 and H >= 3 and W >= 3:
        r, c = random.randint(1, H - 2), random.randint(1, W - 2)
        for rr in range(r - 1, r + 2):
            for cc in range(c - 1, c + 2):
                rows[rr][cc] = "."
        rows[r][c] = "O"
    if rules and random.random() < 0.25:
        sym = random.choice("hv")
        r0, c0 = random.randrange(H), random.randrange(W)
        for rr in range(r0, min(H, r0 + random.randint(1, 2))):
            for cc in range(c0, min(W, c0 + random.randint(1, 3))):
                if rows[rr][cc] == ".":
                    rows[rr][cc] = sym
    border = [(r, c) for r in range(H) for c in range(W) if (r in (0, H - 1) or c in (0, W - 1)) and rows[r][c] in ".hv"]
    if not border:
        return None
    gr, gc = random.choice(border)
    sides = [s for s, ok in (("N", gr == 0), ("S", gr == H - 1), ("W", gc == 0), ("E", gc == W - 1)) if ok]
    g = Garden(["".join(r) for r in rows], (gr, gc, random.choice(sides)))
    sand = g.sand_cells()
    if rules:
        if random.random() < 0.2:
            g.clues[random.choice(sand)] = random.choice(["EW", "NS", "NE", "ES", "SW", "NW"])
        g.nohairpin = random.choice([0, 0, 1, 2])
        g.braced_start = random.random() < 0.3
        if random.random() < 0.15:
            g.end = random.choice(sand)
    for _ in range(random.choice([1, 1, 2, 3])):
        cell = random.choice(sand)
        if cell not in g.clues:
            g.marks[cell] = random.choice("SB")
    try:
        g.ring_sets()
    except ValueError:
        return None
    return g


ok = bad = solvable = 0
for t in range(trials):
    g = random_garden()
    if g is None:
        continue
    res = g.solve(mode=0, cap=10**9, stream=True)
    n_bf, hist_bf = brute_force(g)
    draws = [tuple(map(tuple, s["draw"])) for s in res["sols"]]
    good = (res["nsol"] == n_bf and res["hist"] == hist_bf and len(draws) == res["nsol"]
            and len(set(draws)) == len(draws) and all(check_solution(g, list(d)) for d in draws))
    if good:
        ok += 1
        solvable += res["nsol"] > 0
    else:
        bad += 1
        print("MISMATCH", g, res["nsol"], n_bf, res["hist"], hist_bf)
print(f"small (C vs brute force, streamed sols checked): ok {ok} bad {bad} solvable {solvable}")

# medium gardens: C vs CP-SAT on counts/histograms (when small enough) and on par
ok2 = bad2 = 0
random.seed(seed + 1000)
for t in range(max(20, trials // 50)):
    g = random_garden(Hr=(6, 7), Wr=(6, 7), maxcells=49, rules=False)
    if g is None:
        continue
    for _ in range(random.randint(1, 4)):
        g.marks[random.choice(g.sand_cells())] = random.choice("SB")
    res = g.solve(mode=0, cap=3000)
    if not res["complete"] or res["capped"]:
        continue
    if res["nsol"] == 0:
        st = optimize(g, time_limit=60)[1]
        good = st == "INFEASIBLE"
    else:
        n_cp, hist_cp, st = count_solutions(g, limit=10000)
        par = optimize(g, time_limit=60)
        good = n_cp == res["nsol"] and hist_cp == res["hist"] and par[0] == min(res["hist"]) and par[1] == "OPTIMAL"
    if good:
        ok2 += 1
    else:
        bad2 += 1
        print("CP-SAT MISMATCH", g, res["nsol"], res["hist"])
print(f"medium (C vs CP-SAT): ok {ok2} bad {bad2}")
src/test_side_pivot.py · 83 lines · 4.2 kB — Cross-check the two rule changes of 2026-10-04 across all three solvers: 
"""Cross-check the two rule changes of 2026-10-04 across all three solvers:
(1) with a side gate, the bend where the stroke enters through it counts as a turn;
(2) the abbot's finished sand ('R') is not walkable and is no pivot for a wide-rake U-turn or the rake's start.
Small gardens: C solver vs brute force (counts, turn histograms, every solution valid). Medium gardens: C solver vs
CP-SAT (minimum turns; counts on a subset)."""
import random
import sys
from zg import Garden, brute_force, check_solution
from zcp import optimize, count_solutions

rng = random.Random(int(sys.argv[1]) if len(sys.argv) > 1 else 1)
N_SMALL = int(sys.argv[2]) if len(sys.argv) > 2 else 1500
N_MEDIUM = int(sys.argv[3]) if len(sys.argv) > 3 else 60


def garden(Hr, Wr):
    H, W = rng.randint(*Hr), rng.randint(*Wr)
    rows = [["." for _ in range(W)] for _ in range(H)]
    for _ in range(rng.randint(0, 3)):
        rows[rng.randrange(H)][rng.randrange(W)] = rng.choice("##R")
    if rng.random() < 0.3:                                     # a stone with some finished sand beside it
        r, c = rng.randrange(H), rng.randrange(W)
        rows[r][c] = "#"
        for dr, dc in ((0, 1), (1, 0), (0, -1), (-1, 0)):
            if 0 <= r + dr < H and 0 <= c + dc < W and rng.random() < 0.5:
                rows[r + dr][c + dc] = "R"
    if rng.random() < 0.2:
        sym = rng.choice("hv"); r0, c0 = rng.randrange(H), rng.randrange(W)
        for rr in range(r0, min(H, r0 + 2)):
            for cc in range(c0, min(W, c0 + 2)):
                if rows[rr][cc] == ".":
                    rows[rr][cc] = sym
    border = [(r, c) for r in range(H) for c in range(W) if (r in (0, H - 1) or c in (0, W - 1)) and rows[r][c] in ".hv"]
    if len(border) < 2:
        return None
    gr, gc = rng.choice(border)
    sides = [s for s, ok in (("N", gr == 0), ("S", gr == H - 1), ("W", gc == 0), ("E", gc == W - 1)) if ok]
    g = Garden(["".join(r) for r in rows], (gr, gc, rng.choice(sides)),
               nohairpin=rng.choice([0, 1, 2, 2]), braced_start=rng.random() < 0.4)
    if rng.random() < 0.6:                                     # a side gate in another border cell
        er, ec = rng.choice([b for b in border if b != (gr, gc)])
        g.end = (er, ec)
        g.end_side = rng.choice([s for s, ok in (("N", er == 0), ("S", er == H - 1), ("W", ec == 0), ("E", ec == W - 1)) if ok])
    if rng.random() < 0.2:
        sand = [c for c in g.sand_cells() if c != (gr, gc) and c != g.end]
        if sand:
            c = rng.choice(sand); g.marks = {c: rng.choice("SB")}
    return g


ok = bad = solvable = side = soft = 0
for _ in range(N_SMALL):
    g = garden((3, 5), (3, 5))
    if g is None or len(g.sand_cells()) > 16:
        continue
    res = g.solve(mode=0, cap=10**9)
    n_bf, hist_bf = brute_force(g)
    good = res["nsol"] == n_bf and res["hist"] == hist_bf and all(check_solution(g, s["draw"]) for s in res["sols"])
    ok += good; bad += not good
    solvable += n_bf > 0; side += g.end_side is not None and n_bf > 0; soft += any("R" in r for r in g.rows) and n_bf > 0
    if not good:
        print("MISMATCH small", g, res["nsol"], n_bf, res["hist"], hist_bf)
print(f"small (C vs brute force): ok {ok} bad {bad} solvable {solvable} (side gate {side}, finished sand {soft})")

okm = badm = countchecks = 0
for i in range(N_MEDIUM):
    g = None
    while g is None or not g.sand_cells():
        g = garden((5, 6), (6, 7))
    best = g.solve(mode=1, cap=10**9, nodelimit=2 * 10**8)
    opt = optimize(g, time_limit=60)
    c_min = best["best"]["turns"] if "best" in best else None
    good = (c_min is None and opt[1] == "INFEASIBLE") or (c_min == opt[0] and opt[1] == "OPTIMAL")
    if good:                                                   # full counts and histograms where enumeration is cheap
        c = g.solve(mode=0, cap=10**9)
        if c["nsol"] <= 20000:
            n, h, st = count_solutions(g, workers=1)
            good = c["nsol"] == n and c["hist"] == h
            countchecks += 1
    okm += good; badm += not good
    if not good:
        print("MISMATCH medium", g, c_min, opt[:2])
print(f"medium (C vs CP-SAT): ok {okm} bad {badm} (counts and histograms compared on {countchecks})")
src/verify_doc.py · 355 lines · 31.0 kB — Independent re-derivation of every number stated in samon.md (C solver for counts; CP-SAT for pars/uniqueness). 
"""Independent re-derivation of every number stated in samon.md (C solver for counts; CP-SAT for pars/uniqueness)."""
import json, collections, re
from zg import Garden, turns_of, hairpin_list, check_solution
from zcp import optimize, count_solutions
from levels import from_dict
L = json.load(open('/home/claude/zen/out/levels.json'))
fails = []
def check(name, got, want):
    ok = got == want
    print(f"{'OK ' if ok else 'BAD'} {name}: got {got} want {want}")
    if not ok: fails.append(name)

def med(h):
    tot = sum(h.values()); acc = 0
    for t in sorted(h):
        acc += h[t]
        if acc >= tot / 2: return t

# Table 1
T1 = {(3,3): (8,4,2,5,6,2), (4,4): (52,6,4,8,13,4), (5,5): (824,8,10,13,20,4), (6,6): (22144,10,24,20,31,6),
      (7,7): (1510446,12,66,27,42,6), (6,8): (899432,11,66,26,42,8)}
for (H, W), want in T1.items():
    g = Garden(["."*W]*H, (H-1, 0, "S")); r = g.solve(mode=0, cap=10**12); h = r['hist']
    gw = Garden(["."*W]*H, (H-1, 0, "S"), nohairpin=2, braced_start=True); rw = gw.solve(mode=0, cap=10**9)
    got = (r['nsol'], min(h), h[min(h)], med(h), max(h), rw['nsol'])
    check(f"table1 {H}x{W}", got, want)
    if H*W <= 36:
        check(f"table1 {H}x{W} CP-SAT par", optimize(g)[0], want[1])
# A145157 9x9 is cited from OEIS (not recomputed: 5.5e10 paths)

# parity
check("parity ok", Garden(["....."]*5, (4,0,"S")).solve(mode=0, cap=10**9)['nsol'], 824)
check("parity bad", Garden(["....."]*5, (4,1,"S")).solve(mode=0, cap=10**9)['nsol'], 0)

# anatomy / first garden
g = from_dict(L['first']['garden']); r = g.solve(mode=0, cap=10**9)
check("first n", r['nsol'], 1023); check("first par", min(r['hist']), 10); check("first at par", r['hist'][10], 2)
check("first CP-SAT par", optimize(g)[0], 10)

# calm garden
g = from_dict(L['calm']['garden']); r = g.solve(mode=0, cap=10**9); h = r['hist']
check("calm n", r['nsol'], 103812); check("calm par", min(h), 17); check("calm at par", h[17], 6)
check("calm median", med(h), 31); check("calm max", max(h), 48); check("calm at max", h[48], 5)
check("calm CP-SAT par", optimize(g)[0], 17)

# ripple
g = from_dict(L['ripple']['garden']); r = g.solve(mode=0, cap=10**9); h = r['hist']
check("ripple n", r['nsol'], 35570); check("ripple par", min(h), 12); check("ripple at par", h[12], 4)
g0 = Garden([x.replace('O', '#') for x in g.rows], g.gate); r0 = g0.solve(mode=0, cap=10**9)
check("ripple plain n", r0['nsol'], 183986); check("ripple plain par", min(r0['hist']), 12); check("ripple plain at par", r0['hist'][12], 5)
d = [tuple(c) for c in L['ripple']['intended']]
check("ripple intended valid", check_solution(g, d), True)
check("ripple start adjacent to stone", any(abs(d[-1][0]-4)<=1 and abs(d[-1][1]-2)<=1 for _ in [0]), True)

# waves
g = from_dict(L['waves']['garden']); r = g.solve(mode=0, cap=10**9); h = r['hist']
check("waves n", r['nsol'], 2984); check("waves par", min(h), 17); check("waves at par", h[17], 34)
g0 = Garden([x.replace('h', '.') for x in g.rows], g.gate); r0 = g0.solve(mode=0, cap=10**9)
check("waves plain n", r0['nsol'], 1492008); check("waves plain par", min(r0['hist']), 16)

# ichimatsu
gi = Garden(["".join("=" if ((rr//3)+(cc//3))%2==0 else "|" for cc in range(9)) for rr in range(9)], (8,0,"S"))
ri = gi.solve(mode=0, cap=100)
check("ichimatsu 9x9 n", ri['nsol'], 2); check("ichimatsu turns", sorted(s['turns'] for s in ri['sols']), [45, 45])
n_cp, _, st = count_solutions(gi, limit=10)
check("ichimatsu CP-SAT count", n_cp, 2)
g6 = ["===|||"]*3 + ["|||==="]*3
uniq = [gc for gc in range(6) if Garden(g6, (5, gc, "S")).solve(mode=0, cap=5)['nsol'] == 1]
check("ichimatsu 6x6 unique gate cols", uniq, [0, 1, 2, 3, 5])

# grooves
g = from_dict(L['grooves']['garden'])
check("grooves count", len(g.clues), 5)
check("grooves unique", g.solve(mode=0, cap=5)['nsol'], 1)
check("grooves CP-SAT unique", count_solutions(g, limit=5)[0], 1)
g0 = Garden(g.rows, g.gate); r0 = g0.solve(mode=0, cap=10**9)
check("grooves plain n", r0['nsol'], 12375); check("grooves plain par", min(r0['hist']), 21); check("grooves plain at par", r0['hist'][21], 11)
d = [tuple(c) for c in L['grooves']['intended']]
check("grooves answer turns", turns_of(g, d), 21)

# wide rake
w = json.load(open('/home/claude/zen/out/wide_pick.json'))
g = from_dict(w['garden'])
check("wide unique (C)", g.solve(mode=0, cap=5)['nsol'], 1)
check("wide unique (CP-SAT)", count_solutions(g, limit=5)[0], 1)
d = [tuple(c) for c in w['draw']]
check("wide answer turns", turns_of(g, d), 17)
gp = Garden(g.rows, g.gate); rp = gp.solve(mode=0, cap=10**9)
check("wide plain n", rp['nsol'], 6310226); check("wide plain par", min(rp['hist']), 17); check("wide plain at par", rp['hist'][17], 12)
pr = gp.solve(mode=2, bound=17, cap=100)
graceful = [s for s in pr['sols'] if not any(not br for *_, br in hairpin_list(gp, s['draw']))]
check("wide: graceful par rakings", len(graceful), 1)
check("wide: graceful par == wide answer", graceful[0]['draw'] == d, True)

# two gates
t = json.load(open('/home/claude/zen/out/two_pick.json'))
g = from_dict(t['garden'])
check("two-gate wide unique", g.solve(mode=0, cap=5)['nsol'], 1)
check("two-gate CP-SAT unique", count_solutions(g, limit=5)[0], 1)
d = [tuple(c) for c in t['draw']]; check("two-gate answer turns (the bend entering through the side gate counts)", turns_of(g, d), 20)
gp = Garden(g.rows, g.gate, end=g.end, end_side=g.end_side); rp = gp.solve(mode=0, cap=10**9)
check("two-gate plain n", rp['nsol'], 311080); check("two-gate plain par", min(rp['hist']), 20); check("two-gate plain at par", rp['hist'][20], 32)
check("two-gate plain par by CP-SAT", optimize(gp, time_limit=120)[:2], (20, 'OPTIMAL'))

# the abbot's pebbles: the example garden
from grader import grade
mp = json.load(open('/home/claude/zen/out/marks_pick.json'))
gm = from_dict(mp['garden']); dm = [tuple(c) for c in mp['draw']]
gro = {tuple(map(int, k.split(','))): v for k, v in mp['grooves'].items()}
gg = Garden(gm.rows, gm.gate, clues=gro); gp = Garden(gm.rows, gm.gate)
rp = gp.solve(mode=0, cap=10**8)
check("pebbles garden plain n", rp['nsol'], 59498); check("pebbles garden par", min(rp['hist']), 13); check("pebbles garden at par", rp['hist'][13], 6)
check("pebbles count", len(gm.marks), 8); check("pebbles unique (C)", gm.solve(mode=0, cap=5)['nsol'], 1)
check("pebbles unique (CP-SAT)", count_solutions(gm, limit=5)[0], 1)
check("pebbles answer turns", turns_of(gm, dm), 13); check("pebbles answer valid", check_solution(gm, dm), True)
check("grooves version count", len(gro), 6); check("grooves version unique", gg.solve(mode=0, cap=5)['nsol'], 1)
gr1, gr2 = grade(gm), grade(gg)
check("pebbles tier/trials", (gr1['tier'], gr1['trials']), (3, 14)); check("grooves version tier", gr2['tier'], 1)

# pebbles vs grooves experiment: re-check every saved garden's uniqueness and grade, then the statistics
import statistics
rows = json.load(open('/home/claude/zen/out/marks_vs_grooves.json'))
check("experiment gardens", len(rows), 88)
uniq_ok, gts, mts = True, [], []
for r in rows:
    base = dict(rows=r['rows'], gate=r['gate'], clues={}, end=None, nohairpin=0, braced_start=False, marks={})
    g1 = from_dict({**base, 'clues': r['grooves']}); g2 = from_dict({**base, 'marks': r['marks']})
    tgt = [tuple(c) for c in r['target']]
    for gx in (g1, g2):
        rr = gx.solve(mode=0, cap=3, stream=True)
        uniq_ok &= rr['nsol'] == 1 and [tuple(c) for c in rr['sols'][0]['draw']] == tgt
    a, b = grade(g1), grade(g2)
    gts.append((a['tier'], a['trials'])); mts.append((b['tier'], b['trials']))
check("experiment: every garden unique both ways, with the intended answer", uniq_ok, True)
check("experiment: recomputed grades match saved", [(r['g_tier'], r['g_trials']) for r in rows] == gts and [(r['m_tier'], r['m_trials']) for r in rows] == mts, True)
ng, nm = [len(r['grooves']) for r in rows], [len(r['marks']) for r in rows]
check("median grooves / pebbles", (statistics.median(ng), statistics.median(nm)), (7, 11))
check("share of gardens needing more pebbles than grooves (%)", round(100 * sum(b > a for a, b in zip(ng, nm)) / 88), 92)
check("lookahead share grooves / pebbles (%)", (round(100 * sum(t >= 3 for t, _ in gts) / 88), round(100 * sum(t >= 3 for t, _ in mts) / 88)), (47, 92))
check("nested lookahead grooves / pebbles (%)", (sum(t >= 4 for t, _ in gts), round(100 * sum(t >= 4 for t, _ in mts) / 88, 1)), (0, 12.5))
check("median lookahead steps grooves / pebbles", (statistics.median([n for t, n in gts if t >= 3]), statistics.median([n for t, n in mts if t >= 3])), (2, 7))

# prototype: the full campaign (each garden's par, raking count, par rakings and hint are re-derived, with asserts, by export_proto.py)
P = json.load(open('/home/claude/zen/out/proto_gardens.json'))
check("prototype gardens", len(P), 148)
check("gardens per world", [sum(p['world'] == w for p in P) for w in range(1, 8)], [19, 20, 20, 25, 26, 18, 20])
check("main path / extras per world", ([sum(p['world'] == w and not p['extra'] for p in P) for w in range(1, 8)],
      [sum(p['world'] == w and p['extra'] for p in P) for w in range(1, 8)]), ([9, 10, 11, 16, 9, 9, 8], [10, 10, 9, 9, 17, 9, 12]))
check("main path / extras totals", (sum(not p['extra'] for p in P), sum(p['extra'] for p in P)), (72, 76))
check("the first five gardens", [(p['name'], p['role']) for p in P[:5]], [('The corridor', 'intro'), ('Nine squares', 'intro'), ('Morning dew', 'application'),
      ('The pocket', 'intro'), ('Two tones', 'intro')])
check("the third garden is the first generated one, 5x5 with one stone", (sum(ch == '#' for r in P[2]['rows'] for ch in r) > 0, len(P[2]['rows']), len(P[2]['rows'][0])), (True, 5, 5))
check("the 2x3 garden is gone", any(p['name'] == 'Two rows' for p in P), False)
import math
need = [math.ceil(2 / 3 * sum(p['world'] == w and not p['extra'] for p in P)) for w in range(1, 7)]
check("gardens raked before grooves (world 4) / the wide rake (world 5) open", (sum(need[:3]), sum(need[:4])), (21, 32))
old_need = [math.ceil(2 / 3 * n) for n in (20, 18, 18, 20)]          # the first campaign: every garden on the path
check("first campaign: grooves / wide rake after (Astra's count)", (sum(old_need[:3]), sum(old_need)), (38, 52))
check("demonstrations", [p['name'] for p in P if p['demo']], ['The corridor', 'Nine squares', 'First ripple', 'First wave', 'Ichimatsu',
      'First grooves', 'The finished ring', 'His pebbles', 'Wide blade', 'Two gates'])
check("the corridor's par (quoted in the demonstration)", next(p['par'] for p in P if p['name'] == 'The corridor'), 3)
check("no demonstration among the extras", any(p['demo'] and p['extra'] for p in P), False)
check("every garden has a first hint", all(isinstance(p['hint'], str) and len(p['hint']) > 10 for p in P), True)
check("every garden's par rakings stored", all(len(p['par_moves']) == p['at_par'] for p in P), True)
CS = json.load(open('/home/claude/zen/out/campaign_stats.json'))
specs = {k: v for k, v in CS.items() if isinstance(v, dict) and 'found' in v}
check("campaign: generation specs (47, and one fallback)", len(specs), 48)
check("campaign: 9x9 pebble gardens found / layouts tried / minutes", (specs['w7-p99']['found'], specs['w7-p99']['tried'], round(specs['w7-p99']['seconds'] / 60)), (0, 227, 16))
check("campaign: 8x8 pebble fallback found enough", specs['w7-p88']['found'] >= 2 and specs['w7-p88'].get('fallback_for') == 'w7-p99', True)
import glob, pickle
cand_keys = set()
for f in glob.glob('/home/claude/zen/out/campaign_cache/w*.pkl'):
    for c in pickle.load(open(f, 'rb'))[1]:
        cand_keys.add((tuple(c['garden'].rows), tuple(c['garden'].gate), tuple(c['garden'].end) if c['garden'].end else None))
C = json.load(open('/home/claude/zen/out/campaign.json'))
ngen = sum((tuple(e['garden']['rows']), tuple(e['garden']['gate']), tuple(e['garden']['end']) if e['garden']['end'] else None) in cand_keys for e in C)
check("campaign: generated / teaching gardens / gardens in contrasting pairs", (ngen, len(C) - ngen, sum(1 for p in P if p['pair'])), (114, 34, 13))
check("campaign: exported in the same order", [e['name'] for e in C] == [p['name'] for p in P], True)
from campaign import canon
same_layout = collections.Counter(canon(from_dict(e['garden'])) for e in C)
twins = sorted(sorted(e['name'] for e in C if canon(from_dict(e['garden'])) == k) for k, n in same_layout.items() if n > 1)
check("campaign: no two gardens alike up to rotation and reflection, but the pairs that share a layout", twins,
      [['First grooves', 'Open ground'], ['His grooves', 'His pebbles'], ['Narrow rake', 'Wide blade']])
check("campaign: candidates / layouts sampled / minutes of computing", (sum(v['found'] for v in specs.values()), sum(v['tried'] for v in specs.values()), round(sum(v['seconds'] for v in specs.values()) / 60)), (235, 16437, 33))
check("campaign: about half the time on 9x9 pebbles", round(specs['w7-p99']['seconds'] / sum(v['seconds'] for v in specs.values()), 1), 0.5)
check("campaign: world 7 pebble gardens are 8x8", sorted({(len(p['rows']), len(p['rows'][0])) for p in P if p['world'] == 7 and p['marks']}), [(8, 8)])
check("campaign: hints with only generic advice", sum('Par is' in p['hint'] for p in P), 0)
check("campaign: template roles", sorted(collections.Counter(p['role'] for p in P).items()),
      [('application', 93), ('calm', 7), ('capstone', 7), ('combination', 15), ('intro', 13), ('pair', 6), ('trap', 7)])
mains = {w: [p['role'] for p in P if p['world'] == w and not p['extra']] for w in range(1, 8)}
check("campaign: each world's main path opens with its introduction (a pair, or for world 7 an intro), has one trap, "
      "and ends calm, capstone", all(m[0] in ('pair', 'intro') and 'intro' in m and m.count('trap') == 1 and m[-2:] == ['calm', 'capstone']
                                     for m in mains.values()), True)
check("campaign: extras come after each world's main path", all(
    [p['extra'] for p in P if p['world'] == w] == sorted(p['extra'] for p in P if p['world'] == w) for w in range(1, 8)), True)
ft = [p['trap'] for p in P if p['role'] == 'trap' and p['world'] <= 3]
check("campaign: free traps: share of near-misses that miss the key observation (min, max)", (round(min(ft), 2), round(max(ft), 2)), (0.74, 0.94))
HS = json.load(open('/home/claude/zen/out/hint_stats.json'))
hf = [h for h in HS if h['kind'] == 'free']; hs = [h for h in HS if h['kind'] == 'strict' and not h.get('trap')]
check("hints: free gardens, median share of near-misses that get the hint wrong", (len(hf), round(statistics.median(h['miss'] for h in hf), 2)), (50, 0.5))
tr = collections.Counter((h['tier'], h['tier_after']) for h in hs)
check("hints: strict gardens; need lookahead; then none / nested->one level / fewer steps", (len(hs), sum(h['tier'] >= 3 for h in hs), tr[(3, 1)] + tr[(3, 2)], tr[(4, 3)],
      sum(h['tier'] == h['tier_after'] == 3 and h['trials_after'] < h['trials'] for h in hs)), (80, 62, 16, 10, 36))
check("hints: no strict hint leaves lookahead as it was", sum(h['tier'] == h['tier_after'] and h['tier'] >= 3 and h['trials_after'] >= h['trials'] for h in hs), 0)
check("hints: strict gardens without lookahead; made local / already local", (tr[(2, 1)], tr[(1, 1)]), (4, 14))
check("hints: end hints that say the cell has one way in", sorted(p['name'] for p in P if 'only one way in, so the rake' in p['hint']),
      ['First grooves', 'His grooves', 'His pebbles', 'Interrupted', 'The finished ring'])   # (names follow layouts: out/garden_names.json)
check("campaign: teaching gardens include 13 of the original 14", {p['name'] for p in P} >= {"Nine squares", "The pocket", "First stroke", "Calm", "Ripples", "Waves", "Ichimatsu", "The master's grooves", "The abbot's pebbles", "Wide rake", "Spiral", "Side gate", "Silver sand"}, True)

# the abbot's taste vs the wide rake (exp_continuity.py): par rakings with a U-turn in open sand, free gardens of worlds 1-3
CT = json.load(open('/home/claude/zen/out/continuity.json'))
check("continuity: free gardens; all par rakings graceful; mixed; none graceful",
      (CT['free_gardens'], CT['all_graceful'], CT['mixed'], CT['none_graceful']), (54, 11, 42, 1))
check("continuity: par rakings; with an open-sand U-turn", (CT['par_rakings'], CT['par_rakings_with_open_u']), (407, 327))
wv = next(r for r in CT['gardens'] if r['name'] == 'Waves')
check("continuity: Waves' par rakings, and how many turn back in open sand", (wv['at_par'], wv['with_open_u']), (34, 32))
check("continuity: the one free garden with no graceful par raking is a hand-made one", [r['name'] for r in CT['gardens'] if r['with_open_u'] == r['at_par']], ['The pocket'])
# the contrasting pairs (pairs.py), re-checked from the exported gardens
PP = {p['name']: p for p in P}
from zg import check_solution as _cs
def g_of(p):
    marks = {tuple(map(int, k.split(','))): v for k, v in (p.get('marks') or {}).items()}
    clues = {tuple(map(int, k.split(','))): v for k, v in (p['clues'] or {}).items()}
    return Garden(p['rows'], tuple(p['gate']), clues=clues, marks=marks, end=tuple(p['end']) if p['end'] else None,
                  nohairpin=2 if p['wide'] else 0, braced_start=p['wide'], end_side=p['end_side'])
STEP = {"N": (-1, 0), "S": (1, 0), "E": (0, 1), "W": (0, -1)}
def cells_of(p, ms):
    r, c = p['gate'][0], p['gate'][1]; out = [(r, c)]
    for ch in ms:
        r, c = r + STEP[ch][0], c + STEP[ch][1]; out.append((r, c))
    return out
pair_names = [("Plain stone", "First ripple"), ("Calm water", "First wave"), ("Open ground", "First grooves"), ("His grooves", "His pebbles"),
              ("Narrow rake", "Wide blade"), ("One gate", "Two gates")]
check("pairs: sand cells of every pair garden (and the finished ring's)", sorted({sum(ch in '.hv=|' for r in PP[n]['rows'] for ch in r)
      for ab in pair_names + [("The finished ring",)] for n in ab}), [17, 18, 21, 22, 23])
for a, b in [("Plain stone", "First ripple"), ("Calm water", "First wave")]:
    check(f"pairs: every par raking of {a} breaks the rule added in {b}", any(_cs(g_of(PP[b]), cells_of(PP[a], ms)) for ms in PP[a]['par_moves']), False)
check("pairs: every par raking of Narrow rake turns back in open sand", all(any(not br for *_, br in hairpin_list(g_of(PP['Narrow rake']), cells_of(PP['Narrow rake'], ms)))
      for ms in PP['Narrow rake']['par_moves']), True)
check("pairs: Open ground's rakings; First grooves' grooves and rakings", (PP['Open ground']['rakings'], len(PP['First grooves']['clues']), PP['First grooves']['rakings']), (8, 1, 1))
check("pairs: His grooves' grooves; His pebbles' pebbles (pale, dark), at the grooves' cells; rakings",
      (len(PP['His grooves']['clues']), sorted(PP['His pebbles']['marks'].values()), set(PP['His pebbles']['marks']) == set(PP['His grooves']['clues']),
       PP['His pebbles']['rakings']), (2, ['B', 'S'], True, 1))
og, tg = PP['One gate'], PP['Two gates']
check("pairs: One gate has two wide-rake rakings; Two gates one, which is not One gate's par raking",
      (og['rakings'], tg['rakings'], turns_of(g_of(og), cells_of(tg, tg['par_moves'][0])) > og['par']), (2, 1, True))
check("pairs: which gardens of the wide and side pairs use the wide rake (Narrow rake, Wide blade, One gate, Two gates)", [PP[n]['wide'] for n in ("Narrow rake", "Wide blade", "One gate", "Two gates")], [False, True, True, True])
check("first campaign's big introductions, now applications: Waves; The master's grooves; The abbot's pebbles (sand cells)",
      [sum(ch in '.hv=|' for r in PP[n]['rows'] for ch in r) for n in ("Waves", "The master's grooves", "The abbot's pebbles")], [55, 47, 47])
check("side-gate gardens", sum(1 for p in P if p['end']), 19)
small = solv = side = soft = medium = 0
for f in ('/home/claude/zen/out/test_side_pivot_seed1.log', '/home/claude/zen/out/test_side_pivot_seed2.log'):
    t = open(f).read()
    a = re.search(r'ok (\d+) bad (\d+) solvable (\d+) \(side gate (\d+), finished sand (\d+)\)', t)
    b = re.search(r'medium \(C vs CP-SAT\): ok (\d+) bad (\d+) \(counts and histograms compared on (\d+)\)', t)
    check(f"side/pivot cross-check log {f[-9:]}: no disagreements", (a.group(2), b.group(2)), ('0', '0'))
    small += int(a.group(1)); solv += int(a.group(3)); side += int(a.group(4)); soft += int(a.group(5)); medium += int(b.group(3))
check("side/pivot cross-check: small gardens; solvable; with a side gate; with finished sand; medium (counts compared)", (small, solv, side, soft, medium), (3192, 411, 132, 149, 120))

# garden-like stones vs single-cell stones (exp_sampler.py, 1,000 random 7x7 layouts each)
Y = json.load(open('/home/claude/zen/out/sampler_yields.json'))['summary']
check("sampler layouts", (Y['garden']['layouts'], Y['single']['layouts']), (1000, 1000))
check("sampler stone cells", (round(Y['garden']['mean_stone_cells'], 1), round(Y['single']['mean_stone_cells'], 1)), (4.7, 2.0))
check("sampler solvable %", (round(100 * Y['garden']['solvable']), round(100 * Y['single']['solvable'])), (28, 39))
check("sampler usable free %", (round(100 * Y['garden']['graceful_free'], 1), round(100 * Y['single']['graceful_free'], 1)), (4.7, 3.8))
check("sampler wide-unique %", (round(100 * Y['garden']['wide_unique'], 1), round(100 * Y['single']['wide_unique'], 1)), (2.2, 2.7))
check("sampler side-unique %", (round(100 * Y['garden']['side_unique'], 1), round(100 * Y['single']['side_unique'], 1)), (1.8, 1.3))
check("sampler grooves median / lookahead %", (Y['garden']['median_grooves'], Y['single']['median_grooves'], round(100 * Y['garden']['grooves_lookahead']), round(100 * Y['single']['grooves_lookahead'])), (6, 6, 28, 84))
H = json.load(open('/home/claude/zen/out/hint_check.json'))
check("hint check: routes, disagreements, timeouts", (H['routes'], H['bad'], H['states']['unknown']), (4440, 0, 0))
PG = {p['name']: p for p in P}
check("hint check: no search ran out of budget", H['unknown_in'], [])
check("hint check: stuck routes; rewinds proven nearest / unproven", (H['states']['stuck'], H['proven_rewinds'], H['unproven_rewinds']), (2011, 1989, 22))
check("hint check: the unproven rewinds are in three of the largest gardens (8x8 or 9x9)", sorted({(len(PG[n]['rows']), len(PG[n]['rows'][0])) for n in H['unproven_in']}), [(8, 8), (9, 9)])
check("hint check: unproven rewinds are in three gardens", len(set(H['unproven_in'])), 3)
check("hint check: slowest analysis 0.7 s", round(H['max_ms'] / 1000, 1), 0.7)
import re
small = medium = 0
for f in ('/home/claude/zen/out/test_marks_seed1.log', '/home/claude/zen/out/test_marks_seed2.log'):
    t = open(f).read()
    a = re.search(r'ok (\d+) bad (\d+) solvable', t); b = re.search(r'medium \(C vs CP-SAT\): ok (\d+) bad (\d+)', t)
    check(f"marks cross-check log {f[-9:]}: no disagreements", (a.group(2), b.group(2)), ('0', '0'))
    small += int(a.group(1)); medium += int(b.group(1))
check("pebbles cross-check: small / medium gardens", (small, medium), (6000, 108))
tot = 0
for f in ('/home/claude/zen/out/test_grader_marks_seed7.log', '/home/claude/zen/out/test_grader_marks_seed11.log'):
    m = re.search(r'ok (\d+) bad (\d+)', open(f).read())
    check(f"grader soundness log {f[-10:]}: no failures", m.group(2), '0'); tot += int(m.group(1))
print("grader soundness gardens:", tot)

# poems (Presentation; Credits)
PO = json.load(open('/home/claude/zen/poems/samon-poems.json'))
AS = json.load(open('/home/claude/zen/poems/astra-samon-poem-selection.json'))
pp = PO['poems']; used = list(PO['gardens'].values()) + [PO['practice']]
DOC = open('/home/claude/zen/samon.md').read()
check("poems: one for each garden and one for the practice garden, all different", (len(PO['gardens']), len(set(used)), set(PO['gardens']) == {p['name'] for p in P}), (148, 149, True))
check("poems: every text re-checked against a page that prints it", sum(pp[i]['reverified'].startswith('yes') and pp[i]['source_url'].startswith('https://') for i in used), 149)
pc = collections.Counter(pp[i]['poet'] for i in used)
check("poems: poets; Bashō, Teika, Buson, Rengetsu", (len(pc), pc['Matsuo Bashō'], pc['Fujiwara no Teika'], pc['Yosa Buson'], pc['Ōtagaki Rengetsu']), (25, 34, 23, 20, 13))
check("poems: Tsurayuki, Izumi Shikibu and Shiki have one each", (pc['Ki no Tsurayuki'], pc['Izumi Shikibu'], pc['Masaoka Shiki']), (1, 1, 1))
check("poems: doc states the counts", all(s in DOC for s in ("149 (one for each garden, and one for the practice garden) by 25 poets", "Bashō (34), Teika (23), Buson (20) and Rengetsu (13)")), True)
anames = {l['name']: l['poem_id'] for l in AS['levels']}
check("poems: Astra's selection covered 136 gardens", len(AS['levels']), 136)
rev = [r for r in PO['about']['revisions'] if 'to' in r]
check("poems: second and third passes: 25 swaps, each in force, to a poem used by that garden alone (Owl's hour took the capstone's)", (len(rev), all(PO['gardens'][r['garden']] == r['to'] for r in rev), len({r['to'] for r in rev}), sum('translation_repair' in r for r in PO['about']['revisions'])), (25, True, 25, 4))
check("poems: new gardens paired / poems new to the selection / pairings changed since Astra",
      (sum(g not in anames for g in PO['gardens']), sum(i not in AS['poems'] for i in used), sum(g in anames and anames[g] != PO['gardens'][g] for g in PO['gardens'])), (13, 23, 23))
check("poems: doc says 25 pairings were replaced in two further passes", "replaced 25 pairings judged weak" in DOC, True)
check("poems: translations revised / new", (sum('revised' in pp[i]['translation_credit'] for i in used), sum('New translation' in pp[i]['translation_credit'] for i in used)), (25, 17))
check("poems: doc states translations revised (first pass)", "revised 22 of the translations" in DOC, True)
check("poems: all written before 1900 (poets' death years or floruit; Shiki's by its year), and one poet who outlived it", (max(min(int(re.findall(r'\d{3,4}', pp[i]['author_dates'])[-1]), pp[i].get('composed', 9999)) for i in used) < 1900, sum(int(re.findall(r'\d{3,4}', pp[i]['author_dates'])[-1]) >= 1900 for i in used)), (True, 1))
G1 = PO['gardens']
check("poems: first garden = sweeping the temple garden; new pond then old pond at the first ripple stone; wave pair = the two Kiyotaki drafts; last main-path garden = the sand at Kehi",
      (pp[G1['The corridor']]['japanese'], pp[G1['Plain stone']]['japanese'], pp[G1['First ripple']]['japanese'], pp[G1['Calm water']]['japanese'], pp[G1['First wave']]['japanese'], pp[G1['Silver sand']]['japanese'],
       [p['name'] for p in P if p['world'] == 7 and not p['extra']][-1], P[0]['name'], PG['First ripple']['role'], PG['First wave']['pair'], PG['Calm water']['pair'], PG['Plain stone']['pair']),
      ('庭掃て出ばや寺に散柳', '新池や蛙とびこむ音もなし', '古池や蛙飛びこむ水の音', '清滝や波に塵なき夏の月', '清滝や波に散り込む青松葉', '月清し遊行のもてる砂の上', 'Silver sand', 'The corridor', 'intro', 'band', 'band', 'ripple'))
check("poems: the first and last poems quoted as translated", all(pp[G1[g]]['translation'].replace('\n', ' / ') in DOC for g in ('The corridor', 'Silver sand')), True)

# the daily garden (Progression; Prototype)
DY = json.load(open('/home/claude/zen/out/daily_gardens.json'))
DS = json.load(open('/home/claude/zen/out/daily_stats.json'))
from campaign import canon as canon_
camp_canon = {canon_(from_dict(e['garden'])) for e in C}
def garden_of_entry(e):
    return from_dict({"rows": e["rows"], "gate": e["gate"], "clues": e["clues"], "end": e["end"], "nohairpin": 2 if e["wide"] else 0,
                      "braced_start": bool(e["wide"]), "end_side": e["end_side"], "marks": e["marks"], "name": e["name"]})
check("daily: a year of gardens, even days free and odd days strict", (len(DY), all((e['rakings'] > 1) == (k % 2 == 0) for k, e in enumerate(DY))), (366, True))
check("daily: none shares a layout with the campaign, and no two share one", (sum(canon_(garden_of_entry(e)) in camp_canon for e in DY), len({canon_(garden_of_entry(e)) for e in DY})), (0, 366))
check("daily: sizes 5x5 to 8x8", (min(len(e['rows']) * len(e['rows'][0]) for e in DY), max(len(e['rows']) for e in DY), max(len(e['rows'][0]) for e in DY)), (25, 8, 8))
check("daily: every mechanic appears", all(DS[k] > 0 for k in ('with_ripple', 'with_band', 'with_grooves', 'with_pebbles', 'wide', 'side_gate', 'closed_ring')), True)
check("daily: generation took about 8 minutes on two cores", round(sum(v['seconds'] for v in DS['specs'].values()) / 60 / 2), 8)
check("daily: doc states the year and the timing", all(x in DOC for x in ("a year of 366 gardens", "took 8 minutes of generation on two cores")), True)
import random as random_
smp = random_.Random(7).sample(range(366), 12)
okd = []
for k in smp:                                   # a sample re-derived: the exact count, CP-SAT par, the intended raking at par
    e = DY[k]; g = garden_of_entry(e); r = g.solve(mode=0, cap=10**9); d = [tuple(c) for c in e['intended']]
    okd.append(r['nsol'] == e['rakings'] and optimize(g)[0] == e['par'] and check_solution(g, d) and turns_of(g, d) == e['par'] and r['hist'].get(e['par']) == e['at_par'] == len(e['par_moves']))
check("daily: a sample of 12 re-derived (count, par, intended raking, par rakings)", all(okd), True)

# the sound (Presentation; Prototype): the recording carried in the page, as the document describes it
import base64 as base64_, subprocess as subprocess_
SP = json.load(open('/home/claude/zen/out/sound_pieces.json'))
ST = json.load(open('/home/claude/zen/sound/tracks.json'))
check("sound: one recording, Shika no Tōne, credited to Araki Kodō III with its Commons source, public domain",
      (len(SP), SP[0]['title'][:13], 'Araki Kodō III' in SP[0]['player'], SP[0]['source'], SP[0]['license'].startswith('public domain'), SP[0]['title'] == ST['tracks'][0]['title']),
      (1, 'Shika no Tōne', True, 'https://commons.wikimedia.org/wiki/File:Shikanotoone_new.ogg', True, True))
mp3 = base64_.b64decode(SP[0]['b64'])
open('/tmp/claude-0/verify_piece.mp3', 'wb').write(mp3)   # (probed from a file: piped input hides the duration and bit rate)
pr = json.loads(subprocess_.run(['ffprobe', '-hide_banner', '-loglevel', 'error', '-show_entries', 'format=duration,bit_rate:stream=codec_name,channels', '-of', 'json', '/tmp/claude-0/verify_piece.mp3'], capture_output=True, check=True).stdout)
check("sound: mono MP3, 6 min 15 s, 32 kbps, 1.5 MB, as stated", (pr['streams'][0]['codec_name'], pr['streams'][0]['channels'], round(float(pr['format']['duration'])), round(int(pr['format']['bit_rate']) / 1000), round(len(mp3) / 1e6, 1)), ('mp3', 1, 375, 32, 1.5))
check("sound: doc states the encoding, the source and the credit", all(x in DOC for x in ("re-encoded as mono MP3 at 32 kbps (1.5 MB)", "File:Shikanotoone_new.ogg", "Araki Kodō III (1879--1935) on Victor 13029")), True)
check("sound: the page carries the recording and its credits", all(x in open('/home/claude/zen/proto/samon.html').read() for x in ('class="piece"', 'data-title="Shika no Tōne', 'data:audio/mpeg;base64,' + SP[0]['b64'][:64])), True)

print("FAILS:", fails)
src/xc_cpsat.py · 20 lines · 1.2 kB 
import random, sys, time
from zg import Garden
from zcp import count_solutions, optimize

cases = [
    Garden([".....", ".....", ".....", ".....", "....."], (4, 0, "W")),
    Garden(["......", "..#...", "......", "...#..", "......"], (4, 0, "S")),
    Garden(["......", ".O....", "......", "....#.", "......"], (0, 5, "E")),
    Garden(["......", "..hhh.", "..hhh.", "......", "......"], (4, 2, "S")),
    Garden(["......", ".O....", "......", "....#.", "......"], (0, 5, "E"), nohairpin=True),
    Garden(["......", "..#...", "......", "...#..", "......", "......"], (5, 0, "W"), nohairpin=True),
    Garden(["......", "......", "......", "......"], (3, 0, "W"), clues={(1, 2): "NS", (2, 4): "EW"}),
    Garden(["......", "......", "......", "......"], (3, 0, "W"), end=(0, 5)),
]
for g in cases:
    t0 = time.time(); c = g.solve(mode=0, cap=10**9); t1 = time.time()
    n, h, st = count_solutions(g, workers=1); t2 = time.time()
    b = g.solve(mode=1, cap=10**9)
    opt = optimize(g)
    print(f"C={c['nsol']} ({t1-t0:.2f}s)  CPSAT={n} {st} ({t2-t1:.1f}s)  hist_eq={c['hist']==h}  Cmin={b['best']['turns'] if 'best' in b else None} CPmin={opt[0]} {opt[1]}")
src/xc_cpsat2.py · 17 lines · 1.1 kB 
import time
from zg import Garden
from zcp import count_solutions, optimize
cases = [
    Garden(["......", "......", "......", "......", "......"], (4, 0, "W"), nohairpin=2),
    Garden(["......", "..#...", "......", "...#..", "......"], (4, 0, "S"), nohairpin=2),
    Garden(["......", "......", "......", "......", "......"], (4, 0, "W"), braced_turns=True),
    Garden([".......", "..#....", ".......", "....#..", ".......", "......."], (5, 0, "W"), braced_turns=True),
    Garden(["......", ".O....", "......", "....#.", "......"], (0, 5, "E"), nohairpin=2),
    Garden([".......", ".......", "..#....", ".......", "....#..", "......."], (5, 0, "W"), nohairpin=2, braced_turns=True),
]
for g in cases:
    t0 = time.time(); c = g.solve(mode=0, cap=10**9); t1 = time.time()
    n, h, st = count_solutions(g, workers=1); t2 = time.time()
    b = g.solve(mode=1, cap=10**9)
    opt = optimize(g)
    print(f"C={c['nsol']} ({t1-t0:.2f}s)  CPSAT={n} {st} ({t2-t1:.1f}s)  hist_eq={c['hist']==h}  Cmin={b['best']['turns'] if 'best' in b else None} CPmin={opt[0]} {opt[1]}")
src/xc_stats.py · 28 lines · 1.3 kB 
import sys, random, time
sys.path.insert(0, '/home/claude/zen/src')
from zg import Garden, brute_force, check_solution
src = open('/home/claude/zen/src/test_crosscheck.py').read().split('ok = bad = 0')[0]
ns = {}
exec(src.replace('random.seed(int(sys.argv[1]) if len(sys.argv) > 1 else 1)', f'random.seed({sys.argv[1]})'), ns)
rg = ns['random_garden']
tot = nz = bad = 0; maxsol = 0
featcount = {'ring':0,'zone':0,'clue':0,'nohp':0,'end':0}
t0 = time.time()
while time.time() - t0 < float(sys.argv[2]):
    g = rg()
    if g is None or len(g.sand_cells()) > 18: continue
    try: g.ring_sets()
    except ValueError: continue
    res = g.solve(mode=0, cap=10**9)
    n, h = brute_force(g)
    if res['nsol'] != n or res['hist'] != h or not all(check_solution(g, s['draw']) for s in res['sols']):
        bad += 1; print('BAD', g, res['nsol'], n, flush=True)
    tot += 1
    if n:
        nz += 1; maxsol = max(maxsol, n)
        if g.ring_sets(): featcount['ring'] += 1
        if any(ch in 'hv' for row in g.rows for ch in row): featcount['zone'] += 1
        if g.clues: featcount['clue'] += 1
        if g.nohairpin: featcount['nohp'] += 1
        if g.end: featcount['end'] += 1
print('total', tot, 'solvable', nz, 'bad', bad, 'max sols', maxsol, featcount, flush=True)
src/zcp.py · 209 lines · 8.8 kB — CP-SAT model of the one-stroke raking puzzle (independent of the C solver). 
"""zcp.py: CP-SAT model of the one-stroke raking puzzle (independent of the C solver).

Used for (1) cross-validating solution counts, (2) proving 'par' (minimum turns), and
(3) multi-objective trade-offs (turns vs hairpins).
Arcs are in *raking* order: the stroke runs from the rake-start P to the gate cell, then exits.
"""
from __future__ import annotations

from ortools.sat.python import cp_model
from zg import Garden, DIRS, DR, DC, OPP

PERP = {"N": "EW", "S": "EW", "E": "NS", "W": "NS"}


class Model:
    def __init__(self, g: Garden, nohairpin=None):
        self.g = g
        m = self.m = cp_model.CpModel()
        cells = g.sand_cells()
        self.cells = cells
        idx = self.idx = {c: i for i, c in enumerate(cells)}
        n = len(cells)
        D = n
        gate = (g.gate[0], g.gate[1])
        arcs = []
        self.x = {}  # (cell, d) -> literal for rake moving from cell in direction d
        for c in cells:
            for d in DIRS:
                nb = (c[0] + DR[d], c[1] + DC[d])
                if nb in idx:
                    lit = m.NewBoolVar(f"x{c}{d}")
                    self.x[(c, d)] = lit
                    arcs.append((idx[c], idx[nb], lit))
        self.start = {}
        for c in cells:
            if g.end is not None and c != tuple(g.end):
                continue
            if g.braced_start and not any(g.pivot(c[0] + DR[d], c[1] + DC[d]) for d in DIRS):
                continue
            lit = m.NewBoolVar(f"s{c}")
            self.start[c] = lit
            arcs.append((D, idx[c], lit))
        exit_lit = m.NewConstant(1)
        arcs.append((idx[gate], D, exit_lit))
        m.AddCircuit(arcs)
        # AddCircuit only covers nodes that have arcs: a sand cell with none (walled in by stones and finished sand)
        # must still make the garden infeasible, so every cell needs exactly one way in (an arc, or being the start)
        into = {i: [] for i in range(n)}
        for a, b, lit in arcs:
            if b < n:
                into[b].append(lit)
        for i in range(n):
            m.Add(sum(into[i]) == 1)
        false = m.NewConstant(0)
        # direction literals
        self.ind = {}
        self.outd = {}
        for c in cells:
            for d in DIRS:
                prev = (c[0] - DR[d], c[1] - DC[d])
                self.ind[(c, d)] = self.x.get((prev, d), false)
                self.outd[(c, d)] = self.x.get((c, d), false)
            if c == gate:
                self.outd[(c, g.gate[2])] = exit_lit
        # straightness indicators (with a side gate, the stroke enters its cell through the wall: a bend there is a turn)
        side_in = None
        if g.end is not None and g.end_side:
            side_in = (tuple(g.end), OPP[g.end_side])
        one = m.NewConstant(1)
        self.straight = {}
        for c in cells:
            for d in DIRS:
                a, b = (one if (c, d) == side_in else self.ind[(c, d)]), self.outd[(c, d)]
                s = m.NewBoolVar(f"st{c}{d}")
                m.AddImplication(s, a)
                m.AddImplication(s, b)
                m.AddBoolOr([a.Not(), b.Not(), s])
                self.straight[(c, d)] = s
        self.turns = m.NewIntVar(0, n, "turns")
        m.Add(self.turns == (n if side_in else n - 1) - sum(self.straight.values()))
        # hairpins: rake moves a into u, b from u to v, then -a out of v
        self.hp = []
        self.hp_braced = []
        for (u, b), lit in self.x.items():
            v = (u[0] + DR[b], u[1] + DC[b])
            for a in PERP[b]:
                self.hp_braced.append(g.pivot(u[0] + DR[a], u[1] + DC[a]) and g.pivot(v[0] + DR[a], v[1] + DC[a]))
                i1, i3 = self.ind[(u, a)], self.outd[(v, OPP[a])]
                h = m.NewBoolVar(f"hp{u}{b}{a}")
                m.AddImplication(h, i1)
                m.AddImplication(h, lit)
                m.AddImplication(h, i3)
                m.AddBoolOr([i1.Not(), lit.Not(), i3.Not(), h])
                self.hp.append(h)
        self.hairpins = m.NewIntVar(0, n, "hairpins")
        m.Add(self.hairpins == sum(self.hp))
        nh = g.nohairpin if nohairpin is None else nohairpin
        if nh == 1:
            m.Add(self.hairpins == 0)
        elif nh == 2:
            for h, braced in zip(self.hp, self.hp_braced):
                if not braced:
                    m.Add(h == 0)
        for c in g.noturn_cells():
            m.Add(sum(self.straight[(c, d)] for d in DIRS) + (self.start[c] if c in self.start else 0) == 1)
        # ripple rings: exactly one entry into the ring set (from outside, or by starting there)
        for ring in g.ring_sets():
            S = set(ring)
            entries = []
            for c in S:
                for d in DIRS:
                    prev = (c[0] - DR[d], c[1] - DC[d])
                    if prev in idx and prev not in S:
                        entries.append(self.x[(prev, d)])
                if c in self.start:
                    entries.append(self.start[c])
            m.Add(sum(entries) == 1)
        # wave bands / ichimatsu / given grooves: required & forbidden edges
        def edge_used(c, d):
            nb = (c[0] + DR[d], c[1] + DC[d])
            if c == gate and d == g.gate[2]:
                return None  # the exit itself: always used
            if nb not in idx:
                return false
            return self.x[(c, d)] + self.x[(nb, OPP[d])]
        for c, need in g.zone_requirements().items():
            for d in need:
                e = edge_used(c, d)
                if e is not None:
                    m.Add(e == 1)
        for c, want in g.clues.items():
            for d in DIRS:
                e = edge_used(c, d)
                if e is None:
                    if d not in want:
                        m.Add(false == 1)  # clue forbids the exit direction at the gate: infeasible
                    continue
                m.Add(e == (1 if d in want else 0))

        # shape marks: 'S' = straight through (never the rake start); 'B' = a turn (never the rake start)
        for c, t in g.marks.items():
            m.Add(sum(self.straight[(c, d)] for d in DIRS) == (1 if t == "S" else 0))
            if c in self.start:
                m.Add(self.start[c] == 0)

    def path_from(self, solver):
        """Recover the rake path (P ... gate) from a solved model."""
        g = self.g
        nxt = {}
        for (c, d), lit in self.x.items():
            if solver.BooleanValue(lit):
                nxt[c] = (c[0] + DR[d], c[1] + DC[d])
        start = [c for c, lit in self.start.items() if solver.BooleanValue(lit)][0]
        p = [start]
        while p[-1] in nxt:
            p.append(nxt[p[-1]])
        return p


class _Counter(cp_model.CpSolverSolutionCallback):
    def __init__(self, model: Model, limit):
        super().__init__()
        self.model, self.limit, self.n, self.hist = model, limit, 0, {}

    def on_solution_callback(self):
        self.n += 1
        t = self.Value(self.model.turns)
        self.hist[t] = self.hist.get(t, 0) + 1
        if self.n >= self.limit:
            self.StopSearch()


def count_solutions(g: Garden, limit=10**7, workers=1, time_limit=600):
    mdl = Model(g)
    solver = cp_model.CpSolver()
    solver.parameters.enumerate_all_solutions = True
    solver.parameters.num_workers = workers
    solver.parameters.max_time_in_seconds = time_limit
    cb = _Counter(mdl, limit)
    st = solver.Solve(mdl.m, cb)
    return cb.n, cb.hist, solver.StatusName(st)


def optimize(g: Garden, objective="turns", nohairpin=None, time_limit=120, workers=2, fix=None):
    """Minimise turns (default) or hairpins; returns (value, status, rake path, turns, hairpins).
    `fix`: dict like {'hairpins': 0} or {'turns_le': 12} adds constraints first."""
    mdl = Model(g, nohairpin=nohairpin)
    if fix:
        if "hairpins" in fix:
            mdl.m.Add(mdl.hairpins == fix["hairpins"])
        if "hairpins_le" in fix:
            mdl.m.Add(mdl.hairpins <= fix["hairpins_le"])
        if "turns_le" in fix:
            mdl.m.Add(mdl.turns <= fix["turns_le"])
        if "turns_eq" in fix:
            mdl.m.Add(mdl.turns == fix["turns_eq"])
    if objective is not None:
        obj = {"turns": mdl.turns, "hairpins": mdl.hairpins}[objective]
        mdl.m.Minimize(obj)
    solver = cp_model.CpSolver()
    solver.parameters.max_time_in_seconds = time_limit
    solver.parameters.num_workers = workers
    st = solver.Solve(mdl.m)
    name = solver.StatusName(st)
    if st in (cp_model.OPTIMAL, cp_model.FEASIBLE):
        val = int(solver.ObjectiveValue()) if objective is not None else solver.Value(mdl.turns)
        return (val, name, mdl.path_from(solver),
                solver.Value(mdl.turns), solver.Value(mdl.hairpins))
    return (None, name, None, None, None)
src/zg.py · 340 lines · 13.0 kB — Garden model, solver driver, and independent brute-force checker. 
"""zg.py: garden model, solver driver, and independent brute-force checker.

Garden text symbols:
  .  plain sand            #  stone            O  ripple stone (its 8 neighbours must be raked as one arc)
  h  sand in a horizontal wave band (lines run the band's full width, unbroken)
  v  sand in a vertical wave band
  =  sand in an ichimatsu block raked horizontally;  |  ... raked vertically
     (blocks are maximal same-symbol rectangles; within one block runs are atomic like wave bands)
Given grooves ("the master's raking") are a dict {(r, c): 'EW'|'NS'|'NE'|...}.
Gate = (r, c, side): the rake stroke ends at sand cell (r, c) and leaves through the wall on `side`.
"""
from __future__ import annotations

import subprocess
import os
from dataclasses import dataclass, field

HERE = os.path.dirname(os.path.abspath(__file__))
SOLVER = os.path.join(HERE, "zgsolve")
DIRS = "NESW"
DR = {"N": -1, "E": 0, "S": 1, "W": 0}
DC = {"N": 0, "E": 1, "S": 0, "W": -1}
OPP = {"N": "S", "S": "N", "E": "W", "W": "E"}
SAND = set(".hv=|")


@dataclass
class Garden:
    rows: list[str]
    gate: tuple[int, int, str]
    clues: dict = field(default_factory=dict)  # (r,c) -> 'EW' etc
    end: tuple[int, int] | None = None
    nohairpin: int = 0          # 0: hairpins allowed; 1: forbidden; 2: allowed only when braced against wall/stone (pivot)
    braced_turns: bool = False  # turns allowed only in cells touching a wall or stone
    braced_start: bool = False  # the rake must be set down in a cell touching a wall or stone
    end_side: str | None = None  # two-gate gardens: wall side of the second gate; a bend where the stroke enters there counts as a turn
    name: str = ""
    marks: dict = field(default_factory=dict)  # (r,c) -> 'S' (stroke runs straight through) or 'B' (stroke turns here)

    @property
    def H(self):
        return len(self.rows)

    @property
    def W(self):
        return len(self.rows[0])

    def sym(self, r, c):
        if 0 <= r < self.H and 0 <= c < self.W:
            return self.rows[r][c]
        return None

    def is_sand(self, r, c):
        s = self.sym(r, c)
        return s is not None and s in SAND

    def blocked(self, r, c):
        return not self.is_sand(r, c)

    def pivot(self, r, c):
        """Something a wide-rake U-turn (or the rake's start) can be braced against: a wall or a stone. Sand the abbot
        has already raked ('R', a finished ring) is not walkable, but it is no pivot: turning against it would rake it."""
        s = self.sym(r, c)
        return s is None or (s not in SAND and s != "R")

    def noturn_cells(self):
        if not self.braced_turns:
            return []
        return [(r, c) for (r, c) in self.sand_cells()
                if all(self.is_sand(r + DR[d], c + DC[d]) for d in DIRS)]

    def sand_cells(self):
        return [(r, c) for r in range(self.H) for c in range(self.W) if self.is_sand(r, c)]

    def ring_sets(self):
        rings = []
        for r in range(self.H):
            for c in range(self.W):
                if self.rows[r][c] == "O":
                    ring = [(r - 1, c), (r - 1, c + 1), (r, c + 1), (r + 1, c + 1),
                            (r + 1, c), (r + 1, c - 1), (r, c - 1), (r - 1, c - 1)]
                    for rr, cc in ring:
                        if not self.is_sand(rr, cc):
                            raise ValueError(f"ripple stone at {(r, c)} lacks full sand ring")
                    rings.append(ring)
        return rings

    def zone_requirements(self):
        """Required directions implied by wave bands / ichimatsu blocks: adjacent same-band cells
        along the band orientation must be joined by the stroke."""
        req = {}
        for r in range(self.H):
            for c in range(self.W):
                s = self.rows[r][c]
                if s in "h=":
                    for d in "EW":
                        if self.sym(r + DR[d], c + DC[d]) == s:
                            req.setdefault((r, c), set()).add(d)
                elif s in "v|":
                    for d in "NS":
                        if self.sym(r + DR[d], c + DC[d]) == s:
                            req.setdefault((r, c), set()).add(d)
        return req

    def to_input(self, mode=0, cap=2, nodelimit=None, bound=None, stream=False):
        lines = [f"{self.H} {self.W}"]
        for row in self.rows:
            lines.append("".join("." if ch in SAND else "R" if ch == "R" else "#" for ch in row))
        gr, gc, gd = self.gate
        lines.append(f"gate {gr} {gc} {gd}")
        if self.end is not None:
            lines.append(f"end {self.end[0]} {self.end[1]}")
            if self.end_side:
                lines.append(f"endside {self.end_side}")
        lines.append(f"nohairpin {int(self.nohairpin)}")
        lines.append(f"mode {mode}")
        lines.append(f"cap {cap}")
        if nodelimit:
            lines.append(f"nodelimit {nodelimit}")
        if bound is not None:
            lines.append(f"bound {bound}")
        if stream:
            lines.append("stream 1")
        if self.marks:
            lines.append(f"shape {len(self.marks)} " + " ".join(f"{r} {c} {t}" for (r, c), t in sorted(self.marks.items())))
        if self.braced_start:
            ec = [(r, c) for (r, c) in self.sand_cells() if any(self.pivot(r + DR[d], c + DC[d]) for d in DIRS)]
            lines.append(f"endcells {len(ec)} " + " ".join(f"{r} {c}" for r, c in ec))
        nt = self.noturn_cells()
        if nt:
            lines.append(f"noturn {len(nt)} " + " ".join(f"{r} {c}" for r, c in nt))
        req = self.zone_requirements()
        reqlines = []
        cells = set(req) | set(self.clues)
        for (r, c) in sorted(cells):
            need = set(req.get((r, c), set()))
            forb = set()
            if (r, c) in self.clues:
                g = self.clues[(r, c)]
                need |= set(g)
                forb |= set(DIRS) - set(g)
            reqlines.append(f"{r} {c} {''.join(sorted(need)) or '-'} {''.join(sorted(forb)) or '-'}")
        lines.append(f"req {len(reqlines)}")
        lines.extend(reqlines)
        rings = self.ring_sets()
        lines.append(f"rings {len(rings)}")
        for ring in rings:
            lines.append(f"{len(ring)} " + " ".join(f"{r} {c}" for r, c in ring))
        return "\n".join(lines) + "\n"

    def solve(self, mode=0, cap=2, nodelimit=None, bound=None, timeout=600, stream=False):
        """stream=True returns every solution found (in 'sols'), not just the first 16."""
        inp = self.to_input(mode=mode, cap=cap, nodelimit=nodelimit, bound=bound, stream=stream)
        p = subprocess.run([SOLVER], input=inp, capture_output=True, text=True, timeout=timeout)
        if p.returncode != 0:
            raise RuntimeError(p.stderr + "\n" + inp)
        return parse_output(p.stdout, self.W)


def parse_output(out, W):
    res = {"sols": [], "hist": {}, "hphist": {}}
    for line in out.strip().split("\n"):
        parts = line.split()
        k = parts[0]
        if k in ("nsol", "complete", "capped", "nodes"):
            res[k] = int(parts[1])
        elif k in ("hist", "hphist"):
            res[k] = {int(a): int(b) for a, b in (x.split(":") for x in parts[1:])}
        elif k in ("best", "worst", "sol"):
            turns, n = int(parts[1]), int(parts[2])
            cells = [divmod(int(x), W) for x in parts[3:3 + n]]
            entry = {"turns": turns, "draw": cells, "rake": list(reversed(cells))}
            if k == "sol":
                res["sols"].append(entry)
            else:
                res[k] = entry
    return res


def move_dir(a, b):
    dr, dc = b[0] - a[0], b[1] - a[1]
    for d in DIRS:
        if DR[d] == dr and DC[d] == dc:
            return d
    raise ValueError((a, b))


def path_moves(g: Garden, draw):
    """Moves in drawing order, starting with the entry move through the gate."""
    moves = [OPP[g.gate[2]]]
    for a, b in zip(draw, draw[1:]):
        moves.append(move_dir(a, b))
    return moves


def turns_of(g: Garden, draw):
    """Turns along a raking, counting the turn out through the veranda gate and, with a side gate, the bend where the
    stroke enters through it (in drawing order: the walk's last move against the way out through the side gate)."""
    m = path_moves(g, draw)
    t = sum(1 for a, b in zip(m, m[1:]) if a != b)
    if g.end is not None and g.end_side and len(draw) > 1 and tuple(draw[-1]) == tuple(g.end):
        t += m[-1] != g.end_side
    return t


def hairpin_list(g: Garden, draw):
    """Hairpins as (turn cell 1, turn cell 2, bottom direction, braced?) in drawing order."""
    m = path_moves(g, draw)
    out = []
    for k in range(2, len(draw) + 1):
        a, b = m[k - 2], m[k - 1]
        c = m[k] if k < len(m) else None
        if c is not None and c == OPP[a] and b != a:
            u, v = draw[k - 2], draw[k - 1]
            braced = g.pivot(u[0] + DR[a], u[1] + DC[a]) and g.pivot(v[0] + DR[a], v[1] + DC[a])
            out.append((u, v, a, braced))
    return out


def hairpins_of(g: Garden, draw):
    return len(hairpin_list(g, draw))


def glyphs(g: Garden, draw):
    """Groove glyph per cell as a pair of directions (sorted string); rake-start cell gets 1 direction."""
    out = {}
    n = len(draw)
    for i, cell in enumerate(draw):
        ds = set()
        if i == 0:
            ds.add(g.gate[2])
        else:
            ds.add(move_dir(cell, draw[i - 1]))
        if i + 1 < n:
            ds.add(move_dir(cell, draw[i + 1]))
        out[cell] = "".join(d for d in DIRS if d in ds)
    return out


def check_solution(g: Garden, draw) -> bool:
    """Independent verification of a solution against every rule (used to cross-check the C solver)."""
    sand = set(g.sand_cells())
    if len(draw) != len(sand) or set(draw) != sand:
        return False
    if draw[0] != (g.gate[0], g.gate[1]):
        return False
    for a, b in zip(draw, draw[1:]):
        if abs(a[0] - b[0]) + abs(a[1] - b[1]) != 1:
            return False
    if g.end is not None and draw[-1] != tuple(g.end):
        return False
    if g.braced_start:
        r, c = draw[-1]
        if not any(g.pivot(r + DR[d], c + DC[d]) for d in DIRS):
            return False
    gl = glyphs(g, draw)
    for cell, need in g.zone_requirements().items():
        if not need <= set(gl[cell]):
            return False
    for cell, want in g.clues.items():
        if set(gl[cell]) != set(want):
            return False
    for cell, t in g.marks.items():
        ds = set(gl[cell])
        straight = ds in ({"E", "W"}, {"N", "S"})
        if len(ds) != 2 or straight != (t == "S"):
            return False
    pos = {c: i for i, c in enumerate(draw)}
    for ring in g.ring_sets():
        ps = sorted(pos[c] for c in ring)
        if ps[-1] - ps[0] != len(ring) - 1:
            return False
    hp = hairpin_list(g, draw)
    if g.nohairpin == 1 and hp:
        return False
    if g.nohairpin == 2 and any(not br for *_, br in hp):
        return False
    m = path_moves(g, draw)
    nt = set(g.noturn_cells())
    for k, cell in enumerate(draw[:-1]):
        if cell in nt and m[k] != m[k + 1]:
            return False
    return True


def brute_force(g: Garden, limit=None):
    """Plain exhaustive DFS (no pruning beyond self-avoidance); returns (count, turn histogram)."""
    sand = set(g.sand_cells())
    start = (g.gate[0], g.gate[1])
    n = len(sand)
    count = 0
    hist = {}
    path = [start]
    seen = {start}

    def rec():
        nonlocal count
        if len(path) == n:
            if check_solution(g, path):
                count += 1
                t = turns_of(g, path)
                hist[t] = hist.get(t, 0) + 1
            return
        r, c = path[-1]
        for d in DIRS:
            nxt = (r + DR[d], c + DC[d])
            if nxt in sand and nxt not in seen:
                seen.add(nxt)
                path.append(nxt)
                rec()
                path.pop()
                seen.discard(nxt)

    rec()
    return count, hist


def ascii_solution(g: Garden, draw):
    """Render the raked pattern with box-drawing glyphs."""
    box = {"EW": "─", "NS": "│", "ES": "┌", "SW": "┐", "NE": "└", "NW": "┘",
           "N": "╵", "E": "╶", "S": "╷", "W": "╴"}
    gl = glyphs(g, draw)
    out = []
    for r in range(g.H):
        line = []
        for c in range(g.W):
            if (r, c) in gl:
                key = "".join(sorted(gl[(r, c)], key=DIRS.index))
                # normalise key ordering for lookup
                k2 = {"NE": "NE", "ES": "ES", "SW": "SW", "NW": "NW", "EW": "EW", "NS": "NS"}
                kk = key
                if len(kk) == 2:
                    for cand in box:
                        if len(cand) == 2 and set(cand) == set(kk):
                            kk = cand
                line.append(box.get(kk, "?"))
            else:
                line.append("●" if g.rows[r][c] == "O" else "■")
        out.append("".join(line))
    return "\n".join(out)
src/zgsolve.c · 294 lines · 14.6 kB — Exact solver for one-stroke Zen-garden raking puzzles. 
/* zgsolve.c: exact solver for one-stroke Zen-garden raking puzzles.
 *
 * A garden is an H x W grid of sand cells and blocked cells. A solution is a
 * Hamiltonian path over the sand cells which ends at the gate cell (and exits
 * through the gate in direction gateDir). We search in *drawing order*: from
 * the gate inward, which is the reverse of the raking order; turn and hairpin
 * counts are invariant under reversal.
 *
 * Optional constraints:
 *   - required/forbidden directions per cell (encodes wave bands & given grooves),
 *   - ring sets (cells that must be raked as one unbroken arc),
 *   - no-hairpin ("wide rake": no move triple a,b,-a with b perpendicular to a), braced U-turns allowed in mode 2:
 *     braced = both cells beyond are wall or stone ('#'); finished sand ('R': not walkable) is no pivot,
 *   - fixed rake-start cell (two-gate gardens); with `endside d`, the stroke enters that cell through a side gate
 *     in wall side d, and a bend there counts as a turn (as leaving through the veranda gate does),
 *   - shape marks per cell: 'S' = the stroke runs straight through, 'B' = it turns (either way);
 *     a marked cell can never be the rake's start.
 *
 * Modes: 0 = count solutions (up to cap), 1 = branch-and-bound min turns,
 *        2 = count solutions with turns <= bound.
 *
 * Input (stdin): see parse() below. Output: key/value lines for the Python driver.
 * Build: gcc -O3 -march=native -Wall -Wextra -o zgsolve zgsolve.c
 */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>

typedef unsigned __int128 u128;
#define MAXN 128
#define MAXR 64
#define MAXSOL 16

static const int DR[4] = {-1, 0, 1, 0}; /* N E S W */
static const int DC[4] = {0, 1, 0, -1};
static inline int opp(int d) { return (d + 2) & 3; }

static int H, W, N;
static int isSoft[MAXN];   /* 'R': sand the abbot has finished; not walkable, and not a pivot */
static int isSand[MAXN];
static int nb[MAXN][4];
static int piv[MAXN][4];   /* 1 if the cell beyond in direction d is a wall or stone: something a U-turn can pivot on */
static u128 sandMask, blackMask, notCol0, notColLast;
static int gateCell, gateDir, endCell = -1, endSideDir = -1, noHairpin = 0;
static u128 endMask; static int endMaskSet = 0;
static int reqm[MAXN], forbm[MAXN], noTurn[MAXN];
static int shapeReq[MAXN], nShape = 0, shapeCells[MAXN]; /* 1 = straight, 2 = turn */
static int streamAll = 0; /* print every solution as it is found (instead of storing the first MAXSOL) */
static int nRings, ringSize[MAXR], ringCount[MAXR];
static uint64_t ringOf[MAXN];
static long long cap = 2, nsol = 0, nodes = 0, nodeLimit = 4000000000LL;
static int aborted = 0, hitCap = 0;
static int mode = 0, turnBound = 1 << 20;
static long long hist[2 * MAXN + 8], hpHist[MAXN + 8], ubHist[MAXN + 8];
static int path[MAXN], plen = 0;
static int bestTurns = 1 << 20, bestPath[MAXN], bestLen = 0;
static int worstTurns = -1, worstPath[MAXN], worstLen = 0;
static int nStored = 0, stored[MAXSOL][MAXN], storedLen[MAXSOL], storedTurns[MAXSOL];
static int moveSeq[MAXN + 2]; /* moves in drawing order; moveSeq[0] = entry move into gate cell */

static inline int pc(u128 x) { return __builtin_popcountll((uint64_t)x) + __builtin_popcountll((uint64_t)(x >> 64)); }
static inline u128 bit(int i) { return ((u128)1) << i; }
static inline u128 spread(u128 m) {
    return m | ((m << 1) & notCol0) | ((m >> 1) & notColLast) | (m << W) | (m >> W);
}

static int countHairpins(int *unbraced) {
    /* moveSeq[0..plen-1]: moveSeq[k] is the move INTO path[k] (k=0: entry into gate cell).
       Triple (a,b,c) entering path[k-2], path[k-1], path[k]: turn cells are path[k-2] and path[k-1]. */
    int h = 0, ub = 0;
    for (int k = 2; k < plen; k++) {
        int a = moveSeq[k - 2], b = moveSeq[k - 1], c = moveSeq[k];
        if (c == opp(a) && b != a) {
            h++;
            if (!(piv[path[k - 2]][a] && piv[path[k - 1]][a])) ub++;
        }
    }
    *unbraced = ub;
    return h;
}

static void record(int turns) {
    nsol++;
    if (turns < 2 * MAXN + 8) hist[turns]++;
    int ub;
    int hp = countHairpins(&ub);
    if (hp < MAXN + 8) hpHist[hp]++;
    if (ub < MAXN + 8) ubHist[ub]++;
    if (streamAll) {
        printf("sol %d %d", turns, plen);
        for (int i = 0; i < plen; i++) printf(" %d", path[i]);
        printf("\n");
    } else if (nStored < MAXSOL) {
        memcpy(stored[nStored], path, plen * sizeof(int));
        storedLen[nStored] = plen; storedTurns[nStored] = turns; nStored++;
    }
    if (turns < bestTurns) { bestTurns = turns; memcpy(bestPath, path, plen * sizeof(int)); bestLen = plen; }
    if (turns > worstTurns) { worstTurns = turns; memcpy(worstPath, path, plen * sizeof(int)); worstLen = plen; }
}

/* v: head cell; back: direction from v toward previous path cell (gateDir at the gate);
 * m1: move into v; m2: move before that (-1 if none); unv: unvisited sand (excluding v);
 * turns: turns at cells before v; active: rings started but not finished. */
static void dfs(int v, int back, int m1, int m2, u128 unv, int turns, uint64_t active) {
    if (aborted) return;
    if (++nodes > nodeLimit) { aborted = 1; return; }
    if (mode == 1 && turns >= bestTurns) return;
    if (mode == 2 && turns > turnBound) return;
    if (forbm[v] & (1 << back)) return;
    int rem = reqm[v] & ~(1 << back);
    int remc = __builtin_popcount(rem);
    if (remc > 1) return;
    if (unv == 0) {
        if (remc != 0) return;
        if (endCell >= 0 && v != endCell) return;
        if (!((endMask >> v) & 1)) return;
        int tf = turns + (endSideDir >= 0 && m1 != endSideDir);   /* the bend where the stroke enters through a side gate */
        if (mode == 2 && tf > turnBound) return;
        record(tf);
        if (mode != 1 && nsol >= cap) { aborted = 1; hitCap = 1; }
        return;
    }
    u128 vb = bit(v), allowed = unv | vb;
    /* connectivity of remaining sand through the head */
    u128 reach = vb, nr;
    for (;;) { nr = spread(reach) & allowed; if (nr == reach) break; reach = nr; }
    if (reach != allowed) return;
    /* degree check: every unvisited cell except the eventual rake-start needs >= 2 usable neighbours */
    u128 a = (allowed << 1) & notCol0, b = (allowed >> 1) & notColLast, c = allowed << W, d = allowed >> W;
    u128 any = a | b | c | d;
    if (unv & ~any) return;
    u128 ge2 = (a & b) | (a & c) | (a & d) | (b & c) | (b & d) | (c & d);
    u128 le1 = unv & ~ge2;
    if (le1 & ~endMask) return;
    if (pc(le1) > 1) return;
    if (!(unv & endMask)) return;
    /* checkerboard parity: remaining cells alternate colours starting with the opposite of the head */
    int R = pc(unv), headBlack = (int)((blackMask >> v) & 1), nBlack = pc(unv & blackMask);
    int expBlack = headBlack ? R / 2 : (R + 1) / 2;
    if (nBlack != expBlack) return;
    if (endCell >= 0 && ((unv >> endCell) & 1) == 0) return;
    /* shape marks on unvisited cells: a straight mark needs both neighbours on one axis usable, a turn mark one on each axis */
    for (int i = 0; i < nShape; i++) {
        int s = shapeCells[i];
        if (!((unv >> s) & 1)) continue;
        int u[4];
        for (int d = 0; d < 4; d++) u[d] = nb[s][d] >= 0 && ((allowed >> nb[s][d]) & 1);
        if (shapeReq[s] == 1 ? !((u[0] && u[2]) || (u[1] && u[3])) : !((u[0] || u[2]) && (u[1] || u[3]))) return;
    }
    int forced = remc == 1 ? __builtin_ctz(rem) : -1;
    for (int e = 0; e < 4; e++) {
        if (forced >= 0 && e != forced) continue;
        if (forbm[v] & (1 << e)) continue;
        int w = nb[v][e];
        if (w < 0 || !((unv >> w) & 1)) continue;
        if (noHairpin && m2 >= 0 && e == opp(m2) && m1 != m2) {
            /* hairpin with turn cells u = v - m1 and v; braced iff both cells beyond in direction m2 are wall or stone */
            if (noHairpin == 1) continue;
            int u = nb[v][opp(m1)];
            if (!(piv[u][m2] && piv[v][m2])) continue;
        }
        if (noTurn[v] && e != m1) continue;
        if (shapeReq[v] == 1 && e != m1) continue;
        if (shapeReq[v] == 2 && e == m1) continue;
        uint64_t rw = ringOf[w], newActive = active;
        for (uint64_t t = rw; t; t &= t - 1) {
            int k = __builtin_ctzll(t);
            ringCount[k]++;
            if (ringCount[k] == ringSize[k]) newActive &= ~(1ULL << k); else newActive |= (1ULL << k);
        }
        if (!(newActive & ~ringOf[w])) {
            path[plen] = w; moveSeq[plen] = e; plen++;
            dfs(w, opp(e), e, m1, unv & ~bit(w), turns + (e != m1), newActive);
            plen--;
        }
        for (uint64_t t = rw; t; t &= t - 1) ringCount[__builtin_ctzll(t)]--;
        if (aborted) return;
    }
}

static int dirOf(char ch) {
    switch (ch) { case 'N': return 0; case 'E': return 1; case 'S': return 2; case 'W': return 3; }
    return -1;
}
static int maskOf(const char *s) {
    int m = 0;
    if (strcmp(s, "-") == 0) return 0;
    for (; *s; s++) { int d = dirOf(*s); if (d < 0) { fprintf(stderr, "bad dir %c\n", *s); exit(2); } m |= 1 << d; }
    return m;
}

static void printPath(const char *tag, int *p, int len, int turns) {
    printf("%s %d %d", tag, turns, len);
    for (int i = 0; i < len; i++) printf(" %d", p[i]);
    printf("\n");
}

/* Read exactly n fields or fail loudly, naming the key being parsed. */
#define READ(n, ...) do { if (scanf(__VA_ARGS__) != (n)) { fprintf(stderr, "malformed input for key '%s'\n", key); exit(2); } } while (0)

int main(void) {
    char key[64], buf[256];
    if (scanf("%d %d", &H, &W) != 2) return 2;
    N = H * W;
    if (N > MAXN) { fprintf(stderr, "garden too large\n"); return 2; }
    for (int r = 0; r < H; r++) {
        if (scanf("%255s", buf) != 1 || (int)strlen(buf) != W) { fprintf(stderr, "bad row %d\n", r); return 2; }
        for (int c = 0; c < W; c++) { isSand[r * W + c] = buf[c] == '.'; isSoft[r * W + c] = buf[c] == 'R'; }
    }
    gateCell = -1;
    while (scanf("%63s", key) == 1) {
        if (!strcmp(key, "gate")) { int r, c; char ds[8]; READ(3, "%d %d %7s", &r, &c, ds); gateCell = r * W + c; gateDir = dirOf(ds[0]); }
        else if (!strcmp(key, "end")) { int r, c; READ(2, "%d %d", &r, &c); endCell = r < 0 ? -1 : r * W + c; }
        else if (!strcmp(key, "endside")) { char ds[8]; READ(1, "%7s", ds); endSideDir = dirOf(ds[0]); if (endSideDir < 0) { fprintf(stderr, "bad endside\n"); return 2; } }
        else if (!strcmp(key, "nohairpin")) READ(1, "%d", &noHairpin);
        else if (!strcmp(key, "mode")) READ(1, "%d", &mode);
        else if (!strcmp(key, "bound")) READ(1, "%d", &turnBound);
        else if (!strcmp(key, "cap")) READ(1, "%lld", &cap);
        else if (!strcmp(key, "nodelimit")) READ(1, "%lld", &nodeLimit);
        else if (!strcmp(key, "req")) {
            int k; READ(1, "%d", &k);
            for (int i = 0; i < k; i++) {
                int r, c; char rs[8], fs[8]; READ(4, "%d %d %7s %7s", &r, &c, rs, fs);
                reqm[r * W + c] |= maskOf(rs); forbm[r * W + c] |= maskOf(fs);
            }
        } else if (!strcmp(key, "endcells")) {
            int k; READ(1, "%d", &k); endMaskSet = 1;
            for (int i = 0; i < k; i++) { int r, c; READ(2, "%d %d", &r, &c); endMask |= bit(r * W + c); }
        } else if (!strcmp(key, "shape")) {
            int k; READ(1, "%d", &k);
            for (int i = 0; i < k; i++) {
                int r, c; char ts[8]; READ(3, "%d %d %7s", &r, &c, ts);
                if (r < 0 || r >= H || c < 0 || c >= W || (ts[0] != 'S' && ts[0] != 'B')) { fprintf(stderr, "bad shape mark %d %d %s\n", r, c, ts); exit(2); }
                shapeReq[r * W + c] = ts[0] == 'S' ? 1 : 2;
            }
        } else if (!strcmp(key, "stream")) READ(1, "%d", &streamAll);
        else if (!strcmp(key, "noturn")) {
            int k; READ(1, "%d", &k);
            for (int i = 0; i < k; i++) { int r, c; READ(2, "%d %d", &r, &c); noTurn[r * W + c] = 1; }
        } else if (!strcmp(key, "rings")) {
            READ(1, "%d", &nRings);
            if (nRings > MAXR) { fprintf(stderr, "too many rings\n"); return 2; }
            for (int k = 0; k < nRings; k++) {
                READ(1, "%d", &ringSize[k]);
                for (int i = 0; i < ringSize[k]; i++) { int r, c; READ(2, "%d %d", &r, &c); ringOf[r * W + c] |= 1ULL << k; }
            }
        } else { fprintf(stderr, "unknown key %s\n", key); return 2; }
    }
    if (gateCell < 0 || !isSand[gateCell]) { fprintf(stderr, "gate must be a sand cell\n"); return 2; }
    sandMask = blackMask = notCol0 = notColLast = 0;
    for (int r = 0; r < H; r++) for (int c = 0; c < W; c++) {
        int v = r * W + c;
        if (isSand[v]) sandMask |= bit(v);
        if (((r + c) & 1) == 0) blackMask |= bit(v);
        if (c != 0) notCol0 |= bit(v);
        if (c != W - 1) notColLast |= bit(v);
        for (int d = 0; d < 4; d++) {
            int rr = r + DR[d], cc = c + DC[d];
            int inside = rr >= 0 && rr < H && cc >= 0 && cc < W;
            nb[v][d] = (inside && isSand[rr * W + cc]) ? rr * W + cc : -1;
            piv[v][d] = !inside || (!isSand[rr * W + cc] && !isSoft[rr * W + cc]);
        }
    }
    if (!endMaskSet) endMask = sandMask;
    if (endCell >= 0) endMask &= bit(endCell);
    for (int v = 0; v < N; v++) if (shapeReq[v]) {
        if (!isSand[v]) { fprintf(stderr, "shape mark on a non-sand cell\n"); return 2; }
        endMask &= ~bit(v);                 /* the rake's start has one connection: never a marked cell */
        shapeCells[nShape++] = v;
    }
    path[0] = gateCell; moveSeq[0] = opp(gateDir); plen = 1;
    uint64_t active = 0;
    for (uint64_t t = ringOf[gateCell]; t; t &= t - 1) {
        int k = __builtin_ctzll(t); ringCount[k]++;
        if (ringCount[k] < ringSize[k]) active |= 1ULL << k;
    }
    dfs(gateCell, gateDir, opp(gateDir), -1, sandMask & ~bit(gateCell), 0, active);
    printf("nsol %lld\n", nsol);
    printf("complete %d\n", (aborted && !hitCap) ? 0 : 1);
    printf("capped %d\n", hitCap);
    printf("nodes %lld\n", nodes);
    printf("hist");
    for (int t = 0; t < 2 * MAXN + 8; t++) if (hist[t]) printf(" %d:%lld", t, hist[t]);
    printf("\nhphist");
    for (int t = 0; t < MAXN + 8; t++) if (hpHist[t]) printf(" %d:%lld", t, hpHist[t]);
    printf("\nubhist");
    for (int t = 0; t < MAXN + 8; t++) if (ubHist[t]) printf(" %d:%lld", t, ubHist[t]);
    printf("\n");
    if (bestLen) printPath("best", bestPath, bestLen, bestTurns);
    if (worstLen) printPath("worst", worstPath, worstLen, worstTurns);
    for (int i = 0; i < nStored; i++) printPath("sol", stored[i], storedLen[i], storedTurns[i]);
    return 0;
}

The prototype and its tests ( proto/ )

proto/build.py · 40 lines · 3.5 kB — Build the prototype: samon.html (artifact page body: no doctype/head, per the artifact page contract) 
"""Build the prototype: samon.html (artifact page body: no doctype/head, per the artifact page contract)
and samon.standalone.html (a complete document, for self-hosting)."""
import json

import sys
sys.path.insert(0, '/home/claude/zen/src')
from poems import attach
from sound_pieces import pieces_html

import os
G = json.load(open('/home/claude/zen/out/proto_gardens.json'))
practice = attach(G)                       # each garden's poem (poems/samon-poems.json), and the practice garden's
DAILY_PATH = '/home/claude/zen/out/daily_gardens.json'   # a year of daily gardens (src/daily.py); the page does without if absent
DAILY = json.load(open(DAILY_PATH)) if os.path.exists(DAILY_PATH) else []
if not DAILY: print('warning: no daily gardens (out/daily_gardens.json missing)')
s = open('/home/claude/zen/proto/samon.template.html').read()
assert s.count('__GARDENS__') == 1 and s.count('__PRACTICE_POEM__') == 1 and s.count('__DAILY__') == 1 and s.count('__PIECES__') == 1
s = s.replace('__GARDENS__', json.dumps(G, separators=(',', ':'), ensure_ascii=False)).replace(
    '__PRACTICE_POEM__', json.dumps(practice, separators=(',', ':'), ensure_ascii=False)).replace(
    '__DAILY__', json.dumps(DAILY, separators=(',', ':'), ensure_ascii=False)).replace(
    '__PIECES__', pieces_html())                                     # the recordings (out/sound_pieces.json; sound/make_sound_bundle.py)
open('/home/claude/zen/proto/samon.html', 'w').write(s)
print(len(s), 'bytes: samon.html')

split = s.index('<div class="wrap">')
head, body = s[:split], s[split:]
SOCIAL = ('<meta name="description" content="Samon (砂紋): a Zen-garden raking puzzle. Walk the monk in from the gate, cover every cell of sand in one stroke, and he rakes his way back out. Playable prototype.">\n'
          '<meta name="author" content="Gwern Branwen">\n'
          '<meta property="og:type" content="website">\n<meta property="og:site_name" content="Gwern.net">\n<meta property="og:title" content="Samon 砂紋">\n'
          '<meta property="og:description" content="A Zen-garden raking puzzle: walk the monk in from the gate, cover every cell of sand in one stroke, and he rakes his way back out. 148 gardens, a daily garden, and a poem for each.">\n'
          '<meta property="og:image" content="https://gwern.net/doc/design/2026-10-05-gwern-samon-thumbnail.png">\n<meta property="og:image:width" content="1200">\n<meta property="og:image:height" content="630">\n'
          '<meta property="og:image:alt" content="A raked Zen garden with the abbot\'s vermilion seal, beside the title Samon 砂紋.">\n'
          '<meta name="twitter:card" content="summary_large_image">\n<meta name="twitter:site" content="@gwern">\n<meta name="twitter:creator" content="@gwern">\n'
          '<meta name="twitter:title" content="Samon 砂紋">\n<meta name="twitter:description" content="A Zen-garden raking puzzle: one stroke, every cell once, ending at the gate. 148 gardens, a daily garden, and a poem for each.">\n'
          '<meta name="twitter:image" content="https://gwern.net/doc/design/2026-10-05-gwern-samon-thumbnail.png">\n')
doc = ('<!doctype html>\n<html lang="en">\n<head>\n<meta charset="utf-8">\n'
       '<meta name="viewport" content="width=device-width, initial-scale=1">\n' + SOCIAL
       + head + '</head>\n<body>\n' + body + '</body>\n</html>\n')
open('/home/claude/zen/proto/samon.standalone.html', 'w').write(doc)
print(len(doc), 'bytes: samon.standalone.html')
proto/samon.template.html · 2,744 lines · 199.8 kB 
<title>Samon</title>
<style>
/* Layout: the garden is the page; one quiet row below it carries the count and the controls.
   Style: gwern.net's (gwern.net/design, /style-guide): black on white, greys for everything secondary, no colour but the
   vermilion (the novice's ensō and the abbot's seal, the one accent the design keeps), serif text with sans controls,
   square corners, hairline rules, no shadows. Dark mode is gwern.net's #161616 / #f1f1f1. No web fonts: system stacks,
   with the system's CJK fonts for 砂紋, the poems' Japanese and the seal's 静. */
:root {
  --bg: #ffffff;          /* paper */
  --fg: #000000;          /* ink */
  --muted: #767676;       /* secondary text (AA on white) */
  --line: #cccccc;        /* hairlines */
  --sand: #f3f3f1;        /* the raked gravel; the grain and grooves are drawn over it */
  --sand-dot: #d9d9d6;
  --groove: #8a8a8a;
  --groove-hi: #ffffff;
  --ghost: #bdbdbd;       /* the abbot's sketched groove */
  --stone: #262626;
  --stone-hi: #5a5a5a;
  --wall: #000000;
  --mark: #000000;        /* the abbot's marks and hint highlights (drawn thin, dashed, or faint) */
  --foot: #9a9a9a;
  --seal: #c23b22;        /* vermilion: the novice's ensō, the abbot's seal, a side gate's exit mark, and nothing else */
  --font-display: "Source Serif 4", "Source Serif Pro", "Apple Garamond", Baskerville, "Libre Baskerville", "Hiragino Mincho ProN", "Yu Mincho", "Noto Serif CJK JP", "Times New Roman", Times, serif;
  --font-ui: "Source Sans 3", "Source Sans Pro", "Lucida Sans Unicode", Helvetica, "Trebuchet MS", "Hiragino Sans", "Noto Sans CJK JP", sans-serif;
  --step--1: 0.8125rem;
  --step-0: 0.9375rem;
  --step-1: 1.125rem;
  --step-2: 1.5rem;
  --step-3: 2.1rem;
}
@media (prefers-color-scheme: dark) {
  :root:not([data-theme="light"]) {
    --bg: #161616; --fg: #f1f1f1; --muted: #9a9a9a; --line: #444444;
    --sand: #343434; --sand-dot: #3c3c3c; --groove: #0e0e0e; --groove-hi: #8a8a8a; --ghost: #666666;
    --stone: #050505; --stone-hi: #3c3c3c; --wall: #000000; --mark: #f1f1f1; --foot: #5e5e5e; --seal: #d9452c;
    color-scheme: dark;
  }
}
:root[data-theme="dark"] {
  --bg: #161616; --fg: #f1f1f1; --muted: #9a9a9a; --line: #444444;
  --sand: #343434; --sand-dot: #3c3c3c; --groove: #0e0e0e; --groove-hi: #8a8a8a; --ghost: #666666;
  --stone: #050505; --stone-hi: #3c3c3c; --wall: #000000; --mark: #f1f1f1; --foot: #5e5e5e; --seal: #d9452c;
  color-scheme: dark;
}
[hidden] { display: none !important; }  /* .seal sets display:grid, which would otherwise beat the UA's [hidden] rule */
body:not(.practice) .practice-only, body.practice .puzzle-only { display: none !important; }
body:not(.poems) .poems-only, body.poems .puzzle-only, body.poems .practice-only, body.poems .stage, body.poems .veranda { display: none !important; }
/* the scroll of poems earned */
.scroll { display: grid; gap: 12px; padding: 2px 0 6px; }
.scroll h3 { font-family: var(--font-ui); font-size: var(--step--1); letter-spacing: 0.12em; text-transform: uppercase; color: var(--muted); margin: 12px 0 0; font-weight: 600; }
.entry { border-top: 1px solid var(--line); padding-top: 10px; }
.entry .verse { font-family: var(--font-display); font-size: var(--step-1); line-height: 1.5; white-space: pre-line; }
.entry .by { font-family: var(--font-ui); font-size: var(--step--1); letter-spacing: 0.06em; color: var(--muted); margin-top: 4px; padding-left: 3.5em; }
.entry .by button { font: inherit; color: inherit; background: none; border: 0; padding: 0; text-decoration: underline dotted; text-underline-offset: 3px; cursor: pointer; }
.entry .ja { font-family: var(--font-display); font-size: var(--step-0); letter-spacing: 0.06em; line-height: 1.6; margin-top: 6px; }
.entry .src { font-family: var(--font-ui); font-size: var(--step--1); color: var(--muted); margin-top: 2px; }
.entry .src a { color: inherit; }
.scroll .left { color: var(--muted); font-size: var(--step--1); margin: 0; }
html, body { height: 100%; }
body {
  margin: 0; background: var(--bg); color: var(--fg);
  font-family: var(--font-display); font-size: var(--step-0); line-height: 1.55;
  padding-inline: 16px; padding-block: 0;
}
.wrap { max-width: 600px; margin: 0 auto; padding-block: 18px 40px; display: grid; gap: 14px; }
header { display: flex; align-items: baseline; justify-content: space-between; gap: 12px; flex-wrap: wrap; }
.brand { font-family: var(--font-display); font-weight: 700; font-size: var(--step-3); letter-spacing: 0.01em; line-height: 1; }
.brand span { font-weight: 400; font-size: var(--step-1); color: var(--muted); margin-left: 0.4em; letter-spacing: 0.15em; }
header .right { margin-left: auto; display: flex; align-items: center; gap: 12px; min-width: 0; position: relative; }
.where { text-align: right; min-width: 0; }
.sound { display: inline-flex; flex: none; align-items: stretch; }
.mute { flex: none; width: 34px; height: 34px; padding: 0; display: inline-grid; place-items: center; color: var(--fg); border: 1px solid var(--line); background: transparent; }
.mute:hover { border-color: var(--fg); }
.more-sound { flex: none; width: 20px; height: 34px; padding: 0 0 2px; border: 1px solid var(--line); border-left: 0; background: transparent; color: var(--muted); font-size: 11px; line-height: 1; }
.more-sound:hover, .more-sound[aria-expanded="true"] { color: var(--fg); background: transparent; }
.sound-panel { position: absolute; right: 0; top: calc(100% + 6px); z-index: 5; display: grid; gap: 8px; justify-items: start; min-width: 13em; padding: 10px 12px; background: var(--bg); border: 1px solid var(--line); }
.vol-row { display: flex; align-items: center; gap: 8px; font-family: var(--font-ui); font-size: var(--step--1); letter-spacing: 0.06em; color: var(--muted); }
.mute .slash { display: none; }
.mute.off .slash { display: block; }
.mute.off .wave { opacity: 0.25; }
.mute.blocked { color: var(--seal); border-color: var(--seal); }
.mute.blocked .slash { display: block; }
.mute-hint { position: absolute; right: 0; bottom: calc(100% + 1px); white-space: nowrap; font-family: var(--font-ui); font-size: var(--step--1); color: var(--muted); letter-spacing: 0.04em; animation: hintfade 7s ease-in forwards; pointer-events: none; }
@keyframes hintfade { 0%, 70% { opacity: 1; } 100% { opacity: 0; } }
.where .time { font-family: var(--font-ui); font-size: var(--step--1); letter-spacing: 0.14em; text-transform: uppercase; color: var(--muted); }
.where .gname { font-family: var(--font-display); font-size: var(--step-2); line-height: 1.15; text-wrap: balance; }
.stage { position: relative; width: 100%; }
canvas { display: block; width: 100%; height: auto; touch-action: none; user-select: none; -webkit-user-select: none; }
canvas:focus-visible { outline: 2px solid var(--fg); outline-offset: 3px; }
.seal {
  position: absolute; right: 6%; bottom: 7%; width: 74px; height: 74px;
  background: var(--seal); color: #fff; display: grid; place-items: center;
  font-family: var(--font-display); font-weight: 700; font-size: 44px; line-height: 1;
  transform: rotate(-6deg); opacity: 0.94; pointer-events: none; user-select: none; -webkit-user-select: none;
}
.seal.grey { background: #8a8a8a; }
.seal.stamp { animation: stamp 420ms cubic-bezier(.2,.9,.3,1.2) both; }
.seal.under { animation: none; opacity: 0.2; }   /* (only when a poem's text cannot be laid clear of it) */
@keyframes stamp { from { transform: scale(1.6) rotate(-14deg); opacity: 0; } to { transform: scale(1) rotate(-6deg); opacity: 0.94; } }
.veranda {
  color: var(--fg); border-bottom: 1px solid var(--line);
  display: flex; align-items: center; justify-content: space-between; gap: 10px; flex-wrap: wrap;
  padding: 4px 0 12px;
}
.count { display: flex; gap: 18px; align-items: baseline; font-variant-numeric: tabular-nums; }
.count b { font-family: var(--font-display); font-size: var(--step-2); font-weight: 700; }
.count small { font-family: var(--font-ui); font-size: var(--step--1); letter-spacing: 0.12em; text-transform: uppercase; color: var(--muted); margin-right: 6px; }
.buttons { display: flex; gap: 6px; flex-wrap: wrap; }
button {
  font: inherit; font-family: var(--font-ui); font-size: var(--step--1); letter-spacing: 0.02em;
  background: transparent; color: var(--fg); border: 1px solid var(--fg);
  border-radius: 0; padding: 5px 11px; cursor: pointer;
}
button:hover { background: color-mix(in srgb, var(--fg) 8%, transparent); }
button:focus-visible { outline: 2px solid var(--fg); outline-offset: 2px; }
button:disabled { opacity: 0.4; cursor: default; }
button:disabled:hover { background: transparent; }
button.primary { background: var(--fg); color: var(--bg); border-color: var(--fg); font-weight: 600; }
button.primary:hover { background: color-mix(in srgb, var(--fg) 82%, var(--bg)); }
.veranda button[aria-pressed="true"] { background: var(--fg); color: var(--bg); }
.status { min-height: 1.55em; font-size: var(--step-0); color: var(--fg); }
.status.warn { font-style: italic; }
.actions { display: flex; gap: 8px; flex-wrap: wrap; margin-top: -6px; }
.actions:empty { display: none; }
.act, .toggle, .pill { color: var(--fg); border: 1px solid var(--line); background: transparent; }
.act { border-color: var(--fg); font-weight: 600; }
.actions .share { flex: 1 0 100%; font-family: ui-monospace, Menlo, Consolas, "DejaVu Sans Mono", monospace; font-size: 13px; line-height: 1.25; color: var(--fg); background: transparent; border: 1px solid var(--line); padding: 6px 8px; resize: vertical; white-space: pre-wrap; overflow-wrap: anywhere; }
.actions .share:focus { outline: 1px solid var(--fg); outline-offset: 0; }
.act:hover, .toggle:hover, .pill:hover { border-color: var(--fg); }
.assists { display: flex; gap: 6px; flex-wrap: wrap; align-items: center; }
.assists .t, nav .world .t { font-family: var(--font-ui); font-size: var(--step--1); letter-spacing: 0.14em; text-transform: uppercase; color: var(--muted); }
.assists .t { margin-right: 4px; }
.toggle[aria-pressed="true"] { background: var(--fg); color: var(--bg); border-color: var(--fg); }
.toggle[aria-pressed="true"]::before { content: "✓ "; }
.rule { border-left: 1px solid var(--line); background: color-mix(in srgb, var(--fg) 3%, transparent); padding: 6px 12px; color: var(--fg); margin: 0; }
.rule .facts { display: block; font-family: var(--font-ui); font-size: var(--step--1); color: var(--muted); letter-spacing: 0.01em; margin-top: 2px; }
nav.gardens { display: grid; gap: 8px; }
nav .world { display: grid; grid-template-columns: 7.5em 1fr; gap: 10px; align-items: center; }
nav .world .pills { display: flex; gap: 6px; flex-wrap: wrap; min-width: 0; }
.pill { font-size: var(--step--1); padding: 4px 10px; cursor: pointer; display: inline-flex; gap: 6px; align-items: center; }
.pill[aria-current="true"] { border-color: var(--fg); font-weight: 600; }
.pill .dot { width: 8px; height: 8px; border-radius: 50%; border: 1px solid var(--muted); }
input.vol { width: 88px; margin: 0 2px; accent-color: var(--fg); align-self: center; cursor: pointer; }
.pill.done .dot { background: var(--muted); }
.pill.sealed .dot, .pill.grey .dot { border-radius: 0; }               /* a seal is square; a garden raked off par is a round dot */
.pill.sealed .dot { background: var(--fg); border-color: var(--fg); }
.pill.grey .dot { background: var(--muted); border-color: var(--muted); }
.pip { font-size: var(--step--1); min-width: 2.1em; padding: 3px 4px; color: var(--fg); border: 1px solid var(--line);
       background: transparent; font-variant-numeric: tabular-nums; cursor: pointer; line-height: 1.3; }
.pip:hover { border-color: var(--fg); background: transparent; }
.pip.done { background: color-mix(in srgb, var(--fg) 12%, transparent); }
.pip.sealed { background: var(--fg); border-color: var(--fg); color: var(--bg); }            /* the seals collected */
.pip.grey { background: color-mix(in srgb, var(--fg) 45%, var(--bg)); border-color: transparent; color: var(--bg); }
.pip[aria-current="true"] { outline: 2px solid var(--fg); outline-offset: 1px; font-weight: 700; }
.pip:disabled { opacity: 0.32; cursor: default; }
.pip.extra { border-style: dashed; }
nav .world .more { font-family: var(--font-ui); font-size: var(--step--1); color: var(--muted); align-self: center; margin-left: 4px; }
.pip:disabled:hover { border-color: var(--line); }
nav .world .lock { font-family: var(--font-ui); font-size: var(--step--1); color: var(--muted); align-self: center; }
nav .world.locked { opacity: 0.75; }
nav .world .fold { font-family: var(--font-ui); font-size: var(--step--1); color: var(--muted); background: none; border: 0; padding: 3px 4px; text-decoration: underline dotted; text-underline-offset: 3px; cursor: pointer; align-self: center; }
nav .world .fold:hover { color: var(--fg); background: none; }
.openall { background: none; border: 0; padding: 2px 0; color: var(--muted); text-decoration: underline; font: inherit; font-family: var(--font-ui); font-size: var(--step--1); cursor: pointer; }
.openall:hover { background: none; color: var(--fg); }
nav .world .pills { gap: 4px; }
.help { font-size: var(--step--1); color: var(--muted); border-top: 1px solid var(--line); padding-top: 10px; margin: 0; }
details.help summary { cursor: pointer; list-style: none; font-family: var(--font-ui); letter-spacing: 0.04em; }
details.help summary::-webkit-details-marker { display: none; }
details.help summary::before { content: "▸ "; display: inline-block; width: 1em; }
details.help[open] summary::before { content: "▾ "; }
details.help p { margin: 6px 0 0; }
.help kbd { font-family: var(--font-ui); border: 1px solid var(--line); padding: 0 5px; }
@media (max-width: 420px) {
  nav .world { grid-template-columns: 1fr; gap: 4px; }
  .seal { width: 58px; height: 58px; font-size: 34px; }
}
/* the poem: whatever the raking, a poem is its reward, laid over the raked garden; then the garden gives way to the next */
.poem { position: absolute; inset: 0; display: grid; place-items: center; padding: 4% 6%; cursor: pointer; overflow: auto; scrollbar-width: none;
  opacity: 0; transition: opacity 1.6s ease; }        /* (inset to the sand by script: the walls stay dark) */
.poem.on { opacity: 1; }
.poem .card { max-width: 30em; font-size: var(--poem-fs, 1.15rem); transform: translateY(6px); transition: transform 1.8s ease;
  padding: 0.75em 1em 0.6em; background: var(--bg); border: 1px solid var(--fg); }
.poem.on .card { transform: none; }
.poem .verse { font-family: var(--font-display); font-weight: 400; line-height: 1.55; white-space: pre-line; color: var(--fg); }
.poem .poet { margin-top: 0.6em; padding-left: 3.5em; font-family: var(--font-ui); font-size: 0.72em; letter-spacing: 0.08em; color: var(--muted); }   /* (the attribution is set in by an indent, with no dash) */
.poem .orig { margin: 0 0 0 0.6em; padding: 0 2px; border: 0; background: none; color: var(--muted); font: inherit;
  letter-spacing: 0.06em; text-decoration: underline dotted; text-underline-offset: 3px; cursor: pointer; }
.poem .orig:hover { background: none; color: var(--fg); }
.poem .ja { margin-top: 0.7em; font-family: var(--font-display); font-size: 0.86em; line-height: 1.6; letter-spacing: 0.06em; color: var(--fg); }
.poem .src { font-family: var(--font-ui); font-size: 0.68em; color: var(--muted); margin-top: 0.2em; }
.poem .src a { color: inherit; }
.poem .tick { height: 1px; width: 0; margin-top: 0.8em; background: var(--fg); opacity: 0.5; }
.poem .tick.run { animation: tick var(--hold, 10s) linear forwards; }
@keyframes tick { to { width: 100%; } }
.stage.leaving, .where.leaving { opacity: 0; transition: opacity var(--fade, 1.2s) ease; }
.stage.arriving, .where.arriving { animation: arrive var(--fade, 1.2s) ease both; }
@keyframes arrive { from { opacity: 0; } to { opacity: 1; } }
.ppoem { margin: 0; font-family: var(--font-display); white-space: pre-line; color: var(--fg); }
.ppoem figcaption { font-family: var(--font-ui); font-size: var(--step--1); color: var(--muted); letter-spacing: 0.1em; margin-top: 0.4em; padding-left: 3.5em; white-space: normal; }
@media (prefers-reduced-motion: reduce) {
  .poem, .poem .card { transition: none; } .poem .tick.run { animation: none; }
  .stage.leaving, .where.leaving { transition: none; } .stage.arriving, .where.arriving { animation: none; }
}
/* the monk waits in the gateway: a ring beckons three times (then rests: the idea has been had), and (in the first garden, or
   until the first walk) a ghostly finger shows the drag, three times too */
.beckon { position: absolute; left: 0; top: 0; border-radius: 50%; border: 1.5px solid var(--fg); pointer-events: none;
  transform: translate(-50%, -50%); animation: beckon 1.9s ease-out 3 forwards; }
@keyframes beckon { from { opacity: 0.9; transform: translate(-50%, -50%) scale(0.7); } to { opacity: 0; transform: translate(-50%, -50%) scale(1.7); } }
.finger { position: absolute; left: 0; top: 0; width: 32px; height: 32px; margin: -16px 0 0 -16px; border-radius: 50%; pointer-events: none;
  background: color-mix(in srgb, var(--fg) 34%, transparent); outline: 1px solid var(--bg); opacity: 0; }
@media (prefers-reduced-motion: reduce) { .seal.stamp { animation: none; } .beckon { animation: none; opacity: 0.75; } .finger { display: none; } }
</style>

<div class="wrap">
  <header>
    <div class="brand" title="Samon (砂紋, “sand pattern”): the lines a rake leaves in the sand of a Zen garden">Samon<span>砂紋</span></div>
    <div class="right">
    <div class="where" id="where" title="The world (a time of day) and this garden’s place in it"><div class="time" id="time">Dawn</div><div class="gname" id="gname">The corridor</div></div>
    <span class="sound">
    <button class="mute" id="mute" type="button" aria-pressed="false" aria-label="Mute" title="Mute (M)">
      <svg viewBox="0 0 24 24" width="22" height="22" aria-hidden="true" focusable="false">
        <path class="horn" d="M4 9.5v5h3.2L12 18.5v-13L7.2 9.5H4z" fill="currentColor"/>
        <path class="wave w1" d="M15 9.4a3.6 3.6 0 0 1 0 5.2" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round"/>
        <path class="wave w2" d="M17.4 7a7 7 0 0 1 0 10" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round"/>
        <path class="slash" d="M4.5 4.5l15 15" fill="none" stroke="currentColor" stroke-width="1.9" stroke-linecap="round"/>
      </svg>
    </button>
    <button class="more-sound" id="soundMore" type="button" aria-expanded="false" aria-controls="soundPanel" aria-label="Sound settings" title="Sound: the switch, the volume, and haptics">▾</button>
    </span>
    <div class="sound-panel" id="soundPanel" hidden>
      <button class="pill" id="soundToggle" type="button" aria-pressed="true" title="The garden's sounds: a bamboo pipe drips, wind, birds by day and crickets by night, the rake, a bell at the close of each world, and a shakuhachi recording now and then">Sound on</button>
      <label class="vol-row" id="volRow"><span>Volume</span><input id="soundVol" class="vol" type="range" min="0" max="100" step="1" title="Volume" aria-label="Volume"></label>
      <button class="pill" id="hapticsToggle" type="button" aria-pressed="true" hidden title="A tick in the hand for each cell walked">Haptics on</button>
    </div>
    </div>
  </header>

  <div class="stage" id="stage">
    <canvas id="garden" tabindex="0" aria-label="Garden. Drag from the gate to walk in."></canvas>
    <div class="seal" id="seal" hidden aria-label="The abbot's seal: raked at par" title="The abbot’s seal (静, “stillness”): raked at par">静</div>
    <div class="beckon" id="beckon" hidden aria-hidden="true"></div>
    <div class="finger" id="finger" hidden aria-hidden="true"></div>
    <div class="poem" id="poem" hidden aria-live="polite" title="Tap beside the poem to stay with this garden">
      <div class="card" title="Tap the poem to go on to the next garden">
        <div class="verse" id="verse" lang="en"></div>
        <div class="poet"><span id="poet"></span><button class="orig" id="orig" type="button" aria-label="Show the original Japanese and the source" title="Show the original Japanese and its source">original</button></div>
        <div class="ja" id="ja" lang="ja" hidden></div>
        <div class="src" id="src" hidden></div>
        <div class="tick" id="tick"></div>
      </div>
    </div>
  </div>
  <section class="scroll poems-only" id="scroll" aria-label="Poems earned"></section>

  <div class="veranda">
    <div class="count puzzle-only">
      <span title="Changes of direction in your walk so far: the fewer, the calmer the raking"><small>Turns</small><b id="turns">0</b></span>
      <span title="The fewest turns any raking of this garden can have (the solver has proved it): rake at par for the abbot’s seal"><small>Par</small><b id="par">0</b></span>
    </div>
    <div class="buttons puzzle-only">
      <button id="undo" type="button" title="Take back the last footprint (Backspace); after raking, walk back one cell">Step back</button>
      <button id="restore" type="button" hidden title="Bring back the walk that Start over, or stepping back to a footprint, discarded">Restore</button>
      <button id="restart" type="button" title="Smooth the sand and begin again at the gate (R); Restore brings the walk back">Start over</button>
      <button id="hint" type="button" title="The abbot judges your own walk: whether it can still reach par, or still be finished at all (H). Where to step back to counts as his help, and his seal is then grey">Ask the abbot</button>
      <button id="watch" type="button" hidden title="See the monk rake this garden once more; it counts for nothing">Watch</button>
      <button id="rakeDay" type="button" class="primary" hidden title="The daily garden is raked once: this raking is today's record">Rake</button>
      <button id="next" type="button" class="primary" hidden title="On to the next garden (Enter)">Next garden</button>
    </div>
    <div class="count practice-only"><span title="A garden to rake freely: nothing is counted here"><small>Practice</small><b>No goal</b></span></div>
    <div class="buttons practice-only">
      <button id="toolRake" type="button" aria-pressed="true" title="Drag to rake the sand">Rake</button>
      <button id="toolStone" type="button" aria-pressed="false" title="Tap to set a stone down, or to lift one; the sand under it keeps its grooves">Stones</button>
      <button id="tines" type="button" title="How many tines the rake has: more tines, more lines to a stroke">3 tines</button>
      <button id="smooth" type="button" title="Rake the whole garden flat again">Smooth</button>
    </div>
  </div>

  <div class="status" id="status" role="status" aria-live="polite"></div>
  <div class="actions" id="actions"></div>

  <div class="assists puzzle-only" role="group" aria-label="Assists">
    <span class="t" title="Ways of seeing the garden that make it easier; none of them counts as the abbot’s help">Assists</span>
    <button class="toggle" id="aPreview" type="button" aria-pressed="false" title="Show the grooves your walk would rake, as you draw it">Preview grooves</button>
    <button class="toggle" id="aChecker" type="button" aria-pressed="false" title="Shade the cells in two colours, like a chessboard: a walk alternates them, so counting what is left tells where it can end">Checkerboard</button>
    <button class="toggle" id="aCut" type="button" aria-pressed="false" title="Mark the cells of sand your walk can no longer reach">Cut-off sand</button>
  </div>

  <p class="rule" id="rule"></p>

  <nav class="gardens" id="nav" aria-label="Gardens"></nav>

  <details class="help puzzle-only" id="howto">
    <summary title="How the game is played, in a few lines; the controls say more when hovered">[How to play]</summary>
    <p>[Drag the monk in from the gate and cover every cell of sand without crossing your path; when you let go, he sets down his rake and walks backwards out, raking. Tap a footprint to step back to it, or the gate to start over; after raking, tap any cell to walk back to it. Fewer turns is calmer: rake at par for the abbot's seal. Each raking brings a poem, then the next garden. Keyboard: arrows walk, <kbd>Backspace</kbd> steps back, <kbd>R</kbd> starts over, <kbd>H</kbd> asks the abbot, <kbd>Enter</kbd> goes on.]</p>
  </details>
  <figure class="ppoem practice-only" id="ppoem"></figure>
  <p class="help practice-only">[Drag anywhere to rake. With <b>Stones</b> selected, tap to place a stone or lift one; the sand under a stone keeps whatever was there. <b>Smooth</b> rakes the garden flat again.]</p>
</div>
<!-- the recordings (sound/make_sound_bundle.py → out/sound_pieces.json): each a data: URL in an inert script, which a gwtar build
     extracts into its archive and the page then reads back through the gwtar loader (see samonAsset below) -->
__PIECES__

<script>
const GARDENS = __GARDENS__;
const DAILY = __DAILY__;                  // a year of daily gardens (out/daily_gardens.json); [] if none were built
const NCAMP = GARDENS.length;             // the campaign; the daily garden, if any, is appended after it
const PRACTICE_POEM = __PRACTICE_POEM__;
const TIMES = ["", "Dawn", "Morning", "Noon", "Afternoon", "Evening", "Dusk", "Night"];
const DN = "NESW";                         // direction indices: 0 N, 1 E, 2 S, 3 W
const DR = [-1, 0, 1, 0], DC = [0, 1, 0, -1];
const DIX = { N: 0, E: 1, S: 2, W: 3 };
const DIRS = { N: [-1, 0], E: [0, 1], S: [1, 0], W: [0, -1] };
const OPP = { N: "S", S: "N", E: "W", W: "E" };
const opp = d => (d + 2) & 3;
const SAND = new Set([".", "h", "v", "=", "|"]);
const TINES = [0.2, 0.5, 0.8];
const RING8 = [[-1, 0], [-1, 1], [0, 1], [1, 1], [1, 0], [1, -1], [0, -1], [-1, -1]];

const VIEW_EL = document.body;             // carries the view classes (practice, poems); the single-file edition puts them on the game's own element
const cv = document.getElementById("garden");
const ctx = cv.getContext("2d");
const el = id => document.getElementById(id);

let view = "puzzle";                       // "puzzle" or "practice"
let idx = 0, G = null, path = [], mode = "draw", dragging = false, anim = null;
let cut = null;                            // the route discarded by the last rewind or restart: one Restore brings it back
let shared = null;                         // the raking on show came from a #r= link: another player's, shown but not credited ({ turns, par, kind })
let rakeTimer = null, doneAt = 0;          // a complete walk is raked after a short beat; when the last raking finished
let started = false, demoOnce = false;     // has this player ever started a walk; show the drag once more (after a press elsewhere)
let obs = [];                              // the cells the abbot pointed at with his observation (until another garden)
let rakedHelped = false, cutHelped = false;  // the last raking was made with help; the route Restore would bring back was
let openAll = false;                       // prototype convenience: every world open
try { openAll = localStorage.getItem("samon-open-all") === "1"; } catch (e) {}
try { started = localStorage.getItem("samon-started") === "1"; } catch (e) {}
if (!started) el("howto").open = true;      // the help stands open until the first walk, and is folded away after (it can be unfolded)
let ghosts = new Map();                    // cell -> Set of directions: grooves the abbot has sketched for you
let marks = { cells: [], target: -1, checker: false };  // what the last hint pointed at
let hintPresses = 0;                       // since the route last changed
let helped = false;                        // the abbot has helped with this garden since it was last raked: its seal can only be grey
let progress = {}, assists = { preview: false, checker: false, cutoff: false };
try { progress = JSON.parse(localStorage.getItem("samon-progress") || "{}") || {}; } catch (e) { progress = {}; }
try { Object.assign(assists, JSON.parse(localStorage.getItem("samon-assists") || "{}") || {}); } catch (e) {}
function saveProgress() { try { localStorage.setItem("samon-progress", JSON.stringify(progress)); } catch (e) {} }
function saveAssists() { try { localStorage.setItem("samon-assists", JSON.stringify(assists)); } catch (e) {} }

/* ---------- garden model ---------- */
function sym(g, r, c) { return (r >= 0 && r < g.rows.length && c >= 0 && c < g.rows[0].length) ? g.rows[r][c] : null; }
function buildGarden(g) {
  const H = g.rows.length, W = g.rows[0].length, N = H * W;
  const sand = new Uint8Array(N), nbr = new Int16Array(N * 4).fill(-1), braced = new Uint8Array(N), black = new Uint8Array(N);
  const piv = new Uint8Array(N * 4);   // a wall or a stone beyond the cell: something a wide-rake U-turn can pivot on
  for (let r = 0; r < H; r++) for (let c = 0; c < W; c++) if (SAND.has(g.rows[r][c])) sand[r * W + c] = 1;
  for (let v = 0; v < N; v++) {
    const r = Math.floor(v / W), c = v % W;
    black[v] = (r + c) % 2 === 0 ? 1 : 0;
    for (let d = 0; d < 4; d++) {
      const rr = r + DR[d], cc = c + DC[d], inside = rr >= 0 && rr < H && cc >= 0 && cc < W;
      if (inside && sand[rr * W + cc]) nbr[v * 4 + d] = rr * W + cc;
      // the abbot's finished sand ('R') can't be walked on, but it is no pivot: turning against it would rake it
      if (!inside || (!sand[rr * W + cc] && g.rows[rr][cc] !== "R")) { piv[v * 4 + d] = 1; braced[v] = 1; }
    }
  }
  const req = new Uint8Array(N), forb = new Uint8Array(N), mark = new Uint8Array(N);
  for (let r = 0; r < H; r++) for (let c = 0; c < W; c++) {
    const s = g.rows[r][c], v = r * W + c;
    if (s === "h" || s === "=") for (const d of [1, 3]) if (sym(g, r + DR[d], c + DC[d]) === s) req[v] |= 1 << d;
    if (s === "v" || s === "|") for (const d of [0, 2]) if (sym(g, r + DR[d], c + DC[d]) === s) req[v] |= 1 << d;
  }
  for (const [k, ds] of Object.entries(g.clues || {})) {
    const [r, c] = k.split(",").map(Number), v = r * W + c;
    for (let d = 0; d < 4; d++) { if (ds.includes(DN[d])) req[v] |= 1 << d; else forb[v] |= 1 << d; }
  }
  const markList = [];
  for (const [k, t] of Object.entries(g.marks || {})) { const [r, c] = k.split(",").map(Number); mark[r * W + c] = t === "S" ? 1 : 2; markList.push(r * W + c); }
  const rings = [], ringOf = Array.from({ length: N }, () => []);
  for (let r = 0; r < H; r++) for (let c = 0; c < W; c++) if (g.rows[r][c] === "O") {
    const ring = RING8.map(([a, b]) => (r + a) * W + (c + b));
    ring.forEach(k => ringOf[k].push(rings.length)); rings.push(ring);
  }
  const sandList = []; for (let v = 0; v < N; v++) if (sand[v]) sandList.push(v);
  const keyOf = ([r, c]) => r * W + c;
  return { ...g, H, W, N, sand, sandList, nSand: sandList.length, nbr, piv, braced, black, req, forb, mark, markList, rings, ringOf,
           gateCell: g.gate[0] * W + g.gate[1], gateDir: DIX[g.gate[2]], endCell: g.end ? g.end[0] * W + g.end[1] : -1,
           intendedK: g.intended.map(keyOf),
           parK: g.par_moves ? g.par_moves.map(ms => { let k = g.gate[0] * W + g.gate[1]; const R = [k]; for (const ch of ms) { const d = DIX[ch]; k += DR[d] * W + DC[d]; R.push(k); } return R; })
                             : [g.intended.map(keyOf)] };
}
const rc = k => [Math.floor(k / G.W), k % G.W];
const key = (r, c) => r * G.W + c;
function isSand(r, c) { return r >= 0 && r < G.H && c >= 0 && c < G.W && G.sand[key(r, c)] === 1; }
function dirTo(v, w) { for (let d = 0; d < 4; d++) if (G.nbr[v * 4 + d] === w) return d; return -1; }
function dirBetween(a, b) { const d = dirTo(a, b); return d < 0 ? null : DN[d]; }

/* ---------- route state: the same rules serve the player's moves and the abbot's search ---------- */
function newState() {
  return { P: new Int16Array(G.N + 1), mv: new Int8Array(G.N + 1), vis: new Uint8Array(G.N), ringCnt: new Int16Array(G.rings.length), len: 0 };
}
function push(S, w) {
  S.mv[S.len] = S.len === 0 ? opp(G.gateDir) : dirTo(S.P[S.len - 1], w);  // mv[i]: the move INTO P[i] (P[0] is entered through the gate)
  S.P[S.len++] = w; S.vis[w] = 1;
  for (const k of G.ringOf[w]) S.ringCnt[k]++;
}
function pop(S) { const w = S.P[--S.len]; S.vis[w] = 0; for (const k of G.ringOf[w]) S.ringCnt[k]--; }
function stateOf(P) { const S = newState(); for (const v of P) push(S, v); return S; }
const ctz4 = m => (m & 1) ? 0 : (m & 2) ? 1 : (m & 4) ? 2 : 3;
function reqMsg(v) { const [r, c] = rc(v); return G.rows[r][c] === "." ? "The abbot's groove runs straight on from here." : "Bands are crossed in one straight run, edge to edge."; }

/* Why stepping from the head to w is refused (null = allowed; "" = silently ignored). */
function check(S, w) {
  const v = S.P[S.len - 1], e = dirTo(v, w);
  if (e < 0) return "";
  if (S.vis[w]) return "Never step on your own footprints.";
  const m1 = S.mv[S.len - 1], back = opp(m1);
  if ((G.forb[v] >> e) & 1) return "Follow the abbot's groove through this cell.";
  if (G.req[v] & ~(1 << back) & ~(1 << e)) return reqMsg(v);
  if (G.mark[v] === 1 && e !== m1) return "A pale pebble: the line runs straight through it.";
  if (G.mark[v] === 2 && e === m1) return "A dark pebble: the line turns at it.";
  const bw = opp(e);
  if ((G.forb[w] >> bw) & 1) return "The abbot's groove does not run this way.";
  const rest = G.req[w] & ~(1 << bw);
  if (rest & (rest - 1)) return reqMsg(w);
  if (rest) { const n = G.nbr[w * 4 + ctz4(rest)]; if (n < 0 || S.vis[n]) return reqMsg(w); }
  if (G.mark[w] === 1) { const n = G.nbr[w * 4 + e]; if (n < 0 || S.vis[n]) return "A pale pebble: the line would have to run straight on through it, and can't."; }
  if (G.mark[w] === 2) {
    const a = G.nbr[w * 4 + ((e + 1) & 3)], b = G.nbr[w * 4 + ((e + 3) & 3)];
    if ((a < 0 || S.vis[a]) && (b < 0 || S.vis[b])) return "A dark pebble: the line would have to turn at it, and can't.";
  }
  for (let k = 0; k < G.rings.length; k++) {
    const c = S.ringCnt[k];
    if (c > 0 && c < 8 && !G.ringOf[w].includes(k)) return "Finish the ring around the ripple stone first.";
  }
  if (G.wide && S.len >= 2) {
    const m2 = S.mv[S.len - 2];
    if (e === opp(m2) && m1 !== m2) {
      const u = S.P[S.len - 2];
      if (!(G.piv[u * 4 + m2] && G.piv[v * 4 + m2]))
        return [u, v].some(k => { const [r, c] = rc(k); return sym(G, r + DR[m2], c + DC[m2]) === "R"; })
          ? "The wide rake can't turn back against the abbot's finished sand (it would rake it): a U-turn needs a wall or a stone to pivot on."
          : "The wide rake can't turn back in open sand: a U-turn needs a wall or a stone to pivot on.";
    }
  }
  return null;
}
function canEnd(v) { return !G.mark[v] && (G.endCell < 0 || v === G.endCell) && (!G.wide || G.braced[v] === 1); }
/* Why a route covering every cell can't be raked from where it ends (null = it can). */
function finishCheck(S) {
  const v = S.P[S.len - 1], back = opp(S.mv[S.len - 1]);
  if (G.req[v] & ~(1 << back)) return "The walk can't end here: this cell's groove continues.";
  if (G.mark[v]) return "The walk can't end on a pebble: the line has to pass through it.";
  if (G.endCell >= 0 && v !== G.endCell) return "The walk has to end at the side gate, where the stroke begins.";
  if (G.wide && !G.braced[v]) return "Set the wide rake down against a stone or a wall.";
  return null;
}

/* ---------- the abbot's search: can this route still be finished? ---------- */
const BQ = new Int16Array(256), BSEEN = new Uint8Array(256);
function reachable(S) {   // unvisited cells reachable from the head through unvisited cells
  const head = S.P[S.len - 1]; BSEEN.fill(0, 0, G.N);
  let qh = 0, qt = 0, n = 0; BQ[qt++] = head; BSEEN[head] = 1;
  while (qh < qt) {
    const v = BQ[qh++];
    for (let d = 0; d < 4; d++) { const w = G.nbr[v * 4 + d]; if (w >= 0 && !S.vis[w] && !BSEEN[w]) { BSEEN[w] = 1; BQ[qt++] = w; n++; } }
  }
  return n;
}
function freeNbrs(S, v, head) { let f = 0; for (let d = 0; d < 4; d++) { const w = G.nbr[v * 4 + d]; if (w >= 0 && (!S.vis[w] || w === head)) f++; } return f; }
function feasible(S) {    // necessary conditions only (the same as the offline solver's pruning)
  const head = S.P[S.len - 1], rem = G.nSand - S.len;
  if (G.endCell >= 0 && S.vis[G.endCell]) return false;
  if (reachable(S) !== rem) return false;
  let dead = 0, nb = 0, anyEnd = false;
  for (const v of G.sandList) {
    if (S.vis[v]) continue;
    const f = freeNbrs(S, v, head);
    if (f === 0) return false;
    if (f === 1) { if (++dead > 1 || !canEnd(v)) return false; }
    if (G.black[v]) nb++;
    if (!anyEnd && canEnd(v)) anyEnd = true;
  }
  if (!anyEnd) return false;
  if (nb !== (G.black[head] ? Math.floor(rem / 2) : Math.ceil(rem / 2))) return false;
  for (const v of G.markList) {
    if (S.vis[v]) continue;
    const u = d => { const w = G.nbr[v * 4 + d]; return w >= 0 && (!S.vis[w] || w === head); };
    if (G.mark[v] === 1 ? !((u(0) && u(2)) || (u(1) && u(3))) : !((u(0) || u(2)) && (u(1) || u(3)))) return false;
  }
  return true;
}
/* Depth-first search for one completion. Returns true (S then holds a full raking), false (proved impossible),
   or null (gave up after `budget` nodes: the abbot never claims impossibility without proof). */
/* `dead` remembers positions already proved to have no completion. A position is the footprints, the head, and the
   last two moves: the rules never look further back, so a position reached again by another route is just as dead.
   The keys are exact (no hashing), so a remembered failure is a proof, not a guess. */
function posKey(S) {
  let k = S.P[S.len - 1] + "," + S.mv[S.len - 1] + "," + (S.len >= 2 ? S.mv[S.len - 2] : 9);
  for (let i = 0; i < G.N; i += 30) { let x = 0; const e = Math.min(G.N, i + 30); for (let j = i; j < e; j++) if (S.vis[j]) x |= 1 << (j - i); k += "," + x; }
  return k;
}
function complete(S, budget, dead = new Set()) {
  let nodes = 0;
  const rec = () => {
    if (++nodes > budget) throw complete;
    if (S.len === G.nSand) return finishCheck(S) === null;
    const pk = posKey(S);
    if (dead.has(pk)) return false;
    if (feasible(S)) {
      const v = S.P[S.len - 1];
      for (let d = 0; d < 4; d++) {
        const w = G.nbr[v * 4 + d];
        if (w < 0 || S.vis[w] || check(S, w) !== null) continue;
        push(S, w);
        if (rec()) return true;
        pop(S);
      }
    }
    dead.add(pk);
    return false;
  };
  try { return rec(); } catch (x) { if (x === complete) return null; throw x; }
}
function analyze(P, budget = 250000) {
  const n = P.length;
  // Par: a route can still reach par iff it is a prefix of one of the garden's par rakings (all are precomputed, by proof).
  let lcp = 0, R = null;
  for (const Rk of G.parK) {
    let k = 0; while (k < n && k < Rk.length && Rk[k] === P[k]) k++;
    if (k > lcp || (k === lcp && Rk === G.parK[0])) { lcp = k; R = Rk; }
  }
  if (lcp === n) return { state: "par", lcp, R };
  const S = stateOf(P), dead = new Set();
  const ok = complete(S, budget, dead);
  if (ok === true) return { state: "finish", lcp, completion: Array.from(S.P.slice(0, S.len)) };
  if (ok === null) return { state: "unknown", lcp };
  // Stuck: say why, and find the nearest footprint from which the garden can still be finished. A completion of a
  // longer route completes all its shorter prefixes, so "can be finished" only switches once, from yes to no, along the
  // route: binary search between the par prefix (yes) and the whole route (no, just proved). The answer is proven
  // nearest when the footprint after it is proved impossible (then so is every later one).
  const why = diagnose(P);
  let lo = lcp, hi = n, hiProved = true;
  while (hi - lo > 1) {
    const mid = (lo + hi) >> 1, r = complete(stateOf(P.slice(0, mid)), Math.ceil(budget / 2), dead);
    if (r === true) lo = mid;
    else { hi = mid; hiProved = r === false; }
  }
  return { state: "stuck", lcp, back: lo, proven: hiProved, why };
}
function diagnose(P) {
  const S = stateOf(P), head = P[P.length - 1], rem = G.nSand - S.len;
  if (!rem) return { text: finishCheck(S) || "The walk can't end here.", cells: [head] };
  reachable(S);
  const cutOff = G.sandList.filter(v => !S.vis[v] && !BSEEN[v]);
  if (cutOff.length) return { text: `${cutOff.length === 1 ? "One cell of sand is" : cutOff.length + " cells of sand are"} cut off: the walk can't reach ${cutOff.length === 1 ? "it" : "them"} any more.`, cells: cutOff };
  if (G.endCell >= 0 && S.vis[G.endCell]) return { text: "You have walked through the side gate's cell, where the walk has to end.", cells: [G.endCell] };
  const dead = G.sandList.filter(v => !S.vis[v] && freeNbrs(S, v, head) === 1);
  if (dead.length > 1) return { text: `${dead.length} cells now have only one way in, but the walk can end in only one of them.`, cells: dead };
  if (dead.length === 1 && !canEnd(dead[0])) {
    const v = dead[0], rule = G.endCell >= 0 && v !== G.endCell ? "the walk must end at the side gate"
      : G.mark[v] ? "the line has to pass through a pebble" : "the wide rake must be set down against a stone or a wall";
    return { text: `This cell has only one way in, so the walk would have to end there, and it can't: ${rule}.`, cells: dead };
  }
  let nb = 0; for (const v of G.sandList) if (!S.vis[v] && G.black[v]) nb++;
  if (nb !== (G.black[head] ? Math.floor(rem / 2) : Math.ceil(rem / 2)))
    return { text: "Count the dark and light cells still to cover: a walk alternates colours, and from here one colour has too many, so a cell would be stranded.", cells: [], checker: true };
  return { text: "Every way on from here runs into a dead end somewhere; the abbot has checked them all.", cells: [] };
}

/* ---------- loading a garden ---------- */
/* ---------- the daily garden: one garden for everyone, picked by the local date from a year of them, raked once ---------- */
let todayOverride = null;                  // a test hook sets a date
const pad2 = n => String(n).padStart(2, "0");
function todayDate() { return todayOverride ? new Date(todayOverride) : new Date(); }
function dateKey(d) { return `${d.getFullYear()}-${pad2(d.getMonth() + 1)}-${pad2(d.getDate())}`; }
function dayNumber(d) { return Math.round((Date.UTC(d.getFullYear(), d.getMonth(), d.getDate()) - Date.UTC(2026, 0, 1)) / 86400000); }
function ensureDaily(force = false) {       // the daily slot (after the campaign) holds today's garden; returns its index, or -1
  if (!DAILY.length) return -1;
  const d = todayDate(), k = dateKey(d), n = dayNumber(d);
  const onScreen = GARDENS.length > NCAMP && view === "puzzle" && idx === NCAMP;   // past midnight, the garden being played stays until it is left (so does a shared raking of another day's)
  if (GARDENS.length === NCAMP || (GARDENS[NCAMP].dateKey !== k && (force || !onScreen))) GARDENS[NCAMP] = dailyFor(n);
  return NCAMP;
}
function dailyStreak() {                    // consecutive days raked, ending today (or yesterday, if today is not yet raked)
  let n = 0; const d = todayDate();
  if (!(progress["daily:" + dateKey(d)] || {}).done) d.setDate(d.getDate() - 1);
  while ((progress["daily:" + dateKey(d)] || {}).done) { n++; d.setDate(d.getDate() - 1); }
  return n;
}
function dateLabel(k) { const [y, m, d] = k.split("-").map(Number); return new Date(y, m - 1, d).toLocaleDateString("en-GB", { day: "numeric", month: "long", year: "numeric" }); }
const isDaily = () => !!(GARDENS[idx] && GARDENS[idx].daily);
function dailyReady() { return isDaily() && mode === "draw" && path.length === G.nSand && !finishCheck(stateOf(path)); }
function dailyActions(g, p) {
  shareAction(g, p.kind);
  addAction("Read the poem again", () => { const box = el("poem"); box.hidden = false; fillPoem(g.poem, box); void box.offsetWidth; box.classList.add("on"); el("tick").hidden = true; }, "Lay the poem over the garden again");
}
function pickDaily() { const i = ensureDaily(true); if (i >= 0) { load(i); showStage(); } }

function load(i) {
  cancelPoem();
  stopDemo(false); if (mode === "demo" || (mode === "reveal" && REV && REV.demo)) { mode = "draw"; REV = null; }
  view = "puzzle"; VIEW_EL.classList.remove("practice", "poems");
  idx = i;
  const g = GARDENS[i];
  try { if (!g.daily) localStorage.setItem("samon-last", String(i)); } catch (e) {}
  G = buildGarden(g);
  if (shared) { shared = null; dropHash(); }   // (a shared raking left for another garden is put away, link and all)
  path = []; mode = "draw"; cut = null; ghosts = new Map(); stopAnim(); clearAdvice();
  const draft = (progress[g.name] || {}).draft;
  if (Array.isArray(draft) && draft.length > 1 && draft[0] === G.gateCell) {
    const S = newState(); push(S, draft[0]); let ok = true;
    for (let j = 1; j < draft.length && ok; j++) { if (check(S, draft[j]) === null) push(S, draft[j]); else ok = false; }
    if (ok) path = draft.slice();
  }
  helped = !!(progress[g.name] || {}).helped;
  obs = []; rakedHelped = false; cutHelped = false;
  pzBuild();
  el("seal").hidden = true; el("seal").classList.remove("grey"); el("next").hidden = true; el("restore").hidden = true;
  el("next").textContent = g.daily ? "Back to the gardens" : "Next garden";
  const inPart = GARDENS.filter(x => x.world === g.world && !!x.extra === !!g.extra), pos = inPart.indexOf(g) + 1;
  const streak = g.daily ? dailyStreak() : 0;
  el("time").textContent = g.daily ? `Daily garden${streak ? ` · streak ${streak}` : ""}` : `${TIMES[g.world]} · ${g.extra ? "more, " : ""}${pos} of ${inPart.length}`;
  el("where").title = g.daily ? "Today's garden: one for everyone, from a year of them, raked once; the streak counts the days in a row it has been raked"
    : `${TIMES[g.world]}, the ${["", "first", "second", "third", "fourth", "fifth", "sixth", "seventh"][g.world]} of seven worlds, each a time of day: ${g.extra ? `an extra garden of this world (${pos} of ${inPart.length})` : `garden ${pos} of ${inPart.length} on its main path`}. A world opens when two-thirds of the one before is raked.`;
  el("next").title = g.daily ? "Back to the gardens (Enter)" : "On to the next garden (Enter)";
  el("gname").textContent = g.daily ? dateLabel(g.dateKey) : g.name;
  el("par").textContent = g.par;
  const facts = g.rakings === 1 ? "exactly one valid raking exists" :
    `${g.rakings.toLocaleString("en-US")} valid rakings exist; ${g.at_par === g.rakings ? "all reach par" : `${g.at_par.toLocaleString("en-US")} reach par`}`;
  el("par").parentElement.title = `Par ${g.par}: the fewest turns any raking of this ${G.H}×${G.W} garden can have, proved by the solver (${facts}; a valid raking is a walk in from the gate that covers every cell of sand once and may end where it does). Rake at par for the abbot's seal.`;
  const dailyRule = g.daily ? "Today's garden, the same for everyone. Draw and undo freely; nothing is raked until you press Rake, and that first raking is the day's record. " : "";
  const rule = (dailyRule ? `[${escapeHtml(dailyRule.trim())}] ` : "") + (g.intro ? `[${escapeHtml(g.intro)}]` : ruleSummary(g));   // (only where the garden brings in a rule: a plain garden's rule is the game's)
  el("rule").innerHTML = rule; el("rule").hidden = !rule;
  setStatus(path.length ? "Your unfinished walk is where you left it." : "Drag the monk in from the gate.");
  cv.setAttribute("aria-label", `${g.daily ? "Today's garden" : g.name}: a ${G.H} by ${G.W} garden. Drag from the gate to walk in.`);
  const rec = progress[g.name] || {};
  if (g.daily && rec.done && Array.isArray(rec.path) && rec.path.length === G.nSand) {   // raked already today: show the record
    path = rec.path.slice(); mode = "done"; doneAt = 0; rakedHelped = false;
    const st = strokeOf(path); rakeAlong(PZ, st, 0, st.L); PZ.dirty = null; PZ.feet = null; fieldLight(PZ);
    showSeal(rec.kind);
    setStatus(`Today's record: ${rec.turns} turns, par ${rec.par}${rec.kind === "red" ? ", the abbot's seal" : rec.kind === "grey" ? ", his seal in grey" : ""}. Streak: ${streak} day${streak === 1 ? "" : "s"}. A new garden comes tomorrow.`);
    el("next").hidden = false; dailyActions(g, rec);
  }
  renderNav(); resize(); checkComplete(false); update();
}
function pickGarden(i) { if (worldOpen(GARDENS[i].world)) { load(i); showStage(); maybeDemo(); } }
/* Each world has a short main path (the template) and optional extras. A world opens once two-thirds of the main path
   of the world before it has been raked, at any number of turns, counting extras raked instead: par is never required,
   and any garden can be finished by persistence or with the abbot's help, so nobody is ever stuck. */
const UNLOCK = 2 / 3;
function rakedIn(w) { return GARDENS.filter(g => g.world === w && (progress[g.name] || {}).done).length; }
function needIn(w) { return Math.ceil(UNLOCK * GARDENS.filter(g => g.world === w && !g.extra).length); }
function worldOpen(w) { return w <= 1 || openAll || (worldOpen(w - 1) && rakedIn(w - 1) >= needIn(w - 1)); }
const unraked = g => !(progress[g.name] || {}).done;
function nextGarden(i) {   // along the main path (or through a world's extras, from an extra); then the first unraked garden
  const here = GARDENS[i];
  if (here.daily) return firstUnraked();
  for (let j = i + 1; j < NCAMP; j++) {
    const g = GARDENS[j];
    if (!worldOpen(g.world)) break;
    if (here.extra ? (g.extra && g.world === here.world) : !g.extra) return j;
  }
  return firstUnraked();
}
function firstUnraked() {
  let k = GARDENS.findIndex(g => !g.daily && worldOpen(g.world) && !g.extra && unraked(g));
  if (k < 0) k = GARDENS.findIndex(g => !g.daily && worldOpen(g.world) && unraked(g));
  return k >= 0 ? k : 0;
}
function ruleSummary(g) {
  const parts = [];
  if (g.wide) parts.push("Wide rake: a U-turn needs a wall or a stone to pivot on, and the walk must end against one.");
  if (g.end) parts.push("Begin at the side gate.");
  if (g.marks && Object.keys(g.marks).length) parts.push("Pale pebble: straight through. Dark pebble: turn.");
  if (g.rows.some(r => /[=|]/.test(r))) parts.push("Rake each block across or down, as hatched.");
  if (g.rows.some(r => r.includes("O"))) parts.push("Ripple stones: rake each ring in one unbroken run.");
  if (g.rows.some(r => /[hv]/.test(r))) parts.push("Cross each band in one straight run.");
  if (g.rows.some(r => r.includes("R"))) parts.push("The abbot's finished ring is not to be touched.");
  return parts.join(" ");                 // (empty for a plain garden: one stroke, every cell once, ending at the gate, which the help says)
}
function showStage() {               // picking a garden from the list at the bottom brings the garden itself into view
  const r = cv.getBoundingClientRect();
  if (r.top < 0 || r.top > window.innerHeight * 0.6) cv.scrollIntoView({ block: "start", behavior: matchMedia("(prefers-reduced-motion: reduce)").matches ? "auto" : "smooth" });
}
function escapeHtml(s) { return s.replace(/[&<>"]/g, ch => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;" }[ch])); }
function turnsOf(P = path) {
  if (!P.length) return 0;
  let t = 0, prev = opp(G.gateDir);
  for (let i = 1; i < P.length; i++) { const d = dirTo(P[i - 1], P[i]); if (d !== prev) t++; prev = d; }
  // with a side gate, a complete walk ends at its cell, and the stroke enters there through the wall: a bend there is a turn
  if (G.endCell >= 0 && G.end_side && P.length === G.nSand && P[P.length - 1] === G.endCell && P.length > 1 && prev !== DIX[G.end_side]) t++;
  return t;
}
function backDir(i, P = path) { return i === 0 ? G.gate[2] : dirBetween(P[i], P[i - 1]); }

/* ---------- interaction ---------- */
function cellAt(px, py) {
  const { ox, oy, cs, wall } = geom;
  const c = Math.floor((px - ox) / cs), r = Math.floor((py - oy) / cs);
  if (r < 0 || r >= G.H || c < 0 || c >= G.W) {
    // a touch on the gate's gap in the wall counts as the gate cell
    const [gr, gc, side] = G.gate; const x = ox + gc * cs, y = oy + gr * cs;
    const inGap = side === "S" ? (px >= x && px < x + cs && py >= y + cs && py < y + cs + wall)
      : side === "N" ? (px >= x && px < x + cs && py < y && py >= y - wall)
      : side === "W" ? (py >= y && py < y + cs && px < x && px >= x - wall)
      : (py >= y && py < y + cs && px >= x + cs && px < x + cs + wall);
    return inGap ? G.gateCell : -1;
  }
  return key(r, c);
}
function routeChanged(kind) {        // kind: "start" (first footprint), "step", "back", or "cut" (a rewind or restart)
  clearAdvice();
  if (path.length) { demoOnce = false; if (!started) { started = true; try { localStorage.setItem("samon-started", "1"); } catch (e) {} } }
  if (kind === "step") {
    const head = path[path.length - 1];
    if (ghosts.has(head)) ghosts.delete(head); else ghosts.clear();   // following the abbot's sketch uses it up; leaving it erases it
    cut = null;
  } else if (kind !== "start") ghosts.clear();
  const g = GARDENS[idx]; const p = progress[g.name] || {}; p.draft = path.slice(); progress[g.name] = p; saveProgressSoon();
  if (PZ) pzWalk();
  checkComplete();
  update();
}
let saveTimer = null;
function saveProgressSoon() { clearTimeout(saveTimer); saveTimer = setTimeout(() => { saveTimer = null; saveProgress(); }, 300); }
function saveProgressNow() { if (saveTimer) { clearTimeout(saveTimer); saveTimer = null; saveProgress(); } }   // (a page closed within the debounce would lose the last footprints: the fuzzer found it)
window.addEventListener("pagehide", saveProgressNow);
document.addEventListener("visibilitychange", () => { if (document.hidden) saveProgressNow(); });
function stepTo(w) {                     // one step of the walk to a neighbouring cell (or back off the last one); true if the walk changed
  if (mode !== "draw" || w < 0 || !path.length) return false;
  const head = path[path.length - 1];
  if (w === head) return false;
  if (path.length >= 2 && w === path[path.length - 2]) { path.pop(); setStatus(""); crunch(0.08); haptic(6); routeChanged("back"); return true; }
  if (dirTo(head, w) < 0) return false;
  const why = check(stateOf(path), w);
  if (why === null) { path.push(w); setStatus(""); crunch(0.11); haptic(8); routeChanged("step"); return true; }
  if (why) setStatus(why, true);
  return false;
}
function setHelped(v) {                   // help marks the garden until it is next raked (and travels with a helped route)
  helped = v;
  const p = progress[GARDENS[idx].name] || {};
  if (v) p.helped = true; else delete p.helped;
  progress[GARDENS[idx].name] = p; saveProgressSoon();
}
function rewindTo(k, note = "") {      // keep the first k footprints; the rest can be restored. note: what is true from here, for the status line
  if (k >= path.length) return;
  const n = path.length - k;
  if (n >= 2) { cut = path.slice(); cutHelped = helped; }
  path = path.slice(0, k);
  setStatus(`Stepped back ${n} footprint${n === 1 ? "" : "s"}${note ? `: ${note}` : ""}.${n >= 2 ? " Restore brings them back." : ""}`);
  routeChanged("cut");
}
/* Completing the walk is raking: when the finger lifts from a walk that covers every cell (and may end where it does),
   the monk rakes. From the keyboard there is a longer beat, in which Backspace can still take the last step back. */
function checkComplete(arm = true) {
  clearTimeout(rakeTimer); rakeTimer = null;
  if (mode !== "draw" || path.length !== G.nSand) return;
  const why = finishCheck(stateOf(path));
  if (why) { setStatus(why, true); return; }
  if (isDaily()) { setStatus("Every cell is covered: press Rake, and this raking is today's record."); return; }
  if (arm) armRake(450); else setStatus("Every cell is covered: tap the monk to rake.");
}
function armRake(delay) {
  clearTimeout(rakeTimer); rakeTimer = null;
  if (view !== "puzzle" || mode !== "draw" || dragging || isDaily() || path.length !== G.nSand || finishCheck(stateOf(path))) return;
  rakeTimer = setTimeout(() => { rakeTimer = null; rakeNow(); }, delay);
}
function reopen(k) {                       // after raking: smooth the sand and go back to drawing, keeping the first k footprints
  if (mode !== "done" || k < 1) return;
  if (shared) { leaveShared(); return; }     // (another player's raking is not walked back into: the garden is simply yours again)
  if (isDaily()) { setStatus("Today's raking is made; a new garden comes tomorrow.", true); return; }
  cancelPoem();
  stopAnim(); mode = "draw";
  el("seal").hidden = true; el("seal").classList.remove("grey"); el("next").hidden = true;
  cut = path.slice(); cutHelped = rakedHelped; path = path.slice(0, k);
  // walking back into a raking made with help keeps the help: re-raking the same route can't wash the grey out
  // (walking back all the way to the gate is a fresh start, as Start over is)
  if (rakedHelped && k > 1) setHelped(true);
  if (PZ) PZ.feet = null;                  // the raked grooves go: the whole field is redone from the unraked sand
  setStatus(rakedHelped && k > 1 ? "Smoothed back to your footprints; the abbot's help still counts on this walk. Restore rakes it again as it was."
                                 : "Smoothed back to your footprints; Restore rakes it again as it was.");
  routeChanged("cut");
}
function pointerCell(ev) {
  const rect = cv.getBoundingClientRect();
  return [(ev.clientX - rect.left) * (geom.w / rect.width), (ev.clientY - rect.top) * (geom.h / rect.height)];
}
/* Dragging. The walk follows the pointer along the axis the hand is moving on, not the cell the pointer happens to be in:
   a hand that drifts off the row while sweeping along it (arms are not mounted on tracks) still draws a straight run, and
   the walk turns only when the motion itself turns, to within DRAG_SNAP degrees of the other axis, or when the pointer has
   gone two cells off the run, which is no drift. The direction is read over the last DRAG_WIN cells of motion, so a jitter
   in place reads as nothing. The pointer is captured, so the grip survives leaving the field (and the window): a pointer
   beyond the walls is read as the nearest cell. */
const DRAG = { trail: [], axis: null };
const DRAG_WIN = 0.4, DRAG_SNAP = 30;
function dragStart(x, y) { DRAG.trail = [{ x, y, s: 0 }]; DRAG.axis = null; }
function dragDir(x, y) {                    // the vector of the last DRAG_WIN cells of motion, or null while there has not been that much
  const T = DRAG.trail, last = T[T.length - 1], s = last.s + Math.hypot(x - last.x, y - last.y), back = s - DRAG_WIN * geom.cs;
  T.push({ x, y, s });
  while (T.length > 1 && T[1].s <= back) T.shift();
  const p = T[0]; if (p.s > back) return null;
  const dx = x - p.x, dy = y - p.y;
  return Math.hypot(dx, dy) < 0.6 * DRAG_WIN * geom.cs ? null : [dx, dy];   // (a short net distance over that much path is a wobble)
}
function dragAxis(v) {                      // the axis of motion, with hysteresis: the current one holds until the motion is clearly on the other
  const th = Math.atan2(Math.abs(v[1]), Math.abs(v[0])) * 180 / Math.PI;    // 0 = horizontal, 90 = vertical
  if (DRAG.axis === "H") return th > 90 - DRAG_SNAP ? "V" : "H";
  if (DRAG.axis === "V") return th < DRAG_SNAP ? "H" : "V";
  return th < 45 ? "H" : "V";
}
function clampField(x, y) {
  const { ox, oy, cs } = geom;
  return [Math.min(Math.max(x, ox), ox + G.W * cs - 0.01), Math.min(Math.max(y, oy), oy + G.H * cs - 0.01)];
}
function dragTo(x, y) {                     // the pointer is at (x, y) (canvas pixels, possibly beyond the field): walk towards it
  const v = dragDir(x, y);
  if (v) { const ax = dragAxis(v); if (DRAG.axis && ax !== DRAG.axis) settle(DRAG.axis, x, y); DRAG.axis = ax; }
  if (!DRAG.axis || !path.length || mode !== "draw") return;
  const [pr, pc] = rc(cellAt(...clampField(x, y)));
  for (let n = 0; n < G.W + G.H; n++) {
    const [hr, hc] = rc(path[path.length - 1]);
    let ax = DRAG.axis;
    if (ax === "H" && Math.abs(pr - hr) >= 2) ax = "V"; else if (ax === "V" && Math.abs(pc - hc) >= 2) ax = "H";
    const nr = ax === "V" ? hr + Math.sign(pr - hr) : hr, nc = ax === "H" ? hc + Math.sign(pc - hc) : hc;
    if ((nr === hr && nc === hc) || !stepTo(key(nr, nc))) break;
  }
}
function settle(ax, x, y) {                 // the motion has turned: first finish what it was doing along the old axis, as far as the pointer went
  if (!path.length || mode !== "draw") return;
  const [pr, pc] = rc(cellAt(...clampField(x, y)));
  for (let n = 0; n < G.W + G.H; n++) {
    const [hr, hc] = rc(path[path.length - 1]);
    const nr = ax === "V" ? hr + Math.sign(pr - hr) : hr, nc = ax === "H" ? hc + Math.sign(pc - hc) : hc;
    if ((nr === hr && nc === hc) || !stepTo(key(nr, nc))) break;
  }
}
cv.addEventListener("pointerdown", ev => {
  audio();
  if (view === "practice") { prDown(ev); return; }
  clearTimeout(rakeTimer); rakeTimer = null;
  if (demoRun || mode === "demo" || (mode === "reveal" && REV && REV.demo)) { stopDemo(true); return; }
  if (mode === "reveal") { finishAnimNow(); return; }
  if (mode === "done") {
    if (performance.now() - doneAt < 600) return;        // a second tap meant to skip the raking is not a step back
    if (poemRun && poemRun.shown && !poemRun.stay) { stayPoem(false); return; }   // (the walls around the poem: put it away and stay)
    if (shared) { leaveShared(); return; }
    const [x, y] = pointerCell(ev), j = path.indexOf(cellAt(x, y));
    if (j < 0) return;
    dragging = true; reopen(j + 1); dragStart(x, y); cv.setPointerCapture(ev.pointerId); return;
  }
  const [x, y] = pointerCell(ev); const k = cellAt(x, y);
  if (k < 0 && !path.length) return;
  if (k === G.gateCell) { const fresh = !path.length; if (path.length > 1) { cut = path.slice(); cutHelped = helped; } path = [G.gateCell]; dragging = true; crunch(0.1); haptic(8); routeChanged(fresh ? "start" : "cut"); }
  else if (path.includes(k)) { const j = path.indexOf(k); if (j < path.length - 1) rewindTo(j + 1); dragging = true; }
  else if (path.length && k >= 0 && dirTo(path[path.length - 1], k) >= 0) { dragging = true; stepTo(k); }
  else if (!path.length) { setStatus("Start at the gate: drag the monk in from there.", true); demoOnce = true; draw(); return; }
  else return;
  dragStart(x, y); cv.setPointerCapture(ev.pointerId);
});
cv.addEventListener("pointermove", ev => {
  if (view === "practice") { prMove(ev); return; }
  if (!dragging) return;
  dragTo(...pointerCell(ev));
});
const endDrag = () => { const was = dragging; dragging = false; if (view === "practice") prUp(); else if (was) armRake(250); };
cv.addEventListener("pointerup", endDrag); cv.addEventListener("pointercancel", endDrag);
cv.addEventListener("keydown", ev => {
  audio();
  if (view !== "puzzle") return;
  if (demoRun || mode === "demo" || (mode === "reveal" && REV && REV.demo)) { ev.preventDefault(); stopDemo(true); return; }
  const map = { ArrowUp: "N", ArrowDown: "S", ArrowLeft: "W", ArrowRight: "E" };
  if (map[ev.key]) {
    ev.preventDefault();
    if (mode !== "draw") return;
    if (!path.length) { path = [G.gateCell]; routeChanged("start"); return; }
    const [r, c] = rc(path[path.length - 1]); const [a, b] = DIRS[map[ev.key]];
    if (r + a >= 0 && r + a < G.H && c + b >= 0 && c + b < G.W) stepTo(key(r + a, c + b));
  } else if (ev.key === "Backspace") { ev.preventDefault(); undo(); }
  else if (ev.key === "Enter") {
    if (mode === "reveal") finishAnimNow();
    else if (mode === "done") advance(true);
    else if (rakeTimer) { clearTimeout(rakeTimer); rakeTimer = null; rakeNow(); }
    else if (dailyReady()) rakeNow();
  }
  else if (ev.key === "Escape" && poemRun && !poemRun.stay) stayPoem(false);
  else if (ev.key === "r" || ev.key === "R") restart();
  else if (ev.key === "h" || ev.key === "H") askAbbot();
});
function undo() {
  if (mode === "done") { reopen(path.length - 1); return; }
  if (mode === "draw" && path.length > 1) { path.pop(); setStatus(""); routeChanged("back"); } else if (mode === "draw" && path.length) restart();
}
function restart() {
  if (demoRun || mode === "demo" || (mode === "reveal" && REV && REV.demo)) { stopDemo(true); return; }
  if (shared) { leaveShared(); return; }
  if (mode === "done" && isDaily()) { setStatus("Today's raking is made; a new garden comes tomorrow.", true); return; }
  cancelPoem();
  stopAnim();
  if (mode !== "reveal" && path.length > 1) { cut = path.slice(); cutHelped = mode === "done" ? rakedHelped : helped; }
  mode = "draw"; path = []; el("seal").hidden = true; el("seal").classList.remove("grey"); el("next").hidden = true;
  setStatus("Drag the monk in from the gate."); routeChanged("cut");
}
function restore() {
  if (!cut || mode !== "draw") return;
  if (cutHelped && !helped) setHelped(true);   // a route made with help brings its help back with it
  path = cut; cut = null; setStatus("Restored."); routeChanged("cut");
}
el("undo").onclick = undo; el("restart").onclick = restart; el("restore").onclick = restore;
el("next").onclick = () => { if (mode === "done") advance(true); else { load(nextGarden(idx)); maybeDemo(); } };
el("hint").onclick = () => askAbbot();
el("rakeDay").onclick = () => { if (dailyReady()) rakeNow(); };
function rakeNow() {
  if (view !== "puzzle" || mode !== "draw" || path.length !== G.nSand || finishCheck(stateOf(path))) return;
  clearTimeout(rakeTimer); rakeTimer = null; startReveal();
}

/* ---------- the abbot's hint: about your route, not his ---------- */
function clearAdvice() { el("actions").innerHTML = ""; marks = { cells: [], target: -1, checker: false }; hintPresses = 0; }
function showSeal(kind) {                 // the abbot's seal (or none), with what it says
  const sl = el("seal"); sl.hidden = !kind; sl.classList.toggle("grey", kind === "grey");
  const say = kind === "grey" ? "The abbot's seal, in grey: raked at par, with his help" : "The abbot's seal (静, “stillness”): raked at par";
  sl.setAttribute("aria-label", say); sl.title = say; return sl;
}
function addAction(label, fn, title = "") {
  const b = document.createElement("button"); b.type = "button"; b.className = "act"; b.textContent = label; if (title) b.title = title;
  b.onclick = () => { fn(); if (view !== "poems") cv.focus({ preventScroll: true }); };   // (the button may be gone after; the keys should still walk the monk)
  el("actions").appendChild(b);
}
function sketch(R, from) {             // sketch the groove of R at index `from` (the next cell of a raking that fits your route)
  if (from >= R.length) return false;
  const k = R[from], ds = new Set();
  ds.add(from === 0 ? G.gate[2] : dirBetween(k, R[from - 1]));
  if (from + 1 < R.length) ds.add(dirBetween(k, R[from + 1]));
  ghosts.set(k, ds); return true;
}
let abbotBudget = 250000;                 // search nodes per question (a test hook can lower it)
function askAbbot() {
  if (view !== "puzzle" || mode !== "draw") return;
  const g = GARDENS[idx];
  hintPresses++;
  if (path.length <= 1 && hintPresses === 1 && g.hint) {
    obs = (g.hint_cells || []).map(([r, c]) => key(r, c));
    setStatus(`The abbot: “${g.hint}”${obs.length ? " (He points.)" : ""} Ask again and he'll sketch a groove.`); draw(); return;
  }
  const P = path.length ? path : [G.gateCell];
  const a = analyze(P, abbotBudget);
  el("actions").innerHTML = ""; marks = { cells: [], target: -1, checker: false };
  const steps = n => `${n} footprint${n === 1 ? "" : "s"}`;
  if (a.state === "unknown") {                 // no answer, no charge
    setStatus("The abbot can't be sure yet: too many ways on to check quickly. That costs you nothing."); draw(); return;
  }
  const note = helped ? "" : " (With his help, this raking can earn only a grey seal.)";
  const charge = () => { if (!helped) setHelped(true); };
  // a walk that has already stopped (no step on is allowed, or every cell is covered but it can't end there): saying why
  // is free, since it can't guide a search; where to step back to is help
  const S = stateOf(P), head = P[P.length - 1];
  const noMove = P.length < G.nSand && ![0, 1, 2, 3].some(d => { const w = G.nbr[head * 4 + d]; return w >= 0 && !S.vis[w] && check(S, w) === null; });
  if (a.state === "stuck" && (noMove || (P.length === G.nSand && finishCheck(S)))) {
    marks.cells = a.why.cells; marks.checker = !!a.why.checker;
    setStatus(`${a.why.text} (Saying why a walk has stopped is free; where to step back to counts as his help.)`, true);
    addAction("Where can I step back to?", () => {   // (this is help: it marks the footprint the garden can still be finished from)
      charge(); const n = P.length - a.back; el("actions").innerHTML = ""; marks.target = P[a.back - 1];
      setStatus(`Step back ${steps(n)} to the marked footprint: the garden can still be finished from there${a.proven ? "" : " (perhaps from a nearer one too)"}.${note}`, true);
      addAction(`Step back ${n}`, () => rewindTo(a.back, "the garden can be finished from here"), "Step back to the marked footprint; Restore brings the steps back"); draw();
    }, "Counts as the abbot's help: his seal for this raking can then only be grey");
    draw(); return;
  }
  charge();
  if (a.state === "par") {
    let j = P.length; while (j < a.R.length && ghosts.has(a.R[j])) j++;
    const more = sketch(a.R, j);
    setStatus(`Your walk can still be raked at par.${more ? " The abbot sketches the next groove, faintly." : ""}${note}`);
  } else if (a.state === "finish") {
    const n = P.length - a.lcp;
    marks.target = P[a.lcp - 1];
    setStatus(`Your walk can still be finished, but no longer at par: step back ${steps(n)} to the marked footprint for par.${note}`);
    addAction(`Step back ${n} for par`, () => rewindTo(a.lcp, "par is still possible from here"), "Step back to the marked footprint, from which par is still possible; Restore brings the steps back");
    addAction("Show a way on", () => { let j = P.length; while (j < a.completion.length && ghosts.has(a.completion[j])) j++; sketch(a.completion, j); setStatus("The abbot sketches a groove that still finishes the garden."); draw(); },
              "The abbot sketches, faintly, the next groove of a walk that still finishes the garden (not at par)");
  } else if (a.state === "stuck") {
    const n = P.length - a.back;
    marks.cells = a.why.cells; marks.checker = !!a.why.checker; marks.target = P[a.back - 1];
    setStatus(`${a.why.text} Step back ${steps(n)} to the marked footprint: the garden can still be finished from there${a.proven ? "" : " (perhaps from a nearer one too)"}.${note}`, true);
    addAction(`Step back ${n}`, () => rewindTo(a.back, "the garden can be finished from here"));
    if (a.lcp < a.back) addAction(`Step back ${P.length - a.lcp} for par`, () => rewindTo(a.lcp, "par is still possible from here"));
  }
  draw();
}

/* ---------- the demonstration: when a garden brings in a new rule, the monk first rakes it the calm way (walking in,
   then raking out), and then the garden is reset for the player. It plays once by itself; Watch replays it. A tap or a
   key skips it. It is free: it doesn't touch the seal. ---------- */
let demoRun = null, demoWatch = null;
function maybeDemo() {                     // once the garden is in view (on a long page it may not be yet)
  const g = GARDENS[idx], want = idx;
  if (!(g.demo && !path.length && !(progress[g.name] || {}).demoSeen && view === "puzzle")) return;
  const go = () => { demoRun = { timer: setTimeout(() => { demoRun = null; startDemo(); }, 600) }; };
  if (!("IntersectionObserver" in window)) { go(); return; }
  if (demoWatch) demoWatch.disconnect();
  demoWatch = new IntersectionObserver(es => {
    if (!es.some(e => e.isIntersecting)) return;
    demoWatch.disconnect(); demoWatch = null;
    if (idx === want && view === "puzzle" && mode === "draw" && !path.length && !demoRun) go();
  }, { threshold: 0.6 });
  demoWatch.observe(cv);
}
function startDemo() {
  const g = GARDENS[idx];
  if (view !== "puzzle" || !g.demo || (mode !== "draw" && mode !== "done")) return;
  stopDemo(false); stopAnim(); clearTimeout(rakeTimer); rakeTimer = null; cancelPoem();
  mode = "demo"; path = []; ghosts.clear(); clearAdvice(); obs = []; REV = null;
  el("seal").hidden = true; el("next").hidden = true;
  if (PZ) { PZ.feet = null; pzWalk(true); }
  const route = g.intended.map(([r, c]) => key(r, c));
  const ms = Math.max(90, Math.min(260, 3200 / route.length));
  demoRun = { timer: null };
  setStatus("Watch: the monk walks in, then rakes his way back out. (Tap to skip.)");
  const step = () => {
    if (!demoRun) return;
    path.push(route[path.length]); pzWalk(); update(); demoFinger(path[path.length - 1], ms); crunch(0.09);
    demoRun.timer = setTimeout(path.length < route.length ? step : () => { if (demoRun) { demoFinger(-1); startReveal(true); } }, path.length < route.length ? ms : 500);
  };
  step();
}
function demoFinger(k, ms) {               // the fingertip that drags the monk along
  const f = el("finger");
  if (f.anim) { f.anim.cancel(); f.anim = null; }
  if (k < 0 || (window.matchMedia && matchMedia("(prefers-reduced-motion: reduce)").matches)) { f.style.opacity = 0; f.style.transition = ""; return; }
  const [r, c] = rc(k), sc = cv.clientWidth / cv.width;
  f.hidden = false; f.style.transition = `transform ${ms}ms linear, opacity 200ms`; f.style.opacity = 0.9;
  f.style.transform = `translate(${cv.offsetLeft + (geom.ox + c * geom.cs + geom.cs / 2) * sc}px, ${cv.offsetTop + (geom.oy + r * geom.cs + geom.cs / 2) * sc}px)`;
}
function finishDemo() {                    // the raked result, a moment to look at it, then the garden is the player's
  mode = "demo";
  setStatus(`That is a calm way to rake it: ${turnsOf()} turns, which is par. Now you.`);
  demoRun = { timer: setTimeout(() => stopDemo(true), 2200) };
}
function stopDemo(reset) {
  if (demoWatch) { demoWatch.disconnect(); demoWatch = null; }
  if (demoRun && demoRun.timer) clearTimeout(demoRun.timer);
  const was = !!demoRun || mode === "demo" || (mode === "reveal" && REV && REV.demo);
  demoRun = null; demoFinger(-1);
  if (!reset || !was) return;
  stopAnim(); REV = null; mode = "draw"; path = [];
  if (PZ) { PZ.feet = null; pzWalk(true); }
  const g = GARDENS[idx], p = progress[g.name] || {}; p.demoSeen = true; progress[g.name] = p; saveProgress();
  el("finger").style.transition = ""; beckonKey = "";
  setStatus("Your turn: drag the monk in from the gate.");
  update();
}
el("watch").onclick = () => startDemo();

/* ---------- reveal: the monk walks backwards, the rake glides along the stroke ---------- */
let REV = null;
function easeRake(f, a = 0.1) {            // constant speed, with a gentle start and stop
  const s = f < a ? f * f / (2 * a) : f < 1 - a ? a / 2 + (f - a) : (1 - a) - (1 - f) * (1 - f) / (2 * a);
  return s / (1 - a);
}
function startReveal(demo = false) {
  mode = "reveal"; setStatus(demo ? "Now he sets down the rake and walks backwards to the gate, raking." : "The monk sets down his rake and walks backwards to the gate.");
  clearAdvice(); ghosts.clear();
  pzWalk();
  const reduce = window.matchMedia && matchMedia("(prefers-reduced-motion: reduce)").matches;
  const now = performance.now();
  REV = { st: strokeOf(path), s: 0, t0: now, last: now, dur: reduce ? 0 : Math.max(2500, Math.min(6500, 1800 + 75 * path.length)), grains: [], reduce, demo };
  if (!REV.dur) { finishAnimNow(); return; }
  scrapeStart();
  anim = requestAnimationFrame(revealTick);
}
function revealTick(now) {
  const f = Math.min(1, (now - REV.t0) / REV.dur), s = REV.st.L * easeRake(f);
  if (SND.scr) {                            // the scrape follows the rake: its speed, and whether the tines are in a bend
    const dt = Math.max(0.004, (now - REV.last) / 1000), p = strokePoint(REV.st, s);
    scrapeSet((s - REV.s) / dt, REV.st.segs[p.k].type === "arc" ? 1 : 0, p.k >= REV.st.segs.length - 1 ? 0.2 : 0.5);   // (out through the gate: the rake lifts)
    if (Math.floor(s) > Math.floor(REV.s)) crunch(0.045);   // and the monk's backward step, cell by cell
  }
  rakeAlong(PZ, REV.st, REV.s, s);
  REV.s = s;
  spawnGrains(now);
  fieldFlush(PZ);
  draw();
  if (f < 1) anim = requestAnimationFrame(revealTick); else { anim = null; finishReveal(); }
}
function spawnGrains(now) {                // a few grains flicked aside by the tines
  const R = REV, dt = Math.min(0.05, (now - R.last) / 1000); R.last = now;
  R.grains = R.grains.filter(g => now - g.t0 < g.life);
  if (R.reduce || dt <= 0) return;
  const p = strokePoint(R.st, R.s);
  if (p.k >= R.st.segs.length - 1) return;   // out through the gate
  const nx = -p.ty, ny = p.tx;
  for (let j = 0; j < 2; j++) {
    const side = Math.random() < 0.5 ? -1 : 1, off = side * (RAKE_W / 2) * (0.55 + 0.45 * Math.random()), sp = 0.5 + Math.random() * 0.6;
    R.grains.push({ x: p.x + nx * off, y: p.y + ny * off, vx: (nx * side * sp - p.tx * 0.25 + (Math.random() - 0.5) * 0.3), vy: (ny * side * sp - p.ty * 0.25 + (Math.random() - 0.5) * 0.3),
                    t0: now, life: 260 + Math.random() * 240, r: 0.012 + Math.random() * 0.014 });
  }
}
function finishAnimNow() {
  stopAnim();
  if (REV && REV.s < REV.st.L) { rakeAlong(PZ, REV.st, REV.s, REV.st.L); REV.s = REV.st.L; fieldFlush(PZ); }
  finishReveal();
}
function stopAnim() { if (anim) cancelAnimationFrame(anim); anim = null; scrapeStop(); }
function finishReveal() {
  scrapeStop();
  if (REV && REV.demo) { REV.grains = []; finishDemo(); return; }
  mode = "done"; doneAt = performance.now(); if (REV) REV.grains = [];
  const t = turnsOf(), g = GARDENS[idx], wasOpen = worldOpen(g.world + 1);
  const p = progress[g.name] || {};
  p.done = true; p.best = Math.min(p.best ?? 1e9, t); delete p.draft;
  const kind = t <= g.par ? (helped ? "grey" : "red") : null;
  if (kind === "red") p.sealed = true;
  if (kind === "grey") p.grey = true;
  rakedHelped = helped; helped = false; delete p.helped;
  if (g.daily) { p.path = path.slice(); p.turns = t; p.par = g.par; p.kind = kind; p.day = g.dateKey; }
  progress[g.name] = p; saveProgress();
  if (kind) {
    tok();
    const sl = showSeal(kind);
    sl.classList.remove("stamp"); void sl.offsetWidth; sl.classList.add("stamp");
  }
  setStatus(kind === "red" ? `Raked in ${t} turns: par. The abbot's seal.`
          : kind === "grey" ? `Raked in ${t} turns: par, with the abbot's help, so his seal is grey. Rake it again on your own for the vermilion one.`
          : `Raked in ${t} turns; par is ${g.par}: tap a cell to walk back to it and look for a calmer line.`);
  if (g.world < 7 && !wasOpen && worldOpen(g.world + 1))
    appendStatus(` ${TIMES[g.world + 1]} is open: ${GARDENS.filter(x => x.world === g.world + 1).length} new gardens.`);
  if (g.daily) { const n = dailyStreak(); appendStatus(` Today's record. Streak: ${n} day${n === 1 ? "" : "s"}.`); el("time").textContent = `Daily garden · streak ${n}`; dailyActions(g, p); }
  else shareAction(g, kind);
  if (g.role === "capstone" && !g.extra) { later(() => bell(0.55), 350); maybePiece("capstone"); } else maybePiece("garden");   // the close of a world: the temple bell
  el("next").hidden = false; renderNav(); update();
  startPoem();
}

/* ---------- the poem: whatever the raking (off par, or with the abbot's help), a poem is its reward. It is laid over the
   raked garden; about ten seconds later the garden and its poem fade, and the next garden fades in. A tap on the poem
   goes on at once; a tap beside it stays with this garden; "original" shows the Japanese and the source (and stays); Next garden
   or Enter goes on at once. ---------- */
const POEM_T = { delay: 900, hold: 10000, fade: 1200 };   // ms (a test hook can shorten them)
let poemRun = null, leaveTimer = null;
const reduceMotion = () => window.matchMedia && matchMedia("(prefers-reduced-motion: reduce)").matches;
function poemTitles(stay) {               // what a tap does, for the tooltips: on the card, and beside it
  const box = el("poem"), card = box.firstElementChild;
  box.title = stay ? "Tap to put the poem away" : "Tap beside the poem to stay with this garden";
  card.title = stay ? "Tap to put the poem away" : "Tap the poem to go on to the next garden";
}
function fillPoem(p, box) {
  poemTitles(!poemRun || poemRun.stay || isDaily());
  el("verse").textContent = p.en; el("poet").textContent = p.poet;
  el("ja").textContent = p.ja; el("ja").hidden = true;
  const src = el("src"), facts = [p.dates, p.src].filter(Boolean).join(" · ");   // (the poet's name in kanji was here: clutter, at Gwern's eye)
  src.textContent = facts + (facts && p.url ? " · " : "");
  if (p.url) { const a = document.createElement("a"); a.href = p.url; a.target = "_blank"; a.rel = "noopener"; a.textContent = "source"; src.appendChild(a); }
  src.hidden = true; el("orig").hidden = !p.ja;      // (an original poem would have no Japanese to show)
  fitPoem(box);
}
function fitPoem(box) {                      // over the sand only (the walls stay dark), in the largest type that fits there, clear of the seal
  const sc = cv.clientWidth / cv.width, wall = Math.round(geom.wall * sc);
  box.style.inset = `${wall}px`;
  const card = box.firstElementChild, room = box.clientHeight * 0.92, seal = el("seal");
  card.style.maxWidth = ""; card.style.justifySelf = ""; seal.classList.remove("under");
  const top = Math.max(13, Math.min(21, box.clientWidth / 19));
  const tryAll = () => {
    for (let fs = top; fs > 10.5; fs -= 0.5) { box.style.setProperty("--poem-fs", fs + "px"); if (card.scrollHeight <= room && !sealClash()) return true; }
    return false;
  };
  if (tryAll() || seal.hidden) return;
  const b = box.getBoundingClientRect(), pad = parseFloat(getComputedStyle(box).paddingLeft);   // centred, it reaches the seal:
  card.style.justifySelf = "start";                                                            // set it flush left, short of the seal
  card.style.maxWidth = `${Math.max(120, sealBox().left - b.left - pad - 10)}px`;
  if (tryAll()) return;
  card.style.maxWidth = ""; card.style.justifySelf = ""; tryAll();     // and if even that is too tall, the seal steps back under the text
  seal.classList.add("under");
}
function sealBox() {                         // where the seal rests (its layout box: not mid-stamp, when it is scaled up)
  const s = el("seal"), st = el("stage").getBoundingClientRect(), l = st.left + s.offsetLeft, t = st.top + s.offsetTop;
  return { left: l, top: t, right: l + s.offsetWidth, bottom: t + s.offsetHeight };
}
function sealClash() {                       // does any line of the poem run under the seal (drawn above it)?
  const s = el("seal"); if (s.hidden) return false;
  const r = sealBox(), m = 6;                // (the margin covers its tilt)
  for (const id of ["verse", "poet", "orig", "ja", "src"]) {
    const e = el(id); if (e.hidden) continue;
    const rg = document.createRange(); rg.selectNodeContents(e);
    for (const q of rg.getClientRects()) if (q.width && q.right > r.left - m && q.left < r.right + m && q.bottom > r.top - m && q.top < r.bottom + m) return true;
  }
  return false;
}
function startPoem() {
  cancelPoem();
  const g = GARDENS[idx];
  if (!g.poem) return;
  poemRun = { idx, stay: false, shown: false, timer: setTimeout(showPoem, POEM_T.delay) };
}
function showPoem() {
  if (!poemRun || idx !== poemRun.idx || mode !== "done" || view !== "puzzle") return;
  const box = el("poem");
  box.hidden = false; fillPoem(GARDENS[idx].poem, box);       // (shown before it is measured, still transparent)
  void box.offsetWidth; box.classList.add("on");
  poemRun.shown = true;
  if (isDaily()) { poemRun.stay = true; el("tick").hidden = true; appendStatus(" Tap the poem to put it away."); return; }   // the daily garden has no next garden
  const tick = el("tick"); tick.hidden = false; tick.style.setProperty("--hold", POEM_T.hold + "ms");
  tick.classList.remove("run"); void tick.offsetWidth; tick.classList.add("run");
  appendStatus(" The next garden follows: tap the poem to go on now, or beside it to stay.");
  poemRun.timer = setTimeout(tryAdvance, POEM_T.hold);
}
function hidePoem() { const box = el("poem"); box.classList.remove("on"); box.hidden = true; el("seal").classList.remove("under"); }
function cancelPoem() {                      // any change of garden or mode ends the poem and any pending change of garden
  if (poemRun) clearTimeout(poemRun.timer);
  poemRun = null; hidePoem();
  if (leaveTimer) { clearTimeout(leaveTimer); leaveTimer = null; }
  for (const e of [el("stage"), el("where")]) e.classList.remove("leaving");
}
function stayPoem(keep = false) {            // stay with this garden: no next garden; the poem can be read again
  if (!poemRun) return;
  clearTimeout(poemRun.timer); poemRun.stay = true; el("tick").hidden = true; poemTitles(true);
  if (!keep) hidePoem();
  setStatus(`${statusText().replace(" The next garden follows: tap the poem to go on now, or beside it to stay.", "")} Staying: tap a cell to walk back to it, or go on with Next garden.`);
  el("actions").innerHTML = "";
  if (mode === "done") shareAction(GARDENS[idx], el("seal").hidden ? null : el("seal").classList.contains("grey") ? "grey" : "red");
  addAction("Read the poem again", () => { const box = el("poem"); box.hidden = false; fillPoem(GARDENS[idx].poem, box); void box.offsetWidth; box.classList.add("on"); el("tick").hidden = true; }, "Lay the poem over the garden again");
}
function inView() { const r = cv.getBoundingClientRect(); return r.bottom > 0 && r.top < window.innerHeight; }
function tryAdvance() {
  if (!poemRun || poemRun.stay || mode !== "done" || idx !== poemRun.idx || view !== "puzzle") return;
  if (document.hidden || !inView()) { poemRun.timer = setTimeout(tryAdvance, 1000); return; }   // never while nobody is looking
  advance(false);
}
function advance(fast) {                     // the garden and its poem fade out, and the next garden fades in
  if (shared) { leaveShared(); return; }
  const nx = nextGarden(idx);
  const go = () => { leaveTimer = null; load(nx); maybeDemo(); if (!reduceMotion()) arrive(); };
  if (poemRun) clearTimeout(poemRun.timer);
  if (reduceMotion()) { go(); return; }
  const f = fast ? 450 : POEM_T.fade;
  for (const e of [el("stage"), el("where")]) { e.style.setProperty("--fade", f + "ms"); e.classList.remove("arriving"); e.classList.add("leaving"); }
  leaveTimer = setTimeout(go, f);
}
function arrive() {
  for (const e of [el("stage"), el("where")]) {
    e.style.setProperty("--fade", POEM_T.fade + "ms"); e.classList.remove("leaving", "arriving"); void e.offsetWidth; e.classList.add("arriving");
    setTimeout(() => e.classList.remove("arriving"), POEM_T.fade + 50);
  }
}
/* A tap on the poem itself goes on to the next garden at once (what waiting does); a tap beside it, on the sand around the
   card or on the walls, puts the poem away and stays with this garden. (A player's note: interacting with the poem should
   go forth, dismissing it should go back.) Once the poem is staying (the daily garden, or Read the poem again), any tap puts it away. */
el("poem").addEventListener("click", ev => {
  if (ev.target.closest("#orig, a")) return;
  if (poemRun && !poemRun.stay) { if (ev.target.closest(".card")) advance(true); else stayPoem(false); } else hidePoem();
});
el("orig").onclick = ev => {
  ev.stopPropagation();
  el("ja").hidden = false; el("src").hidden = false; el("orig").hidden = true;
  if (poemRun && !poemRun.stay) stayPoem(true); else el("tick").hidden = true;
  fitPoem(el("poem"));
};

/* ---------- sharing: a raking, or a player's whole progress, as a code in the URL's hash ----------
   A walk is written as a number in a mixed radix: at each step the digit is which of the legal moves (by the garden's
   rules, in the order N E S W) was taken and the radix how many there were; so a raking of a 7×7 garden is some 20–50
   bits, and no code decodes to a walk that breaks a rule. A garden is named in a code by a 32-bit hash of its layout and
   rules, never by its place in the campaign: a link outlives a reordering, and a garden changed since is recognized as
   another, not mistaken for the one meant (Astra's point: an 8-bit check would collide once in 256, and a seal must never
   land on a different puzzle). The codes are base64url. #r= is a raking: version 4 bits; daily 1; day number 16 (offset
   32768) if daily; layout 32; raked with the abbot's help 1; the walk's bits 8, then the walk. #p= is progress: version 4;
   raked gardens 8, each layout 32, kind 2 (1 raked, 2 sealed in grey, 3 sealed), best turns over par 6; daily records 9,
   each day number 16, kind 2, turns over par 5, with a walk 1 (the day's raking: layout 32, bits 8, walk); an unfinished
   walk 1 (layout 32, steps 8, with help 1, bits 8, walk). */
const B64 = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_";
const CODE_V = 1;
function bitWriter() {
  const bits = [];
  return { put(v, n) { v = BigInt(v); for (let i = n - 1; i >= 0; i--) bits.push(Number((v >> BigInt(i)) & 1n)); return this; },
           str() { let s = ""; for (let i = 0; i < bits.length; i += 6) { let v = 0; for (let j = 0; j < 6; j++) v = v * 2 + (bits[i + j] || 0); s += B64[v]; } return s; } };
}
function bitReader(s) {
  const bits = []; let p = 0;
  for (const ch of s) { const v = B64.indexOf(ch); if (v < 0) throw new Error("it is not a Samon code"); for (let j = 5; j >= 0; j--) bits.push((v >> j) & 1); }
  return { big(n) { if (p + n > bits.length) throw new Error("it is cut short"); let v = 0n; for (let i = 0; i < n; i++) v = (v << 1n) | BigInt(bits[p++]); return v; },
           num(n) { return Number(this.big(n)); } };
}
const bitLen = n => n ? n.toString(2).length : 0;
function legalMoves(S) { const v = S.P[S.len - 1], out = []; for (let d = 0; d < 4; d++) { const w = G.nbr[v * 4 + d]; if (w >= 0 && check(S, w) === null) out.push(w); } return out; }
function walkEncode(P) {                    // the walk P (from the gate of the garden G) as a number; null if it breaks a rule
  if (!P.length || P[0] !== G.gateCell) return null;
  const S = newState(); push(S, P[0]);
  let n = 0n, m = 1n;
  for (let i = 1; i < P.length; i++) {
    const opts = legalMoves(S), j = opts.indexOf(P[i]);
    if (j < 0) return null;
    n += BigInt(j) * m; m *= BigInt(opts.length); push(S, P[i]);
  }
  return n;
}
function walkDecode(n, steps) {             // the walk of `steps` steps whose number is n; null if there is none
  const S = newState(); push(S, G.gateCell); const P = [G.gateCell];
  for (let i = 0; i < steps; i++) {
    const opts = legalMoves(S); if (!opts.length) return null;
    const r = BigInt(opts.length), j = Number(n % r); n /= r;
    push(S, opts[j]); P.push(opts[j]);
  }
  return n === 0n ? P : null;
}
function withGarden(g, fn) { const keep = G; G = buildGarden(g); try { return fn(); } finally { G = keep; } }   // run fn with g built as the garden
function layoutHash(g) {                    // a garden's identity in a code: 32 bits (FNV-1a) of its layout and rules, in a fixed order
  const kv = o => Object.entries(o || {}).sort().map(([k, v]) => `${k}:${v}`).join(";");
  const s = [g.rows.join("/"), g.gate.join(","), (g.end || []).join(","), g.end_side || "", g.wide ? "w" : "", kv(g.marks), kv(g.clues)].join("|");
  let h = 2166136261; for (let i = 0; i < s.length; i++) { h ^= s.charCodeAt(i); h = Math.imul(h, 16777619) >>> 0; }
  return h >>> 0;
}
let LAYOUTS = null;                         // layout hash → campaign index (built once; a test hook that reorders the campaign resets it)
function byLayout(h) {
  if (!LAYOUTS) { LAYOUTS = new Map(); for (let i = 0; i < NCAMP; i++) LAYOUTS.set(layoutHash(GARDENS[i]), i); }
  const i = LAYOUTS.get(h); return i === undefined ? -1 : i;
}
function putWalk(w, P) { const n = walkEncode(P); if (n === null) return false; w.put(bitLen(n), 8).put(n, bitLen(n)); return true; }
function readWalk(r, steps) { const P = walkDecode(r.big(r.num(8)), steps); if (!P) throw new Error("its walk does not fit the garden"); return P; }
function skipWalk(r) { r.big(r.num(8)); }   // (a walk of a garden not in this edition: read past it)
function dailyFor(n) {                      // the daily garden of day number n (days since 2026-01-01), under its date
  const i = ((n % DAILY.length) + DAILY.length) % DAILY.length, d = new Date(Date.UTC(2026, 0, 1) + n * 86400000);
  const k = `${d.getUTCFullYear()}-${pad2(d.getUTCMonth() + 1)}-${pad2(d.getUTCDate())}`;
  return Object.assign({}, DAILY[i], { name: "daily:" + k, daily: true, dateKey: k, dayNo: n, poem: GARDENS[((n % NCAMP) + NCAMP) % NCAMP].poem });
}
function rakingCode(P = path, grey = false) {   // the current garden's complete raking P as a #r= code; null if P is not one
  const g = GARDENS[idx];
  if (P.length !== G.nSand || finishCheck(stateOf(P))) return null;
  const w = bitWriter().put(CODE_V, 4).put(g.daily ? 1 : 0, 1);
  if (g.daily) w.put(g.dayNo + 32768, 16);
  w.put(layoutHash(g), 32).put(grey ? 1 : 0, 1);
  return putWalk(w, P) ? w.str() : null;
}
function readRaking(code) {                 // a #r= code: the garden's index (a daily one is put in the daily slot), whether helped, and the walk's number
  const r = bitReader(code);
  if (r.num(4) !== CODE_V) throw new Error("it was made by another edition of the game");
  let i;
  if (r.num(1)) {
    if (!DAILY.length) throw new Error("this edition has no daily gardens");
    const g = dailyFor(r.num(16) - 32768);
    if (r.num(32) !== layoutHash(g)) throw new Error("the garden of that day has changed since");
    GARDENS[NCAMP] = g; i = NCAMP;
  } else { i = byLayout(r.num(32)); if (i < 0) throw new Error("its garden is not in this edition, or has changed since"); }
  const grey = r.num(1) === 1, n = r.big(r.num(8));
  return { i, grey, n };
}
function showShared(h) {
  load(h.i);
  const g = GARDENS[idx], P = walkDecode(h.n, G.nSand - 1);
  if (!P || finishCheck(stateOf(P))) throw new Error("its walk does not fit the garden");
  cancelPoem(); clearAdvice();
  path = P; mode = "done"; doneAt = 0; rakedHelped = false; cut = null;
  const st = strokeOf(path); rakeAlong(PZ, st, 0, st.L); PZ.dirty = null; PZ.feet = null; fieldLight(PZ);
  const t = turnsOf(), kind = t <= g.par ? (h.grey ? "grey" : "red") : null;
  shared = { turns: t, par: g.par, kind };
  showSeal(kind);
  el("next").hidden = true;
  const today = g.daily && g.dateKey === dateKey(todayDate()), mine = g.daily && (progress[g.name] || {}).done;
  setStatus(`A raking shared with you${g.daily ? `, of the daily garden of ${dateLabel(g.dateKey)}` : ""}: ${t} turn${t === 1 ? "" : "s"}, par ${g.par}${kind === "red" ? ", the abbot's seal" : kind === "grey" ? ", his seal in grey (raked with his help)" : ""}.`
    + (g.daily && !today ? " Today's garden is another." : mine ? ` Your own raking today: ${progress[g.name].turns} turns.` : " Rake it yourself from the gate: a tap smooths the sand."));
  addAction(g.daily && !today ? "Today's garden" : mine ? "Your own raking" : "Rake it yourself", leaveShared,
            g.daily && !today ? "This was an earlier day's garden; today's is another, and yours to rake" : mine ? "Smooth this away and see your own raking of today's garden" : "Smooth this raking away and rake the garden yourself, from the gate");
  addAction("Share it on", () => share(g, path, kind), "Pass this raking on as it came: drawn in lines, with a link that shows it");
  update();
}
function leaveShared() {                    // the shared raking is put away, and the garden is the viewer's own (today's, after a past daily garden)
  if (!shared) return;
  const g = GARDENS[idx]; shared = null; dropHash();
  if (g.daily && g.dateKey !== dateKey(todayDate())) { pickDaily(); return; }
  load(idx);
  if (!path.length && mode === "draw") setStatus("Now you: drag the monk in from the gate.");
  maybeDemo();
}
function dropHash() { if (/^#([rp]=|daily$)/.test(location.hash)) history.replaceState(null, "", location.pathname + location.search); }
function progressCode() {                   // all of this player's progress as a #p= code
  const w = bitWriter().put(CODE_V, 4), raked = [];
  for (let i = 0; i < NCAMP; i++) { const g = GARDENS[i], p = progress[g.name] || {}, kind = p.sealed ? 3 : p.grey ? 2 : p.done ? 1 : 0; if (kind) raked.push([g, p, kind]); }
  w.put(Math.min(255, raked.length), 8);
  for (const [g, p, kind] of raked.slice(0, 255)) w.put(layoutHash(g), 32).put(kind, 2).put(Math.max(0, Math.min(63, (p.best ?? g.par) - g.par)), 6);
  const days = DAILY.length ? Object.keys(progress).filter(k => /^daily:\d{4}-\d\d-\d\d$/.test(k) && progress[k].done).sort().slice(-511) : [];
  w.put(days.length, 9);
  for (const k of days) {
    const [y, m, d] = k.slice(6).split("-").map(Number), p = progress[k], n = dayNumber(new Date(y, m - 1, d)), g = dailyFor(n);
    w.put(n + 32768, 16).put(p.kind === "red" ? 3 : p.kind === "grey" ? 2 : 1, 2).put(Math.max(0, Math.min(31, (p.turns ?? g.par) - g.par)), 5);
    const walk = Array.isArray(p.path) && withGarden(g, () => p.path.length === G.nSand && walkEncode(p.path) !== null);
    w.put(walk ? 1 : 0, 1); if (walk) { w.put(layoutHash(g), 32); withGarden(g, () => putWalk(w, p.path)); }
  }
  let oi = GARDENS[idx] && !GARDENS[idx].daily ? idx : -1;   // the garden open (or last open) and its unfinished walk
  if (oi < 0) try { const i = parseInt(localStorage.getItem("samon-last")); if (GARDENS[i] && !GARDENS[i].daily) oi = i; } catch (e) {}
  const p = oi >= 0 ? progress[GARDENS[oi].name] || {} : {}, P = oi >= 0 ? (oi === idx && mode === "draw" ? path : p.draft) : null;
  const ok = Array.isArray(P) && P.length > 1 && withGarden(GARDENS[oi], () => walkEncode(P) !== null);
  w.put(ok ? 1 : 0, 1);
  if (ok) { w.put(layoutHash(GARDENS[oi]), 32).put(P.length - 1, 8).put(oi === idx ? (helped ? 1 : 0) : (p.helped ? 1 : 0), 1); withGarden(GARDENS[oi], () => putWalk(w, P)); }
  return w.str();
}
function readProgress(code) {               // a #p= code → { gardens: [{ i, kind, over }], days: [{ g, kind, over, P }], draft: { i, P, helped } | null, lost }
  const r = bitReader(code);                 // (lost: entries of gardens not in this edition, or changed since, which are left out rather than guessed at)
  if (r.num(4) !== CODE_V) throw new Error("it was made by another edition of the game");
  const nr = r.num(8), gardens = []; let lost = 0;
  for (let k = 0; k < nr; k++) { const i = byLayout(r.num(32)), kind = r.num(2), over = r.num(6); if (i >= 0 && kind) gardens.push({ i, kind, over }); else lost++; }
  const nd = r.num(9), days = [];
  for (let k = 0; k < nd; k++) {
    const n = r.num(16) - 32768, kind = r.num(2), over = r.num(5), g = DAILY.length ? dailyFor(n) : null;
    let P = null;
    if (r.num(1)) { const h = r.num(32); if (g && h === layoutHash(g)) P = withGarden(g, () => readWalk(r, G.nSand - 1)); else skipWalk(r); }
    if (g) days.push({ g, kind, over, P }); else lost++;
  }
  let draft = null;
  if (r.num(1)) {
    const i = byLayout(r.num(32)), steps = r.num(8), helped = r.num(1) === 1;
    if (i >= 0) draft = { i, helped, P: withGarden(GARDENS[i], () => readWalk(r, steps)) }; else { skipWalk(r); lost++; }
  }
  return { gardens, days, draft, lost };
}
function importProgress(h) {                // the progress of a #p= code joins this player's: whichever is better stands
  for (const e of h.gardens) {
    const g = GARDENS[e.i]; if (!g || g.daily || !e.kind) continue;
    const p = progress[g.name] || {};
    p.done = true; p.best = Math.min(p.best ?? 1e9, g.par + e.over);
    if (e.kind === 3) p.sealed = true; else if (e.kind === 2) p.grey = true;
    progress[g.name] = p;
  }
  for (const d of h.days) {
    const p = progress[d.g.name] || {};
    if (p.done) continue;                   // the first raking of a day is its record
    Object.assign(p, { done: true, turns: d.g.par + d.over, par: d.g.par, kind: d.kind === 3 ? "red" : d.kind === 2 ? "grey" : null, day: d.g.dateKey });
    if (d.P) p.path = d.P;
    progress[d.g.name] = p;
  }
  if (h.draft) {
    const g = GARDENS[h.draft.i], p = progress[g.name] || {};
    if (!p.done && !(Array.isArray(p.draft) && p.draft.length > 1)) {
      p.draft = h.draft.P; if (h.draft.helped) p.helped = true; progress[g.name] = p;
      try { localStorage.setItem("samon-last", String(h.draft.i)); } catch (e) {}
    }
  }
  saveProgress();
}
function offerProgress(h) {
  const raked = h.gardens.filter(e => e.kind).length, sealed = h.gardens.filter(e => e.kind === 3).length, days = h.days.length;
  const sum = `${raked} of ${NCAMP} gardens raked, ${sealed} sealed${days ? `, ${days} daily garden${days === 1 ? "" : "s"}` : ""}${h.draft ? ", and an unfinished walk" : ""}`;
  setStatus(`This link carries a player's progress: ${sum}. Take it over here? It joins what is here, and whichever is better stands.${h.lost ? ` (${h.lost} of its entries belong to gardens not in this edition, or changed since, and are left out.)` : ""}`);
  el("actions").innerHTML = "";
  addAction("Take it over", () => { importProgress(h); dropHash(); load(lastGarden()); setStatus(`Taken over: ${sum}.`); maybeDemo(); }, "Join the link's progress with what is here: every garden keeps its better state, and nothing here is lost");
  addAction("Leave it", () => { dropHash(); clearAdvice(); setStatus("Left where it was. This is your own garden."); }, "Keep only what is here; the link is dropped");
}
function applyHash() {                      // a #r= or #p= code in the URL: show the raking, or offer the progress; true if the hash was one
  if (location.hash === "#daily") { if (DAILY.length) { pickDaily(); dropHash(); } return DAILY.length > 0; }
  const m = /^#([rp])=(.*)$/.exec(location.hash);
  if (!m) return false;
  try {
    if (m[1] === "r") showShared(readRaking(m[2]));
    else { const h = readProgress(m[2]); load(lastGarden()); offerProgress(h); }
  } catch (e) { if (!G) load(lastGarden()); setStatus(`The link's code can't be read: ${e.message}.`, true); dropHash(); }
  showStage();
  return true;
}
window.addEventListener("hashchange", () => { if (/^#([rp]=|daily$)/.test(location.hash)) applyHash(); });
/* The share: the raking drawn in box-drawing characters (light lines for the stroke, heavy for the walls, blocks for the
   stones), with a link that opens it; through the share sheet where there is one, else to the clipboard. */
function gardenLabel(g) {
  if (g.daily) return `daily garden ${g.dateKey}`;
  const inPart = GARDENS.filter(x => x.world === g.world && !!x.extra === !!g.extra);
  return `${TIMES[g.world]}${g.extra ? ", more" : ""} ${inPart.indexOf(g) + 1} · ${g.name}`;
}
function gridText(P = path) {
  const CH = { 0: "░", 1: "╵", 2: "╶", 4: "╷", 8: "╴", 5: "│", 10: "─", 3: "└", 9: "┘", 6: "┌", 12: "┐" };   // bits: 1 N, 2 E, 4 S, 8 W
  const conn = new Array(G.N).fill(0);
  for (let i = 0; i < P.length; i++) {
    conn[P[i]] |= 1 << (i === 0 ? G.gateDir : dirTo(P[i], P[i - 1]));
    if (i + 1 < P.length) conn[P[i]] |= 1 << dirTo(P[i], P[i + 1]);
  }
  if (G.endCell >= 0 && G.end_side && P.length === G.nSand && P[P.length - 1] === G.endCell) conn[G.endCell] |= 1 << DIX[G.end_side];
  const gap = (side, r, c) => (G.gate[2] === side && (side === "N" || side === "S" ? G.gate[1] === c : G.gate[0] === r))
                           || (!!G.end && G.end_side === side && (side === "N" || side === "S" ? G.end[1] === c : G.end[0] === r));
  const across = side => [...Array(G.W)].map((_, c) => gap(side, -1, c) ? (side === "N" ? "┯" : "┷") : "━").join("");
  const lines = ["┏" + across("N") + "┓"];
  for (let r = 0; r < G.H; r++) {
    let s = gap("W", r, -1) ? "┠" : "┃";
    for (let c = 0; c < G.W; c++) { const v = key(r, c); s += G.sand[v] ? (CH[conn[v]] || "┼") : G.rows[r][c] === "R" ? "▒" : "█"; }
    lines.push(s + (gap("E", r, -1) ? "┨" : "┃"));
  }
  lines.push("┗" + across("S") + "┛");
  return lines.join("\n");
}
function shareText(g, P, kind) {            // the daily garden's share shows its shape and the score, not the stroke: today everyone is still raking it
  const t = turnsOf(P), seal = kind === "red" ? " · 静" : kind === "grey" ? " · 静 (with the abbot's help)" : "", streak = g.daily ? dailyStreak() : 0;
  return `Samon · ${gardenLabel(g)} · ${t} turn${t === 1 ? "" : "s"}, par ${g.par}${seal}${streak ? ` · streak ${streak}` : ""}\n${gridText(g.daily ? [] : P)}`;
}
const pageURL = () => window.top === window ? location.href.replace(/#.*$/, "") : "";   // (framed, as the artifact is, the page's own address is not one to pass on)
async function shareOut(text, url, title, clip) {   // the share sheet where there is one; else the clipboard (clip, or the text with the link); else the text, to copy by hand
  if (navigator.share) { try { await navigator.share(url ? { title, text, url } : { title, text }); return "shared"; } catch (e) { if (e && e.name === "AbortError") return "cancelled"; } }
  try { await navigator.clipboard.writeText(clip ?? (url ? `${text}\n${url}` : text)); return "copied"; } catch (e) { return "shown"; }
}
function showShareText(t) {
  const ta = document.createElement("textarea"); ta.className = "share"; ta.readOnly = true; ta.value = t; ta.rows = Math.min(16, t.split("\n").length + 2);
  ta.setAttribute("aria-label", "Text to share"); el("actions").appendChild(ta); ta.focus(); ta.select();
}
async function share(g, P, kind) {
  const code = g.daily ? "daily" : rakingCode(P, kind === "grey"), text = shareText(g, P, kind), base = pageURL();
  const url = code && base ? `${base}#${g.daily ? "daily" : "r=" + code}` : "";
  const how = await shareOut(text, url, "Samon");
  const what = g.daily ? "today's garden and your score" : "the raking drawn in lines", opens = g.daily ? "opens today's garden" : "opens it";
  if (how === "copied") setStatus(`Copied: ${what}${url ? `, with a link that ${opens}` : ""}. Paste it anywhere.`);
  else if (how === "shown") { setStatus(`Copy this to share it${url ? `; the link ${opens}` : ""}.`); showShareText(url ? `${text}\n${url}` : text); }
}
async function carryProgress() {
  const base = pageURL(), url = `${base || location.href.replace(/#.*$/, "")}#p=${progressCode()}`;
  const how = await shareOut("Samon: my progress, to take over on another device.", url, "Samon progress", url);
  const note = base ? "" : " (Shown inside another page, as here, the link may not be the page's public address: paste the code after # onto that.)";
  if (how === "copied") setStatus(`Copied a link that carries your progress: open it on the other device and choose Take it over.${note}`);
  else if (how === "shown") { setStatus(`Open this link on the other device and choose Take it over.${note}`); showShareText(url); }
}
function shareAction(g, kind) {
  addAction(g.daily ? "Share today's score" : "Share this raking", () => share(g, path.slice(), kind),
            `${navigator.share ? "Through the share sheet" : "To the clipboard"}: ` + (g.daily ? "your turns against par, the garden's shape without your stroke (everyone is still raking it today), and a link that opens today's garden"
                                                                                : "the raking drawn in lines, and a link that shows it in the garden (shown, not credited: whoever opens it rakes it for themselves)"));
}


/* ---------- assets carried in the page. A recording (or the single-file edition's .zip) is written into the page as a
   data: URL inside an inert <script type="text/plain">. Gwern's gwtar tool (deconstruct_singlefile.php, gwern.net/gwtar)
   pulls every data: URL out of such a page into an appended tar and leaves its path in its place, and its loader (gwtar.js)
   fetches members by HTTP range request on demand; so an asset's text is either the data: URL itself, a path the gwtar
   loader can serve (resolved through its globals, in either of its two modes), or a plain path beside an extracted page. */
const samonAsset = {
  gwtar() { return typeof assetInfoFromResourceURLString === "function" && typeof assets === "object"; },
  bytes(ref) {                               // Promise of a Uint8Array
    if (ref.startsWith("data:")) { const s = atob(ref.slice(ref.indexOf(",") + 1)), b = new Uint8Array(s.length); for (let i = 0; i < s.length; i++) b[i] = s.charCodeAt(i); return Promise.resolve(b); }
    const info = this.gwtar() ? assetInfoFromResourceURLString(ref) : null;
    if (!info) return fetch(ref).then(r => { if (!r.ok) throw new Error(r.status); return r.arrayBuffer(); }).then(a => new Uint8Array(a));
    return new Promise((res, rej) => {
      const got = () => { if (info.data) { clearInterval(t); res(new Uint8Array(info.data)); return true; } return false; };
      if (got()) return;
      const t = setInterval(got, 250);
      try {   // range mode: ask for the member; full-response mode: mark it wanted and let the stream fill it in
        if (typeof loadedResponseDataLength === "undefined" || loadedResponseDataLength === 0) getResources([info], { onFailure: () => { clearInterval(t); rej(new Error("gwtar")); } });
        else markAssetWaiting(info);
      } catch (e) { clearInterval(t); rej(e); }
    });
  },
  url(ref, type) {                           // Promise of a URL a media element or a link can use
    if (ref.startsWith("data:")) return Promise.resolve(ref);
    if (!this.gwtar() || !assetInfoFromResourceURLString(ref)) return Promise.resolve(ref);
    return this.bytes(ref).then(b => URL.createObjectURL(new Blob([b], { type: type || assetInfoFromResourceURLString(ref)["content-type"] || "" })));
  },
  text(el) { return (el.textContent || "").trim().replace(/^["']|["']$/g, ""); }   // (the gwtar tool quotes the path it writes)
};

/* ---------- sound: synthesized in the page (a bamboo pipe that tips and drips now and then, a temple bell at the close
   of a world, the seal's wooden tok), and shakuhachi from the recordings carried in the page (script.piece, see above).
   Nothing sounds until the player has touched the page (browsers require it), and Sound off in the list silences all. ---------- */
const SND = { on: true, vol: 1, haptics: true, ctx: null, out: null, master: null, basin: null, meter: null, wind: null, scr: null, timers: new Set(), tracks: null, cur: null, lastPiece: 0, played: [], offline: null };
try {
  SND.on = localStorage.getItem("samon-sound") !== "0"; SND.haptics = localStorage.getItem("samon-haptics") !== "0";
  const v = parseFloat(localStorage.getItem("samon-sound-vol")); if (v >= 0 && v <= 1) SND.vol = v;
} catch (e) {}
function later(fn, ms) {                     // a delay in wall-clock time; in an offline render (the test hook), in rendered time
  if (SND.offline) { offlineAt(SND.ctx.currentTime + ms / 1000, fn); return 0; }
  const id = setTimeout(() => { SND.timers.delete(id); fn(); }, ms); SND.timers.add(id); return id;
}
function offlineAt(when, fn) {               // run fn with the offline context suspended at `when` (one suspend per render quantum), then go on rendering
  const ctx = SND.ctx, q = 128 / ctx.sampleRate; let k = Math.max(1, Math.ceil(when / q - 1e-6));   // (never at frame 0, which cannot be suspended)
  while (SND.offline.has(k)) k++; SND.offline.add(k);
  ctx.suspend(k * q).then(() => { fn(); ctx.resume(); }).catch(() => {});
}
function audio() {                           // the context, made on a gesture; null when sound is off or unavailable
  if (!SND.on) return null;
  if (!SND.ctx) {
    const C = window.AudioContext || window.webkitAudioContext; if (!C) return null;
    const ctx = SND.ctx = new C();
    SND.master = ctx.createGain(); SND.master.gain.value = SND.vol; SND.master.connect(ctx.destination);   // the player's volume
    SND.out = ctx.createGain(); SND.out.gain.value = 0.8; SND.out.connect(SND.master);
    SND.meter = ctx.createAnalyser(); SND.meter.fftSize = 2048; SND.out.connect(SND.meter);   // (tests read the output level from it)
    const d = ctx.createDelay(0.3), fb = ctx.createGain(), lp = ctx.createBiquadFilter(), wet = ctx.createGain();   // the stone basin's echo
    d.delayTime.value = 0.057; fb.gain.value = 0.4; lp.type = "lowpass"; lp.frequency.value = 1600; wet.gain.value = 0.45;
    d.connect(lp); lp.connect(fb); fb.connect(d); lp.connect(wet); wet.connect(SND.out);
    SND.basin = d;
    ctx.onstatechange = () => { if (SND.ctx === ctx) renderMute(); };
    windStart(); ambientStart(); loadPieces();
    let seen = true; try { seen = !!localStorage.getItem("samon-sound-hint"); localStorage.setItem("samon-sound-hint", "1"); } catch (e) {}
    if (!seen) {                               // the first time sound starts, a word beside the speaker, briefly
      const h = document.createElement("span"); h.className = "mute-hint"; h.id = "muteHint"; h.textContent = "[Sound on · M mutes]"; el("mute").insertAdjacentElement("beforebegin", h);   // (beside the speaker, in the header's right-hand group)
      setTimeout(() => h.remove(), 7000);
    }
  }
  if (!SND.offline) {
    SND.gesture = performance.now(); later(renderMute, 1600);
    if (SND.ctx.state === "suspended") SND.ctx.resume().catch(() => {});   // (an offline render resumes itself once the sound is scheduled)
  }
  return SND.ctx;
}
function noiseBuf(ctx, secs) { const b = ctx.createBuffer(1, Math.ceil(ctx.sampleRate * secs), ctx.sampleRate), x = b.getChannelData(0); for (let i = 0; i < x.length; i++) x[i] = Math.random() * 2 - 1; return b; }
function partial(ctx, dest, f, amp, dec, t, type = "sine") {   // one decaying sine
  const o = ctx.createOscillator(), g = ctx.createGain(); o.type = type; o.frequency.value = f;
  g.gain.setValueAtTime(0, t); g.gain.linearRampToValueAtTime(amp, t + 0.004); g.gain.exponentialRampToValueAtTime(0.0004, t + dec);
  o.connect(g); g.connect(dest); o.start(t); o.stop(t + dec + 0.05);
}
function click(ctx, dest, amp, dec, t, cutoff) {               // a short burst of noise
  const s = ctx.createBufferSource(); s.buffer = noiseBuf(ctx, dec + 0.01); const lp = ctx.createBiquadFilter(); lp.type = "lowpass"; lp.frequency.value = cutoff;
  const g = ctx.createGain(); g.gain.setValueAtTime(amp, t); g.gain.exponentialRampToValueAtTime(0.0005, t + dec);
  s.connect(lp); lp.connect(g); g.connect(dest); s.start(t); s.stop(t + dec + 0.02);
}
function drip(vol = 0.2) {                   // one drop into the basin: a short rising sine, and the basin's echo
  const ctx = audio(); if (!ctx) return; const t = ctx.currentTime + 0.01;
  const o = ctx.createOscillator(), g = ctx.createGain(); o.type = "sine";
  const f0 = 450 + Math.random() * 450, dur = 0.08 + Math.random() * 0.09;
  o.frequency.setValueAtTime(f0, t); o.frequency.exponentialRampToValueAtTime(f0 * (2.1 + Math.random() * 0.9), t + 0.035);
  g.gain.setValueAtTime(0, t); g.gain.linearRampToValueAtTime(vol, t + 0.003); g.gain.exponentialRampToValueAtTime(0.0004, t + dur);
  o.connect(g); g.connect(SND.out); g.connect(SND.basin); o.start(t); o.stop(t + dur + 0.05);
}
function tock(vol = 0.4) {                   // the bamboo pipe tipping onto its stone, then the water it spills
  const ctx = audio(); if (!ctx) return; const t = ctx.currentTime + 0.01, k = 0.94 + Math.random() * 0.12;
  click(ctx, SND.out, vol * 0.8, 0.012, t, 2600);
  for (const [f, a, dec] of [[165, 0.9, 0.11], [610, 0.55, 0.16], [1230, 0.35, 0.12], [2210, 0.16, 0.07]]) partial(ctx, SND.out, f * k, vol * a, dec, t);
  const n = 2 + Math.floor(Math.random() * 3);
  for (let i = 0; i < n; i++) later(() => drip(0.1 + Math.random() * 0.1), 220 + i * (150 + Math.random() * 200));
}
function tok(vol = 0.35) {                   // the seal: a dull wooden knock
  const ctx = audio(); if (!ctx) return; const t = ctx.currentTime + 0.01;
  click(ctx, SND.out, vol, 0.01, t, 1800); partial(ctx, SND.out, 330, vol * 0.8, 0.07, t); partial(ctx, SND.out, 1050, vol * 0.3, 0.035, t);
}
function bell(vol = 0.5, f0 = 108) {         // a bonshō: inharmonic partials, the low ones beating slowly and lasting long, under the log's strike
  const ctx = audio(); if (!ctx) return; const t = ctx.currentTime + 0.02;
  const mix = ctx.createGain(); mix.gain.value = vol * 0.32; mix.connect(SND.out);
  for (const [r, a, dec] of [[1, 1, 17], [1.004, 0.6, 15], [2.0, 0.55, 9], [2.012, 0.3, 8], [2.4, 0.35, 6], [3.01, 0.3, 5], [3.5, 0.2, 3.5], [4.2, 0.15, 2.5], [5.4, 0.1, 1.6], [6.8, 0.07, 1.0], [8.3, 0.05, 0.6]])
    partial(ctx, mix, f0 * r, a, dec, t);
  click(ctx, SND.out, vol * 0.5, 0.045, t, 900);
}
/* the rake's scrape: gravel under the tines, as grains. A looping buffer of crackle (random short bursts over a hiss) runs
   through a bandpass; the stroke's speed sets its loudness, its brightness and its density (the buffer's playback rate),
   and a bend adds the rasp of tines dragged sideways. Nothing is recorded. */
function gravelBuf(ctx, secs) {
  const n = Math.ceil(ctx.sampleRate * secs), b = ctx.createBuffer(1, n, ctx.sampleRate), x = b.getChannelData(0);
  for (let i = 0; i < n; i++) x[i] = (Math.random() * 2 - 1) * 0.12;                 // the hiss of fine sand
  const grains = Math.floor(secs * 900);                                              // and the grains
  for (let k = 0; k < grains; k++) {
    const at = Math.floor(Math.random() * n), len = 8 + Math.floor(Math.random() * 60), a = 0.25 + Math.random() * 0.75, ph = Math.random() * 6.283;
    for (let i = 0; i < len && at + i < n; i++) x[at + i] += a * Math.exp(-i / (len * 0.35)) * Math.sin(ph + i * (0.9 + Math.random() * 0.6)) * (Math.random() * 2 - 1);
  }
  let m = 0; for (let i = 0; i < n; i++) m = Math.max(m, Math.abs(x[i])); for (let i = 0; i < n; i++) x[i] /= m;
  return b;
}
function scrapeStart() {
  const ctx = audio(); if (!ctx || SND.scr) return;
  const src = ctx.createBufferSource(); src.buffer = SND.gravel || (SND.gravel = gravelBuf(ctx, 2.5)); src.loop = true;
  const bp = ctx.createBiquadFilter(); bp.type = "bandpass"; bp.frequency.value = 1400; bp.Q.value = 0.7;
  const hp = ctx.createBiquadFilter(); hp.type = "highpass"; hp.frequency.value = 300;
  const g = ctx.createGain(); g.gain.value = 0;
  const rasp = ctx.createBiquadFilter(); rasp.type = "bandpass"; rasp.frequency.value = 3400; rasp.Q.value = 2.5;   // the sideways rasp of a bend
  const rg = ctx.createGain(); rg.gain.value = 0;
  src.connect(hp); hp.connect(bp); bp.connect(g); g.connect(SND.out); bp.connect(rasp); rasp.connect(rg); rg.connect(SND.out);
  src.start();
  SND.scr = { src, bp, g, rg, step: 0 };
}
const SCRAPE_LEVEL = 0.85;                  // the rake's scrape against the rest (dialled down 15% at Gwern's ear, 2026-10-05)
function scrapeSet(speed, bend, vol = 0.5) {   // speed in cells per second; bend 1 on an arc
  const S = SND.scr, ctx = SND.ctx; if (!S) return;
  const v = Math.min(1, speed / 5), t = ctx.currentTime, lv = vol * SCRAPE_LEVEL;
  S.g.gain.setTargetAtTime(3 * lv * (0.15 + 0.85 * v) * (speed > 0.02 ? 1 : 0), t, 0.04);
  S.rg.gain.setTargetAtTime(lv * 0.9 * v * (bend ? 1 : 0.1), t, 0.06);
  S.bp.frequency.setTargetAtTime(900 + 1300 * v, t, 0.08);
  S.src.playbackRate.setTargetAtTime(0.6 + 0.7 * v, t, 0.08);
}
function scrapeStop() {
  const S = SND.scr, ctx = SND.ctx; if (!S) return; SND.scr = null;
  const t = ctx.currentTime; S.g.gain.setTargetAtTime(0, t, 0.05); S.rg.gain.setTargetAtTime(0, t, 0.05);
  try { S.src.stop(t + 0.4); } catch (e) {}
}
function crunch(vol = 0.12) {                // a footstep on gravel: a burst of grains over a soft thump
  const ctx = audio(); if (!ctx) return; const t = ctx.currentTime + 0.005;
  const s = ctx.createBufferSource(); s.buffer = SND.gravel || (SND.gravel = gravelBuf(ctx, 2.5)); s.playbackRate.value = 0.9 + Math.random() * 0.5;
  const bp = ctx.createBiquadFilter(); bp.type = "bandpass"; bp.frequency.value = 900 + Math.random() * 1400; bp.Q.value = 0.8;
  const g = ctx.createGain(), dur = 0.05 + Math.random() * 0.04;
  g.gain.setValueAtTime(0, t); g.gain.linearRampToValueAtTime(vol, t + 0.006); g.gain.exponentialRampToValueAtTime(0.0005, t + dur);
  s.connect(bp); bp.connect(g); g.connect(SND.out); s.start(t, Math.random() * 2); s.stop(t + dur + 0.02);
  partial(ctx, SND.out, 75 + Math.random() * 40, vol * 0.5, 0.045, t);
}
/* the garden's own air: a wind bed always (filtered noise, slowly wandering, with a gust now and then), small birds by day,
   crickets after dark; by the local clock, read each time something is scheduled, so an open page follows the evening. */
function dayPart(h = new Date().getHours()) { return h >= 6 && h < 19 ? "day" : (h >= 20 || h < 5) ? "night" : "dusk"; }
function windStart(vol = 0.1) {
  const ctx = SND.ctx; if (!ctx || SND.wind) return;
  const src = ctx.createBufferSource(); src.buffer = noiseBuf(ctx, 3); src.loop = true;
  const lp = ctx.createBiquadFilter(); lp.type = "lowpass"; lp.frequency.value = 500; lp.Q.value = 0.9;
  const g = ctx.createGain(); g.gain.value = 0;
  src.connect(lp); lp.connect(g); g.connect(SND.out); src.start();
  SND.wind = { src, lp, g, vol };
  g.gain.setTargetAtTime(vol * 0.5, ctx.currentTime, 2);
}
function windWander() {                      // the bed drifts: a new level and colour every few seconds
  const W = SND.wind; if (!W || !SND.on) return; const t = SND.ctx.currentTime;
  W.g.gain.setTargetAtTime(W.vol * (0.3 + Math.random() * 0.7), t, 1.5); W.lp.frequency.setTargetAtTime(350 + Math.random() * 450, t, 1.5);
}
function gust(vol = 0.16, dur = 4) {         // a gust: the bed swells and brightens, then settles
  const ctx = audio(); if (!ctx) return; windStart(); const W = SND.wind, t = ctx.currentTime;
  W.g.gain.cancelScheduledValues(t); W.g.gain.setTargetAtTime(vol, t, dur * 0.3); W.g.gain.setTargetAtTime(W.vol * 0.5, t + dur * 0.45, dur * 0.3);
  W.lp.frequency.cancelScheduledValues(t); W.lp.frequency.setTargetAtTime(1100, t, dur * 0.3); W.lp.frequency.setTargetAtTime(500, t + dur * 0.45, dur * 0.3);
}
function birds(vol = 0.06) {                 // one small bird, a phrase of two to six notes: glides on a sine, a little vibrato
  const ctx = audio(); if (!ctx) return; let t = ctx.currentTime + 0.05;
  const high = Math.random() < 0.5, base = high ? 3200 + Math.random() * 900 : 1900 + Math.random() * 600, n = 2 + Math.floor(Math.random() * 5);
  const g = ctx.createGain(); g.gain.value = vol * (0.7 + Math.random() * 0.3); g.connect(SND.out);
  for (let i = 0; i < n; i++) {
    const o = ctx.createOscillator(), e = ctx.createGain(), f0 = base * (0.94 + Math.random() * 0.12), up = Math.random() < 0.6, d = 0.06 + Math.random() * 0.09;
    o.type = "sine"; o.frequency.setValueAtTime(f0, t); o.frequency.exponentialRampToValueAtTime(f0 * (up ? 1.25 + Math.random() * 0.2 : 0.78), t + d * 0.7);
    const vib = ctx.createOscillator(), vg = ctx.createGain(); vib.frequency.value = 28 + Math.random() * 20; vg.gain.value = f0 * 0.015; vib.connect(vg); vg.connect(o.frequency);
    e.gain.setValueAtTime(0, t); e.gain.linearRampToValueAtTime(1, t + 0.008); e.gain.exponentialRampToValueAtTime(0.001, t + d);
    o.connect(e); e.connect(g); o.start(t); vib.start(t); o.stop(t + d + 0.02); vib.stop(t + d + 0.02);
    t += d + 0.05 + Math.random() * 0.16;
  }
}
function crickets(vol = 0.035, secs = 0.8) { // one cricket's trill: a high sine pulsed at about 28 Hz, in soft pulses
  const ctx = audio(); if (!ctx) return; const t0 = ctx.currentTime + 0.02, f = 3900 + Math.random() * 800, rate = 24 + Math.random() * 10, p = 1 / rate;
  const o = ctx.createOscillator(), g = ctx.createGain(); o.type = "sine"; o.frequency.value = f; g.gain.value = 0;
  const o2 = ctx.createOscillator(), g2 = ctx.createGain(); o2.type = "sine"; o2.frequency.value = f * 2.01; g2.gain.value = 0.18; o2.connect(g2); g2.connect(g);
  const n = Math.floor(secs / p), a = vol * (0.7 + Math.random() * 0.3);
  for (let i = 0; i < n; i++) { const t = t0 + i * p, env = Math.sin(Math.PI * (i + 0.5) / n); g.gain.setTargetAtTime(a * env, t, p * 0.12); g.gain.setTargetAtTime(0, t + p * 0.45, p * 0.1); }
  o.connect(g); g.connect(SND.out); o.start(t0); o2.start(t0); o.stop(t0 + secs + 0.1); o2.stop(t0 + secs + 0.1);
}
function ambientStart() {
  const loop = (fn, lo, hi) => { const go = () => later(() => { if (SND.on && !document.hidden) fn(); go(); }, lo + Math.random() * (hi - lo)); return go; };
  later(loop(() => drip(0.1 + Math.random() * 0.12), 7000, 23000), 2500 + Math.random() * 4000);
  later(loop(() => tock(0.4), 50000, 130000), 15000 + Math.random() * 25000);
  later(loop(windWander, 2500, 6000), 500);
  later(loop(() => gust(0.12 + Math.random() * 0.08, 3 + Math.random() * 4), 25000, 70000), 8000 + Math.random() * 20000);
  later(loop(() => { if (dayPart() === "day") birds(); }, 12000, 45000), 4000 + Math.random() * 8000);
  later(loop(() => { if (dayPart() === "night") crickets(0.035, 0.5 + Math.random() * 0.6); }, 900, 2200), 3000 + Math.random() * 3000);
}
function setSound(on) {
  SND.on = on; try { localStorage.setItem("samon-sound", on ? "1" : "0"); } catch (e) {}
  if (!on) { for (const id of SND.timers) clearTimeout(id); SND.timers.clear(); stopPiece(); scrapeStop(); if (SND.ctx) SND.ctx.suspend().catch(() => {}); }
  else { const again = !!SND.ctx && !SND.timers.size; audio(); if (again) ambientStart(); }   // (audio() makes or resumes the context; a new one starts its own ambience)
  renderMute();
}
/* Silence, as the player and the browser want it. The mute button (and M) is the page's own switch. Beyond it: the page never
   sounds before a gesture, as browsers require; its audio session is "ambient" where the browser has the Audio Session API
   (Safari 17+), so a phone's silent switch mutes it and it mixes with whatever the player was already listening to instead of
   stopping it; the OS's media keys (pause, stop) mute it while a recording plays, through the Media Session API; and when the
   browser keeps the context from running after a gesture (a site set to block sound, or Firefox's autoplay policy saying
   "disallowed"), the button shows that in red rather than pretending. There is no CSS media query for a sound preference
   (Media Queries 5 has prefers-reduced-motion, -transparency, -contrast, -data, forced-colors; nothing for audio). */
function soundBlocked() {                    // the browser is keeping sound from playing, as far as the page can tell
  if (SND.on && navigator.getAutoplayPolicy) { try { if (navigator.getAutoplayPolicy("audiocontext") === "disallowed") return true; } catch (e) {} }
  return !!(SND.on && SND.ctx && SND.gesture && SND.ctx.state !== "running" && performance.now() - SND.gesture > 1500);
}
function renderMute() {
  const b = el("mute"); if (!b) return;
  const blocked = soundBlocked();
  b.classList.toggle("off", !SND.on); b.classList.toggle("blocked", blocked);
  b.setAttribute("aria-pressed", String(!SND.on));
  b.setAttribute("aria-label", !SND.on ? "Unmute" : blocked ? "Sound is blocked by the browser" : "Mute");
  b.title = !SND.on ? "Unmute (M)" : blocked ? "Your browser is not letting this page play sound: allow sound for it in the site settings, then press here or any key." : "Mute (M)";
  const sw = el("soundToggle"), hb = el("hapticsToggle");   // the panel under the chevron: the switch, the volume while on, and haptics where the device can tick
  sw.textContent = SND.on ? "Sound on" : "Sound off"; sw.setAttribute("aria-pressed", String(SND.on));
  el("volRow").hidden = !SND.on; el("soundVol").value = String(Math.round(SND.vol * 100));
  hb.hidden = !navigator.vibrate; hb.textContent = SND.haptics ? "Haptics on" : "Haptics off"; hb.setAttribute("aria-pressed", String(SND.haptics));
}
function soundPanel(open) {
  const p = el("soundPanel"), t = el("soundMore");
  p.hidden = !open; t.setAttribute("aria-expanded", String(open));
}
el("soundMore").onclick = () => soundPanel(el("soundPanel").hidden);
document.addEventListener("pointerdown", ev => { if (!el("soundPanel").hidden && !ev.target.closest("#soundPanel, #soundMore")) soundPanel(false); });
document.addEventListener("keydown", ev => { if (ev.key === "Escape" && !el("soundPanel").hidden) soundPanel(false); });
el("soundToggle").onclick = () => setSound(!SND.on);
el("soundVol").oninput = () => { setVolume(el("soundVol").value / 100); audio(); };   // (moving it is a gesture, so the sound may start here)
el("soundVol").onchange = () => drip(0.22);                                           // and a drop, to hear the level
el("hapticsToggle").onclick = () => setHaptics(!SND.haptics);
el("mute").onclick = () => { if (SND.on && soundBlocked()) { audio(); renderMute(); return; } setSound(!SND.on); };
document.addEventListener("keydown", ev => {
  if ((ev.key === "m" || ev.key === "M") && !ev.ctrlKey && !ev.metaKey && !ev.altKey && !/^(INPUT|TEXTAREA|SELECT)$/.test(ev.target.tagName)) { ev.preventDefault(); setSound(!SND.on); }
});
if (navigator.audioSession) { try { navigator.audioSession.type = "ambient"; } catch (e) {} }
if (navigator.mediaSession) {
  try {
    navigator.mediaSession.setActionHandler("pause", () => setSound(false));
    navigator.mediaSession.setActionHandler("stop", () => setSound(false));
    navigator.mediaSession.setActionHandler("play", () => setSound(true));
  } catch (e) {}
}
function setVolume(v) {                      // 0..1, remembered
  SND.vol = Math.max(0, Math.min(1, v)); try { localStorage.setItem("samon-sound-vol", String(SND.vol)); } catch (e) {}
  if (SND.master) SND.master.gain.setTargetAtTime(SND.vol, SND.ctx.currentTime, 0.03);
}
function setHaptics(on) { SND.haptics = on; try { localStorage.setItem("samon-haptics", on ? "1" : "0"); } catch (e) {} renderMute(); }
function haptic(ms = 8) { if (SND.haptics && navigator.vibrate) { try { navigator.vibrate(ms); } catch (e) {} } }   // a tick per cell, where the device can
function loadPieces() {                      // the recordings carried in the page, those this browser can play
  if (SND.tracks) return;
  const probe = document.createElement("audio");
  SND.tracks = [...document.querySelectorAll("script.piece")].map(el => ({ src: samonAsset.text(el), title: el.dataset.title || "", player: el.dataset.player || "",
    license: el.dataset.license || "", source: el.dataset.source || "", mime: el.dataset.mime || "", gain: parseFloat(el.dataset.gain) || 0.35, dur: parseFloat(el.dataset.dur) || 0 }))
    .filter(tr => tr.src && (!tr.mime || probe.canPlayType(tr.mime) !== ""));
  if (!SND.tracks.length) SND.tracks = null;
}
function maybePiece(reason) {                // after a world's bell, a piece follows; otherwise one now and then, never within a quarter-hour of the last
  if (!SND.on || !SND.tracks || SND.cur) return;
  if (reason !== "capstone" && (Date.now() - SND.lastPiece < 15 * 60000 || Math.random() > 0.3)) return;
  const unplayed = SND.tracks.filter(tr => !SND.played.includes(tr.title)), pool = unplayed.length ? unplayed : SND.tracks;
  const tr = pool[Math.floor(Math.random() * pool.length)];
  later(() => playPiece(tr), reason === "capstone" ? 7000 : 2500);
}
function playPiece(tr, from = 0) {           // from: seconds into the piece (tests)
  const ctx = audio(); if (!ctx || SND.cur) return;
  const a = new Audio(); a.preload = "auto";
  let node; try { node = ctx.createMediaElementSource(a); } catch (e) { return; }
  const g = ctx.createGain(); g.gain.value = 0; node.connect(g); g.connect(SND.out);
  const cur = SND.cur = { a, g, tr }; SND.lastPiece = Date.now(); SND.played.push(tr.title);
  if (navigator.mediaSession && window.MediaMetadata) { try { navigator.mediaSession.metadata = new MediaMetadata({ title: tr.title, artist: tr.player, album: "Samon" }); } catch (e) {} }
  const vol = tr.gain || 0.35;
  a.addEventListener("playing", () => { const t = ctx.currentTime; g.gain.cancelScheduledValues(t); g.gain.setValueAtTime(0, t); g.gain.linearRampToValueAtTime(vol, t + 3); }, { once: true });
  a.addEventListener("timeupdate", () => { if (a.duration && a.duration - a.currentTime < 4) g.gain.setValueAtTime(Math.max(0, vol * (a.duration - a.currentTime) / 4), ctx.currentTime); });
  const done = () => { if (SND.cur === cur) SND.cur = null; try { node.disconnect(); } catch (e) {} if (a.src.startsWith("blob:")) URL.revokeObjectURL(a.src); };
  a.addEventListener("ended", done); a.addEventListener("error", done);
  if (from) a.addEventListener("loadedmetadata", () => { try { a.currentTime = from; } catch (e) {} }, { once: true });
  samonAsset.url(tr.src, tr.mime).then(url => { if (SND.cur !== cur) return; a.src = url; return a.play(); }).catch(done);
}
function stopPiece() { const c = SND.cur; if (!c) return; SND.cur = null; try { const u = c.a.src; c.a.pause(); c.a.removeAttribute("src"); c.a.load(); if (u.startsWith("blob:")) URL.revokeObjectURL(u); } catch (e) {} }
document.addEventListener("visibilitychange", () => { if (document.hidden) stopPiece(); });
async function renderSound(script, secs) {   // (test hook) render a sequence of the synthesized sounds offline, as 16-bit mono WAV
  const O = window.OfflineAudioContext || window.webkitOfflineAudioContext, ctx = new O(1, Math.ceil(44100 * secs), 44100);
  const keep = { ctx: SND.ctx, out: SND.out, master: SND.master, basin: SND.basin, meter: SND.meter, wind: SND.wind, scr: SND.scr, gravel: SND.gravel, on: SND.on, offline: null };
  SND.on = true; SND.ctx = ctx; SND.offline = new Set(); SND.wind = SND.scr = SND.gravel = null; SND.master = null;
  SND.out = ctx.createGain(); SND.out.gain.value = 0.8; SND.out.connect(ctx.destination);
  const d = ctx.createDelay(0.3), fb = ctx.createGain(), lp = ctx.createBiquadFilter(), wet = ctx.createGain();
  d.delayTime.value = 0.057; fb.gain.value = 0.4; lp.type = "lowpass"; lp.frequency.value = 1600; wet.gain.value = 0.45;
  d.connect(lp); lp.connect(fb); fb.connect(d); lp.connect(wet); wet.connect(SND.out); SND.basin = d;
  const F = { drip, tock, tok, bell, crunch, birds, crickets, gust, wind: (vol) => windStart(vol),
    scrape: (dur = 3, bends = 2) => {        // a whole pass of the rake: start, a steady run with a bend or two, stop (as revealTick drives it)
      scrapeStart(); const n = Math.round(dur / 0.05);
      for (let i = 0; i <= n; i++) { const f = i / n, v = 6 * (easeRake(Math.min(1, f + 0.01)) - easeRake(f)) / 0.01;   // 6 cells/s at full speed
        offlineAt(ctx.currentTime + i * 0.05, i < n ? () => scrapeSet(Math.max(0, v), bends && Math.abs(((f * (bends + 1)) % 1) - 0.5) < 0.08) : scrapeStop); }
    } };
  for (const [at, kind, ...args] of script) offlineAt(at, () => F[kind](...args));   // (each sound is scheduled with the render suspended at its moment, then rendering goes on)
  const buf = await ctx.startRendering();
  Object.assign(SND, keep);
  const x = buf.getChannelData(0), n = x.length, w = new DataView(new ArrayBuffer(44 + n * 2));
  const str = (o, t) => { for (let i = 0; i < t.length; i++) w.setUint8(o + i, t.charCodeAt(i)); };
  str(0, "RIFF"); w.setUint32(4, 36 + n * 2, true); str(8, "WAVE"); str(12, "fmt "); w.setUint32(16, 16, true); w.setUint16(20, 1, true); w.setUint16(22, 1, true);
  w.setUint32(24, 44100, true); w.setUint32(28, 88200, true); w.setUint16(32, 2, true); w.setUint16(34, 16, true); str(36, "data"); w.setUint32(40, n * 2, true);
  for (let i = 0; i < n; i++) { const v = Math.max(-1, Math.min(1, x[i])); w.setInt16(44 + i * 2, v < 0 ? v * 32768 : v * 32767, true); }
  return Array.from(new Uint8Array(w.buffer));
}

/* ---------- drawing ---------- */
let geom = { ox: 0, oy: 0, cs: 40, w: 0, h: 0, wall: 10 };
let sandTex = null;
function resize() {
  const css = cv.parentElement.clientWidth;
  const dpr = Math.min(3, window.devicePixelRatio || 1);
  const wallFrac = 0.32;
  const Wc = view === "practice" ? PR.W : G.W, Hc = view === "practice" ? PR.H : G.H;
  const cs = Math.floor((css * dpr) / (Wc + 2 * wallFrac));
  const wall = Math.round(cs * wallFrac);
  geom = { cs, wall, ox: wall, oy: wall, w: cs * Wc + 2 * wall, h: cs * Hc + 2 * wall };
  cv.width = geom.w; cv.height = geom.h;
  cv.style.aspectRatio = `${geom.w} / ${geom.h}`;
  sandTex = null; draw();
}
let PAL = null;
function color(name) {
  if (!PAL) PAL = {};
  if (!(name in PAL)) PAL[name] = getComputedStyle(document.documentElement).getPropertyValue(name).trim();
  return PAL[name];
}
function rng(seed) { let s = seed >>> 0; return () => ((s = Math.imul(s ^ (s >>> 15), 2246822507) ^ Math.imul(s ^ (s >>> 13), 3266489909)) >>> 0) / 4294967296; }
function makeSandTex() {
  const t = document.createElement("canvas"); t.width = geom.w; t.height = geom.h;
  const c = t.getContext("2d"); const R = rng(freshSeed());
  // specks at device resolution over the (slightly upscaled) field: dark ones and a few light ones, soft and sparse
  const n = Math.floor(geom.w * geom.h / 700), dark = hexRGB(color("--sand")).reduce((a, b) => a + b) < 384;
  for (let i = 0; i < n; i++) {
    const light = R() < 0.3;
    c.fillStyle = light ? color("--groove-hi") : color("--sand-dot"); c.globalAlpha = (light ? (dark ? 0.06 : 0.16) : 0.28) + R() * (light && dark ? 0.1 : 0.28);
    c.beginPath(); c.ellipse(R() * geom.w, R() * geom.h, 0.45 + R() * 0.6, 0.35 + R() * 0.45, R() * 3.14, 0, 6.283); c.fill();
  }
  return t;
}
function glyphFor(i, full) { // directions of the groove at path index i
  const ds = new Set();
  ds.add(backDir(i, full));
  if (i + 1 < full.length) ds.add(dirBetween(full[i], full[i + 1]));
  else if (G.endCell >= 0 && full[i] === G.endCell && G.end_side) ds.add(G.end_side);
  return ds;
}
function grooveCell(c2, r, c, ds) {
  const { ox, oy, cs } = geom; const x = ox + c * cs, y = oy + r * cs;
  const d = [...ds];
  for (const o of TINES) {
    c2.beginPath();
    if (d.length === 2 && ds.has("E") && ds.has("W")) { c2.moveTo(x, y + o * cs); c2.lineTo(x + cs, y + o * cs); }
    else if (d.length === 2 && ds.has("N") && ds.has("S")) { c2.moveTo(x + o * cs, y); c2.lineTo(x + o * cs, y + cs); }
    else if (d.length === 2) {
      const cx = x + (ds.has("E") ? cs : 0), cy = y + (ds.has("S") ? cs : 0);
      const a0 = ds.has("N") ? (ds.has("E") ? Math.PI / 2 : 0) : (ds.has("E") ? Math.PI : 1.5 * Math.PI);
      c2.arc(cx, cy, o * cs, a0, a0 + Math.PI / 2);
    } else {
      const s = d[0];
      if (s === "E") { c2.moveTo(x + cs, y + o * cs); c2.lineTo(x + 0.03 * cs, y + o * cs); }
      if (s === "W") { c2.moveTo(x, y + o * cs); c2.lineTo(x + 0.97 * cs, y + o * cs); }
      if (s === "S") { c2.moveTo(x + o * cs, y + cs); c2.lineTo(x + o * cs, y + 0.03 * cs); }
      if (s === "N") { c2.moveTo(x + o * cs, y); c2.lineTo(x + o * cs, y + 0.97 * cs); }
    }
    c2.stroke();
  }
}
function strokeGrooves(cells, alpha = 1, ghost = false) {
  const lw = Math.max(1.2, geom.cs / 24);
  ctx.save(); ctx.globalAlpha = alpha; ctx.lineCap = "butt";
  if (!ghost) {
    ctx.strokeStyle = color("--groove-hi"); ctx.lineWidth = lw * 1.25;
    ctx.translate(-lw * 0.45, -lw * 0.45); for (const [r, c, ds] of cells) grooveCell(ctx, r, c, ds);
    ctx.translate(lw * 0.45, lw * 0.45);
  }
  ctx.strokeStyle = color(ghost ? "--ghost" : "--groove"); ctx.lineWidth = lw;
  for (const [r, c, ds] of cells) grooveCell(ctx, r, c, ds);
  ctx.restore();
}
function stone(x, y, rad, seed, c2 = ctx) {
  const R = rng(seed); const n = 9, ph = R() * 6.283; const pts = [];
  for (let i = 0; i < n; i++) { const a = ph + i * 6.283 / n, rr = rad * (0.8 + 0.28 * R()); pts.push([x + rr * Math.cos(a), y + rr * Math.sin(a) * 0.86]); }
  c2.beginPath();
  for (let i = 0; i < n; i++) { const p = pts[i], q = pts[(i + 1) % n], m = [(p[0] + q[0]) / 2, (p[1] + q[1]) / 2];
    if (i === 0) { const z = pts[n - 1]; c2.moveTo((z[0] + p[0]) / 2, (z[1] + p[1]) / 2); }
    c2.quadraticCurveTo(p[0], p[1], m[0], m[1]); }
  c2.closePath();
}
/* Stones as sumi-e. A contour that is lumpy and a little rough, flatter at its base where it sits in the sand; a soft
   contact shadow cast down and to the right (the sand's light is from the upper left), drawn as the shadow of one path so
   the parts of a bent stone never darken each other; an ink wash, lighter toward the light; and a brushed edge, one band
   of varying weight: heavy where the ink pools on the shadow side, thinning to nothing where the brush ran dry on the lit
   side. All of it smooth: no segments, no steps. */
function contour(cx, cy, hx, hy, seed, p = 2, wob = 0.1) {   // sampled points of the contour, with outward normals (p = 2 an oval; 3 a boxier stone)
  const R = rng(seed), n = 72, ph = R() * 6.283, lumps = Array.from({ length: 7 }, () => (R() - 0.5) * 2), rough = Array.from({ length: 19 }, () => (R() - 0.5) * 2);
  const smooth = (arr, u) => { const m = arr.length, k = u * m, i0 = Math.floor(k) % m, t = k - Math.floor(k), tt = t * t * (3 - 2 * t); return arr[i0] * (1 - tt) + arr[(i0 + 1) % m] * tt; };
  const pts = [];
  for (let i = 0; i < n; i++) {
    const u = i / n, a = ph + u * 6.283, ca = Math.abs(Math.cos(a)), sa = Math.abs(Math.sin(a));
    const re = Math.pow(Math.pow(ca / hx, p) + Math.pow(sa / hy, p), -1 / p);          // the superellipse's radius this way
    const rr = re * (1 + wob * smooth(lumps, u) + 0.018 * smooth(rough, u) + 0.03 * Math.sin(3 * a + ph));
    let x = cx + rr * Math.cos(a), y = cy + rr * Math.sin(a);
    if (Math.sin(a) > 0.5) y = cy + (y - cy) * (1 - 0.12 * (Math.sin(a) - 0.5) / 0.5);   // the base sits lower and flatter
    pts.push({ x, y });
  }
  for (let i = 0; i < n; i++) {               // outward normals, from the neighbours
    const a = pts[(i + n - 1) % n], b = pts[(i + 1) % n], tx = b.x - a.x, ty = b.y - a.y, l = Math.hypot(tx, ty) || 1;
    pts[i].nx = ty / l; pts[i].ny = -tx / l;
  }
  return pts;
}
function contourPath(pts, dx = 0, dy = 0, k = 1, cx = 0, cy = 0) {   // the contour as a smooth closed path, scaled about (cx, cy) and shifted
  const P = new Path2D(), n = pts.length, Q = pts.map(q => [cx + (q.x - cx) * k + dx, cy + (q.y - cy) * k + dy]);
  for (let i = 0; i < n; i++) {
    const q = Q[i], w = Q[(i + 1) % n], m = [(q[0] + w[0]) / 2, (q[1] + w[1]) / 2];
    if (i === 0) { const z = Q[n - 1]; P.moveTo((z[0] + q[0]) / 2, (z[1] + q[1]) / 2); }
    P.quadraticCurveTo(q[0], q[1], m[0], m[1]);
  }
  P.closePath(); return P;
}
function isDark() { const c = hexRGB(color("--bg")); return c[0] + c[1] + c[2] < 384; }
function paintStone(parts, cs) {           // parts: [{cx, cy, hx, hy, seed, p, wob}], one stone (a bent stone is two overlapping parts)
  const pts = parts.map(q => contour(q.cx, q.cy, q.hx, q.hy, q.seed, q.p, q.wob));
  const union = (dx, dy, k) => { const P = new Path2D(); parts.forEach((q, j) => P.addPath(contourPath(pts[j], dx, dy, k, q.cx, q.cy))); return P; };
  const dark = isDark(), hi = hexRGB(color("--stone-hi")), lo = hexRGB(color("--stone")), rgba = (c, a) => `rgba(${c[0]},${c[1]},${c[2]},${a})`;
  const mix = (a, b, t) => a.map((v, i) => Math.round(v * (1 - t) + b[i] * t));
  const body = union(0, 0, 1);
  const x0 = Math.min(...parts.map(q => q.cx - q.hx)), y0 = Math.min(...parts.map(q => q.cy - q.hy)), x1 = Math.max(...parts.map(q => q.cx + q.hx)), y1 = Math.max(...parts.map(q => q.cy + q.hy));
  ctx.save();
  ctx.fillStyle = color("--stone");           // the contact shadow: the body's own shadow, soft and cast down-right, and a tighter darker one at the base
  ctx.shadowColor = `rgba(0,0,0,${dark ? 0.55 : 0.32})`; ctx.shadowBlur = cs * 0.16; ctx.shadowOffsetX = cs * 0.08; ctx.shadowOffsetY = cs * 0.11; ctx.fill(body);
  ctx.shadowColor = `rgba(0,0,0,${dark ? 0.6 : 0.4})`; ctx.shadowBlur = cs * 0.05; ctx.shadowOffsetX = cs * 0.02; ctx.shadowOffsetY = cs * 0.035; ctx.fill(body);
  ctx.shadowColor = "transparent"; ctx.shadowBlur = 0; ctx.shadowOffsetX = ctx.shadowOffsetY = 0;
  const g = ctx.createLinearGradient(x0 + (x1 - x0) * 0.1, y0, x1, y1);   // the wash: pale where the light falls, the mass of the ink below
  g.addColorStop(0, rgba(mix(hi, lo, dark ? 0 : 0.1), 1)); g.addColorStop(0.3, rgba(mix(hi, lo, 0.6), 1)); g.addColorStop(0.62, color("--stone")); g.addColorStop(1, rgba(mix(lo, [0, 0, 0], 0.35), 1));
  ctx.fillStyle = g; ctx.fill(body);
  // the ink edge, one band astride the contour: its weight follows the light (heavy on the shadow side), wanders a little,
  // and thins to nothing through one or two breaks on the lit side where the brush ran dry; in dark mode it is the pale rim
  // light instead, and so heavy on the lit side
  const base = Math.max(1, cs / 30), paths = parts.map((q, j) => contourPath(pts[j], 0, 0, 1, q.cx, q.cy));
  ctx.fillStyle = dark ? color("--stone-hi") : "#000";
  parts.forEach((q, j) => {
    const R = rng(q.seed + 101), P = pts[j], n = P.length, w0 = R() * 6.283, w1 = R() * 6.283;
    const breaks = Array.from({ length: 1 + Math.floor(R() * 2) }, () => [R() * 6.283, 0.35 + R() * 0.45]);   // [where, half-width] in angle
    const widths = P.map((pt, i) => {
      const facing = (1 + (-0.6 * pt.nx - 0.8 * pt.ny)) / 2, lit = dark ? 1 - facing : facing;   // 1 where the edge is thin
      let w = base * (0.25 + 2.1 * Math.pow(1 - lit, 1.4)) * (1 + 0.3 * Math.sin(w0 + i * 0.35) * Math.cos(w1 + i * 0.13));
      const ang = Math.atan2(pt.ny, pt.nx);
      for (const [at, hw] of breaks) { const d = Math.abs(((ang - at + 9.4248) % 6.2832) - 3.1416); if (d < hw) w *= Math.min(1, Math.pow(d / hw, 2) * (0.4 + 1.6 * (1 - lit))); }   // the dry stretch, deeper where lit
      if (paths.some((o, k) => k !== j && ctx.isPointInPath(o, pt.x, pt.y))) w = 0;    // inside the other part of a bent stone: no edge
      return w;
    });
    const sm = widths.map((w, i) => (widths[(i + n - 1) % n] + 2 * w + widths[(i + 1) % n]) / 4);   // (smoothed once)
    const outer = P.map((pt, i) => [pt.x + pt.nx * sm[i] * 0.6, pt.y + pt.ny * sm[i] * 0.6]), inner = P.map((pt, i) => [pt.x - pt.nx * sm[i] * 0.4, pt.y - pt.ny * sm[i] * 0.4]);
    const band = new Path2D();
    outer.forEach((pt, i) => i ? band.lineTo(pt[0], pt[1]) : band.moveTo(pt[0], pt[1])); band.closePath();
    const hole = new Path2D(); inner.forEach((pt, i) => i ? hole.lineTo(pt[0], pt[1]) : hole.moveTo(pt[0], pt[1])); hole.closePath();
    band.addPath(hole);
    ctx.globalAlpha = dark ? 0.9 : 0.95; ctx.fill(band, "evenodd");
  });
  ctx.restore();
}
function inkStone(X, Y, rad, seed) { paintStone([{ cx: X, cy: Y, hx: rad, hy: rad * 0.88, seed, p: 2, wob: 0.13 }], rad / 0.41); }
function frame() {
  const { w, h, wall } = geom;
  ctx.clearRect(0, 0, w, h);
  ctx.fillStyle = color("--wall"); ctx.beginPath(); ctx.roundRect ? ctx.roundRect(0, 0, w, h, Math.max(2, wall * 0.2)) : ctx.rect(0, 0, w, h); ctx.fill();
}
function stoneGroups() {               // 4-connected groups of stone cells (# and O; a ripple stone is always alone)
  if (G.groups) return G.groups;
  const seen = new Set(), out = [];
  for (let r = 0; r < G.H; r++) for (let c = 0; c < G.W; c++) {
    const s = G.rows[r][c];
    if ((s !== "#" && s !== "O") || seen.has(r * G.W + c)) continue;
    const comp = [], stack = [[r, c]]; seen.add(r * G.W + c);
    while (stack.length) {
      const [a, b] = stack.pop(); comp.push([a, b]);
      if (G.rows[a][b] === "O") continue;
      for (let d = 0; d < 4; d++) { const aa = a + DR[d], bb = b + DC[d];
        if (aa >= 0 && aa < G.H && bb >= 0 && bb < G.W && G.rows[aa][bb] === "#" && !seen.has(aa * G.W + bb)) { seen.add(aa * G.W + bb); stack.push([aa, bb]); } }
    }
    out.push(comp);
  }
  return (G.groups = out);
}
function cutOffCells() {
  if (!path.length) return [];
  const S = stateOf(path); reachable(S);
  return G.sandList.filter(v => !S.vis[v] && !BSEEN[v]);
}
function draw() {
  if (view === "practice") { drawPractice(); return; }
  if (!G || !PZ) return;
  const { ox, oy, cs } = geom;
  frame();
  // gate gaps
  const gap = (r, c, side) => {
    const x = ox + c * cs, y = oy + r * cs, m = cs * 0.1, wall = geom.wall;
    if (side === "S") return [x + m, y + cs, cs - 2 * m, wall];
    if (side === "N") return [x + m, y - wall, cs - 2 * m, wall];
    if (side === "W") return [x - wall, y + m, wall, cs - 2 * m];
    return [x + cs, y + m, wall, cs - 2 * m];
  };
  const [gx, gy, gw, gh] = gap(G.gate[0], G.gate[1], G.gate[2]);
  ctx.fillStyle = color("--sand"); ctx.fillRect(gx, gy, gw, gh);
  let sx, sy, sw, sh;
  if (G.end) { [sx, sy, sw, sh] = gap(G.end[0], G.end[1], G.end_side); ctx.fillRect(sx, sy, sw, sh); }
  // the sand itself: a lit heightfield (footprints, the abbot's grooves, the rake's), with crisp grain on top
  ctx.imageSmoothingEnabled = true; ctx.imageSmoothingQuality = "high";
  ctx.drawImage(PZ.cvs, ox, oy, cs * G.W, cs * G.H);
  if (!sandTex) sandTex = makeSandTex();
  ctx.save(); ctx.beginPath(); ctx.rect(ox, oy, cs * G.W, cs * G.H); ctx.clip(); ctx.globalAlpha = 0.4; ctx.drawImage(sandTex, 0, 0); ctx.restore();
  // two-tone gravel (the parity lesson)
  if (G.tone > 0) {
    ctx.save(); ctx.fillStyle = color("--fg"); ctx.globalAlpha = 0.13 * G.tone;
    for (const v of G.sandList) if (G.black[v]) { const [r, c] = rc(v); ctx.fillRect(ox + c * cs, oy + r * cs, cs, cs); }
    ctx.restore();
  }
  // assists: checkerboard and cut-off sand
  if (mode === "draw" && (assists.checker || marks.checker)) {
    ctx.save(); ctx.fillStyle = color("--fg"); ctx.globalAlpha = 0.075;
    for (const v of G.sandList) if (G.black[v]) { const [r, c] = rc(v); ctx.fillRect(ox + c * cs, oy + r * cs, cs, cs); }
    ctx.restore();
  }
  if (mode === "draw" && assists.cutoff) {
    const cells = cutOffCells();
    if (cells.length) {
      ctx.save(); ctx.strokeStyle = color("--fg"); ctx.globalAlpha = 0.35; ctx.lineWidth = Math.max(1, cs / 30);
      for (const v of cells) { const [r, c] = rc(v); const x = ox + c * cs, y = oy + r * cs;
        ctx.save(); ctx.beginPath(); ctx.rect(x, y, cs, cs); ctx.clip(); ctx.beginPath();
        for (let o = -cs; o < cs; o += cs / 5) { ctx.moveTo(x + o, y + cs); ctx.lineTo(x + o + cs, y); } ctx.stroke(); ctx.restore(); }
      ctx.restore();
    }
  }
  // sketch marks: bands, ripple rings
  const markC = color("--mark");
  if (mode === "draw") {
    ctx.save(); ctx.strokeStyle = markC; ctx.globalAlpha = 0.75; ctx.lineWidth = Math.max(1, cs / 44); ctx.setLineDash([cs / 14, cs / 10]);
    for (let r = 0; r < G.H; r++) for (let c = 0; c < G.W; c++) {
      const s = G.rows[r][c], x = ox + c * cs, y = oy + r * cs;
      ctx.beginPath();
      if (s === "h" || s === "=") for (const o of [0.25, 0.5, 0.75]) { ctx.moveTo(x, y + o * cs); ctx.lineTo(x + cs, y + o * cs); }
      if (s === "v" || s === "|") for (const o of [0.25, 0.5, 0.75]) { ctx.moveTo(x + o * cs, y); ctx.lineTo(x + o * cs, y + cs); }
      ctx.stroke();
    }
    ctx.setLineDash([]); ctx.globalAlpha = 1; ctx.lineWidth = Math.max(1.2, cs / 30);
    for (let r = 0; r < G.H; r++) for (let c = 0; c < G.W; c++) {
      const s = G.rows[r][c]; if (!"hv=|".includes(s)) continue;
      const x = ox + c * cs, y = oy + r * cs; ctx.beginPath();
      if (sym(G, r - 1, c) !== s) { ctx.moveTo(x, y); ctx.lineTo(x + cs, y); }
      if (sym(G, r + 1, c) !== s) { ctx.moveTo(x, y + cs); ctx.lineTo(x + cs, y + cs); }
      if (sym(G, r, c - 1) !== s) { ctx.moveTo(x, y); ctx.lineTo(x, y + cs); }
      if (sym(G, r, c + 1) !== s) { ctx.moveTo(x + cs, y); ctx.lineTo(x + cs, y + cs); }
      ctx.stroke();
    }
    ctx.setLineDash([cs / 22, cs / 9]); ctx.lineWidth = Math.max(1, cs / 36);
    for (let r = 0; r < G.H; r++) for (let c = 0; c < G.W; c++) if (G.rows[r][c] === "O") {
      for (const rad of [0.95, 1.25]) { ctx.beginPath(); ctx.arc(ox + c * cs + cs / 2, oy + r * cs + cs / 2, rad * cs, 0, 6.283); ctx.stroke(); }
    }
    ctx.restore();
  }
  // the abbot's sketched grooves (hints), and a preview of your own
  if (mode === "draw") {
    const ghostCells = [...ghosts].filter(([k]) => !path.includes(k)).map(([k, ds]) => { const [r, c] = rc(k); return [r, c, ds]; });
    if (ghostCells.length) strokeGrooves(ghostCells, 0.95, true);
    if (assists.preview && path.length) strokeGrooves(path.map((k, i) => { const [r, c] = rc(k); return [r, c, glyphFor(i, path)]; }), 0.6, true);
  }
  // where the rake is, during the reveal
  let rk = null, passed = new Set();
  if (mode === "reveal" && REV) {
    rk = strokePoint(REV.st, REV.s);
    for (let j = 0; j < rk.k; j++) passed.add(REV.st.segs[j].cell);
  }
  // the abbot's pebbles (small irregular stones, unlike the monk's ensō): pale = straight through, dark = turn; the rake gathers them
  if (mode !== "done") for (const v of G.markList) {
    if (passed.has(v)) continue;
    const [r, c] = rc(v); const X = ox + c * cs + cs / 2 + cs * 0.17, Y = oy + r * cs + cs / 2 - cs * 0.17, rad = cs * 0.13, seed = v * 7919 + 3;
    ctx.save();
    ctx.fillStyle = color("--sand-dot"); stone(X + rad * 0.15, Y + rad * 0.2, rad * 1.08, seed); ctx.fill();   // its shadow
    stone(X, Y, rad, seed);
    if (G.mark[v] === 1) { ctx.fillStyle = color("--groove-hi"); ctx.fill(); ctx.lineWidth = Math.max(1.2, cs / 30); ctx.strokeStyle = color("--stone"); ctx.stroke(); }
    else { ctx.fillStyle = color("--stone"); ctx.fill(); }
    ctx.restore();
  }
  // the cells the abbot pointed at with his observation: a dotted ring, until another garden
  if (mode === "draw" && obs.length) {
    ctx.save(); ctx.strokeStyle = color("--fg"); ctx.globalAlpha = 0.8; ctx.lineWidth = Math.max(1.5, cs / 24); ctx.setLineDash([cs / 30, cs / 14]);
    for (const v of obs) { const [r, c] = rc(v); ctx.beginPath(); ctx.arc(ox + c * cs + cs / 2, oy + r * cs + cs / 2, cs * 0.42, 0, 6.283); ctx.stroke(); }
    ctx.restore();
  }
  // hint highlights
  if (mode === "draw" && marks.cells.length) {
    ctx.save(); ctx.strokeStyle = color("--fg"); ctx.fillStyle = color("--fg"); ctx.lineWidth = Math.max(1.5, cs / 22); ctx.setLineDash([cs / 9, cs / 12]);
    for (const v of marks.cells) { const [r, c] = rc(v); const x = ox + c * cs, y = oy + r * cs, m = cs * 0.1;
      ctx.globalAlpha = 0.08; ctx.fillRect(x + m, y + m, cs - 2 * m, cs - 2 * m); ctx.globalAlpha = 0.9; ctx.strokeRect(x + m, y + m, cs - 2 * m, cs - 2 * m); }
    ctx.restore();
  }
  // the faint route through the footprints (it fades as the monk sets down his rake)
  const routeA = (mode === "draw" || (mode === "demo" && !REV)) ? 0.22 : mode === "reveal" && REV ? 0.22 * Math.max(0, 1 - (performance.now() - REV.t0) / 450) : 0;
  if (path.length > 1 && routeA > 0) {
    ctx.save(); ctx.strokeStyle = color("--fg"); ctx.globalAlpha = routeA; ctx.lineWidth = Math.max(1.5, cs / 16); ctx.lineJoin = "round"; ctx.lineCap = "round";
    ctx.beginPath(); path.forEach((k, i) => { const [r, c] = rc(k); const X = ox + c * cs + cs / 2, Y = oy + r * cs + cs / 2; i ? ctx.lineTo(X, Y) : ctx.moveTo(X, Y); }); ctx.stroke(); ctx.restore();
  }
  if (mode === "draw" && marks.target >= 0 && path.includes(marks.target)) {
    const [r, c] = rc(marks.target); ctx.save(); ctx.strokeStyle = color("--fg"); ctx.lineWidth = Math.max(2, cs / 16);
    ctx.beginPath(); ctx.arc(ox + c * cs + cs / 2, oy + r * cs + cs / 2, cs * 0.36, 0, 6.283); ctx.stroke(); ctx.restore();
  }
  // stones: a stone of several cells is drawn as one stone
  for (const comp of stoneGroups()) {
    const seed = idx * 131 + comp[0][0] * 17 + comp[0][1];
    if (comp.length === 1) { const [r, c] = comp[0]; inkStone(ox + c * cs + cs / 2, oy + r * cs + cs / 2, cs * 0.41, seed); continue; }
    // one stone, not several touching ones: an irregular elongated stone along each straight run of cells
    // (a straight piece is one long stone; an L-shaped piece, two that share the corner cell, which reads as one bent stone)
    const inG = new Set(comp.map(([r, c]) => r * G.W + c)), has = (r, c) => r >= 0 && c >= 0 && r < G.H && c < G.W && inG.has(r * G.W + c);
    const runs = [];
    for (const [r, c] of comp) {
      if (!has(r, c - 1) && has(r, c + 1)) { let e = c; while (has(r, e + 1)) e++; runs.push([r, c, r, e]); }
      if (!has(r - 1, c) && has(r + 1, c)) { let e = r; while (has(e + 1, c)) e++; runs.push([r, c, e, c]); }
    }
    paintStone(runs.map(([r1, c1, r2, c2], j) => {   // each straight run fills its cells but for a thin bed of sand; the runs are one stone
      const along = ((Math.max(r2 - r1, c2 - c1) + 1) / 2 - 0.07) * cs, across = 0.44 * cs;
      return { cx: ox + (c1 + c2 + 1) / 2 * cs, cy: oy + (r1 + r2 + 1) / 2 * cs, hx: c2 > c1 ? along : across, hy: r2 > r1 ? along : across, seed: seed + j, p: 3, wob: 0.1 };
    }), cs);
  }
  // gate markers
  const tri = (x, y, rot, filled) => {
    ctx.save(); ctx.translate(x, y); ctx.rotate(rot); const s = Math.max(5, cs * 0.12);
    ctx.beginPath(); ctx.moveTo(-s, -s * 0.6); ctx.lineTo(s, -s * 0.6); ctx.lineTo(0, s * 0.8); ctx.closePath();
    ctx.lineWidth = Math.max(1.5, cs / 28); ctx.strokeStyle = color("--seal"); ctx.fillStyle = color("--seal");
    filled ? ctx.fill() : ctx.stroke(); ctx.restore();
  };
  const rotOut = { S: 0, N: Math.PI, W: Math.PI / 2, E: -Math.PI / 2 };
  if (G.end) tri(sx + sw / 2, sy + sh / 2, rotOut[G.end_side] + Math.PI, false);   // (the gate itself needs none: the gap shows it, and the monk waits in it)
  // the monk (and, as he walks backwards, the rake he pulls)
  const monkAt = (X, Y) => enso(X, Y, cs * 0.24);
  if ((mode === "draw" || (mode === "demo" && !REV)) && path.length) { const [r, c] = rc(path[path.length - 1]); monkAt(ox + c * cs + cs / 2, oy + r * cs + cs / 2); }
  if (rk) {
    const m = strokePoint(REV.st, REV.s + 0.5), X = ox + rk.x * cs, Y = oy + rk.y * cs, MX = ox + m.x * cs, MY = oy + m.y * cs;
    const nx = -rk.ty, ny = rk.tx, half = RAKE_W / 2 * cs;
    const now = performance.now();
    ctx.save(); ctx.fillStyle = color("--stone-hi");
    for (const g of REV.grains) {
      const age = (now - g.t0) / 1000, k = 1 - age / (g.life / 1000);
      if (k <= 0) continue;
      ctx.globalAlpha = 0.75 * k; ctx.beginPath();
      ctx.arc(ox + (g.x + g.vx * age * (1 - age)) * cs, oy + (g.y + g.vy * age * (1 - age)) * cs, Math.max(0.8, g.r * cs), 0, 6.283); ctx.fill();
    }
    ctx.restore();
    ctx.save(); ctx.strokeStyle = color("--fg"); ctx.lineCap = "round";
    ctx.globalAlpha = 0.55; ctx.lineWidth = Math.max(1.5, cs / 30);                   // the handle, from the rake to the monk's hands
    ctx.beginPath(); ctx.moveTo(X, Y); ctx.lineTo(MX, MY); ctx.stroke();
    ctx.globalAlpha = 0.85; ctx.lineWidth = Math.max(2, cs / 15);                     // the rake's head
    ctx.beginPath(); ctx.moveTo(X - nx * half, Y - ny * half); ctx.lineTo(X + nx * half, Y + ny * half); ctx.stroke();
    ctx.lineWidth = Math.max(1.2, cs / 34);                                            // and its tines, trailing in the sand
    ctx.beginPath();
    for (const o of [-0.3, 0, 0.3]) { const tx0 = X + nx * o * cs, ty0 = Y + ny * o * cs; ctx.moveTo(tx0, ty0); ctx.lineTo(tx0 - rk.tx * cs * 0.09, ty0 - rk.ty * cs * 0.09); }
    ctx.stroke(); ctx.restore();
    monkAt(MX, MY);
  }
  if (!path.length && mode === "draw") monkAt(...gatePoint());   // before the walk, the monk waits in the gateway: drag him in
  syncBeckon();
}
/* The monk is an ensō: one brushed circle, landing heavy, thinning as it goes round, and left open before it closes, with a
   dry end. It is painted once per size into a small canvas (its wobble is seeded, so it is the same monk every frame). */
let ensoCache = null;
function enso(X, Y, rad) {                 // rubricated: the novice is the one vermilion thing in the sand (the seal is the other)
  const key = Math.round(rad * 4) + ":" + color("--seal");
  if (!ensoCache || ensoCache.key !== key) ensoCache = { key, img: makeEnso(rad, color("--seal")) };
  const img = ensoCache.img; ctx.drawImage(img, X - img.width / 2, Y - img.height / 2);
}
function makeEnso(rad, ink) {
  const pad = rad * 0.7, size = Math.ceil(2 * (rad + pad)), c = document.createElement("canvas"); c.width = c.height = size;
  const g = c.getContext("2d"), cx = size / 2, cy = size / 2, a0 = -2.35, span = 5.5, n = 96, outer = [], inner = [];
  for (let i = 0; i <= n; i++) {   // from the upper left, round clockwise, to a gap at the upper right
    const t = i / n, a = a0 + span * t;
    const land = Math.min(1, t / 0.07), lift = Math.min(1, (1 - t) / 0.08);   // the brush comes down and lifts: rounded, not cut
    const w = rad * (0.26 * Math.pow(1 - t, 0.75) + 0.07) * Math.sqrt(land) * Math.sqrt(lift) * (1 + 0.08 * Math.sin(t * 11 + 1));   // heavy at the landing, thinning
    const r = rad * (1 + 0.035 * Math.sin(t * 6.1 + 0.4) + 0.015 * Math.sin(t * 10.7 + 2));             // the hand's slow wobble
    outer.push([cx + (r + w / 2) * Math.cos(a), cy + (r + w / 2) * Math.sin(a)]);
    inner.push([cx + (r - w / 2) * Math.cos(a), cy + (r - w / 2) * Math.sin(a)]);
  }
  g.fillStyle = ink; g.beginPath();
  outer.forEach((p, i) => i ? g.lineTo(p[0], p[1]) : g.moveTo(p[0], p[1]));
  for (let i = n; i >= 0; i--) g.lineTo(inner[i][0], inner[i][1]);
  g.closePath(); g.fill();
  g.strokeStyle = ink; g.lineCap = "round";                      // the dry end: two thin strands carrying on past the lift, fading
  for (const [off, len, al, lw] of [[0.02, 0.36, 0.55, 0.045], [-0.06, 0.24, 0.3, 0.03]]) {
    const a1 = a0 + span - 0.06, r1 = rad * (1 + off), lg = g.createLinearGradient(cx + r1 * Math.cos(a1), cy + r1 * Math.sin(a1), cx + r1 * Math.cos(a1 + len), cy + r1 * Math.sin(a1 + len));
    const rgb = hexRGB(ink); lg.addColorStop(0, ink); lg.addColorStop(1, `rgba(${rgb[0]},${rgb[1]},${rgb[2]},0)`);
    g.strokeStyle = lg; g.globalAlpha = al; g.lineWidth = Math.max(0.8, rad * lw);
    g.beginPath(); for (let k = 0; k <= 8; k++) { const a = a1 + len * k / 8; k ? g.lineTo(cx + r1 * Math.cos(a), cy + r1 * Math.sin(a)) : g.moveTo(cx + r1 * Math.cos(a), cy + r1 * Math.sin(a)); } g.stroke();
  }
  return c;
}
function gatePoint() {                     // the veranda gate's threshold (where the gap meets the sand), in canvas pixels
  const { ox, oy, cs } = geom, [r, c, side] = G.gate, x = ox + c * cs, y = oy + r * cs;
  return side === "S" ? [x + cs / 2, y + cs] : side === "N" ? [x + cs / 2, y] : side === "W" ? [x, y + cs / 2] : [x + cs, y + cs / 2];
}
let beckonKey = "";
function syncBeckon() {                    // the ring around the waiting monk, and the ghostly finger that shows the first drag
  if (demoRun || mode === "demo" || (mode === "reveal" && REV && REV.demo)) { el("beckon").hidden = true; beckonKey = "demo"; return; }
  const show = view === "puzzle" && mode === "draw" && !path.length && !!G;
  const demo = show && (idx === 0 || !started || demoOnce) && !(window.matchMedia && matchMedia("(prefers-reduced-motion: reduce)").matches);
  const key = show ? [idx, geom.cs, cv.clientWidth, cv.offsetLeft, cv.offsetTop, demo].join() : "";
  if (key === beckonKey) return;
  beckonKey = key;
  const b = el("beckon"), f = el("finger");
  if (f.anim) { f.anim.cancel(); f.anim = null; }
  b.hidden = !show; f.hidden = !demo;
  if (!show) return;
  const sc = cv.clientWidth / cv.width, at = ([X, Y]) => [cv.offsetLeft + X * sc, cv.offsetTop + Y * sc];
  const [bx, by] = at(gatePoint());
  b.style.left = bx + "px"; b.style.top = by + "px"; b.style.width = b.style.height = geom.cs * 0.95 * sc + "px";
  if (!demo || !f.animate) return;
  const cells = (G.parK[0] || []).slice(0, 4).map(k => { const [r, c] = rc(k); return at([geom.ox + c * geom.cs + geom.cs / 2, geom.oy + r * geom.cs + geom.cs / 2]); });
  const pts = [[bx, by], ...cells], tr = ([x, y]) => `translate(${x}px, ${y}px)`;
  const kf = [{ transform: tr(pts[0]), opacity: 0, offset: 0 }, { transform: tr(pts[0]), opacity: 0.95, offset: 0.12 }];
  pts.slice(1).forEach((p, i) => kf.push({ transform: tr(p), opacity: 0.95, offset: 0.2 + 0.55 * (i + 1) / (pts.length - 1) }));
  kf.push({ transform: tr(pts[pts.length - 1]), opacity: 0, offset: 0.92 }, { transform: tr(pts[pts.length - 1]), opacity: 0, offset: 1 });
  f.anim = f.animate(kf, { duration: 2800, iterations: 3, easing: "ease-in-out", fill: "forwards" });
}
function update() {
  if (view !== "puzzle") return;
  el("turns").textContent = turnsOf();
  el("undo").disabled = !((mode === "done" && (!isDaily() || shared)) || (mode === "draw" && path.length));
  el("restart").disabled = mode === "done" && isDaily() && !shared;
  el("rakeDay").hidden = !dailyReady();
  el("hint").disabled = mode !== "draw";
  el("watch").hidden = !(GARDENS[idx] && GARDENS[idx].demo) || (mode !== "draw" && mode !== "done");
  el("restore").hidden = !(cut && mode === "draw");
  draw();
}
/* Messages to the player (the status line, the help, the notes in the list and on the scroll) are editorial, and set in
   square brackets, as gwern.net sets what the editor says to the reader; rules, poems, counts and controls are not. */
let statusRaw = "", statusWarn = false;
function setStatus(msg, warn = false) { statusRaw = msg; statusWarn = warn; const s = el("status"); s.textContent = msg ? `[${msg}]` : ""; s.className = "status" + (warn ? " warn" : ""); }
function appendStatus(more) { setStatus(statusRaw + more, statusWarn); }
function statusText() { return statusRaw; }
const extrasOpen = new Set();              // worlds whose extra gardens the player has unfolded in the list
function renderNav() {
  const nav = el("nav"); nav.innerHTML = "";
  const pill = (label, current, cls, title, fn) => {
    const b = document.createElement("button"); b.type = "button"; b.className = "pill " + cls;
    if (current) b.setAttribute("aria-current", "true");
    b.innerHTML = `<span class="dot" aria-hidden="true"></span>${escapeHtml(label)}`; b.title = title; b.onclick = fn; return b;
  };
  /* Each open world: its main path as pips, and its extras behind a toggle ("+10 more"), unfolded while an extra of that
     world is the garden open, or when the player unfolds it. A locked world is one muted line: its count and what opens it. */
  for (let wld = 1; wld <= 7; wld++) {
    const items = GARDENS.map((g, i) => [g, i]).filter(([g]) => g.world === wld);
    if (!items.length) continue;
    const row = document.createElement("div"); row.className = "world";
    row.innerHTML = `<span class="t">${TIMES[wld]}</span>`;
    const pills = document.createElement("div"); pills.className = "pills";
    const open = worldOpen(wld);
    if (!open) {
      row.classList.add("locked");
      const note = document.createElement("span"); note.className = "lock";
      note.innerHTML = `<span class="n">${items.length} gardens ·</span> <span class="note"></span>`;
      note.querySelector(".note").textContent = worldOpen(wld - 1) ? `[Opens when ${needIn(wld - 1)} gardens of ${TIMES[wld - 1]} are raked (${rakedIn(wld - 1)} so far; extras count).]` : "[Locked.]";
      note.title = "Worlds open in turn, each a time of day; the gardens of a world not yet open are not shown";
      pills.appendChild(note); row.appendChild(pills); nav.appendChild(row); continue;
    }
    const extras = items.filter(([g]) => g.extra), here = view === "puzzle" && GARDENS[idx] && GARDENS[idx].world === wld && GARDENS[idx].extra;
    const unfolded = extras.length && (extrasOpen.has(wld) || here);
    let n = 0, m = 0;
    items.forEach(([g, i]) => {
      if (g.extra && !unfolded) return;
      const p = progress[g.name] || {};
      if (g.extra && !m) { const sp = document.createElement("span"); sp.className = "more"; sp.textContent = "more:"; sp.title = "Extra gardens of this world: beside its main path, and counting toward opening the next world"; pills.appendChild(sp); }
      const b = document.createElement("button"); b.type = "button";
      b.className = "pip " + (g.extra ? "extra " : "") + (p.sealed ? "sealed" : p.grey ? "grey" : p.done ? "done" : "");
      if (view === "puzzle" && i === idx) b.setAttribute("aria-current", "true");
      b.textContent = g.extra ? ++m : ++n; b.setAttribute("aria-label", g.name);
      b.title = `${g.name}: ` + (p.sealed ? "sealed" : p.grey ? "sealed in grey (raked at par with the abbot's help)" : p.done ? `raked (best ${p.best} turns)` : "not yet raked");
      b.onclick = () => pickGarden(i);
      pills.appendChild(b);
    });
    if (extras.length) {
      const raked = extras.filter(([g]) => (progress[g.name] || {}).done).length;
      const t = document.createElement("button"); t.type = "button"; t.className = "fold"; t.setAttribute("aria-expanded", String(!!unfolded));
      t.textContent = unfolded ? "fewer" : `+${extras.length} more`;
      t.title = unfolded ? "Fold the extra gardens away" : `${extras.length} extra gardens beside this world's main path (${raked} raked); they count toward opening the next world`;
      t.onclick = () => { if (unfolded) extrasOpen.delete(wld); else extrasOpen.add(wld); renderNav(); };
      pills.appendChild(t);
    }
    row.appendChild(pills); nav.appendChild(row);
  }
  const row = document.createElement("div"); row.className = "world";
  row.innerHTML = `<span class="t">Any time</span>`;
  const pills = document.createElement("div"); pills.className = "pills";
  if (DAILY.length) {
    const di = ensureDaily(), rec = progress[GARDENS[di].name] || {}, streak = dailyStreak();
    pills.appendChild(pill(`Today's garden${streak ? ` · streak ${streak}` : ""}`, view === "puzzle" && idx === di, rec.sealed ? "sealed" : rec.grey ? "grey" : rec.done ? "done" : "",
      rec.done ? `Raked today: ${rec.turns} turns, par ${rec.par}` : "One garden for everyone today, raked once", pickDaily));
  }
  const earned = GARDENS.filter(g => !g.daily && (progress[g.name] || {}).done).length;
  pills.appendChild(pill(`Poems · ${earned} of ${NCAMP}`, view === "poems", "", "Every poem you have earned, with its Japanese and its source", () => { enterPoems(); }));
  pills.appendChild(pill("Practice garden", view === "practice", "", "Rake freely: no goal, no score", () => { enterPractice(); showStage(); }));
  const cp = pill("Carry my progress", false, "", "A link that carries all your progress to another device or browser: open it there and take it over", carryProgress);
  cp.id = "carry"; pills.appendChild(cp);
  row.appendChild(pills); nav.appendChild(row);   // (sound, volume and haptics live under the speaker's chevron in the header)
  const oa = document.createElement("button"); oa.type = "button"; oa.className = "openall"; oa.id = "openAll";
  oa.textContent = openAll ? "Lock the worlds again (prototype)" : "Open all worlds (prototype)";
  oa.title = openAll ? "Close the worlds not yet earned, as a player would find them" : "A convenience of the prototype: every world open at once, without raking the ones before";
  oa.onclick = () => { openAll = !openAll; try { localStorage.setItem("samon-open-all", openAll ? "1" : "0"); } catch (e) {} renderNav(); };
  nav.appendChild(oa);
}
function syncAssists() {
  for (const [id, k] of [["aPreview", "preview"], ["aChecker", "checker"], ["aCut", "cutoff"]]) el(id).setAttribute("aria-pressed", String(!!assists[k]));
}
for (const [id, k] of [["aPreview", "preview"], ["aChecker", "checker"], ["aCut", "cutoff"]])
  el(id).onclick = () => { assists[k] = !assists[k]; saveAssists(); syncAssists(); draw(); };

/* ---------- sand as a lit heightfield: the gardens and the practice garden share it ---------- */
function hexRGB(s) {
  s = (s || "").trim();
  if (/^#[0-9a-f]{3}$/i.test(s)) s = "#" + [...s.slice(1)].map(ch => ch + ch).join("");
  const m = /^#([0-9a-f]{6})$/i.exec(s);
  if (!m) return [200, 200, 200];
  const n = parseInt(m[1], 16); return [(n >> 16) & 255, (n >> 8) & 255, n & 255];
}
function fieldRes(Wc) {            // field samples per cell: about one per device pixel (so the sand is never upscaled much), within 32..144
  const css = cv.parentElement.clientWidth, dpr = Math.min(3, window.devicePixelRatio || 1);
  return Math.max(32, Math.min(144, Math.floor((css * dpr) / (Wc + 0.64))));
}
const freshSeed = () => (Math.random() * 4294967296) >>> 0;   // the sand is laid out afresh each time: no two visits to a garden share its grain
function makeField(Wc, Hc, res, seed) {
  const w = Wc * res, h = Hc * res, R = rng(seed), sc = res / 32;
  const F = { Wc, Hc, res, sc, w, h, seed, hf: new Float32Array(w * h), base: new Float32Array(w * h), grain: new Float32Array(w * h), fine: new Float32Array(w * h), dirty: null, feet: null };
  F.cvs = document.createElement("canvas"); F.cvs.width = w; F.cvs.height = h; F.c2 = F.cvs.getContext("2d");
  F.img = F.c2.createImageData(w, h);
  // height grain: two octaves (32 and 64 per cell) of value noise, interpolated, so the gravel reads the same at any resolution;
  // and a per-sample albedo grain (the fleck of individual grains, in the colour only, so the lighting stays smooth)
  for (const [per, amp] of [[32, 0.07], [64, 0.035]]) {
    const lw = Wc * per + 2, lh = Hc * per + 2, lat = new Float32Array(lw * lh), k = res / per;
    for (let i = 0; i < lat.length; i++) lat[i] = (R() - 0.5) * amp;
    for (let y = 0; y < h; y++) for (let x = 0; x < w; x++) {
      const fx = x / k, fy = y / k, ix = Math.floor(fx), iy = Math.floor(fy), ux = fx - ix, uy = fy - iy, tx = ux * ux * (3 - 2 * ux), ty = uy * uy * (3 - 2 * uy);
      F.grain[y * w + x] += (lat[iy * lw + ix] * (1 - tx) + lat[iy * lw + ix + 1] * tx) * (1 - ty) + (lat[(iy + 1) * lw + ix] * (1 - tx) + lat[(iy + 1) * lw + ix + 1] * tx) * ty;
    }
  }
  for (let i = 0; i < w * h; i++) { const u = R(); F.fine[i] = (u - 0.5) * (u < 0.04 ? 0.14 : 0.05); }   // mostly faint, a few brighter or darker grains
  { // and a slow mottle (3 per cell) in the colour, so no two patches of sand are quite the same shade
    const per = 3, lw = Wc * per + 2, lh = Hc * per + 2, lat = new Float32Array(lw * lh), k = res / per;
    for (let i = 0; i < lat.length; i++) lat[i] = (R() - 0.5) * 0.07;
    for (let y = 0; y < h; y++) for (let x = 0; x < w; x++) {
      const fx = x / k, fy = y / k, ix = Math.floor(fx), iy = Math.floor(fy), ux = fx - ix, uy = fy - iy, tx = ux * ux * (3 - 2 * ux), ty = uy * uy * (3 - 2 * uy);
      F.fine[y * w + x] += (lat[iy * lw + ix] * (1 - tx) + lat[iy * lw + ix + 1] * tx) * (1 - ty) + (lat[(iy + 1) * lw + ix] * (1 - tx) + lat[(iy + 1) * lw + ix + 1] * tx) * ty;
    }
  }
  fieldUnraked(F);
  return F;
}
function fieldUnraked(F) {     // unraked gravel: low lumps plus grain (into the base and the live field)
  const { w, h } = F, R = rng(F.seed * 31 + 977), G8 = F.res / 4, gw = Math.ceil(w / G8) + 2, gh = Math.ceil(h / G8) + 2, coarse = new Float32Array(gw * gh);
  const lump = 0.26 + R() * 0.16;          // how lumpy this garden's unraked gravel is
  for (let i = 0; i < coarse.length; i++) coarse[i] = (R() - 0.5) * lump;
  for (let y = 0; y < h; y++) for (let x = 0; x < w; x++) {
    const fx = x / G8, fy = y / G8, ix = Math.floor(fx), iy = Math.floor(fy), tx = fx - ix, ty = fy - iy;
    const a = coarse[iy * gw + ix], b = coarse[iy * gw + ix + 1], c = coarse[(iy + 1) * gw + ix], d = coarse[(iy + 1) * gw + ix + 1];
    F.base[y * w + x] = (a * (1 - tx) + b * tx) * (1 - ty) + (c * (1 - tx) + d * tx) * ty + F.grain[y * w + x];
  }
  F.hf.set(F.base);
}
function fieldLight(F, x0 = 0, y0 = 0, x1 = F.w, y1 = F.h) {   // a low sun from the upper left lights the slopes
  const w = F.w, h = F.h, hf = F.hf, data = F.img.data;
  const base = hexRGB(color("--sand")), hi = hexRGB(color("--groove-hi")), sh = hexRGB(color("--groove"));
  const L = [-0.55, -0.62, 0.56], ln = Math.hypot(...L); L[0] /= ln; L[1] /= ln; L[2] /= ln;
  const dark = base[0] + base[1] + base[2] < 384;
  const flat = L[2], k = 2.4 * F.sc, gain = 3.2, dim = dark ? 0.7 : 1, gr = dark ? 0.55 : 1;   // slopes measured per 1/32 of a cell; dark sand is lit more softly, its grain fainter (it is gravel by night, not asphalt)
  x0 = Math.max(0, x0 | 0); y0 = Math.max(0, y0 | 0); x1 = Math.min(w, Math.ceil(x1)); y1 = Math.min(h, Math.ceil(y1));
  for (let y = y0; y < y1; y++) for (let x = x0; x < x1; x++) {
    const i = y * w + x;
    const gx = (hf[y * w + Math.min(w - 1, x + 1)] - hf[y * w + Math.max(0, x - 1)]) * 0.5;
    const gy = (hf[Math.min(h - 1, y + 1) * w + x] - hf[Math.max(0, y - 1) * w + x]) * 0.5;
    const nx = -gx * k, ny = -gy * k, nn = Math.hypot(nx, ny, 1);
    let t = ((nx * L[0] + ny * L[1] + L[2]) / nn - flat) * gain;
    t = t > 1 ? 1 : t < -1 ? -1 : t;
    const tgt = t >= 0 ? hi : sh, f = (t >= 0 ? t * 0.85 : -t * 0.8) * dim, a = 1 + (F.grain[i] * 0.3 + F.fine[i]) * gr;
    const o = i * 4;
    data[o] = (base[0] + (tgt[0] - base[0]) * f) * a; data[o + 1] = (base[1] + (tgt[1] - base[1]) * f) * a;
    data[o + 2] = (base[2] + (tgt[2] - base[2]) * f) * a; data[o + 3] = 255;
  }
  if (x1 > x0 && y1 > y0) F.c2.putImageData(F.img, 0, 0, x0, y0, x1 - x0, y1 - y0);
}
function fieldFlush(F) { const d = F.dirty; if (d) { F.dirty = null; fieldLight(F, d[0] - 1, d[1] - 1, d[2] + 1, d[3] + 1); } }
/* The rake head, held perpendicular to its motion from (ax, ay) to (bx, by) (field samples), sets the sand's
   cross-section: a groove under each tine, a ridge between. weight < 1 presses lightly; mask(x, y) keeps sand untouched. */
function fieldRake(F, ax, ay, bx, by, Wr, tines, weight = 1, mask = null) {
  const dx = bx - ax, dy = by - ay, L = Math.hypot(dx, dy); if (L < 0.25) return false;
  const tx = dx / L, ty = dy / L, nx = -ty, ny = tx;
  const half = Wr / 2, sp = Wr / tines, m1 = 0.07 * F.res, pad = half + m1 + 1, w = F.w;   // m1: the swath's soft outer margin
  const x0 = Math.max(0, Math.floor(Math.min(ax, bx) - pad)), x1 = Math.min(F.w, Math.ceil(Math.max(ax, bx) + pad));
  const y0 = Math.max(0, Math.floor(Math.min(ay, by) - pad)), y1 = Math.min(F.h, Math.ceil(Math.max(ay, by) + pad));
  if (x1 <= x0 || y1 <= y0) return true;
  for (let y = y0; y < y1; y++) for (let x = x0; x < x1; x++) {
    const qx = x + 0.5 - ax, qy = y + 0.5 - ay, b = qx * tx + qy * ty;
    if (b < -0.7 || b > L + 0.7) continue;
    const a = qx * nx + qy * ny, aa = Math.abs(a);
    if (aa > half + m1 || (mask && mask(x + 0.5, y + 0.5))) continue;
    // a groove under each tine, a ridge between; the outer berms are lower, and slope gently into the sand beside the swath
    let prof = 0.5 * Math.cos(2 * Math.PI * (a + half) / sp);
    if (aa > half) prof = 0.5 * 0.65 * (1 - (aa - half) / m1);
    else if (aa > half - sp / 2 && prof > 0) prof *= 0.65;
    const t = Math.min(1, (half + m1 - aa) / m1), wgt = weight * t * t * (3 - 2 * t), i = y * w + x;
    F.hf[i] = F.hf[i] * (1 - wgt) + (prof + F.grain[i] * 0.25) * wgt;
  }
  const d = F.dirty; F.dirty = d ? [Math.min(d[0], x0), Math.min(d[1], y0), Math.max(d[2], x1), Math.max(d[3], y1)] : [x0, y0, x1, y1];
  return true;
}

/* ---------- a garden's sand: the abbot's grooves, the footprints of the walk in, and the rake ---------- */
let PZ = null;
const RAKE_W = 0.9;                         // the rake spans 0.9 of a cell, so its 3 tines sit at 0.2, 0.5, 0.8
function cellSegment(r, c, e, x) {          // the groove's centre line through cell (r, c), from side e to side x (e null: the rake is set down here)
  const port = d => [c + 0.5 + DC[d] * 0.5, r + 0.5 + DR[d] * 0.5];
  const p1 = port(x);
  if (e === null) { const p0 = [c + 0.5 - DC[x] * 0.5, r + 0.5 - DR[x] * 0.5]; return { type: "line", p0, p1, len: 1 }; }   // set down at the back edge: flush with the wall, stone or lane behind it
  const p0 = port(e);
  if (e === opp(x)) return { type: "line", p0, p1, len: 1 };
  const C = [c + ((e === 1 || x === 1) ? 1 : 0), r + ((e === 2 || x === 2) ? 1 : 0)];
  const a0 = Math.atan2(p0[1] - C[1], p0[0] - C[0]);
  let da = Math.atan2(p1[1] - C[1], p1[0] - C[0]) - a0;
  while (da > Math.PI) da -= 2 * Math.PI;
  while (da < -Math.PI) da += 2 * Math.PI;
  return { type: "arc", C, a0, da, len: Math.abs(da) * 0.5 };
}
function strokeOf(P) {                      // the raking stroke, in raking order (from where the rake is set down, out through the gate)
  const n = P.length, segs = [];
  for (let i = n - 1; i >= 0; i--) {
    const [r, c] = rc(P[i]);
    const x = i === 0 ? G.gateDir : dirTo(P[i], P[i - 1]);
    const e = i < n - 1 ? dirTo(P[i], P[i + 1]) : (G.endCell >= 0 && P[i] === G.endCell && G.end_side ? DIX[G.end_side] : null);
    const g = cellSegment(r, c, e, x); g.cell = P[i]; segs.push(g);
  }
  const [gr, gc] = rc(G.gateCell), d = G.gateDir, p0 = [gc + 0.5 + DC[d] * 0.5, gr + 0.5 + DR[d] * 0.5];
  segs.push({ type: "line", p0, p1: [p0[0] + DC[d] * 0.6, p0[1] + DR[d] * 0.6], len: 0.6, cell: -1 });
  let acc = 0; for (const g of segs) { g.at = acc; acc += g.len; }
  return { segs, L: acc };
}
function strokePoint(st, s) {               // position (in cells) and unit tangent at arc length s along the stroke
  s = Math.max(0, Math.min(st.L, s));
  let k = st.segs.length - 1; while (k > 0 && st.segs[k].at > s) k--;
  const g = st.segs[k], u = g.len ? (s - g.at) / g.len : 0;
  if (g.type === "line") {
    const dx = g.p1[0] - g.p0[0], dy = g.p1[1] - g.p0[1], l = Math.hypot(dx, dy) || 1;
    return { x: g.p0[0] + dx * u, y: g.p0[1] + dy * u, tx: dx / l, ty: dy / l, k };
  }
  const a = g.a0 + g.da * u, sg = Math.sign(g.da), rad = g.rad || 0.5;
  return { x: g.C[0] + rad * Math.cos(a), y: g.C[1] + rad * Math.sin(a), tx: -Math.sin(a) * sg, ty: Math.cos(a) * sg, k };
}
function rakeAlong(F, st, s0, s1) {         // draw the rake along the stroke from s0 to s1, in short chords; it is set down gently
  const step = Math.max(0.6, 0.012 * F.res) / F.res;
  let p = strokePoint(st, s0);
  for (let s = s0 + step; ; s += step) {
    const ss = Math.min(s, s1), q = strokePoint(st, ss);
    fieldRake(F, p.x * F.res, p.y * F.res, q.x * F.res, q.y * F.res, RAKE_W * F.res, 3, Math.min(1, 0.4 + 0.6 * ss / 0.05));
    p = q;
    if (ss >= s1) break;
  }
}
function footDirs() {                       // which way the footprints in each cell of the route point
  const m = new Map();
  path.forEach((k, i) => m.set(k, i + 1 < path.length ? dirTo(k, path[i + 1]) : (i ? dirTo(path[i - 1], k) : opp(G.gateDir))));
  return m;
}
function pressFoot(F, k, d) {               // two footprints per cell, as in the walk in (they stay inside the cell)
  const [r, c] = rc(k), res = F.res, ax = DC[d], ay = DR[d], px = -ay, py = ax;
  for (const side of [-1, 1]) {
    const cx = (c + 0.5 + ax * side * 0.09 + px * side * 0.11) * res, cy = (r + 0.5 + ay * side * 0.09 + py * side * 0.11) * res;
    const ru = 0.13 * res, rv = 0.07 * res, R = ru * 1.4 + 1;
    for (let y = Math.max(0, Math.floor(cy - R)); y < Math.min(F.h, Math.ceil(cy + R)); y++)
      for (let x = Math.max(0, Math.floor(cx - R)); x < Math.min(F.w, Math.ceil(cx + R)); x++) {
        const qx = x + 0.5 - cx, qy = y + 0.5 - cy, u = (qx * ax + qy * ay) / ru, v = (qx * px + qy * py) / rv, r2 = u * u + v * v;
        if (r2 < 1) F.hf[y * F.w + x] -= 0.42 * Math.sqrt(1 - r2);
        else if (r2 < 1.9) F.hf[y * F.w + x] += 0.07 * (1 - (r2 - 1) / 0.9);
      }
  }
}
function pzBuild() {                        // a fresh garden: unraked sand, plus the grooves the abbot has already raked
  PZ = makeField(G.W, G.H, fieldRes(G.W), freshSeed());
  for (const [k, ds] of Object.entries(G.clues || {})) {
    const [r, c] = k.split(",").map(Number), g = cellSegment(r, c, DIX[ds[0]], DIX[ds[1]]);
    g.at = 0; rakeAlong(PZ, { segs: [g], L: g.len }, 0, g.len);
  }
  // finished rings: a stone whose eight neighbours are already raked, as one circle around it
  for (let r = 1; r < G.H - 1; r++) for (let c = 1; c < G.W - 1; c++) {
    if (G.rows[r][c] !== "#" || !RING8.every(([a, b]) => G.rows[r + a][c + b] === "R")) continue;
    const ring = { type: "arc", C: [c + 0.5, r + 0.5], a0: Math.PI * 0.75, da: 2 * Math.PI, rad: 1.0, len: 2 * Math.PI, at: 0 };
    rakeAlong(PZ, { segs: [ring], L: ring.len }, 0, ring.len);
  }
  PZ.dirty = null; PZ.base.set(PZ.hf);
  pzWalk(true);
}
function pzWalk(full = false) {             // the sand as the walk in leaves it; only cells whose footprints changed are redone
  const want = footDirs(), F = PZ;
  if (full || !F.feet) {
    F.hf.set(F.base);
    for (const [k, d] of want) pressFoot(F, k, d);
    F.feet = want; fieldLight(F); return;
  }
  const changed = new Set();
  for (const [k, d] of F.feet) if (want.get(k) !== d) changed.add(k);
  for (const [k, d] of want) if (F.feet.get(k) !== d) changed.add(k);
  for (const k of changed) {
    const [r, c] = rc(k), x0 = c * F.res, y0 = r * F.res;
    for (let y = y0; y < y0 + F.res; y++) F.hf.set(F.base.subarray(y * F.w + x0, y * F.w + x0 + F.res), y * F.w + x0);
    if (want.has(k)) pressFoot(F, k, want.get(k));
    fieldLight(F, x0 - 1, y0 - 1, x0 + F.res + 1, y0 + F.res + 1);
  }
  F.feet = want;
}

/* ---------- the practice garden: free raking on the same sand ---------- */
const PR = { W: 8, H: 6, res: 32, F: null, stones: [], tool: "rake", tines: 3, sm: null, head: null, down: false };
function prInit() {
  PR.res = fieldRes(PR.W);
  PR.F = makeField(PR.W, PR.H, PR.res, freshSeed());
  PR.stones = [{ x: 2.1, y: 2.3, r: 0.5, s: 11 }, { x: 5.8, y: 3.9, r: 0.42, s: 23 }, { x: 6.5, y: 1.6, r: 0.3, s: 37 }];
  fieldLight(PR.F);
}
function prSmooth() { fieldUnraked(PR.F); fieldLight(PR.F); }
function prInStone(x, y) {
  for (const s of PR.stones) { const dx = x / PR.res - s.x, dy = (y / PR.res - s.y) / 0.86; if (dx * dx + dy * dy < (s.r * 0.97) ** 2) return true; }
  return false;
}
function prStamp(ax, ay, bx, by) { return fieldRake(PR.F, ax, ay, bx, by, PR.res * (PR.tines === 3 ? 0.8 : 1.25), PR.tines, 1, prInStone); }
function prPoint(ev) { const [x, y] = pointerCell(ev); return [(x - geom.ox) / geom.cs * PR.res, (y - geom.oy) / geom.cs * PR.res]; }
function prDown(ev) {
  const [x, y] = prPoint(ev);
  if (PR.tool === "stone") {
    const cx = x / PR.res, cy = y / PR.res;
    const hit = PR.stones.findIndex(s => Math.hypot(cx - s.x, (cy - s.y) / 0.86) < s.r);
    if (hit >= 0) { PR.stones.splice(hit, 1); setStatus("Stone lifted: the sand under it was never raked."); }
    else if (cx > 0.45 && cx < PR.W - 0.45 && cy > 0.45 && cy < PR.H - 0.45 && PR.stones.every(s => Math.hypot(cx - s.x, cy - s.y) > s.r + 0.5)) {
      PR.stones.push({ x: cx, y: cy, r: 0.42, s: (Math.random() * 1e6) | 0 }); setStatus("");
    }
    drawPractice(); return;
  }
  PR.down = true; PR.sm = [x, y]; PR.head = null; cv.setPointerCapture(ev.pointerId);
}
function prMove(ev) {
  if (!PR.down) return;
  const evs = ev.getCoalescedEvents ? ev.getCoalescedEvents() : [ev];
  for (const e of (evs.length ? evs : [ev])) {
    const [x, y] = prPoint(e);
    const nx = PR.sm[0] + (x - PR.sm[0]) * 0.5, ny = PR.sm[1] + (y - PR.sm[1]) * 0.5;   // light smoothing: a rake doesn't jitter
    if (prStamp(PR.sm[0], PR.sm[1], nx, ny)) { PR.head = [nx, ny, nx - PR.sm[0], ny - PR.sm[1]]; PR.sm = [nx, ny]; }
  }
  fieldFlush(PR.F);
  drawPractice();
}
function prUp() { PR.down = false; PR.head = null; drawPractice(); }
function drawPractice() {
  const { ox, oy, cs } = geom;
  frame();
  ctx.imageSmoothingEnabled = true; ctx.imageSmoothingQuality = "high";
  ctx.drawImage(PR.F.cvs, ox, oy, cs * PR.W, cs * PR.H);
  if (!sandTex) sandTex = makeSandTex();
  ctx.save(); ctx.beginPath(); ctx.rect(ox, oy, cs * PR.W, cs * PR.H); ctx.clip(); ctx.globalAlpha = 0.4; ctx.drawImage(sandTex, 0, 0); ctx.restore();
  for (const s of PR.stones) inkStone(ox + s.x * cs, oy + s.y * cs, s.r * cs, s.s);
  if (PR.head) {                      // the rake head, with its tines
    const [x, y, dx, dy] = PR.head, L = Math.hypot(dx, dy) || 1, nx = -dy / L, ny = dx / L, sc = cs / PR.res;
    const half = PR.res * (PR.tines === 3 ? 0.8 : 1.25) / 2 * sc, X = ox + x * sc, Y = oy + y * sc;
    ctx.save(); ctx.strokeStyle = color("--fg"); ctx.globalAlpha = 0.7; ctx.lineWidth = Math.max(2, cs / 18); ctx.lineCap = "round";
    ctx.beginPath(); ctx.moveTo(X - nx * half, Y - ny * half); ctx.lineTo(X + nx * half, Y + ny * half); ctx.stroke();
    ctx.restore();
  }
}
function enterPoems() {                     // the poems earned so far, in the order of the gardens, with their Japanese and sources
  cancelPoem();
  view = "poems"; VIEW_EL.classList.remove("practice"); VIEW_EL.classList.add("poems"); stopAnim(); dragging = false; clearAdvice();
  el("time").textContent = "Any time"; el("gname").textContent = "Poems"; el("seal").hidden = true;
  const sc = el("scroll"); sc.innerHTML = "";
  const earned = GARDENS.filter(g => !g.daily && (progress[g.name] || {}).done).length;
  el("rule").hidden = false; el("rule").innerHTML = `[${earned ? `${earned} of ${NCAMP} poems earned` : "No poems yet"}: each garden you rake, at par or not, gives one.]<span class="facts">[The texts are classical waka and haiku; the English versions are working translations. Tap a garden's name to go back to it.]</span>`;
  setStatus("");
  const esc = escapeHtml;
  { const lead = document.createElement("p"); lead.className = "left"; lead.textContent = earned ? `[${earned} of ${NCAMP} poems earned, one for each garden raked.]` : "[Rake a garden, and its poem is kept here.]"; sc.appendChild(lead); }
  const entry = (p, who, onName) => {
    const a = document.createElement("article"); a.className = "entry";
    a.innerHTML = `<div class="verse" lang="en">${esc(p.en)}</div><div class="by">${esc(p.poet)}<span class="g"> · </span></div>` +
      `<div class="ja" lang="ja">${esc(p.ja)}</div><div class="src">${esc([p.dates, p.src].filter(Boolean).join(" · "))}${p.url ? `${p.dates || p.src ? " · " : ""}<a href="${esc(p.url)}" target="_blank" rel="noopener">source</a>` : ""}</div>`;
    const g = a.querySelector(".g");
    if (onName) { const b = document.createElement("button"); b.type = "button"; b.textContent = who; b.title = "Go back to this garden"; b.onclick = onName; g.appendChild(b); } else g.append(who);
    return a;
  };
  for (let w = 1; w <= 7; w++) {
    const items = GARDENS.map((g, i) => [g, i]).filter(([g]) => g.world === w && !g.daily), have = items.filter(([g]) => (progress[g.name] || {}).done);
    if (!have.length) continue;
    const h = document.createElement("h3"); h.textContent = TIMES[w]; sc.appendChild(h);
    for (const [g, i] of have) sc.appendChild(entry(g.poem, g.name, () => { pickGarden(i); }));
    if (have.length < items.length) { const p = document.createElement("p"); p.className = "left"; p.textContent = `[${items.length - have.length} more in the gardens of ${TIMES[w]}.]`; sc.appendChild(p); }
  }
  if (PRACTICE_POEM && PRACTICE_POEM.en) { const h = document.createElement("h3"); h.textContent = "Practice garden"; sc.appendChild(h); sc.appendChild(entry(PRACTICE_POEM, "Practice garden", () => { enterPractice(); showStage(); })); }
  { const h = document.createElement("h3"); h.textContent = "Sounds"; sc.appendChild(h); const p = document.createElement("p"); p.className = "left"; p.id = "soundCredits";
    p.append("[The bamboo pipe and its drips, the wind, the birds by day and the crickets by night, the rake's scrape, the footsteps, the temple bell and the seal are all synthesized in the page; nothing of them is recorded. ");
    if (!SND.tracks) p.append("No shakuhachi recording is carried in this copy of the page" + (document.querySelector("script.piece") ? ", or this browser cannot play its encoding.]" : ".]"));
    else {
      p.append("Shakuhachi, now and then: ");
      SND.tracks.forEach((tr, i) => {
        if (tr.source) { const a = document.createElement("a"); a.href = tr.source; a.target = "_blank"; a.rel = "noopener"; a.textContent = tr.title; p.appendChild(a); } else p.append(tr.title);
        p.append(`${tr.player ? `, ${tr.player}` : ""}${tr.license ? ` (${tr.license})` : ""}${i < SND.tracks.length - 1 ? "; " : ".]"}`);
      });
    }
    sc.appendChild(p); }
  renderNav();
  const r = sc.getBoundingClientRect(); if (r.top < 0 || r.top > window.innerHeight * 0.6) sc.scrollIntoView({ block: "start" });
}
function enterPractice() {
  cancelPoem();
  view = "practice"; VIEW_EL.classList.remove("poems"); VIEW_EL.classList.add("practice"); stopAnim(); dragging = false; clearAdvice();
  el("time").textContent = "Any time"; el("gname").textContent = "Practice garden"; el("seal").hidden = true;
  el("rule").hidden = false; el("rule").innerHTML = "[No goal and no score: rake as you like.]<span class=\"facts\">[The same sand as the gardens, raked freely: grooves are carved into a heightfield and lit by a low sun.]</span>";
  setStatus("");
  if (!PR.F) prInit();
  cv.setAttribute("aria-label", "Practice garden: drag to rake freely.");
  renderNav(); resize(); syncBeckon();
}
el("toolRake").onclick = () => { PR.tool = "rake"; el("toolRake").setAttribute("aria-pressed", "true"); el("toolStone").setAttribute("aria-pressed", "false"); };
el("toolStone").onclick = () => { PR.tool = "stone"; el("toolStone").setAttribute("aria-pressed", "true"); el("toolRake").setAttribute("aria-pressed", "false"); };
el("tines").onclick = () => { PR.tines = PR.tines === 3 ? 5 : 3; el("tines").textContent = `${PR.tines} tines`; };
el("smooth").onclick = () => { prSmooth(); drawPractice(); setStatus(""); };

(() => {                                   // the practice garden's poem
  const f = el("ppoem");
  if (!PRACTICE_POEM) { f.remove(); return; }
  f.textContent = PRACTICE_POEM.en;
  const c = document.createElement("figcaption"); c.textContent = PRACTICE_POEM.poet; f.appendChild(c);
})();

/* ---------- start-up ---------- */
window.addEventListener("resize", () => {
  if (view === "puzzle" && G && PZ && Math.abs(fieldRes(G.W) - PZ.res) / PZ.res > 0.3) {
    if (mode === "reveal") finishAnimNow();
    pzBuild();
    if (mode === "done") { const st = strokeOf(path); rakeAlong(PZ, st, 0, st.L); PZ.dirty = null; fieldLight(PZ); }
  }
  if (G || view === "practice") resize();
});
const retheme = () => { PAL = null; sandTex = null; if (PR.F) fieldLight(PR.F); if (PZ) fieldLight(PZ); draw(); };
if (window.matchMedia) matchMedia("(prefers-color-scheme: dark)").addEventListener?.("change", retheme);
new MutationObserver(retheme).observe(document.documentElement, { attributes: true, attributeFilter: ["data-theme"] });
syncAssists();

function lastGarden() {                    // the garden last open, if its walk is unfinished (else the first unraked one)
  try {
    const i = parseInt(localStorage.getItem("samon-last")), g = GARDENS[i];
    if (g && !g.daily && worldOpen(g.world) && Array.isArray((progress[g.name] || {}).draft) && progress[g.name].draft.length > 1) return i;
  } catch (e) {}
  return firstUnraked();
}
function start(data) {
  ensureDaily(); renderMute();
  if (applyHash()) return;                 // a link to a raking, or carrying progress
  const i = (data && Number.isInteger(data.idx) && data.idx < GARDENS.length) ? data.idx : lastGarden();
  load(i);
  if (!(data && Array.isArray(data.path) && data.path.length)) maybeDemo();
  if (data && Array.isArray(data.path) && data.path.length) { path = data.path; routeChanged("cut"); }
  if (data && data.view === "practice") enterPractice();
  if (data && data.view === "poems") enterPoems();
}
window.claude?.hot?.snapshot?.(() => ({ idx, view, path: mode === "draw" ? path : [] }));
window.claude?.hot?.ready ? window.claude.hot.ready(start) : start(window.claude?.hot?.data ?? {});

/* test hooks */
window.__samon = {
  get path() { return path; }, get mode() { return mode; }, get G() { return G; }, get idx() { return idx; }, get view() { return view; }, get ghosts() { return ghosts; },
  geom: () => geom, gate: () => gatePoint(), worldOpen, startDemo, get demoRunning() { return !!demoRun || mode === "demo" || (mode === "reveal" && !!REV && !!REV.demo); }, get obs() { return obs; }, set budget(v) { abbotBudget = v; }, set poemTiming(v) { Object.assign(POEM_T, v); },
  get poem() { const b = el("poem"); return { state: poemRun ? (poemRun.stay ? "stay" : poemRun.shown ? "shown" : "waiting") : "none", visible: !b.hidden && b.classList.contains("on"),
    verse: el("verse").textContent, ja: el("ja").hidden ? "" : el("ja").textContent, leaving: el("stage").classList.contains("leaving"), clash: !b.hidden && sealClash() }; },
  poemFit(i, orig) {                        // lay garden i's poem over it as if raked at par (seal shown), with or without the original: does it fit?
    load(i); const s = el("seal"), box = el("poem"); s.hidden = false; box.hidden = false; fillPoem(GARDENS[i].poem, box);
    if (orig) { el("ja").hidden = false; el("src").hidden = false; el("orig").hidden = true; fitPoem(box); }
    const res = { clash: sealClash(), under: s.classList.contains("under"), left: box.firstElementChild.style.justifySelf === "start",
      over: box.firstElementChild.scrollHeight > box.clientHeight * 0.92, fs: parseFloat(box.style.getPropertyValue("--poem-fs")) };
    hidePoem(); s.hidden = true; return res; },
  poems: () => ({ gardens: GARDENS.filter(g => !g.daily && g.poem && g.poem.en && g.poem.ja && g.poem.poet).length, practice: !!(PRACTICE_POEM && PRACTICE_POEM.en) }), get helped() { return helped; },
  set today(v) { todayOverride = v; }, pickDaily, enterPoems,
  sound: { get on() { return SND.on; }, get ready() { return !!SND.ctx; }, get state() { return SND.ctx ? SND.ctx.state : "none"; }, get tracks() { return SND.tracks; }, get playing() { return SND.cur ? SND.cur.tr.file : null; }, get timers() { return SND.timers.size; },
    set: setSound, drip, tock, tok, bell, piece: maybePiece, play: (i, from) => playPiece(SND.tracks[i], from), render: renderSound, crunch, birds, crickets, gust, dayPart,
    get scraping() { return !!SND.scr; }, get vol() { return SND.vol; }, get haptics() { return SND.haptics; }, setVolume, setHaptics, get master() { return SND.master ? SND.master.gain.value : -1; },
    get blocked() { return soundBlocked(); }, renderMute,
    level() { if (!SND.meter) return -1; const a = new Float32Array(SND.meter.fftSize); SND.meter.getFloatTimeDomainData(a); return Math.sqrt(a.reduce((s, v) => s + v * v, 0) / a.length); } }, get daily() { const i = ensureDaily(); if (i < 0) return null; const g = GARDENS[i], p = progress[g.name] || {}; return { idx: i, key: g.dateKey, dayNo: g.dayNo, year: DAILY.length, name: g.name, done: !!p.done, turns: p.turns, kind: p.kind, streak: dailyStreak(), rakeShown: !el("rakeDay").hidden }; },
  strokeBends(cells) { const P = cells.map(([r, c]) => key(r, c)); return [strokeOf(P).segs.filter(s => s.type === "arc").length, turnsOf(P)]; }, stepTo, load, key: (r, c) => key(r, c), turnsOf, analyze, rakeNow, askAbbot, enterPractice,
  setPath(cells) { if (view !== "puzzle") return; cancelPoem(); mode = "draw"; stopAnim(); el("seal").hidden = true; el("next").hidden = true; if (PZ) PZ.feet = null; path = cells.map(([r, c]) => key(r, c)); routeChanged("cut"); },
  analyzeFor(i, cells) { if (idx !== i || view !== "puzzle") load(i); return analyze(cells.map(([r, c]) => key(r, c))); },
  randomWalk(i, seed, maxLen) {   // a random legal route (by the page's own rules), for testing the abbot's search
    if (idx !== i || view !== "puzzle") load(i);
    const R = rng(seed), S = newState(); push(S, G.gateCell);
    while (S.len < maxLen) {
      const v = S.P[S.len - 1], opts = [];
      for (let d = 0; d < 4; d++) { const w = G.nbr[v * 4 + d]; if (w >= 0 && check(S, w) === null) opts.push(w); }
      if (!opts.length) break;
      push(S, opts[Math.floor(R() * opts.length)]);
    }
    return Array.from(S.P.slice(0, S.len)).map(k => rc(k));
  },
  practice: () => ({ stones: PR.stones.length, sum: PR.F ? PR.F.hf.reduce((a, b) => a + Math.abs(b), 0) : 0 }),
  sand: () => PZ ? PZ.hf.reduce((a, b) => a + Math.abs(b), 0) : 0,
  share: { code: (cells, grey) => rakingCode(cells ? cells.map(([r, c]) => key(r, c)) : path, grey), progress: progressCode, read: readRaking, readProgress, importProgress,
    grid: cells => gridText(cells ? cells.map(([r, c]) => key(r, c)) : path), text: (cells, kind) => shareText(GARDENS[idx], cells ? cells.map(([r, c]) => key(r, c)) : path, kind),
    get shared() { return shared; }, walk: cells => { const n = walkEncode(cells.map(([r, c]) => key(r, c))); return n === null ? null : [n.toString(), bitLen(n)]; },
    unwalk: (n, steps) => { const P = walkDecode(BigInt(n), steps); return P && P.map(k => rc(k)); }, applyHash, leaveShared,
    layouts: () => ({ campaign: GARDENS.slice(0, NCAMP).map(layoutHash), daily: DAILY.map(layoutHash) }),
    swap(i, j) { [GARDENS[i], GARDENS[j]] = [GARDENS[j], GARDENS[i]]; LAYOUTS = null; renderNav(); } },
};
</script>
proto/test/crisp.py · 27 lines · 1.7 kB — Zoomed crops of finished rakings (device pixels, 1:1) and frame timing during the reveal. 
"""Zoomed crops of finished rakings (device pixels, 1:1) and frame timing during the reveal."""
import asyncio, json, sys
from playwright.async_api import async_playwright
G = json.load(open('/home/claude/zen/out/proto_gardens.json'))
URL = 'file:///home/claude/zen/proto/test/index.html'
async def run(b, name, w, dpr, tag):
    page = await (await b.new_context(viewport={'width': w, 'height': 1000}, device_scale_factor=dpr)).new_page()
    await page.goto(URL); await page.wait_for_timeout(300)
    i = next(k for k, g in enumerate(G) if g['name'] == name)
    await page.evaluate(f"window.__samon.load({i})")
    res = await page.evaluate("window.__samon.G && document.getElementById('garden').width")
    await page.evaluate("(w) => window.__samon.setPath(w)", G[i]['intended'])
    # frame timing during the reveal
    times = await page.evaluate("""() => new Promise(done => { window.__samon.rakeNow(); const ts = []; let last = performance.now();
        const f = now => { ts.push(now - last); last = now; if (ts.length < 120) requestAnimationFrame(f); else done(ts); }; requestAnimationFrame(f); })""")
    await page.wait_for_timeout(7000)
    el = await page.query_selector('#garden')
    await el.screenshot(path=f'/home/claude/zen/proto/test/crisp-{tag}.png')
    ts = sorted(times[1:])
    print(tag, name, 'canvas px', res, f'frame ms median {ts[len(ts)//2]:.1f} p95 {ts[int(len(ts)*0.95)]:.1f} max {ts[-1]:.1f}')
async def main():
    async with async_playwright() as p:
        b = await p.chromium.launch()
        await run(b, 'Spiral', 1200, 2, 'desktop2x')
        await run(b, 'Calm', 390, 3, 'phone3x')
        await b.close()
asyncio.run(main())
proto/test/frames.py · 29 lines · 1.7 kB — Capture frames of the raking reveal (and the walk-in sand) for a visual check. 
"""Capture frames of the raking reveal (and the walk-in sand) for a visual check."""
import asyncio, json, sys
from playwright.async_api import async_playwright
G = json.load(open('/home/claude/zen/out/proto_gardens.json'))
URL = 'file:///home/claude/zen/proto/test/index.html'
NAME = sys.argv[1] if len(sys.argv) > 1 else 'Calm'
SCHEME = sys.argv[2] if len(sys.argv) > 2 else 'light'
async def main():
    async with async_playwright() as p:
        b = await p.chromium.launch()
        page = await (await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2, color_scheme=SCHEME)).new_page()
        await page.goto(URL); await page.wait_for_timeout(300)
        i = next(k for k, g in enumerate(G) if g['name'] == NAME)
        await page.evaluate(f"window.__samon.load({i})")
        cells = G[i]['intended']
        await page.evaluate("(w) => window.__samon.setPath(w)", cells[:len(cells) * 2 // 3])
        el = await page.query_selector('#garden')
        await el.screenshot(path=f'/home/claude/zen/proto/test/frame-{NAME}-{SCHEME}-walk.png')
        await page.evaluate("(w) => window.__samon.setPath(w)", cells)
        await page.evaluate("window.__samon.rakeNow()")
        t = 0
        for k, ms in enumerate([700, 1300, 1600]):
            await page.wait_for_timeout(ms); t += ms
            await el.screenshot(path=f'/home/claude/zen/proto/test/frame-{NAME}-{SCHEME}-{k}.png')
        await page.wait_for_timeout(5000)
        await el.screenshot(path=f'/home/claude/zen/proto/test/frame-{NAME}-{SCHEME}-done.png')
        print(await page.evaluate("[window.__samon.mode, document.getElementById('status').textContent]"))
        await b.close()
asyncio.run(main())
proto/test/fuzz.py · 183 lines · 12.1 kB — Gesture fuzzer for the prototype: random drags, taps, keys, button presses and links (shared rakings, carried progress) for a while, against invariants. 
"""Gesture fuzzer for the prototype: random drags, taps, keys, button presses and links (shared rakings, carried progress) for a while, against invariants.

Invariants, after every action: the walk is a legal chain of sand cells from the gate with no repeats; the mode, view and
the controls agree (Next only when done, Rake only in a finishable daily walk, Restore only with something to restore,
Step back and Start over disabled when the daily raking is final); the status is empty or in editorial brackets; the
saved progress parses and, after a reload, brings the garden and its draft back; the sound engine keeps a bounded number of
timers; no page errors, no console errors.

Usage: python3 proto/test/fuzz.py [seconds=180] [seed=1]
"""
import asyncio, json, random, sys, time
from playwright.async_api import async_playwright

URL = 'file:///home/claude/zen/proto/test/index.html'
SECS = float(sys.argv[1]) if len(sys.argv) > 1 else 180
SEED = int(sys.argv[2]) if len(sys.argv) > 2 else 1
R = random.Random(SEED)

STATE_JS = """() => {
  const s = window.__samon, G = s.G, el = id => document.getElementById(id);
  const vis = id => { const e = el(id); return !!e && !e.hidden && getComputedStyle(e).display !== 'none'; };
  return { mode: s.mode, view: s.view, path: s.path.slice(), W: G.W, H: G.H, rows: G.rows, gate: G.gateCell, nSand: G.nSand, daily: !!G.daily,
    status: el('status').textContent, next: vis('next'), rakeDay: vis('rakeDay'), restore: vis('restore'), undoOff: el('undo').disabled, restartOff: el('restart').disabled,
    timers: s.sound.timers, soundOn: s.sound.on, actions: [...document.querySelectorAll('#actions button')].map(b => b.textContent),
    poem: s.poem.state, demo: s.demoRunning, idx: window.__samon.G.name, shared: !!s.share.shared, hash: location.hash };
}"""


def legal(st):
    P, W, rows = st['path'], st['W'], st['rows']
    if not P:
        return True, ''
    if P[0] != st['gate']:
        return False, f'walk starts at {P[0]}, not the gate {st["gate"]}'
    if len(set(P)) != len(P):
        return False, 'walk repeats a cell'
    for i, k in enumerate(P):
        r, c = divmod(k, W)
        if rows[r][c] in '#OR':
            return False, f'walk stands on {rows[r][c]!r} at {(r, c)}'
        if i and abs(k // W - P[i - 1] // W) + abs(k % W - P[i - 1] % W) != 1:
            return False, f'walk jumps from {P[i - 1]} to {k}'
    return True, ''


def consistent(st):
    if st['mode'] not in ('draw', 'reveal', 'done', 'demo'):
        return f'mode {st["mode"]!r}'
    if st['view'] not in ('puzzle', 'practice', 'poems'):
        return f'view {st["view"]!r}'
    if st['view'] == 'puzzle':
        if st['status'] and not (st['status'].startswith('[') and st['status'].endswith(']')):
            return f'status not bracketed: {st["status"][:60]!r}'
        if st['next'] and st['mode'] != 'done':
            return 'Next shown outside done'
        if st['rakeDay'] and not (st['daily'] and st['mode'] == 'draw' and len(st['path']) == st['nSand']):
            return 'Rake shown when the daily walk is not complete'
        if st['mode'] == 'done' and len(st['path']) != st['nSand']:
            return f'done with {len(st["path"])} of {st["nSand"]} cells'
        if st['daily'] and st['mode'] == 'done' and not st['shared'] and not (st['undoOff'] and st['restartOff']):
            return 'daily raking final but Step back / Start over enabled'
        if st['shared'] and st['mode'] != 'done':
            return f'a shared raking on show but mode {st["mode"]!r}'
        if st['hash'] and not st['shared'] and not st['hash'].startswith('#p='):
            return f'a hash left over: {st["hash"][:30]!r}'
    if st['timers'] > 60:
        return f'{st["timers"]} sound timers'
    return ''


async def main():
    fails, log = [], []
    async with async_playwright() as p:
        b = await p.chromium.launch()
        ctx = await b.new_context(viewport={'width': 420, 'height': 900}, device_scale_factor=1)
        page = await ctx.new_page()
        errors = []
        page.on('pageerror', lambda e: errors.append('pageerror: ' + str(e)))
        page.on('console', lambda m: errors.append('console: ' + m.text) if m.type == 'error' and 'Failed to load resource' not in m.text else None)
        await page.goto(URL); await page.wait_for_timeout(300)
        await page.evaluate("localStorage.setItem('samon-open-all', '1'); localStorage.setItem('samon-started', '1'); localStorage.setItem('samon-sound-hint', '1')")
        await page.reload(); await page.wait_for_timeout(300)
        await page.evaluate("window.__samon.poemTiming = {delay: 150, hold: 700, fade: 120}")
        ngardens = await page.evaluate("window.__samon.share.layouts().campaign.length")

        async def rect():
            return await page.evaluate("(() => { const r = document.getElementById('garden').getBoundingClientRect(); return [r.left, r.top, r.width, r.height]; })()")

        async def check(what):
            st = await page.evaluate(STATE_JS)
            ok, why = legal(st)
            bad = ('' if ok else why) or consistent(st)
            if bad:
                fails.append((what, bad, st['idx'], st['mode'], st['view']))
                print('FAIL', what, '->', bad, '| garden', st['idx'], st['mode'], st['view'])
                print('   last actions:', ' | '.join(log[-12:]))
            if errors:
                fails.append((what, errors[:]))
                print('FAIL', what, '-> errors', errors[:3]); errors.clear()
            return st

        t0 = time.time(); n = 0
        while time.time() - t0 < SECS:
            n += 1
            kind = R.choices(['drag', 'tap', 'key', 'button', 'pip', 'pill', 'reload', 'wait', 'link'], weights=[34, 14, 16, 14, 7, 5, 3, 7, 4])[0]
            try:
                if kind == 'drag':
                    L, T, W, H = await rect()
                    x, y = L + R.uniform(-0.1, 1.1) * W, T + R.uniform(-0.1, 1.1) * H
                    await page.mouse.move(x, y); await page.mouse.down()
                    vx, vy = R.uniform(-60, 60), R.uniform(-60, 60)
                    for _ in range(R.randint(1, 14)):
                        if R.random() < 0.3: vx, vy = R.uniform(-80, 80), R.uniform(-80, 80)
                        x = min(max(x + vx + R.uniform(-8, 8), L - 60), L + W + 60); y = min(max(y + vy + R.uniform(-8, 8), T - 60), T + H + 60)
                        await page.mouse.move(x, y, steps=R.randint(1, 4))
                    await page.mouse.up()
                    log.append(f'drag@{x - L:.0f},{y - T:.0f}')
                elif kind == 'tap':
                    L, T, W, H = await rect()
                    x, y = L + R.uniform(-0.05, 1.05) * W, T + R.uniform(-0.05, 1.05) * H
                    await page.mouse.click(x, y); log.append(f'tap@{x - L:.0f},{y - T:.0f}')
                elif kind == 'key':
                    k = R.choice(['ArrowUp', 'ArrowDown', 'ArrowLeft', 'ArrowRight', 'ArrowUp', 'ArrowRight', 'Backspace', 'r', 'h', 'Enter', 'Escape', 'm'])
                    await page.focus('#garden'); await page.keyboard.press(k); log.append('key ' + k)
                elif kind == 'button':
                    SEL = '.veranda button, #actions button, .assists button, #mute, #soundMore, #soundPanel button, #nav .fold, .help summary, #orig, .poem .card'
                    ids = await page.evaluate(f"[...document.querySelectorAll('{SEL}')].filter(e => !e.hidden && !e.disabled && e.offsetParent !== null).map((e, i) => e.id || (e.className + '#' + i))")
                    if ids:
                        pick = R.randrange(len(ids)); which = ids[pick]
                        els = await page.query_selector_all(SEL)
                        vis = [e for e in els if await e.is_visible() and await e.is_enabled()]
                        if vis:
                            e = vis[min(pick, len(vis) - 1)]
                            await e.click(timeout=2000); log.append('button ' + which)
                elif kind == 'pip':
                    pips = await page.query_selector_all('#nav .pip:not(:disabled)')
                    if pips:
                        e = R.choice(pips); await e.scroll_into_view_if_needed(); await e.click(timeout=2000); log.append('pip')
                elif kind == 'pill':
                    pills = await page.query_selector_all('#nav .pill')
                    if pills:
                        e = R.choice(pills); await e.scroll_into_view_if_needed(); await e.click(timeout=2000); log.append('pill ' + (await e.text_content() or '')[:14])
                        if R.random() < 0.5:
                            await page.evaluate("window.__samon.G && window.__samon.load(window.__samon.G.daily ? 0 : Math.floor(Math.random() * 20))")
                elif kind == 'reload':
                    before = await page.evaluate(STATE_JS)
                    await page.reload(); await page.wait_for_timeout(400)
                    await page.evaluate("window.__samon.poemTiming = {delay: 150, hold: 700, fade: 120}")
                    after = await page.evaluate(STATE_JS)
                    if before['view'] == 'puzzle' and before['mode'] == 'draw' and len(before['path']) > 1 and after['view'] == 'puzzle' and after['idx'] == before['idx'] and after['path'] != before['path'] and not after['demo']:   # (a walk of one footprint, the gate, is not kept: it is no walk)
                        fails.append(('reload', f'draft lost: {len(before["path"])} -> {len(after["path"])}', before['idx']))
                        print('FAIL reload -> draft lost', before['idx'], len(before['path']), len(after['path']))
                    log.append('reload')
                elif kind == 'link':   # a shared raking, today's garden, or carried progress, by its hash
                    what = R.choice(['raking', 'raking', 'daily', 'progress', 'bad'])
                    if what == 'raking':
                        gi = R.randrange(ngardens)
                        code = await page.evaluate(f"(() => {{ const s = window.__samon, here = s.G.name; s.load({gi}); const c = s.share.code(s.G.intended); return c; }})()")
                        await page.evaluate(f"location.hash = '#r={code}'"); await page.wait_for_timeout(300); log.append(f'link raking {gi}')
                    elif what == 'daily':
                        await page.evaluate("location.hash = '#daily'"); await page.wait_for_timeout(300); log.append('link daily')
                    elif what == 'progress':
                        code = await page.evaluate("window.__samon.share.progress()")
                        await page.evaluate(f"location.hash = '#p={code}'"); await page.wait_for_timeout(300); log.append(f'link progress ({len(code)})')
                        if R.random() < 0.7:
                            btn = await page.query_selector('#actions button:has-text("Take it over")') if R.random() < 0.5 else await page.query_selector('#actions button:has-text("Leave it")')
                            if btn: await btn.click(timeout=2000); log.append('link answered')
                    else:
                        await page.evaluate("location.hash = '#r=' + Math.random().toString(36).slice(2)"); await page.wait_for_timeout(300); log.append('link bad')
                else:
                    await page.wait_for_timeout(R.randint(100, 900)); log.append('wait')
            except Exception as e:   # a click that could not land (an element gone mid-way) is not a page failure
                log.append(f'({kind}: {type(e).__name__})')
            await check(f'#{n} {kind}')
            if len(log) > 60: del log[:-60]
        prog = await page.evaluate("(() => { try { return JSON.stringify(JSON.parse(localStorage.getItem('samon-progress') || '{}')).length; } catch (e) { return -1; } })()")
        if prog < 0:
            fails.append(('progress', 'does not parse'))
        print(f'{n} actions in {time.time() - t0:.0f} s, progress {prog} chars, gardens {ngardens}; fails: {len(fails)}')
        await b.close()
    sys.exit(1 if fails else 0)

asyncio.run(main())
proto/test/grey_shot.py · 16 lines · 991 bytes 
import asyncio, json
from playwright.async_api import async_playwright
G = json.load(open('/home/claude/zen/out/proto_gardens.json'))
async def main():
    async with async_playwright() as p:
        b = await p.chromium.launch()
        page = await (await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2)).new_page()
        await page.goto('file:///home/claude/zen/proto/test/index.html'); await page.wait_for_timeout(300)
        i = next(k for k, g in enumerate(G) if g['name'] == 'Spiral')
        await page.evaluate(f"window.__samon.load({i})")
        await page.click('#hint'); await page.click('#hint')
        await page.evaluate("(w) => window.__samon.setPath(w)", G[i]['intended'])
        await page.evaluate("window.__samon.rakeNow()"); await page.wait_for_timeout(7000)
        await page.screenshot(path='/home/claude/zen/proto/test/shot-grey.png', clip={'x': 0, 'y': 0, 'width': 390, 'height': 760})
        await b.close()
asyncio.run(main())
proto/test/gt_check.py · 89 lines · 5.3 kB — Check the page's own hint analysis against the C solver on random legal routes. 
"""Check the page's own hint analysis against the C solver on random legal routes.
For each route: can it still reach par? can it still be finished? if not, which is the nearest footprint from which
it can? (and is the par rewind point exact?) Ground truth: the C solver, with the route fixed by exact groove clues."""
import asyncio, json, random, sys, time
sys.path.insert(0, '/home/claude/zen/src')
from zg import Garden
from playwright.async_api import async_playwright

URL = 'file:///home/claude/zen/proto/test/index.html'
P = json.load(open('/home/claude/zen/out/proto_gardens.json'))
DIRS = {(-1, 0): 'N', (0, 1): 'E', (1, 0): 'S', (0, -1): 'W'}
N_WALKS = int(sys.argv[1]) if len(sys.argv) > 1 else 40

def garden_of(p, extra=None):
    clues = {tuple(map(int, k.split(','))): v for k, v in (p['clues'] or {}).items()}
    clues.update(extra or {})
    marks = {tuple(map(int, k.split(','))): v for k, v in (p.get('marks') or {}).items()}
    return Garden(p['rows'], tuple(p['gate']), clues=clues, marks=marks, end=tuple(p['end']) if p['end'] else None,
                  nohairpin=2 if p['wide'] else 0, braced_start=p['wide'], end_side=p['end_side'])

def d(a, b): return DIRS[(b[0] - a[0], b[1] - a[1])]

def prefix_clues(p, cells):
    out = {}
    for i in range(len(cells) - 1):
        ds = {p['gate'][2] if i == 0 else d(cells[i], cells[i - 1]), d(cells[i], cells[i + 1])}
        out[cells[i]] = ''.join(sorted(ds))
    return out

cache = {}
def truth(p, cells, kind):
    k = (p['name'], tuple(cells), kind)
    if k not in cache:
        g = garden_of(p, prefix_clues(p, cells))
        r = g.solve(mode=0, cap=1) if kind == 'comp' else g.solve(mode=2, bound=p['par'], cap=1)
        assert r['complete'] or r['capped'], 'solver incomplete'
        cache[k] = r['nsol'] >= 1
    return cache[k]

async def main():
    # the flags the page assumes reproduce the exported counts
    for p in P:
        r = garden_of(p).solve(mode=0, cap=10**7)
        assert r['nsol'] == p['rakings'], (p['name'], r['nsol'], p['rakings'])
    async with async_playwright() as pw:
        b = await pw.chromium.launch()
        page = await (await b.new_context(viewport={'width': 390, 'height': 900})).new_page()
        await page.goto(URL); await page.wait_for_timeout(300)
        stats = {'par': 0, 'finish': 0, 'stuck': 0, 'unknown': 0}; bad = 0; tmax = 0; proven_back = 0; unproven_back = 0
        unknown_in, unproven_in, slowest = [], [], ''
        rng = random.Random(1)
        for i, p in enumerate(P):
            nsand = sum(ch in '.hv=|' for row in p['rows'] for ch in row)
            sols = garden_of(p).solve(mode=0, cap=3000, stream=True)['sols']   # prefixes of real rakings: never stuck
            for w in range(2 * N_WALKS):
                if w % 2 == 0:
                    maxlen = rng.randint(2, nsand)
                    cells = await page.evaluate(f"window.__samon.randomWalk({i}, {rng.randrange(10**9)}, {maxlen})")
                    cells = [tuple(c) for c in cells]
                else:
                    s_ = rng.choice(sols)['draw']
                    cells = [tuple(c) for c in s_[:rng.randint(1, len(s_))]]
                t0 = time.time()
                a = await page.evaluate("""([i, cells]) => { const t = performance.now(); const a = window.__samon.analyzeFor(i, cells);
                    return {state: a.state, lcp: a.lcp, back: a.back ?? null, proven: a.proven ?? null, ms: performance.now() - t}; }""", [i, cells])
                if a['ms'] > tmax: tmax, slowest = a['ms'], p['name']
                stats[a['state']] += 1
                if a['state'] == 'unknown': unknown_in.append(p['name'])
                n = len(cells)
                gpar, gcomp = truth(p, cells, 'par'), truth(p, cells, 'comp')
                ok = True
                # the par rewind point is exact: cells[:lcp] can reach par, cells[:lcp+1] cannot (when lcp < n)
                ok &= truth(p, cells[:a['lcp']], 'par') and (a['lcp'] == n or not truth(p, cells[:a['lcp'] + 1], 'par'))
                if a['state'] == 'par': ok &= gpar
                else: ok &= not gpar
                if a['state'] == 'finish': ok &= gcomp
                if a['state'] == 'stuck':
                    ok &= not gcomp
                    gback = next(k for k in range(n - 1, 0, -1) if truth(p, cells[:k], 'comp'))
                    if a['proven']: ok &= a['back'] == gback; proven_back += 1
                    else: ok &= truth(p, cells[:a['back']], 'comp') and a['back'] <= gback; unproven_back += 1; unproven_in.append(p['name'])
                if not ok:
                    bad += 1; print('BAD', p['name'], cells, a, 'truth par/comp', gpar, gcomp)
        print('states', stats, 'bad', bad, f'max analysis time {tmax:.0f} ms ({slowest})', 'stuck rewinds proven/unproven', proven_back, unproven_back)
        print('unknown in', unknown_in, 'unproven rewinds in', sorted(set(unproven_in)))
        json.dump({'routes': sum(stats.values()), 'states': stats, 'bad': bad, 'max_ms': round(tmax), 'proven_rewinds': proven_back,
                   'unproven_rewinds': unproven_back, 'slowest': slowest, 'unknown_in': unknown_in, 'unproven_in': unproven_in}, open('/home/claude/zen/out/hint_check.json', 'w'), indent=1)
        await b.close()
asyncio.run(main())
proto/test/run.py · 896 lines · 91.1 kB — Browser test of the prototype: every garden raked at par with real mouse drags (raked on letting go); refusals; 
"""Browser test of the prototype: every garden raked at par with real mouse drags (raked on letting go); refusals;
Restore after a rewind and after Start over; the abbot's hints (first hint, sketched groove, 'no longer at par' with its
rewind, 'stuck' with its rewind); the grey seal; unlocking; assists; the poems and the scroll of poems earned; the daily
garden; drag intent (drift, the grip beyond the walls, arcs, reversals, wobble); sound; sharing (a raking as a code in the URL's hash,
progress carried between devices, the share text); the practice garden; screenshots in light and dark."""
import asyncio, json, math, os, sys
from playwright.async_api import async_playwright
sys.path.insert(0, '/home/claude/zen/src')
from zg import Garden, check_solution, turns_of

G = json.load(open('/home/claude/zen/out/proto_gardens.json'))
URL = 'file:///home/claude/zen/proto/test/index.html'
SHOT = '/home/claude/zen/proto/test/'
fails = []

def garden_of(p, braced=True):
    marks = {tuple(map(int, k.split(','))): v for k, v in (p.get('marks') or {}).items()}
    clues = {tuple(map(int, k.split(','))): v for k, v in (p['clues'] or {}).items()}
    return Garden(p['rows'], tuple(p['gate']), clues=clues, marks=marks, end=tuple(p['end']) if p['end'] else None,
                  nohairpin=2 if p['wide'] else 0, braced_start=p['wide'] and braced, end_side=p['end_side'])
def expect(cond, what):
    print(('ok   ' if cond else 'FAIL ') + what)
    if not cond: fails.append(what)

async def centers(page, cells):
    return await page.evaluate("""(path) => { const cv = document.getElementById('garden'); const r = cv.getBoundingClientRect();
        const s = r.width / cv.width; const g = window.__samon.geom();
        return path.map(([rr, cc]) => [r.left + (g.ox + cc * g.cs + g.cs / 2) * s, r.top + (g.oy + rr * g.cs + g.cs / 2) * s]); }""", cells)

async def drag(page, cells):
    pts = await centers(page, cells)
    await page.mouse.move(*pts[0]); await page.mouse.down()
    for (x, y) in pts[1:]: await page.mouse.move(x, y, steps=3)
    await page.mouse.up()

async def quiet(page, **t):   # poems off (they would carry the page on to the next garden), unless a test wants them
    await page.evaluate("(t) => { window.__samon.poemTiming = t; }", t or {'delay': 3600000})

async def state(page):
    return await page.evaluate("() => ({mode: window.__samon.mode, len: window.__samon.path.length, turns: window.__samon.turnsOf(), "
                               "status: document.getElementById('status').textContent.replace(/^\\[|\\]$/g, ''), "
                               "restore: !document.getElementById('restore').hidden, seal: getComputedStyle(document.getElementById('seal')).display, "
                               "ghosts: window.__samon.ghosts.size, actions: [...document.querySelectorAll('#actions button')].map(b => b.textContent)})")

async def main():
    async with async_playwright() as p:
        b = await p.chromium.launch()
        errors = []
        ctx = await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2, color_scheme='light')
        page = await ctx.new_page()
        page.on('console', lambda m: errors.append(m.text) if m.type == 'error' else None)
        page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(300)
        await quiet(page)
        st = await state(page)
        expect(st['seal'] == 'none' and not st['restore'] and not await page.query_selector('#rake'), 'seal and Restore hidden at load; no Rake button')

        # 0. a first visit: the first garden is raked once by the monk (walking in, then raking out) and reset for the player;
        # then the monk waits in the gateway, a ring beckons, and a ghostly finger shows the drag
        onboard = """() => ({beckon: !document.getElementById('beckon').hidden, finger: !document.getElementById('finger').hidden,
            anim: document.getElementById('finger').getAnimations().length, status: document.getElementById('status').textContent.replace(/^\\[|\\]$/g, ''),
            started: localStorage.getItem('samon-started')})"""
        g0 = G[0]
        expect(g0['demo'] and G[1]['demo'] and not G[2]['demo'], 'the first two gardens have demonstrations, the first real garden (third) none')
        await page.wait_for_function("window.__samon.mode === 'demo'", timeout=4000)
        st = await state(page); ob = await page.evaluate(onboard)
        expect(st['status'].startswith('Watch') and not ob['beckon'], f"the first garden demonstrates itself, and nothing beckons meanwhile ({st['status']!r})")
        await page.wait_for_function("window.__samon.mode === 'reveal'", timeout=8000)
        walked = await page.evaluate("window.__samon.path.slice()")
        want = await page.evaluate("(cells) => cells.map(([r, c]) => window.__samon.key(r, c))", g0['intended'])
        expect(walked == want, 'the demonstration walks the intended raking, then rakes it')
        await page.wait_for_function("!window.__samon.demoRunning", timeout=20000)
        st = await state(page); ob = await page.evaluate(onboard)
        pill0 = await page.evaluate(f"[...document.querySelectorAll('#nav .pip')].find(b => b.getAttribute('aria-label') === {json.dumps(g0['name'])}).className")
        expect(st['len'] == 0 and st['mode'] == 'draw' and 'Your turn' in st['status'] and st['seal'] == 'none' and 'done' not in pill0 and 'sealed' not in pill0,
               f"then the garden is reset for the player, and the demonstration counts for nothing ({st['status']!r}, {pill0!r})")
        expect(ob['beckon'] and ob['finger'] and ob['anim'] == 1, f'then the waiting monk beckons, and a finger shows the drag {ob}')
        br = await page.evaluate("[document.getElementById('status').textContent, document.querySelector('.help.puzzle-only').textContent.trim(), document.querySelector('#nav .lock .note').textContent, document.getElementById('rule').firstChild.textContent, document.querySelector('#nav .lock .n').textContent, [...document.querySelectorAll('#actions button, .veranda button')].map(b => b.textContent), document.getElementById('par').parentElement.title]")
        expect(all(t.startswith('[') and t.endswith(']') for t in br[:3]) and br[3].startswith('[') and not br[4].startswith('[') and not any(t.startswith('[') for t in br[5]) and 'valid raking' in br[6] and f"Par {g0['par']}" in br[6],
               f"what the game says to the player is set in editorial brackets (status, help, lock notes, a garden's instructions), and rules, counts and controls are not; the counts of rakings live in Par's tooltip ({br[0][:40]!r}, {br[3][:30]!r}, {br[4][:20]!r})")
        await page.reload(); await page.wait_for_timeout(1500)
        await quiet(page)
        st = await state(page); ob = await page.evaluate(onboard)
        expect(not await page.evaluate("window.__samon.demoRunning") and st['len'] == 0 and 'Drag the monk in' in st['status'] and ob['beckon'] and ob['finger'],
               f"a demonstration seen once does not play again by itself ({st['status']!r})")
        wv = await page.evaluate("!document.getElementById('watch').hidden")
        await page.click('#watch'); await page.wait_for_timeout(700)
        running = await page.evaluate("window.__samon.demoRunning")
        mid = await centers(page, [g0['intended'][1]]); await page.mouse.click(*mid[0]); await page.wait_for_timeout(100)
        st = await state(page)
        expect(wv and running and not await page.evaluate("window.__samon.demoRunning") and st['len'] == 0 and st['mode'] == 'draw' and 'Your turn' in st['status'],
               'Watch replays the demonstration, and a tap skips it')
        await page.evaluate("window.__samon.load(2)")
        expect(await page.evaluate("document.getElementById('watch').hidden"), 'a garden without a demonstration has no Watch button')
        await page.evaluate("window.__samon.load(0)")
        far = await centers(page, [[0, len(g0['rows'][0]) - 1]])
        await page.mouse.click(*far[0]); st = await state(page)
        expect('Start at the gate' in st['status'] and st['len'] == 0, 'a press away from the gate says where to start')
        gp = await page.evaluate("(() => { const cv = document.getElementById('garden'); const r = cv.getBoundingClientRect(); const s = r.width / cv.width; const [x, y] = window.__samon.gate(); return [r.left + x * s, r.top + y * s]; })()")
        pts = await centers(page, g0['intended'][:3])
        await page.mouse.move(*gp); await page.mouse.down()
        for (x, y) in pts: await page.mouse.move(x, y, steps=3)
        await page.mouse.up(); st = await state(page); ob = await page.evaluate(onboard)
        expect(st['len'] == 3 and not ob['beckon'] and not ob['finger'] and ob['started'] == '1', f'dragging the monk in from the gateway starts the walk, and the beckoning stops {ob}')
        await page.screenshot(path=SHOT + 'shot-phone-light-start.png', full_page=False)
        await page.evaluate("window.__samon.load(2)"); ob = await page.evaluate(onboard)
        expect(ob['beckon'] and not ob['finger'], f'after the first walk, fresh gardens still beckon, without the finger {ob}')

        # 1. every garden at par, by mouse: letting go of a complete walk rakes it
        for i, g in enumerate(G if not os.environ.get('SAMON_QUICK') else G[:3]):
            await page.evaluate(f"window.__samon.load({i})")
            await drag(page, g['intended'])
            await page.wait_for_timeout(350)
            st = await state(page)
            ok = st['len'] == len(g['intended']) and st['turns'] == g['par'] and st['mode'] == 'reveal'
            pts = await centers(page, [g['intended'][0]]); await page.mouse.click(*pts[0])   # skip the reveal
            await page.wait_for_timeout(60)
            st2 = await state(page)
            expect(ok and st2['mode'] == 'done' and st2['seal'] == 'grid' and 'par' in st2['status'], f"{g['name']}: drawn, raked on letting go, at par, sealed")

        # 2. refusals: wide rake U-turn; pebbles
        wi = next(i for i, g in enumerate(G) if g['name'] == 'Wide rake')
        await page.evaluate(f"window.__samon.load({wi})")
        await drag(page, [[6, 5], [5, 5], [4, 5], [4, 6], [5, 6]])
        st = await state(page)
        expect('wide rake' in st['status'] and st['len'] == 4, 'wide rake refuses an open-sand U-turn')
        pi = next(i for i, g in enumerate(G) if g['name'] == "The abbot's pebbles")
        pg = G[pi]; marks = {tuple(map(int, k.split(','))): v for k, v in pg['marks'].items()}
        rows = pg['rows']; seq = [tuple(c) for c in pg['intended']]
        j = next(k for k in range(1, len(seq) - 1) if marks.get(seq[k]) == 'S'
                 and any(0 <= seq[k][0] + a < len(rows) and 0 <= seq[k][1] + bb < len(rows[0]) and rows[seq[k][0] + a][seq[k][1] + bb] == '.'
                         and (seq[k][0] + a, seq[k][1] + bb) not in seq[:k + 1]
                         and (a, bb) != (seq[k][0] - seq[k - 1][0], seq[k][1] - seq[k - 1][1])
                         and (a, bb) != (seq[k - 1][0] - seq[k][0], seq[k - 1][1] - seq[k][1]) for a, bb in ((-1, 0), (1, 0), (0, -1), (0, 1))))
        v = seq[j]; din = (v[0] - seq[j - 1][0], v[1] - seq[j - 1][1])
        turn = next((v[0] + a, v[1] + bb) for a, bb in ((-1, 0), (1, 0), (0, -1), (0, 1))
                    if (a, bb) != din and (a, bb) != (-din[0], -din[1]) and 0 <= v[0] + a < len(rows) and 0 <= v[1] + bb < len(rows[0])
                    and rows[v[0] + a][v[1] + bb] == '.' and (v[0] + a, v[1] + bb) not in seq[:j + 1])
        await page.evaluate(f"window.__samon.load({pi})")
        await drag(page, [list(c) for c in seq[:j + 1]] + [list(turn)])
        st = await state(page)
        expect('pale pebble' in st['status'] and st['len'] == j + 1, 'a pale pebble refuses a turn')

        # 3. Restore after a tap-rewind and after Start over
        ci = next(i for i, g in enumerate(G) if g['name'] == 'Calm')
        await page.evaluate(f"window.__samon.load({ci})")
        await page.click('#restart')
        await drag(page, G[ci]['intended'][:12])
        pts = await centers(page, [G[ci]['intended'][3]]); await page.mouse.click(*pts[0])
        st = await state(page)
        expect(st['len'] == 4 and st['restore'], 'tapping a footprint rewinds to it, and offers Restore')
        await page.click('#restore'); st = await state(page)
        expect(st['len'] == 12 and not st['restore'], 'Restore brings the rewound route back')
        await page.click('#restart'); st = await state(page)
        expect(st['len'] == 0 and st['restore'], 'Start over clears the route, and offers Restore')
        await page.click('#restore'); st = await state(page)
        expect(st['len'] == 12, 'Restore after Start over')

        # 3b. after raking: a tap on a cell walks back to it (the sand is smoothed back to the footprints), Restore rakes it again;
        # holding at the last cell does not rake until the mouse lets go; the keyboard rakes after a beat
        await page.evaluate(f"window.__samon.load({ci})"); await page.click('#restart')
        full = G[ci]['intended']
        pts = await centers(page, full)
        await page.mouse.move(*pts[0]); await page.mouse.down()
        for (x, y) in pts[1:]: await page.mouse.move(x, y, steps=3)
        await page.wait_for_timeout(700); st = await state(page)
        expect(st['mode'] == 'draw' and st['len'] == len(full), 'a complete walk is not raked while the mouse is still down')
        await page.mouse.up(); await page.wait_for_timeout(350); st = await state(page)
        expect(st['mode'] == 'reveal', 'letting go starts the raking')
        await page.mouse.click(*pts[0]); await page.wait_for_timeout(700)
        k = len(full) - 5
        await page.mouse.click(*pts[k - 1]); await page.wait_for_timeout(100); st = await state(page)
        ref = await page.evaluate("window.__samon.sand()")
        expect(st['mode'] == 'draw' and st['len'] == k and st['seal'] == 'none' and st['restore'], 'after raking, tapping a cell walks back to it, and offers Restore')
        await page.evaluate("(w) => window.__samon.setPath(w)", full[:k]); fresh = await page.evaluate("window.__samon.sand()")
        expect(abs(ref - fresh) < 1e-6, 'walking back smooths the raked sand back to the footprints')
        await page.evaluate("(w) => window.__samon.setPath(w)", full)
        await page.wait_for_timeout(600); st = await state(page)
        expect(st['mode'] in ('reveal', 'done') and st['len'] == len(full), 'a walk completed any other way is raked after a beat')
        await page.focus('#garden'); await page.keyboard.press('Enter'); await page.wait_for_timeout(60); st = await state(page)
        expect(st['mode'] == 'done', 'Enter skips the raking')
        await page.focus('#garden'); await page.keyboard.press('Backspace'); await page.wait_for_timeout(60); st = await state(page)
        expect(st['mode'] == 'draw' and st['len'] == len(full) - 1, 'Backspace after raking steps back one footprint')
        await page.click('#restore'); await page.wait_for_timeout(600); st = await state(page)
        expect(st['mode'] in ('reveal', 'done') and st['len'] == len(full), 'Restore brings the whole walk back, and it is raked again')
        await page.keyboard.press('Enter'); await page.wait_for_timeout(60)
        # the keyboard: the last arrow rakes after a beat, in which Backspace can still take it back
        ti = next(i for i, g in enumerate(G) if g['name'] == 'Nine squares')
        await page.evaluate(f"window.__samon.load({ti})"); await page.focus('#garden')
        keys = ('ArrowUp', 'ArrowUp', 'ArrowUp', 'ArrowRight', 'ArrowDown', 'ArrowDown', 'ArrowRight', 'ArrowUp', 'ArrowUp')   # in at the gate, then a serpentine
        for kk in keys: await page.keyboard.press(kk)
        await page.keyboard.press('Backspace'); await page.wait_for_timeout(600); st = await state(page)
        expect(st['mode'] == 'draw' and st['len'] == 8, 'Backspace in the beat takes the last step back, and nothing is raked')
        await page.keyboard.press(keys[-1]); await page.wait_for_timeout(600); st = await state(page)
        expect(st['mode'] in ('reveal', 'done') and st['len'] == 9, 'from the keyboard, a complete walk is raked after a beat')
        await page.keyboard.press('Enter'); await page.wait_for_timeout(60); await page.keyboard.press('Enter'); await page.wait_for_timeout(800)
        nm = await page.evaluate("window.__samon.G.name")
        expect(nm == G[ti + 1]['name'], 'Enter after raking goes on to the next garden')
        # a walk that covers every cell but may not end where it does is not raked
        wi2, bad = None, None
        for j, gg in enumerate(G):
            if not gg['wide'] or gg['end']: continue
            gx = garden_of(gg, braced=False)
            for s_ in gx.solve(mode=0, cap=400, stream=True)['sols']:
                if not check_solution(garden_of(gg), s_['draw']): wi2, bad = j, [list(c) for c in s_['draw']]; break
            if bad: break
        await page.evaluate(f"window.__samon.load({wi2})"); await drag(page, bad); await page.wait_for_timeout(600); st = await state(page)
        expect(st['mode'] == 'draw' and st['len'] == len(bad) and 'wide rake down' in st['status'], f"a complete walk ending in open sand under the wide rake is not raked ({G[wi2]['name']})")

        # 4. the abbot's hints
        await page.evaluate(f"window.__samon.load({ci})")
        await page.click('#restart'); await page.click('#hint'); st = await state(page)
        expect(G[ci]['hint'] in st['status'], 'first hint is the authored observation')
        await page.click('#hint'); st = await state(page)
        expect('at par' in st['status'] and st['ghosts'] == 1, 'second hint: route can reach par, one groove sketched')
        found = await page.evaluate(f"""(() => {{ for (let s = 1; s < 4000; s++) {{ const w = window.__samon.randomWalk({ci}, s, 10 + s % 30);
            const a = window.__samon.analyzeFor({ci}, w); if (a.state === 'finish') return w; }} return null; }})()""")
        await page.evaluate("(w) => window.__samon.setPath(w)", found)
        await page.click('#hint'); st = await state(page)
        expect('no longer at par' in st['status'] and any('for par' in a for a in st['actions']), "'no longer at par' with a rewind offer")
        await page.click('#actions button >> nth=0'); st = await state(page)
        a = await page.evaluate("window.__samon.analyze(window.__samon.path).state")
        expect(a == 'par' and st['restore'], 'the par rewind lands on a route that can reach par, and is restorable')
        fk = await page.evaluate("[document.activeElement && document.activeElement.id, document.getElementById('status').textContent.replace(/^\\[|\\]$/g, '')]")
        n0 = await page.evaluate("window.__samon.path.length"); await page.keyboard.press('ArrowUp'); await page.keyboard.press('ArrowRight'); n1 = await page.evaluate("window.__samon.path.length")
        expect(fk[0] == 'garden' and fk[1].startswith('Stepped back') and 'par is still possible' in fk[1] and 'no longer at par' not in fk[1] and n1 > n0,
               f"after the abbot's rewind the garden has the keyboard again, and the old message has given way to what now holds ({fk[1][:70]!r})")
        found = await page.evaluate(f"""(() => {{ for (let s = 1; s < 4000; s++) {{ const w = window.__samon.randomWalk({ci}, s, 8 + s % 30);
            const a = window.__samon.analyzeFor({ci}, w); if (a.state === 'stuck' && a.back < w.length - 1 && a.why.cells.length) return w; }} return null; }})()""")
        await page.evaluate("(w) => window.__samon.setPath(w)", found)
        await page.click('#hint'); st = await state(page)
        expect('to the marked footprint' in st['status'] and any(a.startswith('Step back') for a in st['actions']), "'stuck': reason, marked cells, rewind offer, action first")
        await page.screenshot(path=SHOT + 'shot-phone-light-stuck.png', full_page=True)
        await page.click('#actions button >> nth=0')
        a = await page.evaluate("window.__samon.analyze(window.__samon.path).state")
        fk = await page.evaluate("[document.activeElement && document.activeElement.id, document.getElementById('status').textContent]")
        expect(a in ('par', 'finish') and fk[0] == 'garden' and 'can be finished from here' in fk[1] and 'marked footprint' not in fk[1], f"the rewind lands on a route that can be finished, says so, and keeps the keyboard ({fk[1][:60]!r})")

        # 5. assists
        await page.evaluate(f"window.__samon.load({ci})"); await page.click('#restart')
        await drag(page, G[ci]['intended'][:20] if False else [[6, 3], [5, 3], [5, 2], [5, 1], [5, 0], [4, 0], [3, 0], [3, 1], [4, 1], [4, 2], [4, 3], [4, 4], [3, 4], [3, 5]])
        for bid in ('#aPreview', '#aChecker', '#aCut'): await page.click(bid)
        pressed = await page.evaluate("['aPreview','aChecker','aCut'].map(id => document.getElementById(id).getAttribute('aria-pressed'))")
        expect(pressed == ['true'] * 3, 'assist toggles switch on')
        await page.screenshot(path=SHOT + 'shot-phone-light-assists.png', full_page=True)
        for bid in ('#aPreview', '#aChecker', '#aCut'): await page.click(bid)

        # 6. practice garden
        await page.click('text=Practice garden'); await page.wait_for_timeout(900)   # the page scrolls the garden into view
        s0 = await page.evaluate("window.__samon.practice()")
        box = await page.evaluate("(() => { const r = document.getElementById('garden').getBoundingClientRect(); return [r.left, r.top, r.width, r.height]; })()")
        L, T, W, H = box
        import math
        for k, (cx, cy, rad) in enumerate([(0.27, 0.40, 0.18), (0.72, 0.65, 0.14)]):
            pts = [(L + W * (cx + rad * math.cos(t / 30 * 2 * math.pi) * 0.75), T + H * (cy + rad * math.sin(t / 30 * 2 * math.pi))) for t in range(0, 34)]
            await page.mouse.move(*pts[0]); await page.mouse.down()
            for (x, y) in pts[1:]: await page.mouse.move(x, y, steps=4)
            await page.mouse.up()
        for yy in (0.1, 0.16, 0.86, 0.92):
            await page.mouse.move(L + W * 0.05, T + H * yy); await page.mouse.down()
            await page.mouse.move(L + W * 0.95, T + H * yy, steps=40); await page.mouse.up()
        s1 = await page.evaluate("window.__samon.practice()")
        expect(abs(s1['sum'] - s0['sum']) > 50, 'raking changes the practice sand')
        await page.screenshot(path=SHOT + 'shot-phone-light-practice.png', full_page=True)
        await page.click('#toolStone'); await page.mouse.click(L + W * 0.5, T + H * 0.2)
        s2 = await page.evaluate("window.__samon.practice()")
        expect(s2['stones'] == s0['stones'] + 1, 'Stones tool places a stone')
        await page.mouse.click(L + W * 0.5, T + H * 0.2)
        s3 = await page.evaluate("window.__samon.practice()")
        expect(s3['stones'] == s0['stones'], 'tapping a stone lifts it')
        await page.click('#smooth'); s4 = await page.evaluate("window.__samon.practice()")
        expect(abs(s4['sum'] - s0['sum']) < 1e-6, 'Smooth restores the unraked sand')
        await page.click('#nav [aria-label="Calm"]'); st = await state(page)
        expect(await page.evaluate("window.__samon.view") == 'puzzle', 'back from the practice garden to a garden')
        await ctx.close()

        # 7. dark mode: a finished garden, and the practice garden
        ctx = await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2, color_scheme='dark')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(200)
        await quiet(page, delay=100, hold=3600000)                   # the screenshot shows the raked garden with its poem
        si = next(i for i, g in enumerate(G) if g['name'] == 'Spiral')
        await page.evaluate(f"window.__samon.load({si})"); await drag(page, G[si]['intended'])
        await page.wait_for_timeout(7200)
        await page.screenshot(path=SHOT + 'shot-phone-dark-done.png', full_page=True)
        await page.evaluate("window.__samon.enterPractice()")
        box = await page.evaluate("(() => { const r = document.getElementById('garden').getBoundingClientRect(); return [r.left, r.top, r.width, r.height]; })()")
        L, T, W, H = box
        for yy in (0.25, 0.42, 0.59, 0.76):
            await page.mouse.move(L + W * 0.06, T + H * yy); await page.mouse.down()
            await page.mouse.move(L + W * 0.94, T + H * (yy + 0.04), steps=40); await page.mouse.up()
        await page.screenshot(path=SHOT + 'shot-phone-dark-practice.png', full_page=True)
        await ctx.close()

        # 7b. the grey seal: help beyond the abbot's first observation marks the raking (fresh profile)
        ctx = await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2, color_scheme='light')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(200)
        await quiet(page)
        ci = next(i for i, g in enumerate(G) if g['name'] == 'Calm')
        ni = next(i for i, g in enumerate(G) if g['name'] == 'Nine squares')
        async def rake_intended(i):
            await drag(page, G[i]['intended']); await page.wait_for_timeout(350)
            pts = await centers(page, [G[i]['intended'][0]]); await page.mouse.click(*pts[0]); await page.wait_for_timeout(60)
            return await page.evaluate("[getComputedStyle(document.getElementById('seal')).display, document.getElementById('seal').classList.contains('grey'), document.getElementById('status').textContent]")
        async def pill_class(name):
            return await page.evaluate(f"[...document.querySelectorAll('#nav .pip')].find(b => b.getAttribute('aria-label') === {json.dumps(name)}).className")
        await page.evaluate(f"window.__samon.load({ni})")
        await page.click('#hint')                                   # the authored observation only: free
        r1 = await rake_intended(ni)
        expect(r1[0] == 'grid' and not r1[1] and 'grey' not in r1[2], 'the first observation alone leaves the seal vermilion')
        await page.evaluate(f"window.__samon.load({ci})")
        await page.click('#hint'); await page.click('#hint')        # observation, then a verdict: help
        st = await state(page)
        expect('grey seal' in st['status'], 'the first real help says the seal will be grey')
        await page.click('#restart')                                # starting over does not undo help
        r2 = await rake_intended(ci)
        expect(r2[0] == 'grid' and r2[1] and 'grey' in r2[2], 'a helped raking at par earns the grey seal, even after Start over')
        expect('grey' in await pill_class('Calm'), 'the garden list shows a grey seal')
        await page.click('#restart')
        r3 = await rake_intended(ci)
        expect(r3[0] == 'grid' and not r3[1] and 'grey' not in r3[2], 'raking it again unaided earns the vermilion seal')
        expect('sealed' in await pill_class('Calm'), 'the garden list now shows the vermilion seal')
        sand0 = await page.evaluate("window.__samon.sand()")
        await page.click('#restart')
        sand1 = await page.evaluate("window.__samon.sand()")
        expect(abs(sand0 - sand1) > 1, 'Start over smooths the raked sand back to unraked')
        # help travels with the route it helped: stepping back into a helped raking and restoring it re-rakes it grey
        ti = next(i for i, g in enumerate(G) if g['name'] == 'The pocket')
        await page.evaluate(f"window.__samon.load({ti})")
        await page.click('#hint'); await page.click('#hint')
        r4 = await rake_intended(ti)
        await page.focus('#garden'); await page.keyboard.press('Backspace'); await page.wait_for_timeout(60)
        h = await page.evaluate("window.__samon.helped")
        async def rerake():                                          # Restore rakes again after a beat; Enter (on the garden) skips it
            await page.click('#restore')
            await page.wait_for_function("['reveal', 'done'].includes(window.__samon.mode)", timeout=3000)
            if await page.evaluate("window.__samon.mode") == 'reveal':
                await page.focus('#garden'); await page.keyboard.press('Enter')
            await page.wait_for_function("window.__samon.mode === 'done'", timeout=3000)
        await rerake()
        r5 = await page.evaluate("[getComputedStyle(document.getElementById('seal')).display, document.getElementById('seal').classList.contains('grey'), document.getElementById('status').textContent]")
        expect(r4[1] and h and r5[0] == 'grid' and r5[1] and 'grey' in r5[2], f'Step back, then Restore, re-rakes a helped raking with the grey seal {r5}')
        tp = await centers(page, [G[ti]['intended'][2]]); await page.wait_for_timeout(700); await page.mouse.click(*tp[0])
        await page.wait_for_function("window.__samon.mode === 'draw'", timeout=3000)
        await rerake()
        r6 = await page.evaluate("document.getElementById('seal').classList.contains('grey')")
        expect(r6, 'so does tapping back into it, then Restore')
        # an answer the abbot can't give in time costs nothing
        await page.evaluate(f"window.__samon.load({ci})")
        found = await page.evaluate(f"""(() => {{ for (let s = 1; s < 4000; s++) {{ const w = window.__samon.randomWalk({ci}, s, 10 + s % 30);
            const a = window.__samon.analyzeFor({ci}, w); if (a.state === 'finish') return w; }} return null; }})()""")
        await page.evaluate("(w) => window.__samon.setPath(w)", found)
        h0 = await page.evaluate("window.__samon.helped")
        await page.evaluate("window.__samon.budget = 1"); await page.click('#hint')
        st = await state(page); h1 = await page.evaluate("window.__samon.helped")
        await page.evaluate("window.__samon.budget = 250000")
        expect(not h0 and "can't be sure" in st['status'] and 'costs you nothing' in st['status'] and not h1, f"an inconclusive answer costs nothing ({st['status']!r})")
        # a walk that has stopped: why is free, where to step back to is help
        dead = await page.evaluate(f"""(() => {{ for (let s = 1; s < 4000; s++) {{ const w = window.__samon.randomWalk({ci}, s, 999);
            if (w.length < window.__samon.G.nSand && w.length > 3) return w; }} return null; }})()""")
        await page.evaluate("(w) => window.__samon.setPath(w)", dead); await page.click('#hint')
        st = await state(page); h2 = await page.evaluate("window.__samon.helped")
        expect('Saying why a walk has stopped is free' in st['status'] and not h2 and st['actions'] == ['Where can I step back to?'],
               f"a stopped walk: the abbot says why for free ({st['status']!r}, {st['actions']})")
        await page.click('#actions button >> nth=0')
        st = await state(page); h3 = await page.evaluate("window.__samon.helped")
        expect(h3 and 'to the marked footprint' in st['status'] and any(a.startswith('Step back') for a in st['actions']), 'asking where to step back to is help')
        await ctx.close()

        # 7c. worlds open as the world before them is raked (two-thirds of it, at any turn count); the prototype can open all
        ctx = await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2, color_scheme='light')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(200)
        await quiet(page)
        w1 = [i for i, g in enumerate(G) if g['world'] == 1]
        w1m = [i for i in w1 if not G[i]['extra']]; need = -(-2 * len(w1m) // 3)
        w1x = len(w1) - len(w1m)
        nx = await page.evaluate("[document.querySelectorAll('#nav .pip').length, document.querySelectorAll('#nav .pip.extra').length, [...document.querySelectorAll('#nav .fold')].map(b => b.textContent)]")
        await page.click('#nav .fold'); await page.wait_for_timeout(100)
        nx2 = await page.evaluate("[document.querySelectorAll('#nav .pip.extra').length, document.querySelectorAll('#nav .more').length, document.querySelector('#nav .fold').textContent]")
        await page.evaluate(f"window.__samon.load({w1[-1]})"); await page.wait_for_timeout(100)   # (an extra of Dawn open: its row unfolds by itself)
        nx3 = await page.evaluate("[document.querySelectorAll('#nav .pip.extra').length, document.querySelector('#nav .pip[aria-current=\"true\"]').getAttribute('aria-label')]")
        await page.click('#nav .fold'); await page.evaluate("window.__samon.load(0)"); await page.wait_for_timeout(100)
        nx4 = await page.evaluate("document.querySelectorAll('#nav .pip.extra').length")
        expect(nx == [len(w1m), 0, [f'+{w1x} more']] and nx2 == [w1x, 1, 'fewer'] and nx3 == [w1x, G[w1[-1]]['name']] and nx4 == 0,
               f"a world's extras fold behind '+{w1x} more' (unfolding by themselves while one of them is open), and only open worlds show pips ({nx}, {nx2}, {nx3}, {nx4})")
        lk = await page.evaluate("[...document.querySelectorAll('#nav .world.locked')].map(r => [r.querySelector('.t').textContent, r.querySelector('.lock .n').textContent, r.querySelector('.lock .note').textContent])")
        expect(len(lk) == 6 and lk[0][0] == 'Morning' and f'{need} gardens of Dawn' in lk[0][2] and all(n[1] == f"{sum(1 for g in G if g['world'] == k + 2)} gardens ·" for k, n in enumerate(lk)) and all(n[2] == '[Locked.]' for n in lk[1:]),
               f'at first only Dawn is open; each locked world is one line with its count and what opens it {lk[:2]}')
        await page.click('#openAll')
        locked2 = await page.evaluate("[document.querySelectorAll('#nav .world.locked').length, document.querySelectorAll('#nav .pip').length]")
        await page.click('#openAll')
        locked3 = await page.evaluate("document.querySelectorAll('#nav .world.locked').length")
        expect(locked2 == [0, sum(1 for g in G if not g['extra'])] and locked3 == 6, f'the prototype switch opens every world (main paths shown, extras folded), and locks them again {locked2} {locked3}')
        for k, i in enumerate(w1m[:need]):
            await page.evaluate(f"window.__samon.load({i})"); await page.click('#restart')
            await drag(page, G[i]['intended']); await page.wait_for_timeout(350)
            pts = await centers(page, [G[i]['intended'][0]]); await page.mouse.click(*pts[0]); await page.wait_for_timeout(60)
            if k == need - 2:
                still = await page.evaluate("window.__samon.worldOpen(2)")
        st = await state(page)
        open2 = await page.evaluate("[window.__samon.worldOpen(2), window.__samon.worldOpen(3), document.querySelectorAll('#nav .world.locked').length]")
        expect(not still and open2[0] and not open2[1] and 'Morning is open' in st['status'] and open2[2] == 5,
               f'raking {need} of the {len(w1m)} Dawn gardens on the main path opens Morning, and says so {open2}')
        # Next follows the main path: from the last Dawn garden on it, past the extras, to the first of Morning; that
        # garden introduces a rule, and its demonstration plays by itself
        last1 = w1m[-1]
        await page.evaluate(f"window.__samon.load({last1})"); await page.click('#restart')
        await drag(page, G[last1]['intended']); await page.wait_for_timeout(350)
        pts = await centers(page, [G[last1]['intended'][0]]); await page.mouse.click(*pts[0]); await page.wait_for_timeout(100)
        await page.click('#next'); await page.wait_for_timeout(800)
        nm = await page.evaluate("window.__samon.G.name")
        first2 = next(g for g in G if g['world'] == 2 and not g['extra'])
        expect(nm == first2['name'], f'Next from the end of the main path skips the extras ({nm})')
        if not first2['demo']:
            i2 = G.index(first2)
            await drag(page, first2['intended']); await page.wait_for_timeout(350)
            pts = await centers(page, [first2['intended'][0]]); await page.mouse.click(*pts[0]); await page.wait_for_timeout(100)
            await page.click('#next'); await page.wait_for_timeout(800)
        nm2 = await page.evaluate("window.__samon.G.name")
        dg = next(g for g in G if g['name'] == nm2)
        await page.wait_for_timeout(700)
        expect(dg['demo'] and await page.evaluate("window.__samon.demoRunning"), f'the garden that brings in the ripple stone demonstrates itself ({nm2})')
        await page.mouse.click(*(await centers(page, [dg['intended'][0]]))[0]); await page.wait_for_timeout(100)
        # the abbot's observation points at its cells, and they stay marked while the walk goes on
        pk = next(i for i, g in enumerate(G) if g['name'] == 'The pocket')
        await page.evaluate(f"window.__samon.load({pk})"); await page.click('#restart'); await page.click('#hint')
        st = await state(page); ob = await page.evaluate("window.__samon.obs.length")
        await drag(page, G[pk]['intended'][:3]); ob2 = await page.evaluate("window.__samon.obs.length")
        expect('He points' in st['status'] and ob == 1 and ob2 == 1, 'the observation points at its cell, which stays marked while drawing')
        await ctx.close()

        # 8. desktop: the pebbles garden mid-route
        ctx = await b.new_context(viewport={'width': 1200, 'height': 1000}, color_scheme='light')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(200)
        await quiet(page)
        await page.evaluate(f"window.__samon.load({pi})"); await drag(page, G[pi]['intended'][:15])
        await page.screenshot(path=SHOT + 'shot-desktop-light.png', full_page=True)
        await ctx.close()
        # 9. properties, over every garden
        ctx = await b.new_context(viewport={'width': 390, 'height': 900}, color_scheme='light')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(200)
        await quiet(page)
        bad_bends = []
        for i, g in enumerate(G):
            await page.evaluate(f"window.__samon.load({i})")
            arcs, turns = await page.evaluate("(c) => window.__samon.strokeBends(c)", g['intended'])
            if not (arcs == turns == g['par']): bad_bends.append((g['name'], arcs, turns, g['par']))
        nside = sum(1 for g in G if g['end'])
        expect(not bad_bends, f'every garden: the bends drawn in the raked stroke = the turns counted = par ({nside} with a side gate) {bad_bends[:5]}')
        piv = await page.evaluate("""(n) => { let pairs = 0, bad = 0; for (let i = 0; i < n; i++) { window.__samon.load(i); const G = window.__samon.G;
            if (!G.wide) continue;
            for (const v of G.sandList) for (let d = 0; d < 4; d++) { const r = Math.floor(v / G.W) + [-1, 0, 1, 0][d], c = v % G.W + [0, 1, 0, -1][d];
              if (r >= 0 && r < G.H && c >= 0 && c < G.W && G.rows[r][c] === 'R') { pairs++; if (G.piv[v * 4 + d]) bad++; } } }
            return [pairs, bad]; }""", len(G))
        expect(piv[0] > 0 and piv[1] == 0, f'under the wide rake, finished sand is never a pivot ({piv[0]} cell sides checked)')
        wrong, seen = [], 0
        quick = bool(os.environ.get('SAMON_QUICK'))          # the abbot's explanations: 400 random routes per garden in full, a sample when quick
        for i, g in enumerate(G):
            if not (g['end'] or g['wide'] or g.get('marks')): continue
            res = await page.evaluate(f"""(() => {{ const out = []; for (let s = 1; s < {30 if quick else 400}; s++) {{ const w = window.__samon.randomWalk({i}, s, 999);
                const a = window.__samon.analyzeFor({i}, w); if (a.state === 'stuck' && a.why && /only one way in, so the walk would have to end there/.test(a.why.text))
                out.push([a.why.text, a.why.cells[0]]); }} return out; }})()""")
            W_ = len(g['rows'][0]); end = g['end'][0] * W_ + g['end'][1] if g['end'] else -1
            marks = {int(k.split(',')[0]) * W_ + int(k.split(',')[1]) for k in (g.get('marks') or {})}
            for text, cell in res:
                seen += 1
                want = 'side gate' if end >= 0 and cell != end else 'pebble' if cell in marks else 'wide rake'
                if want not in text: wrong.append((g['name'], text))
        expect(seen > 0 and not wrong, f'a dead-end cell is explained by the rule that actually fails ({seen} explanations checked) {wrong[:3]}')
        await ctx.close()

        # 10. the poem is the reward: after any raking (here off par, and with the abbot's help) a poem is laid over the
        # garden, and then the next garden fades in; a tap on the poem goes on, a tap beside it stays; 'original' shows the Japanese; Enter goes on at once; a
        # demonstration earns no poem; the practice garden has one
        PO = json.load(open('/home/claude/zen/poems/samon-poems.json'))
        ppoem = lambda name: PO['poems'][PO['gardens'][name]]
        ctx = await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2, color_scheme='light')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(200)
        pc = await page.evaluate("window.__samon.poems()")
        expect(pc['gardens'] == len(G) and pc['practice'], f'every garden has a poem, and so does the practice garden {pc}')
        await quiet(page, delay=200, hold=1500, fade=300)
        ni = next(i for i, g in enumerate(G) if g['name'] == 'Nine squares')
        gx = garden_of(G[ni])
        sub = next(s_['draw'] for s_ in gx.solve(mode=0, cap=100, stream=True)['sols'] if turns_of(gx, s_['draw']) > G[ni]['par'])
        await page.evaluate(f"window.__samon.load({ni})")
        await page.click('#hint'); await page.click('#hint')
        await drag(page, [list(c) for c in sub]); await page.wait_for_timeout(350)
        pts = await centers(page, [list(sub[0])]); await page.mouse.click(*pts[0])
        await page.wait_for_function("window.__samon.poem.state === 'shown'", timeout=3000); await page.wait_for_timeout(400)
        po = await page.evaluate("window.__samon.poem")
        expect(po['visible'] and po['verse'] == ppoem('Nine squares')['translation'], f"a raking off par, made with help, still earns its poem ({po['verse'][:40]!r})")
        nxt = G[ni + 1]['name']
        await page.wait_for_function("window.__samon.poem.leaving", timeout=4000)
        await page.wait_for_function(f"window.__samon.G.name === {json.dumps(nxt)}", timeout=3000)
        st = await state(page); po = await page.evaluate("window.__samon.poem")
        expect(st['mode'] == 'draw' and po['state'] == 'none' and not po['visible'], f'then garden and poem fade, and the next garden fades in ({nxt})')
        i2 = ni + 1
        await quiet(page, delay=200, hold=1200, fade=300)
        await drag(page, G[i2]['intended']); await page.wait_for_timeout(350)
        pts = await centers(page, [G[i2]['intended'][0]]); await page.mouse.click(*pts[0])
        await page.wait_for_function("window.__samon.poem.state === 'shown'", timeout=3000); await page.wait_for_timeout(300)
        beside = await page.evaluate("(() => { const r = document.getElementById('poem').getBoundingClientRect(), c = document.querySelector('#poem .card').getBoundingClientRect(); return [r.left + 3, c.top > r.top + 6 ? r.top + 3 : r.top + r.height - 3]; })()")
        await page.mouse.click(*beside); await page.wait_for_timeout(2000)
        st = await state(page); po = await page.evaluate("window.__samon.poem"); nm = await page.evaluate("window.__samon.G.name")
        expect(nm == G[i2]['name'] and po['state'] == 'stay' and not po['visible'] and 'Staying' in st['status'] and 'Read the poem again' in st['actions'],
               f"a tap beside the poem puts it away and stays with the garden ({st['status'][-60:]!r})")
        await page.click('#actions button:has-text("Read the poem again")'); await page.wait_for_timeout(1800)
        po = await page.evaluate("window.__samon.poem")
        expect(po['visible'], 'the poem can be read again')
        await page.click('#orig'); await page.wait_for_timeout(100)
        po = await page.evaluate("window.__samon.poem")
        expect(po['ja'] == ppoem(G[i2]['name'])['japanese'] and not po['clash'], "'original' shows the Japanese (and no line runs under the seal)")
        await page.focus('#garden'); await page.keyboard.press('Enter'); await page.wait_for_timeout(800)
        nm = await page.evaluate("window.__samon.G.name")
        expect(nm == G[i2 + 1]['name'], 'Enter goes on to the next garden at once')
        await page.evaluate("(w) => window.__samon.setPath(w)", G[i2 + 1]['intended'])
        await page.wait_for_function("window.__samon.poem.state === 'shown'", timeout=12000); await page.wait_for_timeout(200)
        card = await page.evaluate("(() => { const c = document.querySelector('#poem .card').getBoundingClientRect(); return [c.left + c.width / 2, c.top + c.height / 2]; })()")
        await page.mouse.click(*card); await page.wait_for_timeout(900)
        nm = await page.evaluate("window.__samon.G.name"); po = await page.evaluate("window.__samon.poem")
        expect(nm == G[i2 + 2]['name'] and po['state'] == 'none', f'a tap on the poem itself goes on to the next garden at once ({nm!r})')
        await page.evaluate("(w) => window.__samon.setPath(w)", G[i2 + 2]['intended'])
        await page.wait_for_function("window.__samon.poem.state === 'shown'", timeout=12000); await page.wait_for_timeout(700)   # (a tap within 0.6 s of the raking is taken as a skip of it)
        wall = await page.evaluate("(() => { const r = document.getElementById('garden').getBoundingClientRect(); return [r.left + 4, r.top + 4]; })()")
        await page.mouse.click(*wall); await page.wait_for_timeout(300)
        nm = await page.evaluate("window.__samon.G.name"); po = await page.evaluate("window.__samon.poem"); st = await state(page)
        expect(nm == G[i2 + 2]['name'] and po['state'] == 'stay' and not po['visible'] and 'Read the poem again' in st['actions'], f'a tap on the walls around the poem puts it away and stays ({nm!r}, {po["state"]})')
        fit = await page.evaluate(f"""(() => {{ const out = [], fs = [], moved = [0, 0]; for (let i = 0; i < {len(G)}; i++) for (const o of [false, true]) {{
            const r = window.__samon.poemFit(i, o); fs.push(r.fs); moved[0] += r.left; moved[1] += r.under;
            if (r.over || (r.clash && !r.under) || (r.under && !o)) out.push([i, o, r]); }} return [out, Math.min(...fs), moved]; }})()""")
        expect(not fit[0], f'on a phone, every poem fits its garden and clears the seal (set flush left {fit[2][0]} times, over a faded seal {fit[2][1]} times, '
               f'only with the original; smallest type {fit[1]:.1f}px) {fit[0][:3]}')
        await page.evaluate("window.__samon.load(0)"); await page.evaluate("window.__samon.startDemo()")
        await page.wait_for_function("document.getElementById('status').textContent.startsWith('[That is')", timeout=15000)
        po = await page.evaluate("window.__samon.poem")
        expect(po['state'] == 'none' and not po['visible'], 'a demonstration earns no poem')
        await page.evaluate("window.__samon.enterPractice()")
        pp = await page.evaluate("document.getElementById('ppoem').textContent")
        expect(PO['poems'][PO['practice']]['translation'].split('\n')[0] in pp, 'the practice garden has its poem')
        await ctx.close()

        # 11. the scroll of poems earned, and the daily garden (one for everyone, picked by the date; raked once, by Rake)
        DAILY = json.load(open('/home/claude/zen/out/daily_gardens.json'))
        ctx = await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2, color_scheme='light')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(200)
        await quiet(page, delay=200, hold=3600000)
        await page.evaluate("window.__samon.enterPoems()"); await page.wait_for_timeout(150)
        sc0 = await page.evaluate("[document.querySelectorAll('#scroll .entry').length, getComputedStyle(document.getElementById('scroll')).display !== 'none', getComputedStyle(document.getElementById('stage')).display, window.__samon.view]")
        expect(sc0 == [1, True, 'none', 'poems'], f'the scroll before any raking: only the practice poem, shown in place of the garden {sc0}')
        for i in (0, 1):
            await page.evaluate(f"window.__samon.load({i})")
            await page.evaluate("(w) => window.__samon.setPath(w)", G[i]['intended'])
            await page.wait_for_function("['reveal', 'done'].includes(window.__samon.mode)", timeout=3000)
            await page.focus('#garden'); await page.keyboard.press('Enter')
            await page.wait_for_function("window.__samon.mode === 'done'", timeout=5000)
        await page.evaluate("window.__samon.enterPoems()"); await page.wait_for_timeout(150)
        sc = await page.evaluate("[[...document.querySelectorAll('#scroll .entry .verse')].map(e => e.textContent), [...document.querySelectorAll('#scroll .entry .ja')].map(e => e.textContent), [...document.querySelectorAll('#scroll .left')].map(e => e.textContent)]")
        expect(sc[0][:2] == [ppoem(G[0]['name'])['translation'], ppoem(G[1]['name'])['translation']] and sc[1][:2] == [ppoem(G[0]['name'])['japanese'], ppoem(G[1]['name'])['japanese']] and len(sc[0]) == 3,
               f'two gardens raked: their poems, in order, with the Japanese, then the practice poem ({len(sc[0])})')
        expect(sc[2][0].startswith('[2 of') and sc[2][1] == f"[{sum(1 for g in G if g['world'] == 1) - 2} more in the gardens of Dawn.]", f'the scroll counts what is earned and what is left, in editorial brackets {sc[2][:2]}')
        pills = await page.evaluate("[...document.querySelectorAll('#nav .pill')].map(b => b.textContent)")
        sw = await page.evaluate("[document.getElementById('soundToggle').textContent, document.getElementById('soundPanel').hidden]")
        expect(any(t.startswith('Poems · 2 of') for t in pills) and pills[0].startswith("Today's garden") and sw[0] in ('Sound on', 'Sound off') and sw[1], f'the nav offers the poems and the daily garden; the sound switch waits under the speaker {pills} {sw}')
        await page.click('#scroll .entry .by button >> nth=1'); await page.wait_for_timeout(150)
        v = await page.evaluate("[window.__samon.view, window.__samon.G.name]")
        expect(v == ['puzzle', G[1]['name']], f"a garden's name in the scroll opens it {v}")
        expect(len(DAILY) == 366 and all(e['world'] == 0 and e['intended'] and e['par_moves'] for e in DAILY), f'a year of daily gardens is embedded ({len(DAILY)})')
        await page.evaluate("window.__samon.today = '2026-10-04T12:00:00'")
        d = await page.evaluate("window.__samon.daily")
        expect(d['year'] == len(DAILY) and d['key'] == '2026-10-04' and d['dayNo'] == 276 and not d['done'], f'the daily slot: day 276 of the year, not yet raked {d}')
        await page.evaluate("window.__samon.pickDaily()"); await page.wait_for_timeout(250)
        g = await page.evaluate("window.__samon.G")
        day = DAILY[276 % len(DAILY)]
        hdr = await page.evaluate("[document.getElementById('time').textContent, document.getElementById('gname').textContent, document.getElementById('next').textContent, document.getElementById('rakeDay').hidden]")
        expect(g['rows'] == day['rows'] and hdr == ['Daily garden', '4 October 2026', 'Back to the gardens', True], f'today\'s garden is the year\'s day 276, under its date {hdr}')
        await page.evaluate("(w) => window.__samon.setPath(w)", day['intended']); await page.wait_for_timeout(900)
        st = await page.evaluate("[window.__samon.mode, document.getElementById('rakeDay').hidden, document.getElementById('status').textContent]")
        expect(st[0] == 'draw' and not st[1] and 'press Rake' in st[2], f'a complete walk is not raked by itself here: it waits for Rake {st}')
        await page.click('#rakeDay')
        await page.wait_for_function("window.__samon.mode === 'done'", timeout=8000)
        await page.wait_for_function("window.__samon.poem.visible", timeout=5000)
        d2 = await page.evaluate("window.__samon.daily")
        st2 = await page.evaluate("[document.getElementById('status').textContent, [...document.querySelectorAll('#actions button')].map(b => b.textContent), window.__samon.poem.state, document.getElementById('seal').hidden]")
        expect(d2['done'] and d2['streak'] == 1 and d2['turns'] == day['par'] and d2['kind'] == 'red' and 'Streak: 1 day' in st2[0] and st2[1] == ["Share today's score", 'Read the poem again'] and st2[2] == 'stay' and not st2[3],
               f'raked at par: the record, a streak of one, the seal, the share action; the poem stays {d2} {st2}')
        box = await page.evaluate("(() => { const r = document.getElementById('garden').getBoundingClientRect(); return [r.left + r.width / 2, r.top + r.height / 2]; })()")
        await page.mouse.click(*box); await page.wait_for_timeout(250); await page.mouse.click(*box); await page.wait_for_timeout(250)
        st3 = await page.evaluate("[window.__samon.mode, document.getElementById('undo').disabled, document.getElementById('restart').disabled]")
        expect(st3 == ['done', True, True], f'the raking is final: no walking back, Step back and Start over off {st3}')
        await page.reload(); await page.wait_for_timeout(400)
        await page.evaluate("window.__samon.today = '2026-10-04T12:00:00'")
        await page.evaluate("window.__samon.pickDaily()"); await page.wait_for_timeout(300)
        st4 = await page.evaluate("[window.__samon.mode, document.getElementById('status').textContent, document.getElementById('seal').hidden, window.__samon.daily.streak, window.__samon.path.length]")
        expect(st4[0] == 'done' and "Today's record" in st4[1] and not st4[2] and st4[3] == 1 and st4[4] == len(day['intended']), f'after a reload the day\'s raking and its record are back {st4[:4]}')
        await page.evaluate("window.__samon.today = '2026-10-05T12:00:00'")
        await page.evaluate("window.__samon.pickDaily()"); await page.wait_for_timeout(300)
        d3 = await page.evaluate("window.__samon.daily"); g3 = await page.evaluate("window.__samon.G.rows")
        expect(d3['key'] == '2026-10-05' and not d3['done'] and d3['streak'] == 1 and g3 == DAILY[277 % len(DAILY)]['rows'] and await page.evaluate("window.__samon.mode") == 'draw', f'the next day is a fresh garden, and the streak counts yesterday {d3}')
        nx = await page.evaluate("(() => { window.__samon.load(0); document.getElementById('next').click(); return window.__samon.G.name; })()")
        expect(nx != d3['name'] and nx == G[1]['name'], f'the daily garden is never the next garden ({nx})')
        await ctx.close()

        # 12. drag intent: the walk follows the axis the hand moves on, not the cell under the pointer; the grip survives the walls
        ctx = await b.new_context(viewport={'width': 700, 'height': 900}, device_scale_factor=2, color_scheme='light')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(200); await quiet(page)
        gi = next(i for i, g in enumerate(G) if len(g['rows']) >= 6 and len(g['rows'][0]) >= 7 and g['gate'][2] == 'S' and g['gate'][0] == len(g['rows']) - 1
                  and set(g['rows'][-1] + g['rows'][-2]) == {'.'} and g['rows'][-3][g['gate'][1] + 3] == '.')
        H, W = len(G[gi]['rows']), len(G[gi]['rows'][0]); gr, gc = G[gi]['gate'][:2]
        async def at(r, c, dx=0.0, dy=0.0):
            return await page.evaluate("""([r, c, dx, dy]) => { const cv = document.getElementById('garden'); const b = cv.getBoundingClientRect();
                const s = b.width / cv.width; const g = window.__samon.geom();
                return [b.left + (g.ox + (c + 0.5 + dx) * g.cs) * s, b.top + (g.oy + (r + 0.5 + dy) * g.cs) * s]; }""", [r, c, dx, dy])
        async def walk():
            return await page.evaluate("window.__samon.path.map(k => [Math.floor(k / window.__samon.G.W), k % window.__samon.G.W])")
        async def begin():
            await page.evaluate(f"window.__samon.load({gi}); window.__samon.setPath([]); document.getElementById('garden').scrollIntoView({{block: 'center'}})"); await page.wait_for_timeout(50)
            x, y = await at(gr, gc); await page.mouse.move(x, y); await page.mouse.down()
        run = [[gr, c] for c in range(gc, gc + 4)]
        await begin(); x, y = await at(gr, gc + 3, 0, -0.9); await page.mouse.move(x, y, steps=40); await page.mouse.up()
        expect(await walk() == run, f'a drag along the row whose hand drifts 0.9 of a cell into the next row still draws the row ({await walk()})')
        await begin(); x, y = await at(gr, gc + 1, 0, 1.6); await page.mouse.move(x, y, steps=12); x, y = await at(gr, gc + 3, 0, 1.8); await page.mouse.move(x, y, steps=30); await page.mouse.up()
        expect(await walk() == run, f'a pointer below the wall keeps its grip and the walk follows along the row ({await walk()})')
        await begin(); x, y = await at(gr, gc + 3, 0.1, 0); await page.mouse.move(x, y, steps=40)
        for k in range(1, 13):   # a U-turn: a semicircle within the last cell of the run, up into the next row
            a = math.pi / 2 - math.pi * k / 12
            x, y = await at(gr, gc + 3, 0.1 + 0.35 * math.cos(a), -0.35 + 0.35 * math.sin(a)); await page.mouse.move(x, y)
        x, y = await at(gr - 1, gc); await page.mouse.move(x, y, steps=40); await page.mouse.up()
        expect(await walk() == run + [[gr - 1, c] for c in range(gc + 3, gc - 1, -1)], f'a U-turn drawn as an arc turns once and comes back along the next row ({await walk()})')
        await begin(); x, y = await at(gr, gc + 3, -0.2, 0.05); await page.mouse.move(x, y, steps=30)
        for k in range(1, 9):    # a rounded corner from heading east to heading north
            a = (math.pi / 2) * k / 8
            x, y = await at(gr, gc + 3, -0.2 + 0.3 * math.sin(a), 0.05 - 0.3 + 0.3 * math.cos(a)); await page.mouse.move(x, y)
        x, y = await at(gr - 2, gc + 3, 0.1, 0); await page.mouse.move(x, y, steps=30); await page.mouse.up()
        expect(await walk() == run + [[gr - 1, gc + 3], [gr - 2, gc + 3]], f'a rounded corner is a run and a turn ({await walk()})')
        await begin(); x, y = await at(gr, gc + 3, 0, -0.6); await page.mouse.move(x, y, steps=30); out = await walk()
        x, y = await at(gr, gc, -0.2, -0.6); await page.mouse.move(x, y, steps=30); await page.mouse.up()
        expect(out == run and await walk() == [[gr, gc]], f'dragging back along a drifted run retraces it ({out} → {await walk()})')
        await begin()
        for k in range(40): x, y = await at(gr, gc, 0.12 * math.cos(k), 0.12 * math.sin(k)); await page.mouse.move(x, y)
        await page.mouse.up()
        expect(await walk() == [[gr, gc]], f'a wobble in place takes no step ({await walk()})')
        await ctx.close()

        # 13. sound: nothing before a gesture; the synthesized sounds render; Sound off persists; the recordings carried in the
        # page are found, credited on the scroll with their source, put down to follow a world's bell, and play
        import struct
        PIECES = json.load(open('/home/claude/zen/out/sound_pieces.json'))
        ctx = await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2, color_scheme='light')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(300); await quiet(page)
        s0 = await page.evaluate("[window.__samon.sound.on, window.__samon.sound.ready, window.__samon.sound.state, window.__samon.sound.tracks]")
        expect(s0 == [True, False, 'none', None], f'before any gesture there is no audio context, and the recordings are not looked at {s0}')
        await page.click('#garden'); await page.wait_for_timeout(200)
        s1 = await page.evaluate("[window.__samon.sound.ready, window.__samon.sound.state, window.__samon.sound.timers, window.__samon.sound.tracks.map(t => [t.title, t.src.slice(0, 22), t.dur])]")
        expect(s1[:2] == [True, 'running'] and s1[2] in (6, 7) and s1[3] == [[p['title'], 'data:audio/mpeg;base64', p['dur']] for p in PIECES], f'a touch starts the audio, the six ambience loops (and, for a moment, the speaker\'s redraw), and finds the recordings in the page {s1}')
        wav = bytes(await page.evaluate("window.__samon.sound.render([[0.05, 'drip', 0.3], [0.3, 'tock', 0.4], [0.7, 'tok', 0.4], [0.9, 'bell', 0.5]], 1.3)"))
        pcm = struct.unpack(f'<{(len(wav) - 44) // 2}h', wav[44:])
        def peak(t0, t1): return max(abs(v) for v in pcm[int(t0 * 44100):int(t1 * 44100)]) / 32768
        pk = [round(peak(*w), 3) for w in ((0.05, 0.15), (0.3, 0.4), (0.7, 0.75), (0.9, 1.2))]
        expect(all(p > 0.03 for p in pk) and peak(0, 0.05) == 0, f'drip, pipe, seal and bell each render audibly, and nothing before the first {pk}')
        wav = bytes(await page.evaluate("window.__samon.sound.render([[0.05, 'scrape', 1.2, 1], [1.5, 'crunch', 0.14], [1.8, 'birds', 0.1], [3.0, 'crickets', 0.08, 0.5], [3.7, 'wind', 0.3], [3.8, 'gust', 0.4, 1]], 5)"))
        pcm = struct.unpack(f'<{(len(wav) - 44) // 2}h', wav[44:])
        pk = [round(peak(*w), 3) for w in ((0.3, 1.1), (1.5, 1.6), (1.85, 2.9), (3.0, 3.5), (4.3, 5.0))]
        expect(all(p > 0.015 for p in pk) and peak(1.44, 1.5) < 0.02, f'the rake\'s scrape, a footstep, a bird, a cricket and the wind each render audibly, and the scrape dies away when the rake stops {pk} (tail {peak(1.44, 1.5):.3f})')
        dp = await page.evaluate("[5, 6, 12, 18, 19, 20, 23, 4].map(h => window.__samon.sound.dayPart(h))")
        expect(dp == ['dusk', 'day', 'day', 'day', 'dusk', 'night', 'night', 'night'], f'birds from 6 to 19, crickets from 20 to 5, by the local clock {dp}')
        await page.evaluate("window.__samon.load(0)"); await page.evaluate("(w) => window.__samon.setPath(w)", G[0]['intended'])
        await page.wait_for_function("window.__samon.mode === 'reveal'", timeout=3000)
        sc1 = await page.evaluate("window.__samon.sound.scraping")
        await page.wait_for_function("window.__samon.mode === 'done'", timeout=12000); await page.wait_for_timeout(100)
        sc2 = await page.evaluate("window.__samon.sound.scraping")
        expect(sc1 and not sc2, f'the scrape runs with the raking and stops with it ({sc1}, {sc2})')
        await page.evaluate("window.__samon.sound.setVolume(0.3)"); await page.wait_for_timeout(300)
        vs = await page.evaluate("[window.__samon.sound.vol, Math.round(window.__samon.sound.master * 100) / 100, localStorage.getItem('samon-sound-vol'), document.getElementById('soundVol') && document.getElementById('soundVol').value, !!document.getElementById('hapticsToggle'), window.__samon.sound.haptics]")
        expect(vs[:3] == [0.3, 0.3, '0.3'] and vs[4] and vs[5], f'the volume is set, remembered and reaches the master gain; the haptics switch shows where the device can tick {vs}')
        pn0 = await page.evaluate("document.getElementById('soundPanel').hidden")
        await page.click('#soundMore'); await page.wait_for_timeout(100)
        pn1 = await page.evaluate("[document.getElementById('soundPanel').hidden, document.getElementById('soundMore').getAttribute('aria-expanded'), document.getElementById('volRow').hidden]")
        await page.click('#hapticsToggle'); await page.wait_for_timeout(100)
        hp = await page.evaluate("[window.__samon.sound.haptics, localStorage.getItem('samon-haptics'), document.getElementById('hapticsToggle').textContent]")
        expect(pn0 and pn1 == [False, 'true', False] and hp == [False, '0', 'Haptics off'], f'the chevron by the speaker opens the sound panel (switch, volume, haptics); haptics can be turned off there, and that is remembered {pn1} {hp}')
        await page.mouse.click(5, 400); await page.wait_for_timeout(100)
        expect(await page.evaluate("document.getElementById('soundPanel').hidden"), 'a press elsewhere closes the panel')
        OPEN_PANEL = "(() => { if (document.getElementById('soundPanel').hidden) document.getElementById('soundMore').click(); })()"
        s2 = await page.evaluate("[window.__samon.sound.ready, window.__samon.sound.state, window.__samon.sound.on]")
        expect(s2 == [True, 'running', True], f'the live context is untouched by the offline render {s2}')
        tm = await page.evaluate("(() => { const a = window.__samon.sound.timers; window.__samon.sound.piece('capstone'); return [a, window.__samon.sound.timers]; })()")
        expect(tm[1] == tm[0] + 1, f'after a world\'s bell a piece is put down to follow {tm}')
        await page.evaluate(f"window.__samon.sound.play(0, {PIECES[0]['dur'] - 12})"); await page.wait_for_function("window.__samon.sound.playing !== null", timeout=5000)
        await page.wait_for_timeout(4000); lv = await page.evaluate("window.__samon.sound.level()")
        expect(lv > 0.002, f'the recording is heard: the output meter reads {lv:.4f} while it plays (silence would mean the media source is muted)')
        await page.evaluate("window.__samon.enterPoems()"); await page.wait_for_timeout(150)
        cr = await page.evaluate("[document.getElementById('soundCredits').textContent, [...document.querySelectorAll('#soundCredits a')].map(a => [a.textContent, a.href])]")
        expect(f"Shakuhachi, now and then: {PIECES[0]['title']}, {PIECES[0]['player']} ({PIECES[0]['license']})." in cr[0] and cr[1] == [[PIECES[0]['title'], PIECES[0]['source']]],
               f'the scroll credits the recording, with its source {cr}')
        await page.wait_for_function("window.__samon.sound.playing === null", timeout=20000)
        expect(True, 'the piece plays to its end and is let go')
        await page.evaluate(OPEN_PANEL); await page.click('#soundToggle'); await page.wait_for_timeout(150)
        s3 = await page.evaluate("[window.__samon.sound.on, window.__samon.sound.state, window.__samon.sound.timers, document.getElementById('soundToggle').textContent, localStorage.getItem('samon-sound')]")
        expect(s3 == [False, 'suspended', 0, 'Sound off', '0'], f'Sound off suspends the context, clears the loops, and is remembered {s3}')
        await page.reload(); await page.wait_for_timeout(300); await quiet(page); await page.click('#garden'); await page.wait_for_timeout(200)
        s4 = await page.evaluate("[window.__samon.sound.on, window.__samon.sound.ready, document.getElementById('soundToggle').textContent]")
        expect(s4 == [False, False, 'Sound off'], f'after a reload sound stays off, and a touch makes no context {s4}')
        noslider = await page.evaluate("document.getElementById('volRow').hidden")
        await page.evaluate(OPEN_PANEL); await page.click('#soundToggle'); await page.wait_for_timeout(200)
        s5 = await page.evaluate("[window.__samon.sound.on, window.__samon.sound.state, window.__samon.sound.timers, document.getElementById('soundVol').value, window.__samon.sound.vol]")
        expect(s5[:2] == [True, 'running'] and s5[2] in (6, 7) and s5[3:] == ['30', 0.3] and noslider, f'Sound on starts it again, with its loops (drip, pipe, wind, gusts, birds, crickets) and the remembered volume; no slider while off {s5}')
        # the mute button and the M key; what the page can tell of the browser's own silence
        mb = await page.evaluate("(() => { const b = document.getElementById('mute'); return [b.className, b.getAttribute('aria-pressed'), b.getAttribute('aria-label'), b.title]; })()")
        await page.click('#mute'); await page.wait_for_timeout(120)
        mb2 = await page.evaluate("(() => { const b = document.getElementById('mute'); return [b.className, b.getAttribute('aria-pressed'), b.getAttribute('aria-label'), window.__samon.sound.on, window.__samon.sound.state, document.getElementById('soundToggle').textContent, localStorage.getItem('samon-sound')]; })()")
        await page.keyboard.press('m'); await page.wait_for_timeout(200)
        mb3 = await page.evaluate("[document.getElementById('mute').className, window.__samon.sound.on, window.__samon.sound.state, window.__samon.sound.timers > 0]")
        expect(mb == ['mute', 'false', 'Mute', 'Mute (M)'] and mb2 == ['mute off', 'true', 'Unmute', False, 'suspended', 'Sound off', '0'] and mb3 == ['mute', True, 'running', True],
               f'the speaker in the header mutes and unmutes, as does M, in step with the list\'s switch and the saved setting {mb} {mb2} {mb3}')
        bl = await page.evaluate("(() => { navigator.getAutoplayPolicy = () => 'disallowed'; window.__samon.sound.renderMute(); const b = document.getElementById('mute'); const r = [window.__samon.sound.blocked, b.className, b.getAttribute('aria-label')]; delete navigator.getAutoplayPolicy; window.__samon.sound.renderMute(); return r.concat([b.className]); })()")
        expect(bl == [True, 'mute blocked', 'Sound is blocked by the browser', 'mute'], f'when the browser says it will not play (Firefox\'s autoplay policy), the speaker says so in red, and recovers {bl}')
        aud = await page.evaluate("[typeof navigator.audioSession, typeof navigator.mediaSession, !!document.getElementById('muteHint') || localStorage.getItem('samon-sound-hint') === '1']")
        expect(aud[2], f'the first start of sound shows a one-time word beside the speaker (audioSession: {aud[0]}, mediaSession: {aud[1]})')
        await ctx.close()

        # 14. sharing: a raking is a code in the URL's hash (the walk as a number in the mixed radix of its legal moves), shown
        # but not credited to whoever opens it; a player's whole progress is another code, offered for taking over; the share
        # text draws the raking in box-drawing characters
        ctx = await b.new_context(viewport={'width': 500, 'height': 900}, device_scale_factor=2, color_scheme='light', permissions=['clipboard-read', 'clipboard-write'])
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(300); await quiet(page)
        lens = []; bits = []; bad = []
        for i in (range(len(G)) if not os.environ.get('SAMON_QUICK') else range(0, len(G), 7)):
            g = G[i]; await page.evaluate(f"window.__samon.load({i})")
            code = await page.evaluate("(w) => window.__samon.share.code(w)", g['intended'])
            w = await page.evaluate("(w) => window.__samon.share.walk(w)", g['intended'])
            back = await page.evaluate("(a) => window.__samon.share.unwalk(a[0], a[1])", [w[0], len(g['intended']) - 1])
            h = await page.evaluate("(c) => { const h = window.__samon.share.read(c); return [h.i, h.grey, h.n.toString()]; }", code)
            if not (code and back == g['intended'] and h == [i, False, w[0]]): bad.append(g['name'])
            lens.append(len(code)); bits.append(w[1])
        expect(not bad and max(lens) <= 18, f'every intended raking round-trips through its code: {len(lens)} gardens, {min(lens)}–{max(lens)} characters (mean {sum(lens) / len(lens):.1f}); the walks take {sum(bits) / len(bits):.1f} bits on average, against {sum(2 * (len(g["intended"]) - 1) for g in G) / len(G):.0f} at two bits a step {bad}')
        lay = await page.evaluate("window.__samon.share.layouts()")
        allh = lay['campaign'] + lay['daily']
        expect(len(set(allh)) == len(allh) == len(G) + 366 and all(0 <= h < 2 ** 32 for h in allh), f'every garden, campaign or daily, has its own 32-bit layout hash ({len(set(allh))} of {len(allh)} distinct)')
        legal = await page.evaluate("(() => { window.__samon.load(0); return window.__samon.share.walk([[2, 0], [2, 1], [1, 1]]); })()")
        expect(legal is None, f'a walk that breaks a rule (onto a stone) has no code {legal}')
        grid = await page.evaluate("(w) => window.__samon.share.text(w, 'red')", G[0]['intended'])
        rows = grid.split('\n')
        expect(rows[0] == f"Samon · Dawn 1 · {G[0]['name']} · {G[0]['par']} turns, par {G[0]['par']} · 静" and len(rows) == len(G[0]['rows']) + 3 and all(len(r) == len(G[0]['rows'][0]) + 2 for r in rows[1:])
               and rows[1] == '┏━━━━┓' and rows[2] == '┃╶──┐┃' and rows[3] == '┃███│┃' and rows[4] == '┃┌──┘┃' and rows[-1] == '┗┷━━━┛', f'the share text: a heading, then the garden in box-drawing characters, the stroke in light lines through the gate in the heavy wall, stones as blocks\n{grid}')
        i2 = next(i for i, g in enumerate(G) if g['end'] and g['end_side'] == 'E')
        await page.evaluate(f"window.__samon.load({i2})")
        grid2 = await page.evaluate("(w) => window.__samon.share.grid(w)", G[i2]['intended'])
        r2 = grid2.split('\n')
        expect(r2[G[i2]['end'][0] + 1].endswith('┨') and sum(1 for ch in grid2 if ch in '┨┠┯┷') == 2, f'a side gate is a second gap in the wall, where the stroke begins\n{grid2}')
        # #r=: the raking is shown, with its seal, but recorded for nobody; Rake it yourself (or any tap) gives the viewer the garden fresh
        gi = 5; g5 = G[gi]
        await page.evaluate(f"window.__samon.load({gi})")
        code = await page.evaluate("(w) => window.__samon.share.code(w)", g5['intended'])
        await page.evaluate("localStorage.clear()")
        await page.goto('about:blank'); await page.goto(URL + '#r=' + code); await page.wait_for_timeout(600); await quiet(page)
        st = await page.evaluate("[window.__samon.mode, window.__samon.G.name, window.__samon.path.length, window.__samon.turnsOf(), document.getElementById('status').textContent, [...document.querySelectorAll('#actions button')].map(b => b.textContent), document.getElementById('seal').hidden, document.getElementById('next').hidden, localStorage.getItem('samon-progress'), !!window.__samon.share.shared, document.getElementById('undo').disabled]")
        expect(st[:4] == ['done', g5['name'], len(g5['intended']), g5['par']] and st[4] == f"[A raking shared with you: {g5['par']} turns, par {g5['par']}, the abbot's seal. Rake it yourself from the gate: a tap smooths the sand.]"
               and st[5] == ['Rake it yourself', 'Share it on'] and not st[6] and st[7] and st[8] is None and st[9] and not st[10], f'a link to a raking opens that garden raked, with the seal, credited to nobody {st[:8]} {st[8:]}')
        pts = await centers(page, [g5['intended'][3]]); await page.mouse.click(*pts[0]); await page.wait_for_timeout(300)
        st = await page.evaluate("[window.__samon.mode, window.__samon.G.name, window.__samon.path.length, document.getElementById('status').textContent, location.hash, !!window.__samon.share.shared, document.getElementById('restore').hidden, document.getElementById('seal').hidden]")
        expect(st == ['draw', g5['name'], 0, '[Now you: drag the monk in from the gate.]', '', False, True, True], f'a tap on the shared raking smooths it away: the garden is the viewer\'s, from the gate, with nothing to Restore, and the hash is gone {st}')
        await page.goto(URL + '#r=' + code); await page.wait_for_timeout(400)   # (a hash change on the open page shows it too)
        st = await page.evaluate("[window.__samon.mode, !!window.__samon.share.shared]")
        await page.click('#actions button >> nth=0'); await page.wait_for_timeout(300)
        st2 = await page.evaluate("[window.__samon.mode, window.__samon.path.length, !!window.__samon.share.shared, location.hash]")
        expect(st == ['done', True] and st2 == ['draw', 0, False, ''], f'the same by Rake it yourself, after the link is followed on the open page {st} {st2}')
        gcode = await page.evaluate("(w) => window.__samon.share.code(w, true)", g5['intended'])
        await page.goto(URL + '#r=' + gcode); await page.wait_for_timeout(400)
        st = await page.evaluate("[document.getElementById('seal').className, document.getElementById('status').textContent.includes('his seal in grey')]")
        expect(st == ['seal grey', True], f'a raking made with the abbot\'s help shows his grey seal {st}')
        # a daily garden's raking: of another day, the viewer is sent on to today's; of today, raked at par, it is shown and then today's is theirs to rake
        DAILY = json.load(open('/home/claude/zen/out/daily_gardens.json'))
        await page.evaluate("window.__samon.today = '2026-10-04T12:00:00'; window.__samon.pickDaily()"); await page.wait_for_timeout(200)
        day = DAILY[276 % len(DAILY)]
        dcode = await page.evaluate("(w) => window.__samon.share.code(w)", day['intended'])
        await page.goto('about:blank'); await page.goto(URL + '#r=' + dcode); await page.wait_for_timeout(600); await quiet(page)
        st = await page.evaluate("[window.__samon.mode, window.__samon.G.name, document.getElementById('gname').textContent, document.getElementById('status').textContent, [...document.querySelectorAll('#actions button')].map(b => b.textContent), window.__samon.daily.key]")
        expect(st[:3] == ['done', 'daily:2026-10-04', '4 October 2026'] and 'of the daily garden of 4 October 2026' in st[3] and "Today's garden is another" in st[3] and st[4][0] == "Today's garden" and st[5] == '2026-10-04',
               f'a shared raking of an earlier day\'s garden is shown under its date, and offers today\'s instead {st}')
        await page.click('#actions button >> nth=0'); await page.wait_for_timeout(300)
        st = await page.evaluate("[window.__samon.mode, window.__samon.G.name, window.__samon.path.length, location.hash, window.__samon.daily.done]")
        expect(st[0] == 'draw' and st[1].startswith('daily:') and st[1] != 'daily:2026-10-04' and st[2] == 0 and st[3] == '' and not st[4], f'Today\'s garden leaves it for today\'s, unraked {st}')
        await page.evaluate("window.__samon.today = '2026-10-04T12:00:00'")
        await page.goto(URL + '#r=' + dcode); await page.wait_for_timeout(400)
        st = await page.evaluate("[window.__samon.mode, window.__samon.G.name, document.getElementById('status').textContent.includes('Rake it yourself'), [...document.querySelectorAll('#actions button')].map(b => b.textContent)[0]]")
        await page.click('#actions button >> nth=0'); await page.wait_for_timeout(300)
        st2 = await page.evaluate("[window.__samon.mode, window.__samon.G.name, window.__samon.path.length, document.getElementById('rakeDay').hidden]")
        expect(st == ['done', 'daily:2026-10-04', True, 'Rake it yourself'] and st2 == ['draw', 'daily:2026-10-04', 0, True], f'the same day\'s raking, shared, is shown and then left for the viewer to rake {st} {st2}')
        # the share text and link: the share action after a raking, to the clipboard where there is no share sheet
        await page.evaluate("window.__samon.load(0)"); await page.evaluate("(w) => window.__samon.setPath(w)", G[0]['intended'])
        await page.wait_for_function("['reveal', 'done'].includes(window.__samon.mode)", timeout=3000)
        await page.focus('#garden'); await page.keyboard.press('Enter'); await page.wait_for_function("window.__samon.mode === 'done'", timeout=5000)
        acts = await page.evaluate("[...document.querySelectorAll('#actions button')].map(b => b.textContent)")
        await page.click('#actions button:has-text("Share this raking")'); await page.wait_for_timeout(400)
        clip = await page.evaluate("navigator.clipboard.readText()")
        st = await page.evaluate("document.getElementById('status').textContent")
        cl = clip.split('\n')
        expect(acts == ['Share this raking'] and cl[0] == f"Samon · Dawn 1 · {G[0]['name']} · {G[0]['par']} turns, par {G[0]['par']} · 静" and cl[1:-1] == grid.split('\n')[1:] and cl[-1].startswith(URL + '#r=') and st.startswith('[Copied: the raking drawn in lines, with a link that opens it'),
               f'after a raking, Share this raking copies the heading, the drawing and a link that opens the raking {cl[0]!r} {cl[-1][-30:]!r} {st!r}')
        await page.goto('about:blank'); await page.goto(cl[-1]); await page.wait_for_timeout(600)
        st = await page.evaluate("[window.__samon.mode, window.__samon.G.name, !!window.__samon.share.shared, window.__samon.turnsOf()]")
        expect(st == ['done', G[0]['name'], True, G[0]['par']], f'and that link shows it {st}')
        # #p=: progress as a code; the link offers it, Take it over joins it with what is here (whichever is better stands)
        await page.evaluate("localStorage.clear()"); await page.goto('about:blank'); await page.goto(URL); await page.wait_for_timeout(300); await quiet(page)
        for i in (0, 1, 2):
            await page.evaluate(f"window.__samon.load({i})"); await page.evaluate("(w) => window.__samon.setPath(w)", G[i]['intended'])
            await page.wait_for_function("['reveal', 'done'].includes(window.__samon.mode)", timeout=3000)
            await page.focus('#garden'); await page.keyboard.press('Enter'); await page.wait_for_function("window.__samon.mode === 'done'", timeout=5000)
        await page.evaluate("window.__samon.today = '2026-10-04T12:00:00'; window.__samon.pickDaily()"); await page.wait_for_timeout(200)
        await page.evaluate("(w) => window.__samon.setPath(w)", day['intended']); await page.wait_for_timeout(300); await page.click('#rakeDay')
        await page.wait_for_function("window.__samon.mode === 'done'", timeout=8000)
        await page.evaluate("window.__samon.load(3)"); await page.evaluate("(w) => window.__samon.setPath(w)", G[3]['intended'][:4]); await page.wait_for_timeout(100)
        await page.evaluate("window.__samon.askAbbot()"); await page.wait_for_timeout(500)   # (help on the unfinished walk travels with it; the save is debounced)
        pcode = await page.evaluate("window.__samon.share.progress()")
        mine = json.loads(await page.evaluate("localStorage.getItem('samon-progress')"))
        expect(len(pcode) < 120 and mine[G[3]['name']].get('helped') and len(mine[G[3]['name']]['draft']) == 4, f'the whole progress is a code of {len(pcode)} characters {pcode}')
        await page.evaluate("localStorage.clear()"); await page.goto('about:blank')
        await page.goto(URL); await page.wait_for_timeout(300); await quiet(page)
        await page.evaluate(f"window.__samon.load(1)"); await page.evaluate("(w) => window.__samon.setPath(w)", G[1]['intended'][:3]); await page.wait_for_timeout(400)   # (something of its own here: an unfinished walk on garden 2)
        await page.evaluate("window.__samon.today = '2026-10-04T12:00:00'")
        await page.goto(URL + '#p=' + pcode); await page.wait_for_timeout(500)
        st = await page.evaluate("[document.getElementById('status').textContent, [...document.querySelectorAll('#actions button')].map(b => b.textContent), localStorage.getItem('samon-progress')]")
        expect(st[0] == "[This link carries a player's progress: 3 of 148 gardens raked, 3 sealed, 1 daily garden, and an unfinished walk. Take it over here? It joins what is here, and whichever is better stands.]" and st[1] == ['Take it over', 'Leave it'] and G[0]['name'] not in (st[2] or ''),
               f'a progress link is offered, not taken, with what it holds {st[:2]}')
        tt = await page.evaluate("[...document.querySelectorAll('.veranda button, .assists button, .assists .t, #nav button, #nav .more, #nav .lock, #actions button, #mute, #soundMore, #soundPanel button, #soundPanel input, #orig, .count span, #where, .brand, #seal, .help summary')].filter(e => !e.title).map(e => e.id || e.className || e.textContent)")
        expect(not tt, f'every control, count and mark has a tooltip (gwern.net wants one wherever a hover could want a word more) {tt}')
        await page.click('#actions button >> nth=0'); await page.wait_for_timeout(400)
        got = json.loads(await page.evaluate("localStorage.getItem('samon-progress')"))
        st = await page.evaluate("[document.getElementById('status').textContent, location.hash, window.__samon.G.name, window.__samon.path.length, window.__samon.helped, [...document.querySelectorAll('#nav .pip.sealed')].length, window.__samon.daily.done, window.__samon.daily.turns, window.__samon.daily.streak]")
        expect(all(got[G[i]['name']] == mine[G[i]['name']] for i in (0, 2)) and got[G[1]['name']]['done'] and got[G[1]['name']]['sealed'] and len(got[G[1]['name']]['draft']) == 3
               and got[G[3]['name']]['draft'] == mine[G[3]['name']]['draft'] and got[G[3]['name']].get('helped') and got['daily:2026-10-04']['path'] == mine['daily:2026-10-04']['path'] and got['daily:2026-10-04']['turns'] == day['par'] and got['daily:2026-10-04']['kind'] == 'red'
               and st[0].startswith('[Taken over: 3 of 148 gardens raked') and st[1] == '' and st[2] == G[3]['name'] and st[3] == 4 and st[4] and st[5] == 3 and st[6] and st[7] == day['par'] and st[8] == 1,
               f'Take it over: the rakings, the seals, the day\'s record with its walk, and the unfinished walk with its help arrive, and the walk of its own here is kept {st}')
        await page.evaluate("window.__samon.pickDaily()"); await page.wait_for_timeout(300)
        st = await page.evaluate("[window.__samon.mode, window.__samon.path.length, document.getElementById('status').textContent.includes(\"Today's record\")]")
        expect(st == ['done', len(day['intended']), True], f'the carried day\'s raking shows as the record {st}')
        await page.click('#actions button:has-text("Share today\'s score")'); await page.wait_for_timeout(400)
        cl = (await page.evaluate("navigator.clipboard.readText()")).split('\n')
        shape = await page.evaluate("window.__samon.share.grid([])")
        expect(cl[0] == f"Samon · daily garden 2026-10-04 · {day['par']} turns, par {day['par']} · 静 · streak 1" and cl[1:-1] == shape.split('\n') and cl[-1] == URL + '#daily'
               and not any(ch in shape for ch in '─│┌┐└┘╴╵╶╷') and shape.count('░') == sum(r.count('.') for r in day['rows']),
               f'the daily garden shares the score and the garden\'s shape, without the stroke, and a link to today\'s garden {cl[0]!r} {cl[-1][-8:]!r}')
        await page.evaluate("window.__samon.load(0)"); await page.goto(URL + '#daily'); await page.wait_for_timeout(400)
        st = await page.evaluate("[window.__samon.G.name, location.hash]")
        expect(st == ['daily:2026-10-04', ''], f'#daily opens today\'s garden {st}')
        await page.click('#carry'); await page.wait_for_timeout(400)
        cl = await page.evaluate("navigator.clipboard.readText()"); st = await page.evaluate("document.getElementById('status').textContent")
        expect(cl.startswith(URL + '#p=') and st.startswith('[Copied a link that carries your progress'), f'Carry my progress copies such a link {cl[:60]!r} {st!r}')
        await page.goto('about:blank'); await page.goto(URL + '#r=AAAA'); await page.wait_for_timeout(500)
        st = await page.evaluate("[document.getElementById('status').textContent, document.getElementById('status').className, location.hash, window.__samon.mode]")
        expect(st == ["[The link's code can't be read: it was made by another edition of the game.]", 'status warn', '', 'draw'], f'a code the page can\'t read is said so, and dropped {st}')
        await page.goto(URL + '#p=not a code'); await page.wait_for_timeout(300)
        st = await page.evaluate("[document.getElementById('status').textContent, location.hash]")
        expect(st == ["[The link's code can't be read: it is not a Samon code.]", ''], f'so is a string that is no code at all {st}')
        # durability (Astra's points): a garden is named by a 32-bit layout hash, not its place, so a code survives a reordering of the
        # campaign and a changed garden is refused, never mistaken for the one meant
        B64 = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_'
        await page.evaluate("localStorage.clear()"); await page.goto('about:blank'); await page.goto(URL); await page.wait_for_timeout(300); await quiet(page)
        await page.evaluate("window.__samon.load(5)")
        code5 = await page.evaluate("(w) => window.__samon.share.code(w)", G[5]['intended'])
        bits = ''.join(format(B64.index(ch), '06b') for ch in code5)
        bent = bits[:5] + ('1' if bits[5] == '0' else '0') + bits[6:]     # one bit of the layout hash
        bent = ''.join(B64[int(bent[i:i + 6], 2)] for i in range(0, len(bent), 6))
        await page.goto(URL + '#r=' + bent); await page.wait_for_timeout(400)
        st = await page.evaluate("[document.getElementById('status').textContent, window.__samon.mode, location.hash]")
        expect(st == ["[The link's code can't be read: its garden is not in this edition, or has changed since.]", 'draw', ''], f'a code whose garden hash matches no garden is refused, not shown over another garden {st}')
        await page.evaluate("(w) => window.__samon.setPath(w)", G[5]['intended'])
        await page.wait_for_function("['reveal', 'done'].includes(window.__samon.mode)", timeout=3000)
        await page.focus('#garden'); await page.keyboard.press('Enter'); await page.wait_for_function("window.__samon.mode === 'done'", timeout=5000)
        pcode5 = await page.evaluate("window.__samon.share.progress()")
        await page.evaluate("window.__samon.share.swap(5, 9); window.__samon.load(0)")     # the campaign reordered under the links
        await page.goto(URL + '#r=' + code5); await page.wait_for_timeout(400)
        st = await page.evaluate("[window.__samon.G.name, window.__samon.mode, !!window.__samon.share.shared, window.__samon.idx, document.querySelector('#nav .pip[aria-current=\"true\"]').getAttribute('aria-label')]")
        expect(st == [G[5]['name'], 'done', True, 9, G[5]['name']], f'after the campaign is reordered, a raking link still opens the garden it was made in, now in its new place {st}')
        await page.evaluate("localStorage.clear()")
        await page.goto(URL + '#p=' + pcode5); await page.wait_for_timeout(400)
        await page.click('#actions button:has-text("Take it over")'); await page.wait_for_timeout(400)
        got = json.loads(await page.evaluate("localStorage.getItem('samon-progress')"))
        expect(got.get(G[5]['name'], {}).get('sealed') and G[9]['name'] not in got, f'and carried progress lands on the garden it was earned in, not on the one now at its index {sorted(got)}')
        await page.evaluate("window.__samon.share.swap(5, 9)")
        await ctx.close()
        await b.close()
        real = [e for e in errors if 'ERR_TUNNEL' not in e and 'Failed to load resource' not in e]
        expect(not real, f'no page errors {real}')
        print('FAILS:', fails)

asyncio.run(main())
proto/test/smoke.py · 20 lines · 1.2 kB 
import asyncio, json
from playwright.async_api import async_playwright
URL = 'file:///home/claude/zen/proto/test/index.html'
async def main():
    async with async_playwright() as p:
        b = await p.chromium.launch()
        page = await (await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2)).new_page()
        errs = []
        page.on('console', lambda m: errs.append(m.text) if m.type == 'error' else None)
        page.on('pageerror', lambda e: errs.append('PAGEERROR ' + str(e)))
        await page.goto(URL); await page.wait_for_timeout(300)
        n = await page.evaluate("GARDENS.length")
        for i in range(n):
            r = await page.evaluate(f"(() => {{ window.__samon.load({i}); const a = window.__samon.analyze([window.__samon.G.gateCell]); return [window.__samon.G.name, a.state, window.__samon.G.parK.length]; }})()")
            print(r)
        await page.evaluate("window.__samon.enterPractice()")
        print("practice", await page.evaluate("window.__samon.practice()"))
        print("errors:", [e for e in errs if 'ERR_TUNNEL' not in e and 'Failed to load resource' not in e])
        await b.close()
asyncio.run(main())
proto/test/thumbnail.py · 68 lines · 4.5 kB — The social-media thumbnail (1200×630): a garden raked at par, with its seal, beside the title, in the page's own style. 
"""The social-media thumbnail (1200×630): a garden raked at par, with its seal, beside the title, in the page's own style.

Usage: python3 proto/test/thumbnail.py [garden index]  ->  deliver/2026-10-05-gwern-samon-thumbnail.png (and a preview of the
candidates in the scratchpad when run with 'pick').
"""
import asyncio, base64, os, sys
from playwright.async_api import async_playwright

URL = 'file:///home/claude/zen/proto/test/index.html'
OUT = '/home/claude/zen/deliver/2026-10-05-gwern-samon-thumbnail.png'
PICK = sys.argv[1] if len(sys.argv) > 1 else '25'

COMPOSE = """<!doctype html><html lang="en"><head><meta charset="utf-8"><style>
:root { --fg: #000; --bg: #fff; --muted: #767676; --seal: #c23b22; }
body { margin: 0; width: 1200px; height: 630px; background: var(--bg); color: var(--fg); font-family: "Source Serif 4", "Apple Garamond", Baskerville, "Libre Baskerville", "Hiragino Mincho ProN", "Yu Mincho", "Noto Serif CJK JP", "Times New Roman", Times, serif; display: grid; grid-template-columns: 630px 1fr; }
.garden { padding: 36px 0 36px 44px; display: grid; align-items: center; }
.garden img { width: 100%; display: block; }
.text { padding: 0 56px 0 48px; display: grid; align-content: center; gap: 18px; }
h1 { margin: 0; font-size: 92px; line-height: 1; font-weight: 700; letter-spacing: 0.01em; }
h1 span { display: block; font-size: 34px; font-weight: 400; color: var(--muted); letter-spacing: 0.22em; margin-top: 14px; }
p { margin: 0; font-size: 27px; line-height: 1.35; }
p.rule { color: var(--fg); }
p.site { font-family: "Source Sans 3", "Source Sans Pro", "Lucida Sans Unicode", Helvetica, sans-serif; font-size: 20px; letter-spacing: 0.14em; text-transform: uppercase; color: var(--muted); margin-top: 10px; }
.mark { display: inline-block; width: 16px; height: 16px; background: var(--seal); vertical-align: -1px; margin-right: 10px; transform: rotate(-6deg); }
</style></head><body>
<div class="garden"><img src="data:image/png;base64,__IMG__" alt=""></div>
<div class="text">
  <h1>Samon<span>砂紋</span></h1>
  <p class="rule">A Zen-garden raking puzzle: walk the monk in from the gate, cover every cell of sand in one stroke, and he rakes his way back out.</p>
  <p class="site"><span class="mark"></span>gwern.net · design gwern.net · design &amp; playable prototypeamp; prototype</p>
</div>
</body></html>"""


async def shot(page, gi, path):
    await page.evaluate(f"window.__samon.load({gi})"); await page.wait_for_timeout(150)
    G = await page.evaluate("window.__samon.G")
    await page.evaluate("(w) => window.__samon.setPath(w)", G['intended'])
    await page.wait_for_function("window.__samon.mode === 'done'", timeout=20000); await page.wait_for_timeout(500)
    r = await page.evaluate("(() => { const r = document.getElementById('garden').getBoundingClientRect(); return [r.left, r.top, r.width, r.height]; })()")
    await page.screenshot(path=path, clip={'x': r[0], 'y': r[1], 'width': r[2], 'height': r[3]})
    return G['name']


async def main():
    async with async_playwright() as p:
        b = await p.chromium.launch()
        ctx = await b.new_context(viewport={'width': 560, 'height': 900}, device_scale_factor=2, color_scheme='light')
        page = await ctx.new_page()
        await page.goto(URL); await page.wait_for_timeout(300)
        await page.evaluate("localStorage.setItem('samon-open-all', '1'); localStorage.setItem('samon-started', '1'); localStorage.setItem('samon-sound-hint', '1'); localStorage.removeItem('samon-assists')")
        await page.reload(); await page.wait_for_timeout(400)
        await page.evaluate("window.__samon.poemTiming = {delay: 3600000}")
        scratch = '/tmp/claude-0/-home-claude/1d7d7911-7ec3-59eb-91d4-be08299e88f7/scratchpad/'
        if PICK == 'pick':
            for gi in (22, 23, 25, 26, 28, 47, 49):
                name = await shot(page, gi, f'{scratch}thumb-cand-{gi}.png'); print(gi, name)
            await b.close(); return
        gi = int(PICK)
        name = await shot(page, gi, scratch + 'thumb-garden.png')
        img = base64.b64encode(open(scratch + 'thumb-garden.png', 'rb').read()).decode()
        comp = await (await b.new_context(viewport={'width': 1200, 'height': 630}, device_scale_factor=1)).new_page()
        await comp.set_content(COMPOSE.replace('__IMG__', img)); await comp.wait_for_timeout(300)
        await comp.screenshot(path=OUT, clip={'x': 0, 'y': 0, 'width': 1200, 'height': 630})
        print('wrote', OUT, 'from', name)
        await b.close()

asyncio.run(main())
proto/test/wrap.py · 3 lines · 239 bytes — Test page = the standalone build, with no extra base styles (so the test can't mask a missing rule in the page). 
"""Test page = the standalone build, with no extra base styles (so the test can't mask a missing rule in the page)."""
import shutil
shutil.copyfile('/home/claude/zen/proto/samon.standalone.html', '/home/claude/zen/proto/test/index.html')

This single-file edition and its test ( standalone/ )

standalone/build_standalone.py · 543 lines · 31.8 kB — Build one fully self-contained HTML file: the Samon design document (from samon.md, via Pandoc) with the playable 
"""Build one fully self-contained HTML file: the Samon design document (from samon.md, via Pandoc) with the playable
prototype embedded in the same page, and an appendix with every file of the code bundle (code shown inline; everything
in an embedded .zip). No external resources: figures are embedded SVG data URIs, no web fonts, no scripts or styles
from elsewhere; only ordinary hyperlinks point outside.

Usage: python3 build_standalone.py  ->  /home/claude/zen/standalone/2026-10-03-gwern-samon.html
"""
import ast
import base64
import hashlib
import html
import io
import json
import os
import re
import subprocess
import sys
import zipfile

from bs4 import BeautifulSoup

ROOT = "/home/claude/zen"
OUT = f"{ROOT}/standalone/2026-10-03-gwern-samon.html"
FIG = f"{ROOT}/fig/final"


def must_replace(s, old, new, count=1):
    n = s.count(old)
    if n != count:
        sys.exit(f"expected {count} occurrence(s) of {old[:70]!r}, found {n}")
    return s.replace(old, new)


# ---------------------------------------------------------------- 1. the document: Gwerndown -> Pandoc Markdown
md = open(f"{ROOT}/samon.md").read()
m = re.match(r"^---\n(.*?)\n\.\.\.\n", md, re.S)
if not m:
    sys.exit("no YAML header")
meta = {}
for line in m.group(1).splitlines():
    k, _, v = line.partition(":")
    meta[k.strip()] = v.strip().strip('"')
body = md[m.end():]
body = re.sub(r"<!--.*?-->\n?", "", body, flags=re.S)          # handoff notes and TODOs are for the source, not the page


def wiki(mo):                                                   # gwern.net's [text](!W) and [text](!W "Article")
    text, title = mo.group(1), mo.group(2)
    article = title or re.sub(r"[_*]", "", text)
    url = "https://en.wikipedia.org/wiki/" + article.replace(" ", "_")
    return f'[{text}]({url} "{article} (Wikipedia)")'


body, nwiki = re.subn(r'\[((?:[^\[\]]|\[[^\]]*\])*)\]\(!W(?:\s+"([^"]*)")?\)', wiki, body)
# point the document's mentions of the prototype at the copy in this page
body = must_replace(body, "A playable prototype implements most of it, with the whole campaign of 148 gardens and their poems.",
                    "A playable prototype implements most of it, with the whole campaign of 148 gardens and their poems: [play it below](#play).")
body = must_replace(body, "[Prototype.]{.margin-note} A one-page HTML prototype, included with the code, implements",
                    "[Prototype.]{.margin-note} The one-page prototype [at the top of this page](#play) implements")
body = must_replace(body, "Everything is reproducible from the attached code.",
                    "Everything is reproducible from the code, which is in the [appendix](#code).")

html_doc = subprocess.run(["pandoc", "-f", "markdown+smart", "-t", "html5", "--section-divs", "--wrap=none"],
                          input=body, capture_output=True, text=True, check=True).stdout
soup = BeautifulSoup(html_doc, "html.parser")

# figures: embed the SVG renders as data URIs
nimg = 0
for img in soup.find_all("img"):
    src = img.get("src", "")
    mo = re.match(r"^/doc/design/(.+)\.png$", src)
    if not mo:
        sys.exit(f"unexpected image {src}")
    svg = open(f"{FIG}/{mo.group(1)}.svg", "rb").read()
    img["src"] = "data:image/svg+xml;base64," + base64.b64encode(svg).decode()
    img["decoding"] = "async"
    if not img.get("alt"):
        img["alt"] = ""
    nimg += 1
# site-relative links -> gwern.net
for a in soup.find_all("a", href=True):
    if a["href"].startswith("/"):
        a["href"] = "https://gwern.net" + a["href"]
# collapsed sections -> <details>
for sec in soup.find_all("section", class_="collapse"):
    head = sec.find(re.compile(r"^h[1-6]$"))
    det = soup.new_tag("details", attrs={"class": "collapse " + " ".join(c for c in sec.get("class", []) if c.startswith("level")), "id": sec.get("id", "")})
    summ = soup.new_tag("summary")
    head.extract()
    if head.get("class"):
        del head["class"]
    head["id"] = (sec.get("id") or "") + "-h"
    summ.append(head)
    det.append(summ)
    for child in list(sec.children):
        det.append(child.extract())
    sec.replace_with(det)
# a footnote's reference shows the footnote itself on hover (gwern.net shows it in a popup); a collapsed section's summary says what it does
for a in soup.find_all("a", class_="footnote-ref", href=True):
    fn = soup.find(id=a["href"].lstrip("#"))
    if fn:
        a["title"] = re.sub(r"\s*↩︎?\s*$", "", " ".join(fn.get_text(" ").split()))
for summ in soup.find_all("summary"):
    summ["title"] = "Click to show or hide this section"
# tables scroll sideways on narrow screens
for t in soup.find_all("table"):
    wrap = soup.new_tag("div", attrs={"class": "table-scroll"})
    t.wrap(wrap)
# the abstract and what follows it
abstract = soup.find("div", class_="abstract")
abstract.extract()
first_p = soup.find("p")
first_p["class"] = first_p.get("class", []) + ["dropcap"]
# table of contents from the level-1 and level-2 headings
toc_items = []
for h in soup.find_all(["h1", "h2"]):
    hid = h.get("id") or (h.find_parent(["section", "details"]) or {}).get("id")
    if h.get("id", "").endswith("-h"):
        hid = h["id"][:-2]
    if not hid:
        continue
    toc_items.append((h.name, hid, h.get_text(" ", strip=True)))
APPENDIX_TITLE = "Appendix: Code and Data"
toc_items.append(("h1", "code", APPENDIX_TITLE))                 # built in part 4
toc = ['<nav class="toc" aria-label="Contents"><div class="toc-title">Contents</div><ol>',
       '<li><a href="#play">Play the prototype</a></li>']
open_sub = False
for lvl, hid, text in toc_items:
    if lvl == "h1":
        if open_sub:
            toc.append("</ol></li>")
            open_sub = False
        toc.append(f'<li><a href="#{hid}">{html.escape(text)}</a>')
        toc.append("<ol>")
        open_sub = True
    else:
        toc.append(f'<li><a href="#{hid}">{html.escape(text)}</a></li>')
if open_sub:
    toc.append("</ol></li>")
toc.append("</ol></nav>")
toc_html = re.sub(r"<li><a ([^>]*)>([^<]*)</a><ol></ol></li>", r"<li><a \1>\2</a></li>", "".join(toc))

# ---------------------------------------------------------------- 2. the game, from the prototype template
tpl = open(f"{ROOT}/proto/samon.template.html").read()
gardens = json.load(open(f"{ROOT}/out/proto_gardens.json"))
sys.path.insert(0, f"{ROOT}/src")
from poems import attach
from sound_pieces import pieces_html
practice_poem = attach(gardens)        # each garden's poem, and the practice garden's (poems/samon-poems.json)
game_css = tpl[tpl.index("<style>") + 7: tpl.index("</style>")]
game_markup = tpl[tpl.index('<div class="wrap">'): tpl.index("<script>")].strip()
game_markup = must_replace(game_markup, "__PIECES__", pieces_html())   # the recordings (out/sound_pieces.json; sound/make_sound_bundle.py)
game_js = tpl[tpl.index("<script>") + 8: tpl.rindex("</script>")]
game_js = must_replace(game_js, "__GARDENS__", json.dumps(gardens, separators=(",", ":"), ensure_ascii=False))
game_js = must_replace(game_js, "__PRACTICE_POEM__", json.dumps(practice_poem, separators=(",", ":"), ensure_ascii=False))
DAILY_PATH = f"{ROOT}/out/daily_gardens.json"
daily = json.load(open(DAILY_PATH)) if os.path.exists(DAILY_PATH) else []
if not daily:
    print("warning: no daily gardens (out/daily_gardens.json missing)")
game_js = must_replace(game_js, "__DAILY__", json.dumps(daily, separators=(",", ":"), ensure_ascii=False))
game_js = must_replace(game_js, "const VIEW_EL = document.body;", "const VIEW_EL = document.getElementById(\"play\");")
if "document.body.classList" in game_js:
    sys.exit("the game puts a view class on document.body; the single-file edition scopes its styles to #play, so use VIEW_EL")
game_js = must_replace(game_js, "window.claude?.hot?.snapshot?.(() => ({ idx, view, path: mode === \"draw\" ? path : [] }));\n", "")
game_js = must_replace(game_js, "window.claude?.hot?.ready ? window.claude.hot.ready(start) : start(window.claude?.hot?.data ?? {});", "start({});")
if "</script" in game_js.lower():
    sys.exit("game script contains a closing script tag")


def prefix_css(css, pre="#play"):
    """Scope the prototype's stylesheet to the game's container (it was written to own the whole page)."""
    css = re.sub(r"/\*.*?\*/", "", css, flags=re.S)
    out, i, n = [], 0, len(css)
    while i < n:
        j = css.find("{", i)
        if j < 0:
            break
        sel = css[i:j].strip()
        depth, k = 0, j
        while k < n:
            depth += css[k] == "{"
            depth -= css[k] == "}"
            if depth == 0:
                break
            k += 1
        block = css[j + 1:k]
        i = k + 1
        if sel.startswith("@media"):
            out.append(f"{sel} {{\n{prefix_css(block, pre)}\n}}")
        elif sel.startswith("@keyframes"):
            out.append(f"{sel} {{{block}}}")
        elif sel in ("html, body", "body"):
            continue                                            # the page styles its own body
        else:
            parts = []
            for s in (x.strip() for x in sel.split(",")):
                if s.startswith(":root") or s.startswith("[hidden]"):
                    parts.append(s)
                elif s.startswith("body"):
                    parts.append(pre + s[4:])
                else:
                    parts.append(f"{pre} {s}")
            out.append(f"{', '.join(parts)} {{{block}}}")
    return "\n".join(out)


game_css_scoped = prefix_css(game_css)

# ---------------------------------------------------------------- 3. the appendix: every file behind the page, readable
# here, and all of them as one .zip (embedded as base64; a .zip because that is a format the claude.ai viewer can save)
BUNDLE = f"{ROOT}/deliver/samon-code" if os.path.isdir(f"{ROOT}/deliver/samon-code") else ROOT
ZIP_NAME = "2026-10-03-gwern-samon-code.zip"
SHOWN_EXT = {".py", ".c", ".md", ".html", ".sh", ".log"}   # code and logs are shown; data, figures and caches are only in the .zip
SHOWN_ALSO = {"poems/samon-poems.json", "sound/tracks.json"}   # (the poems and the recordings' credits: editions rather than machine output)
GROUPS = [("", "The guide and the document"), ("src/", "Solvers, experiments, the level generator, and the document's checks"),
          ("proto/", "The prototype and its tests"), ("standalone/", "This single-file edition and its test"),
          ("poems/", "The poems"), ("sound/", "The sounds: the recordings' credits and their encoder (the synthesis is in the prototype)"),
          ("out/", "Data: levels, experiment results, logs, and the campaign cache"),
          ("fig/", "Figures (shown in the document as SVG; PNG renders for the web)")]


def generated(p):              # this page, the prototype pages build.py writes, and local build or test output
    return (p in {"standalone/2026-10-03-gwern-samon.html", "proto/samon.html", "proto/samon.standalone.html", "src/zgsolve"}
            or "__pycache__" in p.split("/") or p.startswith("standalone/shots/")
            or (p.startswith("proto/test/") and not p.endswith(".py")))


files = sorted(os.path.relpath(os.path.join(d, f), BUNDLE).replace(os.sep, "/") for d, _, fs in os.walk(BUNDLE) for f in fs)
files = [p for p in files if not generated(p)]
for want in ("README.md", "samon.md", "src/zgsolve.c", "src/zg.py", "src/zcp.py", "src/grader.py", "src/verify_doc.py",
             "proto/samon.template.html", "proto/test/run.py", "standalone/build_standalone.py", "poems/samon-poems.json"):
    if want not in files:
        sys.exit(f"the code bundle lacks {want}")
modified = tuple(int(x) for x in meta["modified"].split("-")) + (0, 0, 0)
zbuf = io.BytesIO()
with zipfile.ZipFile(zbuf, "w") as z:              # deterministic: sorted, fixed dates and modes
    for p in files:
        zi = zipfile.ZipInfo(f"samon-code/{p}", date_time=modified)
        zi.compress_type = zipfile.ZIP_DEFLATED
        zi.external_attr = 0o644 << 16
        z.writestr(zi, open(f"{BUNDLE}/{p}", "rb").read(), compresslevel=9)
zip_bytes = zbuf.getvalue()
zip_sha = hashlib.sha256(zip_bytes).hexdigest()
open(f"{ROOT}/standalone/{ZIP_NAME}", "wb").write(zip_bytes)      # (the test compares the page's copy with this one)


def describe(p, text):
    """A file's own one-line description: a Python module's docstring, or a C file's opening comment."""
    first = ""
    if p.endswith(".py"):
        try:
            doc = (ast.get_docstring(ast.parse(text)) or "").strip()
        except SyntaxError:
            doc = ""
        first = doc.splitlines()[0] if doc else ""
    elif p.endswith(".c"):
        mo = re.match(r"\s*/\*+\s*(.*)", text)
        first = mo.group(1).strip() if mo else ""
    first = re.sub(r"^[\w.]+\.(py|c):\s*", "", first)
    return first[:1].upper() + first[1:] if first else ""


def human(n):
    return f"{n / 1e6:.1f} MB" if n >= 1e6 else f"{n / 1e3:.1f} kB" if n >= 1e3 else f"{n} bytes"


def fid(p):
    return "file-" + re.sub(r"[^A-Za-z0-9]+", "-", p).strip("-").lower()


readme_html = subprocess.run(["pandoc", "-f", "markdown+smart", "-t", "html5", "--wrap=none", "--shift-heading-level-by=2", "--id-prefix=readme-"],
                             input=open(f"{BUNDLE}/README.md").read(), capture_output=True, text=True, check=True).stdout
readme_html = re.sub(r"<h3[^>]*>.*?</h3>\s*", "", readme_html, count=1, flags=re.S)       # its title; the appendix has one
grouped = {g: [] for g, _ in GROUPS}
for p in files:
    g = next((g for g, _ in GROUPS[1:] if p.startswith(g)), "")
    grouped[g].append(p)
if sum(map(len, grouped.values())) != len(files):
    sys.exit("a file fell outside the appendix's groups")
shown = archived = 0
parts = []
for g, title in GROUPS:
    if not grouped[g]:
        continue
    gid = "code-" + (g.strip("/") or "guide")
    parts.append(f'<h2 id="{gid}">{html.escape(title)}{f" (<code>{g}</code>)" if g else ""}</h2>')
    rows = []
    for p in grouped[g]:
        raw = open(f"{BUNDLE}/{p}", "rb").read()
        ext = os.path.splitext(p)[1]
        try:
            text = raw.decode("utf-8")
        except UnicodeDecodeError:
            text = None
        inline = text is not None and (ext in SHOWN_EXT or p in SHOWN_ALSO)
        size = human(len(raw))
        if inline:
            shown += 1
            lines = text.count("\n") + (not text.endswith("\n"))
            desc = describe(p, text)
            parts.append(f'<details class="file" id="{fid(p)}" data-path="{html.escape(p)}"><summary title="Show or hide this file (shown exactly as it is in the .zip)"><code>{html.escape(p)}</code>'
                         f'<span class="file-meta"> · {lines:,} line{"s" if lines != 1 else ""} · {size}</span>'
                         f'{f"<span class=file-desc> — {html.escape(desc)}</span>" if desc else ""}</summary>'
                         f'<pre><code>{html.escape(text, quote=False)}</code></pre></details>')
        else:
            archived += 1
            why = ("figure source; the figure is shown in the document" if ext == ".svg" and p.startswith("fig/")
                   else "image" if ext == ".png" else "binary" if text is None else "data")
            rows.append(f'<li><code>{html.escape(p)}</code> <span class="file-meta">· {size} · {why}; in the .zip</span></li>')
    if rows:
        parts.append(f'<ul class="file-list">{"".join(rows)}</ul>')
start = "".join(f'<a href="#{fid(p)}"><code>{p}</code></a>{s}' for p, s in (
    ("src/zgsolve.c", " (exact search), "), ("src/zg.py", " (the rules, the solver driver and a brute-force checker), "),
    ("src/zcp.py", " (CP-SAT par proofs), "), ("src/grader.py", " (the deductive grader), and "),
    ("src/verify_doc.py", ", which re-derives every number in the document")))
appendix_html = f"""<details class="collapse level1 appendix" id="code"><summary><h1 id="code-h">{APPENDIX_TITLE}</h1></summary>
<div class="abstract-collapse">Every file behind this page: the solvers, the experiments, the level generator, the prototype's source and its tests, this edition's build script and test, the poems, and the data they wrote. The code, the logs and the poems are shown below; the data, the figures' sources and the campaign cache are listed, and everything is in the download.</div>
<p class="bundle"><a id="bundle-dl" href="#code" download="{ZIP_NAME}" title="Save the .zip carried in this page: every file behind it, as listed below"><strong>Download all {len(files)} files</strong></a> as one .zip ({human(len(zip_bytes))}; SHA-256 <code class="sha">{zip_sha}</code>).<span id="bundle-msg" role="status"></span></p>
<p>Start with {start}. The build and test commands are in the guide below. Not included: this page itself and the prototype's two pages, which <code>standalone/build_standalone.py</code> and <code>proto/build.py</code> write.</p>
<div class="readme">{readme_html}</div>
{"".join(parts)}
<script type="text/plain" class="asset" id="bundle-zip">data:application/zip;base64,{base64.b64encode(zip_bytes).decode()}</script>
</details>"""
print(f"appendix: {len(files)} files ({shown} shown, {archived} only in the .zip); .zip {len(zip_bytes):,} bytes")

DOWNLOAD_JS = r"""
(function () {   // the appendix's download: the viewer's save prompt inside claude.ai, a plain download elsewhere
  const a = document.getElementById("bundle-dl"), msg = document.getElementById("bundle-msg");
  if (!a) return;
  const name = a.getAttribute("download");
  const viewer = (window.claude && typeof window.claude.use === "function") ? window.claude.use("downloads").catch(() => null) : Promise.resolve(null);
  let blob = null;
  async function zip() {   // the .zip is a data: URL in the page, or (in a gwtar build) a member of the archive beside it: samonAsset reads either
    if (!blob) blob = new Blob([await samonAsset.bytes(samonAsset.text(document.getElementById("bundle-zip")))], { type: "application/zip" });
    return blob;
  }
  a.addEventListener("click", async ev => {
    ev.preventDefault(); msg.textContent = "";
    const dl = await viewer;
    let z;
    try { z = await zip(); } catch (e) { msg.textContent = " [The .zip could not be read from this copy of the page.]"; return; }
    if (dl) {
      try { await dl.save({ filename: name, data: z }); msg.textContent = " [Saved.]"; }
      catch (e) { if (!e || e.code !== "declined") msg.textContent = ` [It could not be saved here (${(e && e.code) || "error"}).]`; }
      return;
    }
    const url = URL.createObjectURL(z), t = document.createElement("a");
    t.href = url; t.download = name; document.body.appendChild(t); t.click(); t.remove();
    setTimeout(() => URL.revokeObjectURL(url), 60000);
  });
  function reveal() {   // a link to a file (or to anything inside a collapsed section) opens what hides it
    const id = decodeURIComponent(location.hash.slice(1)), t = id && document.getElementById(id);
    if (!t) return;
    let opened = false;
    if (t.matches("details.file") && !t.open) { t.open = true; opened = true; }
    for (let el = t.parentElement; el; el = el.parentElement) if (el.tagName === "DETAILS" && !el.open) { el.open = true; opened = true; }
    if (opened) t.scrollIntoView({ block: "start" });
  }
  window.addEventListener("hashchange", reveal); reveal();
})();
"""

# ---------------------------------------------------------------- 4. the page
PAGE_CSS = r"""
/* gwern.net's manner (gwern.net/design): black on white, greys for the secondary, square corners, hairlines, serif text
   with sans apparatus. No web fonts: gwern.net's own stacks, with the system's CJK fonts for 砂紋, the Japanese and 静. */
:root {
  --font-text: var(--font-display);
  --plate: #ffffff;
  --col: 42rem;
}
* { box-sizing: border-box; }
html { -webkit-text-size-adjust: 100%; }
body { margin: 0; background: var(--bg); color: var(--fg); font-family: var(--font-text); font-size: 1.0625rem; line-height: 1.6; padding: 0 16px; }
a { color: inherit; text-decoration: underline; text-decoration-color: color-mix(in srgb, var(--fg) 35%, transparent); text-underline-offset: 2px; }
a:hover { color: var(--muted); text-decoration-color: var(--muted); }
.page { max-width: var(--col); margin: 0 auto; padding: 28px 0 64px; }
.topbar { display: flex; justify-content: space-between; align-items: center; gap: 12px; flex-wrap: wrap; font-family: var(--font-ui); font-size: 0.8125rem; color: var(--muted); }
.topbar .jump a { margin-right: 14px; }
.theme { display: inline-flex; border: 1px solid var(--line); overflow: hidden; }
.theme button { font: inherit; background: transparent; color: var(--muted); border: 0; border-radius: 0; padding: 4px 10px; cursor: pointer; }
.theme button:hover { background: color-mix(in srgb, var(--fg) 8%, transparent); }
.theme button[aria-pressed="true"] { background: var(--fg); color: var(--bg); }
h1.title { font-family: var(--font-display); font-weight: 700; font-size: clamp(1.9rem, 6vw, 2.6rem); line-height: 1.15; margin: 22px 0 8px; text-wrap: balance; }
.subtitle { color: var(--muted); font-style: italic; font-size: 0.98rem; margin: 0 0 10px; }
.meta { font-family: var(--font-ui); font-size: 0.8125rem; color: var(--muted); margin-bottom: 22px; }
.abstract { border-top: 1px solid var(--line); border-bottom: 1px solid var(--line); padding: 14px 0 4px; margin: 0 0 26px; font-size: 1rem; }
.abstract p { margin: 0 0 12px; }
#play { margin: 0 auto 30px; padding: 18px 0 6px; border-bottom: 1px solid var(--line); font-family: var(--font-ui); font-size: var(--step-0); line-height: 1.55; }
#play .wrap { padding-block: 0 18px; }
.play-head { font-family: var(--font-ui); font-size: 0.8125rem; letter-spacing: 0.14em; text-transform: uppercase; color: var(--muted); max-width: 600px; margin: 0 auto 8px; }
.toc { border: 1px solid var(--line); background: color-mix(in srgb, var(--fg) 3%, transparent); padding: 12px 16px; margin: 0 0 28px; font-family: var(--font-ui); font-size: 0.875rem; line-height: 1.5; }
.toc-title { font-size: 0.75rem; letter-spacing: 0.14em; text-transform: uppercase; color: var(--muted); margin-bottom: 6px; }
.toc ol { margin: 0; padding-left: 1.3em; }
.toc ol ol { padding-left: 1.1em; color: var(--muted); }
@media (min-width: 700px) { .toc > ol { columns: 2; column-gap: 28px; } .toc > ol > li { break-inside: avoid; } }
.doc h1, .doc h2, details.collapse summary h1, details.collapse summary h2 { font-family: var(--font-display); font-weight: 700; line-height: 1.25; }
.doc h1 { font-size: 1.6rem; margin: 2.2em 0 0.6em; border-bottom: 1px solid #888; padding-bottom: 4px; }
.doc h2 { font-size: 1.25rem; margin: 1.8em 0 0.5em; }
.doc p { margin: 0 0 0.9em; position: relative; }
.doc p.dropcap::first-letter { float: left; font-family: var(--font-display); font-size: 3.3em; line-height: 0.85; padding: 0.06em 0.08em 0 0; color: var(--fg); }
.margin-note { font-family: var(--font-ui); font-size: 0.78rem; font-weight: 700; letter-spacing: 0.02em; color: var(--muted); }
.margin-note::after { content: " "; }
@media (min-width: 1180px) {
  .doc p > .margin-note:first-child, .doc li > .margin-note:first-child { position: absolute; left: -12.5rem; width: 11rem; text-align: right; top: 0.25em; line-height: 1.35; font-weight: 600; }
}
.doc ul, .doc ol { padding-left: 1.4em; }
.doc li { margin-bottom: 0.35em; }
.doc figure { margin: 1.4em 0; }
.doc figure img { display: block; width: 100%; height: auto; background: var(--plate); border: 1px solid var(--line); }
.doc figure img:not(.width-full) { max-width: 30rem; margin: 0 auto; }
.doc figcaption { font-size: 0.9rem; color: var(--muted); margin-top: 8px; line-height: 1.45; }
.table-scroll { overflow-x: auto; margin: 1.2em 0; }
.doc table { border-collapse: collapse; width: 100%; font-family: var(--font-ui); font-size: 0.82rem; font-variant-numeric: tabular-nums; }
.doc caption { caption-side: bottom; text-align: left; color: var(--muted); font-size: 0.8rem; padding-top: 6px; font-family: var(--font-text); }
.doc th, .doc td { padding: 5px 8px; border-bottom: 1px solid var(--line); vertical-align: top; }
.doc thead th { border-bottom: 2px solid var(--fg); }
.doc tbody tr:nth-child(odd) { background: color-mix(in srgb, var(--fg) 3.5%, transparent); }
details.collapse { margin: 2em 0 1em; border-top: 1px solid var(--line); }
details.collapse > summary { cursor: pointer; list-style: none; display: flex; align-items: baseline; gap: 10px; }
details.collapse > summary::-webkit-details-marker { display: none; }
details.collapse > summary::before { content: "▸"; font-family: var(--font-ui); color: var(--muted); }
details.collapse[open] > summary::before { content: "▾"; }
details.collapse > summary h1 { font-size: 1.6rem; margin: 0.6em 0; }
details.collapse > summary h2 { font-size: 1.25rem; margin: 0.6em 0; }
details.collapse > summary::after { content: "[click to expand]"; font-family: var(--font-ui); font-size: 0.75rem; color: var(--muted); }
details.collapse[open] > summary::after { content: ""; }
.abstract-collapse { font-style: italic; color: var(--muted); }
.footnotes { font-size: 0.9rem; border-top: 1px solid var(--line); margin-top: 2.5em; padding-top: 0.5em; }
.footnotes hr { display: none; }
sup { line-height: 0; }
.colophon { font-family: var(--font-ui); font-size: 0.78rem; color: var(--muted); border-top: 1px solid var(--line); margin-top: 3em; padding-top: 10px; }
/* the appendix: code listings */
.appendix h2 { font-size: 1.15rem; }
.appendix .readme { font-size: 0.92rem; }
.appendix .readme h4 { font-family: var(--font-ui); font-size: 0.8rem; letter-spacing: 0.12em; text-transform: uppercase; color: var(--muted); margin: 1.4em 0 0.4em; }
.appendix code, .appendix pre { font-family: "IBM Plex Mono", "Liberation Mono", Consolas, Menlo, Courier, monospace; }
.appendix :not(pre) > code { font-size: 0.86em; }
.appendix .sha { word-break: break-all; font-size: 0.78em; }
.appendix .bundle { font-family: var(--font-ui); font-size: 0.95rem; }
details.file { border-bottom: 1px solid color-mix(in srgb, var(--line) 60%, transparent); }
details.file > summary { cursor: pointer; font-family: var(--font-ui); font-size: 0.86rem; line-height: 1.45; padding: 5px 0; }
details.file > summary code { font-size: 0.95em; }
.file-meta, .file-desc { color: var(--muted); }
details.file pre { margin: 4px 0 12px; padding: 10px 12px; font-size: 0.74rem; line-height: 1.45; tab-size: 4; overflow-x: auto; background: color-mix(in srgb, var(--fg) 4%, transparent); border: 1px solid var(--line); }
.file-list { font-family: var(--font-ui); font-size: 0.82rem; columns: 1; }
.doc .file-list li { margin-bottom: 0.15em; }
@media (prefers-color-scheme: dark) { :root:not([data-theme="light"]) { --plate: #f4f5f2; } }
:root[data-theme="dark"] { --plate: #f4f5f2; }
"""

THEME_JS = r"""
(function () {
  const root = document.documentElement, KEY = "samon-theme";
  const btns = document.querySelectorAll(".theme button");
  function apply(t) {
    if (t === "light" || t === "dark") root.setAttribute("data-theme", t); else root.removeAttribute("data-theme");
    btns.forEach(b => b.setAttribute("aria-pressed", String(b.dataset.t === (t || "auto"))));
  }
  let saved = "auto"; try { saved = localStorage.getItem(KEY) || "auto"; } catch (e) {}
  apply(saved);
  btns.forEach(b => b.addEventListener("click", () => { apply(b.dataset.t); try { localStorage.setItem(KEY, b.dataset.t); } catch (e) {} }));
})();
"""

dates = f"{meta.get('created', '')}–{meta.get('modified', '')}"
title_html = re.sub(r"_([^_]+)_", r"<em>\1</em>", html.escape(meta["title"]))
title_text = re.sub(r"_", "", meta["title"])
# Social cards (Open Graph; Twitter reads the og: tags too, plus its own card type). The thumbnail is a garden raked at par
# (proto/test/thumbnail.py -> deliver/2026-10-05-gwern-samon-thumbnail.png); crawlers need an absolute URL, so this is where
# gwern.net would host it: adjust THUMB_URL when the page is placed.
THUMB_URL = "https://gwern.net" + meta["thumbnail"]          # (the document's own thumbnail field)
SOCIAL_SUMMARY = ("A Zen-garden raking puzzle: a novice monk walks into the sand and must rake his way back out to the gate in one stroke. "
                  "The full design, solver experiments, 148 gardens and a year of daily ones, and a playable prototype, in one page.")
SOCIAL_META = f"""<meta property="og:type" content="article">
<meta property="og:site_name" content="Gwern.net">
<meta property="og:title" content="{html.escape(title_text)}">
<meta property="og:description" content="{html.escape(SOCIAL_SUMMARY)}">
<meta property="og:image" content="{THUMB_URL}">
<meta property="og:image:width" content="1200">
<meta property="og:image:height" content="630">
<meta property="og:image:alt" content="{html.escape(meta['thumbnail-text'])}">
<meta property="og:locale" content="en_US">
<meta property="article:author" content="{html.escape(meta['author'])}">
<meta property="article:published_time" content="{meta.get('created', '')}">
<meta property="article:modified_time" content="{meta.get('modified', '')}">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:site" content="@gwern">
<meta name="twitter:creator" content="@gwern">
<meta name="twitter:title" content="{html.escape(title_text)}">
<meta name="twitter:description" content="{html.escape(SOCIAL_SUMMARY)}">
<meta name="twitter:image" content="{THUMB_URL}">
<meta name="twitter:image:alt" content="{html.escape(meta['thumbnail-text'])}">"""

page = f"""<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>{html.escape(title_text)}</title>
<meta name="description" content="{html.escape(meta['description'])}">
<meta name="author" content="{html.escape(meta['author'])}">
{SOCIAL_META}
<style>
{game_css_scoped}
{PAGE_CSS}
</style>
</head>
<body>
<div class="page">
  <div class="topbar">
    <span class="jump"><a href="#play" title="The playable prototype, at the top of the page">Play</a><a href="#contents" title="The document’s table of contents">Contents</a><a href="#methods" title="How the gardens were made and checked: solvers, proofs of par, the grader">Methods</a><a href="#code" title="The appendix: every file behind this page, shown and downloadable">Code</a></span>
    <span class="theme" role="group" aria-label="Color theme" title="The page’s colours: as the system has them, or light or dark regardless"><button type="button" data-t="auto" title="Follow the system’s light or dark setting">Auto</button><button type="button" data-t="light" title="Light, whatever the system’s setting">Light</button><button type="button" data-t="dark" title="Dark, whatever the system’s setting">Dark</button></span>
  </div>
  <h1 class="title">{title_html}</h1>
  <p class="subtitle">{html.escape(meta['description'])}</p>
  <div class="meta">{html.escape(meta['author'])} · {dates} · status: {html.escape(meta.get('status', ''))} · confidence: {html.escape(meta.get('confidence', ''))}</div>
  {str(abstract)}
  <div class="play-head">[The prototype: {len(gardens)} gardens, playable here]</div>
  <section id="play" class="game" aria-label="Playable prototype">
{game_markup}
  </section>
  <div id="contents">{toc_html}</div>
  <article class="doc">
{str(soup)}
{appendix_html}
  </article>
  <p class="colophon">[Single-file edition: the design document, the playable prototype, and all the code and data behind them in one self-contained page (figures and files embedded; no fonts, scripts, or images loaded from elsewhere). The solvers, experiments, tests and data are in the <a href="#code">appendix</a>.]</p>
</div>
<script>
{THEME_JS}
</script>
<script>
{DOWNLOAD_JS}
</script>
<script>
{game_js}
</script>
</body>
</html>
"""
open(OUT, "w").write(page)
print(f"wrote {OUT}: {len(page):,} bytes; {nwiki} Wikipedia links resolved; {nimg} figures embedded; {len(toc_items)} TOC entries")
standalone/make_test_standalone.py · 84 lines · 7.3 kB — Generate standalone/test_standalone.py from proto/test/run.py: the same prototype tests, run against the single-file 
"""Generate standalone/test_standalone.py from proto/test/run.py: the same prototype tests, run against the single-file
edition (the game inside the document), plus checks of the document itself (no outside requests, figures, structure,
collapsed sections, contents links, theme switch)."""
import sys

MARGIN_NOTES = 70   # margin notes in samon.md: update when the document gains or loses one

src = open('/home/claude/zen/proto/test/run.py').read()


def rep(s, old, new, count=1):
    n = s.count(old)
    if n != count:
        sys.exit(f"expected {count} of {old[:60]!r}, found {n}")
    return s.replace(old, new)


s = rep(src, "URL = 'file:///home/claude/zen/proto/test/index.html'", "URL = 'file:///home/claude/zen/standalone/2026-10-03-gwern-samon.html'")
s = rep(s, "SHOT = '/home/claude/zen/proto/test/'", "SHOT = '/home/claude/zen/standalone/shots/'")
s = rep(s, "async def centers(page, cells):\n",
        "async def centers(page, cells):\n    await page.evaluate(\"(() => { const cv = document.getElementById('garden'); const r = cv.getBoundingClientRect(); if (r.top < 0 || r.bottom > innerHeight) cv.scrollIntoView({block: 'center'}); })()\")\n")
first_goto = "        await page.goto(URL); await page.wait_for_timeout(300)\n"
i = s.index(first_goto)
block = f'''        outside = []
        page.on('request', lambda r: outside.append(r.url) if not r.url.startswith('file://') and not r.url.startswith('data:') else None)
        await page.goto(URL); await page.wait_for_timeout(500)
        expect(not outside, f'no requests beyond the file itself {{outside[:3]}}')
        imgs = await page.evaluate("[...document.querySelectorAll('.doc img')].map(i => i.complete && i.naturalWidth > 0)")
        expect(len(imgs) == 12 and all(imgs), f'all 12 figures decode ({{sum(imgs)}}/{{len(imgs)}})')
        n = await page.evaluate("[document.querySelectorAll('.doc h1, .doc h2, details summary h1, details summary h2').length, document.querySelectorAll('.margin-note').length, document.querySelectorAll('.footnote-ref').length, document.querySelectorAll('.toc a').length]")
        expect(n[0] >= 20 and n[1] == {MARGIN_NOTES} and n[2] == 2 and n[3] == 23, f'document structure (headings, margin notes, footnote refs, TOC links) {{n}}')
        # the appendix: every file of the code bundle is in the page's .zip, which is the one the build wrote; the code
        # shown inline is byte-for-byte the files'; the download link delivers the .zip
        import base64, hashlib, io, zipfile
        ZP = '/home/claude/zen/standalone/2026-10-03-gwern-samon-code.zip'
        BUNDLE = '/home/claude/zen/deliver/samon-code'
        ap = await page.evaluate("""(() => {{ const shown = {{}};
            document.querySelectorAll('details.file').forEach(d => {{ shown[d.dataset.path] = d.querySelector('pre code').textContent; }});
            return [shown, document.querySelectorAll('.appendix .file-list li').length, document.getElementById('bundle-zip').textContent.trim()]; }})()""")
        expect(ap[2].startswith('data:application/zip;base64,'), 'the .zip is carried as a data: URL (which a gwtar build moves into its archive)')
        zb = base64.b64decode(ap[2].split(',', 1)[1]); zdisk = open(ZP, 'rb').read()
        expect(zb == zdisk, f'the page carries the .zip the build wrote ({{len(zb):,}} bytes)')
        zf = zipfile.ZipFile(io.BytesIO(zb))
        names = [z_[len('samon-code/'):] for z_ in zf.namelist()]
        on_disk = sorted(os.path.relpath(os.path.join(d_, f_), BUNDLE) for d_, _, fs_ in os.walk(BUNDLE) for f_ in fs_)
        gen_ = {{'standalone/2026-10-03-gwern-samon.html', 'proto/samon.html', 'proto/samon.standalone.html'}}
        expect(names == [p_ for p_ in on_disk if p_ not in gen_], f'the .zip holds every file of the bundle but the generated pages ({{len(names)}} files)')
        expect(all(zf.read('samon-code/' + p_) == open(os.path.join(BUNDLE, p_), 'rb').read() for p_ in names), 'every file in the .zip is byte-for-byte the bundle\\'s')
        expect(len(ap[0]) + ap[1] == len(names) and all(zf.read('samon-code/' + p_).decode() == t_ for p_, t_ in ap[0].items()),
               f'the appendix shows or lists every file, and what it shows is the file exactly ({{len(ap[0])}} shown, {{ap[1]}} listed)')
        expect(all(p_ in ap[0] for p_ in names if p_.endswith(('.py', '.c', '.md')) or p_ == 'proto/samon.template.html'), 'all the code is shown')
        await page.click('summary:has(h1:text("Appendix: Code and Data"))')
        async with page.expect_download() as dli_:
            await page.click('#bundle-dl')
        dl_ = await dli_.value
        got_ = open(await dl_.path(), 'rb').read()
        expect(dl_.suggested_filename == '2026-10-03-gwern-samon-code.zip' and hashlib.sha256(got_).hexdigest() == hashlib.sha256(zdisk).hexdigest(), f'the download link delivers the .zip ({{dl_.suggested_filename}})')
        await page.evaluate("location.hash = '#file-src-zgsolve-c'"); await page.wait_for_timeout(200)
        op_ = await page.evaluate("[document.getElementById('file-src-zgsolve-c').open, Math.round(document.getElementById('file-src-zgsolve-c').getBoundingClientRect().top)]")
        expect(op_[0] and abs(op_[1]) < 5, f'a link to a file opens it and scrolls to it {{op_}}')
        await page.evaluate("document.getElementById('file-src-zgsolve-c').open = false; history.replaceState(null, '', location.pathname)")
        await page.click('summary:has(h1:text("Appendix: Code and Data"))')
        await page.click('summary:has(h1:text("Methods"))')
        op = await page.evaluate("document.getElementById('methods').open")
        expect(op, 'Methods expands')
        await page.click('summary:has(h1:text("Methods"))')
        await page.click('.toc >> text=The Wide Rake')
        y = await page.evaluate("document.getElementById('the-wide-rake').getBoundingClientRect().top")
        expect(abs(y) < 5, f'TOC links jump to their sections ({{y:.0f}})')
        await page.evaluate("window.scrollTo(0, 0)")
        await page.click('.theme >> text=Dark')
        th = await page.evaluate("[document.documentElement.getAttribute('data-theme'), getComputedStyle(document.body).backgroundColor]")
        expect(th[0] == 'dark' and th[1] == 'rgb(22, 22, 22)', f'theme switch to dark {{th}}')
        await page.click('.theme >> text=Auto')
        await page.evaluate("document.getElementById('garden').scrollIntoView({{block: 'center'}})")
'''
s = s[:i] + block + s[i + len(first_goto):]
s = rep(s, "        await page.click('text=Practice garden'); await page.wait_for_timeout(900)   # the page scrolls the garden into view\n",
        "        await page.click('#play nav >> text=Practice garden'); await page.wait_for_timeout(900)   # the page scrolls the garden into view\n"
        "        await page.evaluate(\"document.getElementById('garden').scrollIntoView({block: 'center'})\")\n")
s = rep(s, "        await page.evaluate(\"window.__samon.enterPractice()\")\n",
        "        await page.evaluate(\"window.__samon.enterPractice()\")\n        await page.evaluate(\"document.getElementById('garden').scrollIntoView({block: 'center'})\")\n", count=2)
open('/home/claude/zen/standalone/test_standalone.py', 'w').write(s)
print('wrote standalone/test_standalone.py')
standalone/shots.py · 28 lines · 1.9 kB 
import asyncio
from playwright.async_api import async_playwright
URL = 'file:///home/claude/zen/standalone/2026-10-03-gwern-samon.html'
async def main():
    async with async_playwright() as p:
        b = await p.chromium.launch()
        # phone, light: top of page, then the game
        page = await (await b.new_context(viewport={'width': 390, 'height': 844}, device_scale_factor=2)).new_page()
        await page.goto(URL); await page.wait_for_timeout(400)
        await page.screenshot(path='shots/phone-top.png')
        await page.evaluate("document.getElementById('play').scrollIntoView()"); await page.wait_for_timeout(200)
        await page.screenshot(path='shots/phone-game.png')
        await page.evaluate("document.getElementById('the-wide-rake').scrollIntoView()"); await page.wait_for_timeout(200)
        await page.screenshot(path='shots/phone-doc.png')
        # desktop, light: margin notes in the margin, a table
        page = await (await b.new_context(viewport={'width': 1400, 'height': 900})).new_page()
        await page.goto(URL); await page.wait_for_timeout(400)
        await page.evaluate("document.getElementById('the-bare-rule-is-loose').scrollIntoView()"); await page.wait_for_timeout(200)
        await page.screenshot(path='shots/desktop-doc.png')
        # desktop, dark: calm section with figure
        ctx = await b.new_context(viewport={'width': 1400, 'height': 900}, color_scheme='dark')
        page = await ctx.new_page(); await page.goto(URL); await page.wait_for_timeout(400)
        await page.evaluate("document.getElementById('calm-turns-as-score').scrollIntoView()"); await page.wait_for_timeout(200)
        await page.screenshot(path='shots/desktop-dark.png')
        await page.evaluate("window.scrollTo(0,0)"); await page.wait_for_timeout(200)
        await page.screenshot(path='shots/desktop-dark-top.png')
        await b.close()
asyncio.run(main())
standalone/test_standalone.py · 937 lines · 94.1 kB — Browser test of the prototype: every garden raked at par with real mouse drags (raked on letting go); refusals; 
"""Browser test of the prototype: every garden raked at par with real mouse drags (raked on letting go); refusals;
Restore after a rewind and after Start over; the abbot's hints (first hint, sketched groove, 'no longer at par' with its
rewind, 'stuck' with its rewind); the grey seal; unlocking; assists; the poems and the scroll of poems earned; the daily
garden; drag intent (drift, the grip beyond the walls, arcs, reversals, wobble); sound; sharing (a raking as a code in the URL's hash,
progress carried between devices, the share text); the practice garden; screenshots in light and dark."""
import asyncio, json, math, os, sys
from playwright.async_api import async_playwright
sys.path.insert(0, '/home/claude/zen/src')
from zg import Garden, check_solution, turns_of

G = json.load(open('/home/claude/zen/out/proto_gardens.json'))
URL = 'file:///home/claude/zen/standalone/2026-10-03-gwern-samon.html'
SHOT = '/home/claude/zen/standalone/shots/'
fails = []

def garden_of(p, braced=True):
    marks = {tuple(map(int, k.split(','))): v for k, v in (p.get('marks') or {}).items()}
    clues = {tuple(map(int, k.split(','))): v for k, v in (p['clues'] or {}).items()}
    return Garden(p['rows'], tuple(p['gate']), clues=clues, marks=marks, end=tuple(p['end']) if p['end'] else None,
                  nohairpin=2 if p['wide'] else 0, braced_start=p['wide'] and braced, end_side=p['end_side'])
def expect(cond, what):
    print(('ok   ' if cond else 'FAIL ') + what)
    if not cond: fails.append(what)

async def centers(page, cells):
    await page.evaluate("(() => { const cv = document.getElementById('garden'); const r = cv.getBoundingClientRect(); if (r.top < 0 || r.bottom > innerHeight) cv.scrollIntoView({block: 'center'}); })()")
    return await page.evaluate("""(path) => { const cv = document.getElementById('garden'); const r = cv.getBoundingClientRect();
        const s = r.width / cv.width; const g = window.__samon.geom();
        return path.map(([rr, cc]) => [r.left + (g.ox + cc * g.cs + g.cs / 2) * s, r.top + (g.oy + rr * g.cs + g.cs / 2) * s]); }""", cells)

async def drag(page, cells):
    pts = await centers(page, cells)
    await page.mouse.move(*pts[0]); await page.mouse.down()
    for (x, y) in pts[1:]: await page.mouse.move(x, y, steps=3)
    await page.mouse.up()

async def quiet(page, **t):   # poems off (they would carry the page on to the next garden), unless a test wants them
    await page.evaluate("(t) => { window.__samon.poemTiming = t; }", t or {'delay': 3600000})

async def state(page):
    return await page.evaluate("() => ({mode: window.__samon.mode, len: window.__samon.path.length, turns: window.__samon.turnsOf(), "
                               "status: document.getElementById('status').textContent.replace(/^\\[|\\]$/g, ''), "
                               "restore: !document.getElementById('restore').hidden, seal: getComputedStyle(document.getElementById('seal')).display, "
                               "ghosts: window.__samon.ghosts.size, actions: [...document.querySelectorAll('#actions button')].map(b => b.textContent)})")

async def main():
    async with async_playwright() as p:
        b = await p.chromium.launch()
        errors = []
        ctx = await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2, color_scheme='light')
        page = await ctx.new_page()
        page.on('console', lambda m: errors.append(m.text) if m.type == 'error' else None)
        page.on('pageerror', lambda e: errors.append(str(e)))
        outside = []
        page.on('request', lambda r: outside.append(r.url) if not r.url.startswith('file://') and not r.url.startswith('data:') else None)
        await page.goto(URL); await page.wait_for_timeout(500)
        expect(not outside, f'no requests beyond the file itself {outside[:3]}')
        imgs = await page.evaluate("[...document.querySelectorAll('.doc img')].map(i => i.complete && i.naturalWidth > 0)")
        expect(len(imgs) == 12 and all(imgs), f'all 12 figures decode ({sum(imgs)}/{len(imgs)})')
        n = await page.evaluate("[document.querySelectorAll('.doc h1, .doc h2, details summary h1, details summary h2').length, document.querySelectorAll('.margin-note').length, document.querySelectorAll('.footnote-ref').length, document.querySelectorAll('.toc a').length]")
        expect(n[0] >= 20 and n[1] == 69 and n[2] == 2 and n[3] == 23, f'document structure (headings, margin notes, footnote refs, TOC links) {n}')
        # the appendix: every file of the code bundle is in the page's .zip, which is the one the build wrote; the code
        # shown inline is byte-for-byte the files'; the download link delivers the .zip
        import base64, hashlib, io, zipfile
        ZP = '/home/claude/zen/standalone/2026-10-03-gwern-samon-code.zip'
        BUNDLE = '/home/claude/zen/deliver/samon-code'
        ap = await page.evaluate("""(() => { const shown = {};
            document.querySelectorAll('details.file').forEach(d => { shown[d.dataset.path] = d.querySelector('pre code').textContent; });
            return [shown, document.querySelectorAll('.appendix .file-list li').length, document.getElementById('bundle-zip').textContent.trim()]; })()""")
        expect(ap[2].startswith('data:application/zip;base64,'), 'the .zip is carried as a data: URL (which a gwtar build moves into its archive)')
        zb = base64.b64decode(ap[2].split(',', 1)[1]); zdisk = open(ZP, 'rb').read()
        expect(zb == zdisk, f'the page carries the .zip the build wrote ({len(zb):,} bytes)')
        zf = zipfile.ZipFile(io.BytesIO(zb))
        names = [z_[len('samon-code/'):] for z_ in zf.namelist()]
        on_disk = sorted(os.path.relpath(os.path.join(d_, f_), BUNDLE) for d_, _, fs_ in os.walk(BUNDLE) for f_ in fs_)
        gen_ = {'standalone/2026-10-03-gwern-samon.html', 'proto/samon.html', 'proto/samon.standalone.html'}
        expect(names == [p_ for p_ in on_disk if p_ not in gen_], f'the .zip holds every file of the bundle but the generated pages ({len(names)} files)')
        expect(all(zf.read('samon-code/' + p_) == open(os.path.join(BUNDLE, p_), 'rb').read() for p_ in names), 'every file in the .zip is byte-for-byte the bundle\'s')
        expect(len(ap[0]) + ap[1] == len(names) and all(zf.read('samon-code/' + p_).decode() == t_ for p_, t_ in ap[0].items()),
               f'the appendix shows or lists every file, and what it shows is the file exactly ({len(ap[0])} shown, {ap[1]} listed)')
        expect(all(p_ in ap[0] for p_ in names if p_.endswith(('.py', '.c', '.md')) or p_ == 'proto/samon.template.html'), 'all the code is shown')
        await page.click('summary:has(h1:text("Appendix: Code and Data"))')
        async with page.expect_download() as dli_:
            await page.click('#bundle-dl')
        dl_ = await dli_.value
        got_ = open(await dl_.path(), 'rb').read()
        expect(dl_.suggested_filename == '2026-10-03-gwern-samon-code.zip' and hashlib.sha256(got_).hexdigest() == hashlib.sha256(zdisk).hexdigest(), f'the download link delivers the .zip ({dl_.suggested_filename})')
        await page.evaluate("location.hash = '#file-src-zgsolve-c'"); await page.wait_for_timeout(200)
        op_ = await page.evaluate("[document.getElementById('file-src-zgsolve-c').open, Math.round(document.getElementById('file-src-zgsolve-c').getBoundingClientRect().top)]")
        expect(op_[0] and abs(op_[1]) < 5, f'a link to a file opens it and scrolls to it {op_}')
        await page.evaluate("document.getElementById('file-src-zgsolve-c').open = false; history.replaceState(null, '', location.pathname)")
        await page.click('summary:has(h1:text("Appendix: Code and Data"))')
        await page.click('summary:has(h1:text("Methods"))')
        op = await page.evaluate("document.getElementById('methods').open")
        expect(op, 'Methods expands')
        await page.click('summary:has(h1:text("Methods"))')
        await page.click('.toc >> text=The Wide Rake')
        y = await page.evaluate("document.getElementById('the-wide-rake').getBoundingClientRect().top")
        expect(abs(y) < 5, f'TOC links jump to their sections ({y:.0f})')
        await page.evaluate("window.scrollTo(0, 0)")
        await page.click('.theme >> text=Dark')
        th = await page.evaluate("[document.documentElement.getAttribute('data-theme'), getComputedStyle(document.body).backgroundColor]")
        expect(th[0] == 'dark' and th[1] == 'rgb(22, 22, 22)', f'theme switch to dark {th}')
        await page.click('.theme >> text=Auto')
        await page.evaluate("document.getElementById('garden').scrollIntoView({block: 'center'})")
        await quiet(page)
        st = await state(page)
        expect(st['seal'] == 'none' and not st['restore'] and not await page.query_selector('#rake'), 'seal and Restore hidden at load; no Rake button')

        # 0. a first visit: the first garden is raked once by the monk (walking in, then raking out) and reset for the player;
        # then the monk waits in the gateway, a ring beckons, and a ghostly finger shows the drag
        onboard = """() => ({beckon: !document.getElementById('beckon').hidden, finger: !document.getElementById('finger').hidden,
            anim: document.getElementById('finger').getAnimations().length, status: document.getElementById('status').textContent.replace(/^\\[|\\]$/g, ''),
            started: localStorage.getItem('samon-started')})"""
        g0 = G[0]
        expect(g0['demo'] and G[1]['demo'] and not G[2]['demo'], 'the first two gardens have demonstrations, the first real garden (third) none')
        await page.wait_for_function("window.__samon.mode === 'demo'", timeout=4000)
        st = await state(page); ob = await page.evaluate(onboard)
        expect(st['status'].startswith('Watch') and not ob['beckon'], f"the first garden demonstrates itself, and nothing beckons meanwhile ({st['status']!r})")
        await page.wait_for_function("window.__samon.mode === 'reveal'", timeout=8000)
        walked = await page.evaluate("window.__samon.path.slice()")
        want = await page.evaluate("(cells) => cells.map(([r, c]) => window.__samon.key(r, c))", g0['intended'])
        expect(walked == want, 'the demonstration walks the intended raking, then rakes it')
        await page.wait_for_function("!window.__samon.demoRunning", timeout=20000)
        st = await state(page); ob = await page.evaluate(onboard)
        pill0 = await page.evaluate(f"[...document.querySelectorAll('#nav .pip')].find(b => b.getAttribute('aria-label') === {json.dumps(g0['name'])}).className")
        expect(st['len'] == 0 and st['mode'] == 'draw' and 'Your turn' in st['status'] and st['seal'] == 'none' and 'done' not in pill0 and 'sealed' not in pill0,
               f"then the garden is reset for the player, and the demonstration counts for nothing ({st['status']!r}, {pill0!r})")
        expect(ob['beckon'] and ob['finger'] and ob['anim'] == 1, f'then the waiting monk beckons, and a finger shows the drag {ob}')
        br = await page.evaluate("[document.getElementById('status').textContent, document.querySelector('.help.puzzle-only').textContent.trim(), document.querySelector('#nav .lock').textContent, document.getElementById('rule').firstChild.textContent, document.getElementById('rule').querySelector('.facts').textContent, [...document.querySelectorAll('#actions button, .veranda button')].map(b => b.textContent)]")
        expect(all(t.startswith('[') and t.endswith(']') for t in br[:3]) and br[3].startswith('[') and not br[4].startswith('[') and not any(t.startswith('[') for t in br[5]),
               f"what the game says to the player is set in editorial brackets (status, help, lock notes, a garden's instructions), and rules, counts and controls are not ({br[0][:40]!r}, {br[3][:30]!r}, {br[4][:20]!r})")
        await page.reload(); await page.wait_for_timeout(1500)
        await quiet(page)
        st = await state(page); ob = await page.evaluate(onboard)
        expect(not await page.evaluate("window.__samon.demoRunning") and st['len'] == 0 and 'Drag the monk in' in st['status'] and ob['beckon'] and ob['finger'],
               f"a demonstration seen once does not play again by itself ({st['status']!r})")
        wv = await page.evaluate("!document.getElementById('watch').hidden")
        await page.click('#watch'); await page.wait_for_timeout(700)
        running = await page.evaluate("window.__samon.demoRunning")
        mid = await centers(page, [g0['intended'][1]]); await page.mouse.click(*mid[0]); await page.wait_for_timeout(100)
        st = await state(page)
        expect(wv and running and not await page.evaluate("window.__samon.demoRunning") and st['len'] == 0 and st['mode'] == 'draw' and 'Your turn' in st['status'],
               'Watch replays the demonstration, and a tap skips it')
        await page.evaluate("window.__samon.load(2)")
        expect(await page.evaluate("document.getElementById('watch').hidden"), 'a garden without a demonstration has no Watch button')
        await page.evaluate("window.__samon.load(0)")
        far = await centers(page, [[0, len(g0['rows'][0]) - 1]])
        await page.mouse.click(*far[0]); st = await state(page)
        expect('Start at the gate' in st['status'] and st['len'] == 0, 'a press away from the gate says where to start')
        gp = await page.evaluate("(() => { const cv = document.getElementById('garden'); const r = cv.getBoundingClientRect(); const s = r.width / cv.width; const [x, y] = window.__samon.gate(); return [r.left + x * s, r.top + y * s]; })()")
        pts = await centers(page, g0['intended'][:3])
        await page.mouse.move(*gp); await page.mouse.down()
        for (x, y) in pts: await page.mouse.move(x, y, steps=3)
        await page.mouse.up(); st = await state(page); ob = await page.evaluate(onboard)
        expect(st['len'] == 3 and not ob['beckon'] and not ob['finger'] and ob['started'] == '1', f'dragging the monk in from the gateway starts the walk, and the beckoning stops {ob}')
        await page.screenshot(path=SHOT + 'shot-phone-light-start.png', full_page=False)
        await page.evaluate("window.__samon.load(2)"); ob = await page.evaluate(onboard)
        expect(ob['beckon'] and not ob['finger'], f'after the first walk, fresh gardens still beckon, without the finger {ob}')

        # 1. every garden at par, by mouse: letting go of a complete walk rakes it
        for i, g in enumerate(G if not os.environ.get('SAMON_QUICK') else G[:3]):
            await page.evaluate(f"window.__samon.load({i})")
            await drag(page, g['intended'])
            await page.wait_for_timeout(350)
            st = await state(page)
            ok = st['len'] == len(g['intended']) and st['turns'] == g['par'] and st['mode'] == 'reveal'
            pts = await centers(page, [g['intended'][0]]); await page.mouse.click(*pts[0])   # skip the reveal
            await page.wait_for_timeout(60)
            st2 = await state(page)
            expect(ok and st2['mode'] == 'done' and st2['seal'] == 'grid' and 'par' in st2['status'], f"{g['name']}: drawn, raked on letting go, at par, sealed")

        # 2. refusals: wide rake U-turn; pebbles
        wi = next(i for i, g in enumerate(G) if g['name'] == 'Wide rake')
        await page.evaluate(f"window.__samon.load({wi})")
        await drag(page, [[6, 5], [5, 5], [4, 5], [4, 6], [5, 6]])
        st = await state(page)
        expect('wide rake' in st['status'] and st['len'] == 4, 'wide rake refuses an open-sand U-turn')
        pi = next(i for i, g in enumerate(G) if g['name'] == "The abbot's pebbles")
        pg = G[pi]; marks = {tuple(map(int, k.split(','))): v for k, v in pg['marks'].items()}
        rows = pg['rows']; seq = [tuple(c) for c in pg['intended']]
        j = next(k for k in range(1, len(seq) - 1) if marks.get(seq[k]) == 'S'
                 and any(0 <= seq[k][0] + a < len(rows) and 0 <= seq[k][1] + bb < len(rows[0]) and rows[seq[k][0] + a][seq[k][1] + bb] == '.'
                         and (seq[k][0] + a, seq[k][1] + bb) not in seq[:k + 1]
                         and (a, bb) != (seq[k][0] - seq[k - 1][0], seq[k][1] - seq[k - 1][1])
                         and (a, bb) != (seq[k - 1][0] - seq[k][0], seq[k - 1][1] - seq[k][1]) for a, bb in ((-1, 0), (1, 0), (0, -1), (0, 1))))
        v = seq[j]; din = (v[0] - seq[j - 1][0], v[1] - seq[j - 1][1])
        turn = next((v[0] + a, v[1] + bb) for a, bb in ((-1, 0), (1, 0), (0, -1), (0, 1))
                    if (a, bb) != din and (a, bb) != (-din[0], -din[1]) and 0 <= v[0] + a < len(rows) and 0 <= v[1] + bb < len(rows[0])
                    and rows[v[0] + a][v[1] + bb] == '.' and (v[0] + a, v[1] + bb) not in seq[:j + 1])
        await page.evaluate(f"window.__samon.load({pi})")
        await drag(page, [list(c) for c in seq[:j + 1]] + [list(turn)])
        st = await state(page)
        expect('pale pebble' in st['status'] and st['len'] == j + 1, 'a pale pebble refuses a turn')

        # 3. Restore after a tap-rewind and after Start over
        ci = next(i for i, g in enumerate(G) if g['name'] == 'Calm')
        await page.evaluate(f"window.__samon.load({ci})")
        await page.click('#restart')
        await drag(page, G[ci]['intended'][:12])
        pts = await centers(page, [G[ci]['intended'][3]]); await page.mouse.click(*pts[0])
        st = await state(page)
        expect(st['len'] == 4 and st['restore'], 'tapping a footprint rewinds to it, and offers Restore')
        await page.click('#restore'); st = await state(page)
        expect(st['len'] == 12 and not st['restore'], 'Restore brings the rewound route back')
        await page.click('#restart'); st = await state(page)
        expect(st['len'] == 0 and st['restore'], 'Start over clears the route, and offers Restore')
        await page.click('#restore'); st = await state(page)
        expect(st['len'] == 12, 'Restore after Start over')

        # 3b. after raking: a tap on a cell walks back to it (the sand is smoothed back to the footprints), Restore rakes it again;
        # holding at the last cell does not rake until the mouse lets go; the keyboard rakes after a beat
        await page.evaluate(f"window.__samon.load({ci})"); await page.click('#restart')
        full = G[ci]['intended']
        pts = await centers(page, full)
        await page.mouse.move(*pts[0]); await page.mouse.down()
        for (x, y) in pts[1:]: await page.mouse.move(x, y, steps=3)
        await page.wait_for_timeout(700); st = await state(page)
        expect(st['mode'] == 'draw' and st['len'] == len(full), 'a complete walk is not raked while the mouse is still down')
        await page.mouse.up(); await page.wait_for_timeout(350); st = await state(page)
        expect(st['mode'] == 'reveal', 'letting go starts the raking')
        await page.mouse.click(*pts[0]); await page.wait_for_timeout(700)
        k = len(full) - 5
        await page.mouse.click(*pts[k - 1]); await page.wait_for_timeout(100); st = await state(page)
        ref = await page.evaluate("window.__samon.sand()")
        expect(st['mode'] == 'draw' and st['len'] == k and st['seal'] == 'none' and st['restore'], 'after raking, tapping a cell walks back to it, and offers Restore')
        await page.evaluate("(w) => window.__samon.setPath(w)", full[:k]); fresh = await page.evaluate("window.__samon.sand()")
        expect(abs(ref - fresh) < 1e-6, 'walking back smooths the raked sand back to the footprints')
        await page.evaluate("(w) => window.__samon.setPath(w)", full)
        await page.wait_for_timeout(600); st = await state(page)
        expect(st['mode'] in ('reveal', 'done') and st['len'] == len(full), 'a walk completed any other way is raked after a beat')
        await page.focus('#garden'); await page.keyboard.press('Enter'); await page.wait_for_timeout(60); st = await state(page)
        expect(st['mode'] == 'done', 'Enter skips the raking')
        await page.focus('#garden'); await page.keyboard.press('Backspace'); await page.wait_for_timeout(60); st = await state(page)
        expect(st['mode'] == 'draw' and st['len'] == len(full) - 1, 'Backspace after raking steps back one footprint')
        await page.click('#restore'); await page.wait_for_timeout(600); st = await state(page)
        expect(st['mode'] in ('reveal', 'done') and st['len'] == len(full), 'Restore brings the whole walk back, and it is raked again')
        await page.keyboard.press('Enter'); await page.wait_for_timeout(60)
        # the keyboard: the last arrow rakes after a beat, in which Backspace can still take it back
        ti = next(i for i, g in enumerate(G) if g['name'] == 'Nine squares')
        await page.evaluate(f"window.__samon.load({ti})"); await page.focus('#garden')
        keys = ('ArrowUp', 'ArrowUp', 'ArrowUp', 'ArrowRight', 'ArrowDown', 'ArrowDown', 'ArrowRight', 'ArrowUp', 'ArrowUp')   # in at the gate, then a serpentine
        for kk in keys: await page.keyboard.press(kk)
        await page.keyboard.press('Backspace'); await page.wait_for_timeout(600); st = await state(page)
        expect(st['mode'] == 'draw' and st['len'] == 8, 'Backspace in the beat takes the last step back, and nothing is raked')
        await page.keyboard.press(keys[-1]); await page.wait_for_timeout(600); st = await state(page)
        expect(st['mode'] in ('reveal', 'done') and st['len'] == 9, 'from the keyboard, a complete walk is raked after a beat')
        await page.keyboard.press('Enter'); await page.wait_for_timeout(60); await page.keyboard.press('Enter'); await page.wait_for_timeout(800)
        nm = await page.evaluate("window.__samon.G.name")
        expect(nm == G[ti + 1]['name'], 'Enter after raking goes on to the next garden')
        # a walk that covers every cell but may not end where it does is not raked
        wi2, bad = None, None
        for j, gg in enumerate(G):
            if not gg['wide'] or gg['end']: continue
            gx = garden_of(gg, braced=False)
            for s_ in gx.solve(mode=0, cap=400, stream=True)['sols']:
                if not check_solution(garden_of(gg), s_['draw']): wi2, bad = j, [list(c) for c in s_['draw']]; break
            if bad: break
        await page.evaluate(f"window.__samon.load({wi2})"); await drag(page, bad); await page.wait_for_timeout(600); st = await state(page)
        expect(st['mode'] == 'draw' and st['len'] == len(bad) and 'wide rake down' in st['status'], f"a complete walk ending in open sand under the wide rake is not raked ({G[wi2]['name']})")

        # 4. the abbot's hints
        await page.evaluate(f"window.__samon.load({ci})")
        await page.click('#restart'); await page.click('#hint'); st = await state(page)
        expect(G[ci]['hint'] in st['status'], 'first hint is the authored observation')
        await page.click('#hint'); st = await state(page)
        expect('at par' in st['status'] and st['ghosts'] == 1, 'second hint: route can reach par, one groove sketched')
        found = await page.evaluate(f"""(() => {{ for (let s = 1; s < 4000; s++) {{ const w = window.__samon.randomWalk({ci}, s, 10 + s % 30);
            const a = window.__samon.analyzeFor({ci}, w); if (a.state === 'finish') return w; }} return null; }})()""")
        await page.evaluate("(w) => window.__samon.setPath(w)", found)
        await page.click('#hint'); st = await state(page)
        expect('no longer at par' in st['status'] and any('for par' in a for a in st['actions']), "'no longer at par' with a rewind offer")
        await page.click('#actions button >> nth=0'); st = await state(page)
        a = await page.evaluate("window.__samon.analyze(window.__samon.path).state")
        expect(a == 'par' and st['restore'], 'the par rewind lands on a route that can reach par, and is restorable')
        fk = await page.evaluate("[document.activeElement && document.activeElement.id, document.getElementById('status').textContent.replace(/^\\[|\\]$/g, '')]")
        n0 = await page.evaluate("window.__samon.path.length"); await page.keyboard.press('ArrowUp'); await page.keyboard.press('ArrowRight'); n1 = await page.evaluate("window.__samon.path.length")
        expect(fk[0] == 'garden' and fk[1].startswith('Stepped back') and 'par is still possible' in fk[1] and 'no longer at par' not in fk[1] and n1 > n0,
               f"after the abbot's rewind the garden has the keyboard again, and the old message has given way to what now holds ({fk[1][:70]!r})")
        found = await page.evaluate(f"""(() => {{ for (let s = 1; s < 4000; s++) {{ const w = window.__samon.randomWalk({ci}, s, 8 + s % 30);
            const a = window.__samon.analyzeFor({ci}, w); if (a.state === 'stuck' && a.back < w.length - 1 && a.why.cells.length) return w; }} return null; }})()""")
        await page.evaluate("(w) => window.__samon.setPath(w)", found)
        await page.click('#hint'); st = await state(page)
        expect('to the marked footprint' in st['status'] and any(a.startswith('Step back') for a in st['actions']), "'stuck': reason, marked cells, rewind offer, action first")
        await page.screenshot(path=SHOT + 'shot-phone-light-stuck.png', full_page=True)
        await page.click('#actions button >> nth=0')
        a = await page.evaluate("window.__samon.analyze(window.__samon.path).state")
        fk = await page.evaluate("[document.activeElement && document.activeElement.id, document.getElementById('status').textContent]")
        expect(a in ('par', 'finish') and fk[0] == 'garden' and 'can be finished from here' in fk[1] and 'marked footprint' not in fk[1], f"the rewind lands on a route that can be finished, says so, and keeps the keyboard ({fk[1][:60]!r})")

        # 5. assists
        await page.evaluate(f"window.__samon.load({ci})"); await page.click('#restart')
        await drag(page, G[ci]['intended'][:20] if False else [[6, 3], [5, 3], [5, 2], [5, 1], [5, 0], [4, 0], [3, 0], [3, 1], [4, 1], [4, 2], [4, 3], [4, 4], [3, 4], [3, 5]])
        for bid in ('#aPreview', '#aChecker', '#aCut'): await page.click(bid)
        pressed = await page.evaluate("['aPreview','aChecker','aCut'].map(id => document.getElementById(id).getAttribute('aria-pressed'))")
        expect(pressed == ['true'] * 3, 'assist toggles switch on')
        await page.screenshot(path=SHOT + 'shot-phone-light-assists.png', full_page=True)
        for bid in ('#aPreview', '#aChecker', '#aCut'): await page.click(bid)

        # 6. practice garden
        await page.click('#play nav >> text=Practice garden'); await page.wait_for_timeout(900)   # the page scrolls the garden into view
        await page.evaluate("document.getElementById('garden').scrollIntoView({block: 'center'})")
        s0 = await page.evaluate("window.__samon.practice()")
        box = await page.evaluate("(() => { const r = document.getElementById('garden').getBoundingClientRect(); return [r.left, r.top, r.width, r.height]; })()")
        L, T, W, H = box
        import math
        for k, (cx, cy, rad) in enumerate([(0.27, 0.40, 0.18), (0.72, 0.65, 0.14)]):
            pts = [(L + W * (cx + rad * math.cos(t / 30 * 2 * math.pi) * 0.75), T + H * (cy + rad * math.sin(t / 30 * 2 * math.pi))) for t in range(0, 34)]
            await page.mouse.move(*pts[0]); await page.mouse.down()
            for (x, y) in pts[1:]: await page.mouse.move(x, y, steps=4)
            await page.mouse.up()
        for yy in (0.1, 0.16, 0.86, 0.92):
            await page.mouse.move(L + W * 0.05, T + H * yy); await page.mouse.down()
            await page.mouse.move(L + W * 0.95, T + H * yy, steps=40); await page.mouse.up()
        s1 = await page.evaluate("window.__samon.practice()")
        expect(abs(s1['sum'] - s0['sum']) > 50, 'raking changes the practice sand')
        await page.screenshot(path=SHOT + 'shot-phone-light-practice.png', full_page=True)
        await page.click('#toolStone'); await page.mouse.click(L + W * 0.5, T + H * 0.2)
        s2 = await page.evaluate("window.__samon.practice()")
        expect(s2['stones'] == s0['stones'] + 1, 'Stones tool places a stone')
        await page.mouse.click(L + W * 0.5, T + H * 0.2)
        s3 = await page.evaluate("window.__samon.practice()")
        expect(s3['stones'] == s0['stones'], 'tapping a stone lifts it')
        await page.click('#smooth'); s4 = await page.evaluate("window.__samon.practice()")
        expect(abs(s4['sum'] - s0['sum']) < 1e-6, 'Smooth restores the unraked sand')
        await page.click('#nav [aria-label="Calm"]'); st = await state(page)
        expect(await page.evaluate("window.__samon.view") == 'puzzle', 'back from the practice garden to a garden')
        await ctx.close()

        # 7. dark mode: a finished garden, and the practice garden
        ctx = await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2, color_scheme='dark')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(200)
        await quiet(page, delay=100, hold=3600000)                   # the screenshot shows the raked garden with its poem
        si = next(i for i, g in enumerate(G) if g['name'] == 'Spiral')
        await page.evaluate(f"window.__samon.load({si})"); await drag(page, G[si]['intended'])
        await page.wait_for_timeout(7200)
        await page.screenshot(path=SHOT + 'shot-phone-dark-done.png', full_page=True)
        await page.evaluate("window.__samon.enterPractice()")
        await page.evaluate("document.getElementById('garden').scrollIntoView({block: 'center'})")
        box = await page.evaluate("(() => { const r = document.getElementById('garden').getBoundingClientRect(); return [r.left, r.top, r.width, r.height]; })()")
        L, T, W, H = box
        for yy in (0.25, 0.42, 0.59, 0.76):
            await page.mouse.move(L + W * 0.06, T + H * yy); await page.mouse.down()
            await page.mouse.move(L + W * 0.94, T + H * (yy + 0.04), steps=40); await page.mouse.up()
        await page.screenshot(path=SHOT + 'shot-phone-dark-practice.png', full_page=True)
        await ctx.close()

        # 7b. the grey seal: help beyond the abbot's first observation marks the raking (fresh profile)
        ctx = await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2, color_scheme='light')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(200)
        await quiet(page)
        ci = next(i for i, g in enumerate(G) if g['name'] == 'Calm')
        ni = next(i for i, g in enumerate(G) if g['name'] == 'Nine squares')
        async def rake_intended(i):
            await drag(page, G[i]['intended']); await page.wait_for_timeout(350)
            pts = await centers(page, [G[i]['intended'][0]]); await page.mouse.click(*pts[0]); await page.wait_for_timeout(60)
            return await page.evaluate("[getComputedStyle(document.getElementById('seal')).display, document.getElementById('seal').classList.contains('grey'), document.getElementById('status').textContent]")
        async def pill_class(name):
            return await page.evaluate(f"[...document.querySelectorAll('#nav .pip')].find(b => b.getAttribute('aria-label') === {json.dumps(name)}).className")
        await page.evaluate(f"window.__samon.load({ni})")
        await page.click('#hint')                                   # the authored observation only: free
        r1 = await rake_intended(ni)
        expect(r1[0] == 'grid' and not r1[1] and 'grey' not in r1[2], 'the first observation alone leaves the seal vermilion')
        await page.evaluate(f"window.__samon.load({ci})")
        await page.click('#hint'); await page.click('#hint')        # observation, then a verdict: help
        st = await state(page)
        expect('grey seal' in st['status'], 'the first real help says the seal will be grey')
        await page.click('#restart')                                # starting over does not undo help
        r2 = await rake_intended(ci)
        expect(r2[0] == 'grid' and r2[1] and 'grey' in r2[2], 'a helped raking at par earns the grey seal, even after Start over')
        expect('grey' in await pill_class('Calm'), 'the garden list shows a grey seal')
        await page.click('#restart')
        r3 = await rake_intended(ci)
        expect(r3[0] == 'grid' and not r3[1] and 'grey' not in r3[2], 'raking it again unaided earns the vermilion seal')
        expect('sealed' in await pill_class('Calm'), 'the garden list now shows the vermilion seal')
        sand0 = await page.evaluate("window.__samon.sand()")
        await page.click('#restart')
        sand1 = await page.evaluate("window.__samon.sand()")
        expect(abs(sand0 - sand1) > 1, 'Start over smooths the raked sand back to unraked')
        # help travels with the route it helped: stepping back into a helped raking and restoring it re-rakes it grey
        ti = next(i for i, g in enumerate(G) if g['name'] == 'The pocket')
        await page.evaluate(f"window.__samon.load({ti})")
        await page.click('#hint'); await page.click('#hint')
        r4 = await rake_intended(ti)
        await page.focus('#garden'); await page.keyboard.press('Backspace'); await page.wait_for_timeout(60)
        h = await page.evaluate("window.__samon.helped")
        async def rerake():                                          # Restore rakes again after a beat; Enter (on the garden) skips it
            await page.click('#restore')
            await page.wait_for_function("['reveal', 'done'].includes(window.__samon.mode)", timeout=3000)
            if await page.evaluate("window.__samon.mode") == 'reveal':
                await page.focus('#garden'); await page.keyboard.press('Enter')
            await page.wait_for_function("window.__samon.mode === 'done'", timeout=3000)
        await rerake()
        r5 = await page.evaluate("[getComputedStyle(document.getElementById('seal')).display, document.getElementById('seal').classList.contains('grey'), document.getElementById('status').textContent]")
        expect(r4[1] and h and r5[0] == 'grid' and r5[1] and 'grey' in r5[2], f'Step back, then Restore, re-rakes a helped raking with the grey seal {r5}')
        tp = await centers(page, [G[ti]['intended'][2]]); await page.wait_for_timeout(700); await page.mouse.click(*tp[0])
        await page.wait_for_function("window.__samon.mode === 'draw'", timeout=3000)
        await rerake()
        r6 = await page.evaluate("document.getElementById('seal').classList.contains('grey')")
        expect(r6, 'so does tapping back into it, then Restore')
        # an answer the abbot can't give in time costs nothing
        await page.evaluate(f"window.__samon.load({ci})")
        found = await page.evaluate(f"""(() => {{ for (let s = 1; s < 4000; s++) {{ const w = window.__samon.randomWalk({ci}, s, 10 + s % 30);
            const a = window.__samon.analyzeFor({ci}, w); if (a.state === 'finish') return w; }} return null; }})()""")
        await page.evaluate("(w) => window.__samon.setPath(w)", found)
        h0 = await page.evaluate("window.__samon.helped")
        await page.evaluate("window.__samon.budget = 1"); await page.click('#hint')
        st = await state(page); h1 = await page.evaluate("window.__samon.helped")
        await page.evaluate("window.__samon.budget = 250000")
        expect(not h0 and "can't be sure" in st['status'] and 'costs you nothing' in st['status'] and not h1, f"an inconclusive answer costs nothing ({st['status']!r})")
        # a walk that has stopped: why is free, where to step back to is help
        dead = await page.evaluate(f"""(() => {{ for (let s = 1; s < 4000; s++) {{ const w = window.__samon.randomWalk({ci}, s, 999);
            if (w.length < window.__samon.G.nSand && w.length > 3) return w; }} return null; }})()""")
        await page.evaluate("(w) => window.__samon.setPath(w)", dead); await page.click('#hint')
        st = await state(page); h2 = await page.evaluate("window.__samon.helped")
        expect('Saying why a walk has stopped is free' in st['status'] and not h2 and st['actions'] == ['Where can I step back to?'],
               f"a stopped walk: the abbot says why for free ({st['status']!r}, {st['actions']})")
        await page.click('#actions button >> nth=0')
        st = await state(page); h3 = await page.evaluate("window.__samon.helped")
        expect(h3 and 'to the marked footprint' in st['status'] and any(a.startswith('Step back') for a in st['actions']), 'asking where to step back to is help')
        await ctx.close()

        # 7c. worlds open as the world before them is raked (two-thirds of it, at any turn count); the prototype can open all
        ctx = await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2, color_scheme='light')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(200)
        await quiet(page)
        w1 = [i for i, g in enumerate(G) if g['world'] == 1]
        w1m = [i for i in w1 if not G[i]['extra']]; need = -(-2 * len(w1m) // 3)
        nx = await page.evaluate("[document.querySelectorAll('#nav .pip.extra').length, document.querySelectorAll('#nav .more').length]")
        expect(nx[0] == sum(g['extra'] for g in G) and nx[1] == len({g['world'] for g in G if g['extra']}),
               f"the extras are listed after each world's main path, marked as more ({nx})")
        locked = await page.evaluate("[...document.querySelectorAll('#nav .pip')].filter(b => b.disabled).length")
        note = await page.evaluate("document.querySelector('#nav .lock')?.textContent || ''")
        expect(locked == len(G) - len(w1) and f'{need} gardens of Dawn' in note, f'at first only Dawn is open ({locked} locked; {note!r})')
        await page.click('#openAll')
        locked2 = await page.evaluate("[...document.querySelectorAll('#nav .pip')].filter(b => b.disabled).length")
        await page.click('#openAll')
        locked3 = await page.evaluate("[...document.querySelectorAll('#nav .pip')].filter(b => b.disabled).length")
        expect(locked2 == 0 and locked3 == locked, 'the prototype switch opens every world, and locks them again')
        for k, i in enumerate(w1m[:need]):
            await page.evaluate(f"window.__samon.load({i})"); await page.click('#restart')
            await drag(page, G[i]['intended']); await page.wait_for_timeout(350)
            pts = await centers(page, [G[i]['intended'][0]]); await page.mouse.click(*pts[0]); await page.wait_for_timeout(60)
            if k == need - 2:
                still = await page.evaluate("window.__samon.worldOpen(2)")
        st = await state(page)
        open2 = await page.evaluate("[window.__samon.worldOpen(2), window.__samon.worldOpen(3), [...document.querySelectorAll('#nav .pip')].filter(b => b.disabled).length]")
        expect(not still and open2[0] and not open2[1] and 'Morning is open' in st['status'] and open2[2] == locked - len([g for g in G if g['world'] == 2]),
               f'raking {need} of the {len(w1m)} Dawn gardens on the main path opens Morning, and says so {open2}')
        # Next follows the main path: from the last Dawn garden on it, past the extras, to the first of Morning; that
        # garden introduces a rule, and its demonstration plays by itself
        last1 = w1m[-1]
        await page.evaluate(f"window.__samon.load({last1})"); await page.click('#restart')
        await drag(page, G[last1]['intended']); await page.wait_for_timeout(350)
        pts = await centers(page, [G[last1]['intended'][0]]); await page.mouse.click(*pts[0]); await page.wait_for_timeout(100)
        await page.click('#next'); await page.wait_for_timeout(800)
        nm = await page.evaluate("window.__samon.G.name")
        first2 = next(g for g in G if g['world'] == 2 and not g['extra'])
        expect(nm == first2['name'], f'Next from the end of the main path skips the extras ({nm})')
        if not first2['demo']:
            i2 = G.index(first2)
            await drag(page, first2['intended']); await page.wait_for_timeout(350)
            pts = await centers(page, [first2['intended'][0]]); await page.mouse.click(*pts[0]); await page.wait_for_timeout(100)
            await page.click('#next'); await page.wait_for_timeout(800)
        nm2 = await page.evaluate("window.__samon.G.name")
        dg = next(g for g in G if g['name'] == nm2)
        await page.wait_for_timeout(700)
        expect(dg['demo'] and await page.evaluate("window.__samon.demoRunning"), f'the garden that brings in the ripple stone demonstrates itself ({nm2})')
        await page.mouse.click(*(await centers(page, [dg['intended'][0]]))[0]); await page.wait_for_timeout(100)
        # the abbot's observation points at its cells, and they stay marked while the walk goes on
        pk = next(i for i, g in enumerate(G) if g['name'] == 'The pocket')
        await page.evaluate(f"window.__samon.load({pk})"); await page.click('#restart'); await page.click('#hint')
        st = await state(page); ob = await page.evaluate("window.__samon.obs.length")
        await drag(page, G[pk]['intended'][:3]); ob2 = await page.evaluate("window.__samon.obs.length")
        expect('He points' in st['status'] and ob == 1 and ob2 == 1, 'the observation points at its cell, which stays marked while drawing')
        await ctx.close()

        # 8. desktop: the pebbles garden mid-route
        ctx = await b.new_context(viewport={'width': 1200, 'height': 1000}, color_scheme='light')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(200)
        await quiet(page)
        await page.evaluate(f"window.__samon.load({pi})"); await drag(page, G[pi]['intended'][:15])
        await page.screenshot(path=SHOT + 'shot-desktop-light.png', full_page=True)
        await ctx.close()
        # 9. properties, over every garden
        ctx = await b.new_context(viewport={'width': 390, 'height': 900}, color_scheme='light')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(200)
        await quiet(page)
        bad_bends = []
        for i, g in enumerate(G):
            await page.evaluate(f"window.__samon.load({i})")
            arcs, turns = await page.evaluate("(c) => window.__samon.strokeBends(c)", g['intended'])
            if not (arcs == turns == g['par']): bad_bends.append((g['name'], arcs, turns, g['par']))
        nside = sum(1 for g in G if g['end'])
        expect(not bad_bends, f'every garden: the bends drawn in the raked stroke = the turns counted = par ({nside} with a side gate) {bad_bends[:5]}')
        piv = await page.evaluate("""(n) => { let pairs = 0, bad = 0; for (let i = 0; i < n; i++) { window.__samon.load(i); const G = window.__samon.G;
            if (!G.wide) continue;
            for (const v of G.sandList) for (let d = 0; d < 4; d++) { const r = Math.floor(v / G.W) + [-1, 0, 1, 0][d], c = v % G.W + [0, 1, 0, -1][d];
              if (r >= 0 && r < G.H && c >= 0 && c < G.W && G.rows[r][c] === 'R') { pairs++; if (G.piv[v * 4 + d]) bad++; } } }
            return [pairs, bad]; }""", len(G))
        expect(piv[0] > 0 and piv[1] == 0, f'under the wide rake, finished sand is never a pivot ({piv[0]} cell sides checked)')
        wrong, seen = [], 0
        quick = bool(os.environ.get('SAMON_QUICK'))          # the abbot's explanations: 400 random routes per garden in full, a sample when quick
        for i, g in enumerate(G):
            if not (g['end'] or g['wide'] or g.get('marks')): continue
            res = await page.evaluate(f"""(() => {{ const out = []; for (let s = 1; s < {30 if quick else 400}; s++) {{ const w = window.__samon.randomWalk({i}, s, 999);
                const a = window.__samon.analyzeFor({i}, w); if (a.state === 'stuck' && a.why && /only one way in, so the walk would have to end there/.test(a.why.text))
                out.push([a.why.text, a.why.cells[0]]); }} return out; }})()""")
            W_ = len(g['rows'][0]); end = g['end'][0] * W_ + g['end'][1] if g['end'] else -1
            marks = {int(k.split(',')[0]) * W_ + int(k.split(',')[1]) for k in (g.get('marks') or {})}
            for text, cell in res:
                seen += 1
                want = 'side gate' if end >= 0 and cell != end else 'pebble' if cell in marks else 'wide rake'
                if want not in text: wrong.append((g['name'], text))
        expect(seen > 0 and not wrong, f'a dead-end cell is explained by the rule that actually fails ({seen} explanations checked) {wrong[:3]}')
        await ctx.close()

        # 10. the poem is the reward: after any raking (here off par, and with the abbot's help) a poem is laid over the
        # garden, and then the next garden fades in; a tap on the poem goes on, a tap beside it stays; 'original' shows the Japanese; Enter goes on at once; a
        # demonstration earns no poem; the practice garden has one
        PO = json.load(open('/home/claude/zen/poems/samon-poems.json'))
        ppoem = lambda name: PO['poems'][PO['gardens'][name]]
        ctx = await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2, color_scheme='light')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(200)
        pc = await page.evaluate("window.__samon.poems()")
        expect(pc['gardens'] == len(G) and pc['practice'], f'every garden has a poem, and so does the practice garden {pc}')
        await quiet(page, delay=200, hold=1500, fade=300)
        ni = next(i for i, g in enumerate(G) if g['name'] == 'Nine squares')
        gx = garden_of(G[ni])
        sub = next(s_['draw'] for s_ in gx.solve(mode=0, cap=100, stream=True)['sols'] if turns_of(gx, s_['draw']) > G[ni]['par'])
        await page.evaluate(f"window.__samon.load({ni})")
        await page.click('#hint'); await page.click('#hint')
        await drag(page, [list(c) for c in sub]); await page.wait_for_timeout(350)
        pts = await centers(page, [list(sub[0])]); await page.mouse.click(*pts[0])
        await page.wait_for_function("window.__samon.poem.state === 'shown'", timeout=3000); await page.wait_for_timeout(400)
        po = await page.evaluate("window.__samon.poem")
        expect(po['visible'] and po['verse'] == ppoem('Nine squares')['translation'], f"a raking off par, made with help, still earns its poem ({po['verse'][:40]!r})")
        nxt = G[ni + 1]['name']
        await page.wait_for_function("window.__samon.poem.leaving", timeout=4000)
        await page.wait_for_function(f"window.__samon.G.name === {json.dumps(nxt)}", timeout=3000)
        st = await state(page); po = await page.evaluate("window.__samon.poem")
        expect(st['mode'] == 'draw' and po['state'] == 'none' and not po['visible'], f'then garden and poem fade, and the next garden fades in ({nxt})')
        i2 = ni + 1
        await quiet(page, delay=200, hold=1200, fade=300)
        await drag(page, G[i2]['intended']); await page.wait_for_timeout(350)
        pts = await centers(page, [G[i2]['intended'][0]]); await page.mouse.click(*pts[0])
        await page.wait_for_function("window.__samon.poem.state === 'shown'", timeout=3000); await page.wait_for_timeout(300)
        beside = await page.evaluate("(() => { const r = document.getElementById('poem').getBoundingClientRect(), c = document.querySelector('#poem .card').getBoundingClientRect(); return [r.left + 3, c.top > r.top + 6 ? r.top + 3 : r.top + r.height - 3]; })()")
        await page.mouse.click(*beside); await page.wait_for_timeout(2000)
        st = await state(page); po = await page.evaluate("window.__samon.poem"); nm = await page.evaluate("window.__samon.G.name")
        expect(nm == G[i2]['name'] and po['state'] == 'stay' and not po['visible'] and 'Staying' in st['status'] and 'Read the poem again' in st['actions'],
               f"a tap beside the poem puts it away and stays with the garden ({st['status'][-60:]!r})")
        await page.click('#actions button:has-text("Read the poem again")'); await page.wait_for_timeout(1800)
        po = await page.evaluate("window.__samon.poem")
        expect(po['visible'], 'the poem can be read again')
        await page.click('#orig'); await page.wait_for_timeout(100)
        po = await page.evaluate("window.__samon.poem")
        expect(po['ja'] == ppoem(G[i2]['name'])['japanese'] and not po['clash'], "'original' shows the Japanese (and no line runs under the seal)")
        await page.focus('#garden'); await page.keyboard.press('Enter'); await page.wait_for_timeout(800)
        nm = await page.evaluate("window.__samon.G.name")
        expect(nm == G[i2 + 1]['name'], 'Enter goes on to the next garden at once')
        await page.evaluate("(w) => window.__samon.setPath(w)", G[i2 + 1]['intended'])
        await page.wait_for_function("window.__samon.poem.state === 'shown'", timeout=12000); await page.wait_for_timeout(200)
        card = await page.evaluate("(() => { const c = document.querySelector('#poem .card').getBoundingClientRect(); return [c.left + c.width / 2, c.top + c.height / 2]; })()")
        await page.mouse.click(*card); await page.wait_for_timeout(900)
        nm = await page.evaluate("window.__samon.G.name"); po = await page.evaluate("window.__samon.poem")
        expect(nm == G[i2 + 2]['name'] and po['state'] == 'none', f'a tap on the poem itself goes on to the next garden at once ({nm!r})')
        await page.evaluate("(w) => window.__samon.setPath(w)", G[i2 + 2]['intended'])
        await page.wait_for_function("window.__samon.poem.state === 'shown'", timeout=12000); await page.wait_for_timeout(700)   # (a tap within 0.6 s of the raking is taken as a skip of it)
        wall = await page.evaluate("(() => { const r = document.getElementById('garden').getBoundingClientRect(); return [r.left + 4, r.top + 4]; })()")
        await page.mouse.click(*wall); await page.wait_for_timeout(300)
        nm = await page.evaluate("window.__samon.G.name"); po = await page.evaluate("window.__samon.poem"); st = await state(page)
        expect(nm == G[i2 + 2]['name'] and po['state'] == 'stay' and not po['visible'] and 'Read the poem again' in st['actions'], f'a tap on the walls around the poem puts it away and stays ({nm!r}, {po["state"]})')
        fit = await page.evaluate(f"""(() => {{ const out = [], fs = [], moved = [0, 0]; for (let i = 0; i < {len(G)}; i++) for (const o of [false, true]) {{
            const r = window.__samon.poemFit(i, o); fs.push(r.fs); moved[0] += r.left; moved[1] += r.under;
            if (r.over || (r.clash && !r.under) || (r.under && !o)) out.push([i, o, r]); }} return [out, Math.min(...fs), moved]; }})()""")
        expect(not fit[0], f'on a phone, every poem fits its garden and clears the seal (set flush left {fit[2][0]} times, over a faded seal {fit[2][1]} times, '
               f'only with the original; smallest type {fit[1]:.1f}px) {fit[0][:3]}')
        await page.evaluate("window.__samon.load(0)"); await page.evaluate("window.__samon.startDemo()")
        await page.wait_for_function("document.getElementById('status').textContent.startsWith('[That is')", timeout=15000)
        po = await page.evaluate("window.__samon.poem")
        expect(po['state'] == 'none' and not po['visible'], 'a demonstration earns no poem')
        await page.evaluate("window.__samon.enterPractice()")
        await page.evaluate("document.getElementById('garden').scrollIntoView({block: 'center'})")
        pp = await page.evaluate("document.getElementById('ppoem').textContent")
        expect(PO['poems'][PO['practice']]['translation'].split('\n')[0] in pp, 'the practice garden has its poem')
        await ctx.close()

        # 11. the scroll of poems earned, and the daily garden (one for everyone, picked by the date; raked once, by Rake)
        DAILY = json.load(open('/home/claude/zen/out/daily_gardens.json'))
        ctx = await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2, color_scheme='light')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(200)
        await quiet(page, delay=200, hold=3600000)
        await page.evaluate("window.__samon.enterPoems()"); await page.wait_for_timeout(150)
        sc0 = await page.evaluate("[document.querySelectorAll('#scroll .entry').length, getComputedStyle(document.getElementById('scroll')).display !== 'none', getComputedStyle(document.getElementById('stage')).display, window.__samon.view]")
        expect(sc0 == [1, True, 'none', 'poems'], f'the scroll before any raking: only the practice poem, shown in place of the garden {sc0}')
        for i in (0, 1):
            await page.evaluate(f"window.__samon.load({i})")
            await page.evaluate("(w) => window.__samon.setPath(w)", G[i]['intended'])
            await page.wait_for_function("['reveal', 'done'].includes(window.__samon.mode)", timeout=3000)
            await page.focus('#garden'); await page.keyboard.press('Enter')
            await page.wait_for_function("window.__samon.mode === 'done'", timeout=5000)
        await page.evaluate("window.__samon.enterPoems()"); await page.wait_for_timeout(150)
        sc = await page.evaluate("[[...document.querySelectorAll('#scroll .entry .verse')].map(e => e.textContent), [...document.querySelectorAll('#scroll .entry .ja')].map(e => e.textContent), [...document.querySelectorAll('#scroll .left')].map(e => e.textContent)]")
        expect(sc[0][:2] == [ppoem(G[0]['name'])['translation'], ppoem(G[1]['name'])['translation']] and sc[1][:2] == [ppoem(G[0]['name'])['japanese'], ppoem(G[1]['name'])['japanese']] and len(sc[0]) == 3,
               f'two gardens raked: their poems, in order, with the Japanese, then the practice poem ({len(sc[0])})')
        expect(sc[2][0].startswith('[2 of') and sc[2][1] == f"[{sum(1 for g in G if g['world'] == 1) - 2} more in the gardens of Dawn.]", f'the scroll counts what is earned and what is left, in editorial brackets {sc[2][:2]}')
        pills = await page.evaluate("[...document.querySelectorAll('#nav .pill')].map(b => b.textContent)")
        expect(any(t.startswith('Poems · 2 of') for t in pills) and pills[0].startswith("Today's garden") and any(t in ('Sound on', 'Sound off') for t in pills), f'the nav offers the poems, the daily garden and the sound {pills}')
        await page.click('#scroll .entry .by button >> nth=1'); await page.wait_for_timeout(150)
        v = await page.evaluate("[window.__samon.view, window.__samon.G.name]")
        expect(v == ['puzzle', G[1]['name']], f"a garden's name in the scroll opens it {v}")
        expect(len(DAILY) == 366 and all(e['world'] == 0 and e['intended'] and e['par_moves'] for e in DAILY), f'a year of daily gardens is embedded ({len(DAILY)})')
        await page.evaluate("window.__samon.today = '2026-10-04T12:00:00'")
        d = await page.evaluate("window.__samon.daily")
        expect(d['year'] == len(DAILY) and d['key'] == '2026-10-04' and d['dayNo'] == 276 and not d['done'], f'the daily slot: day 276 of the year, not yet raked {d}')
        await page.evaluate("window.__samon.pickDaily()"); await page.wait_for_timeout(250)
        g = await page.evaluate("window.__samon.G")
        day = DAILY[276 % len(DAILY)]
        hdr = await page.evaluate("[document.getElementById('time').textContent, document.getElementById('gname').textContent, document.getElementById('next').textContent, document.getElementById('rakeDay').hidden]")
        expect(g['rows'] == day['rows'] and hdr == ['Daily garden', '4 October 2026', 'Back to the gardens', True], f'today\'s garden is the year\'s day 276, under its date {hdr}')
        await page.evaluate("(w) => window.__samon.setPath(w)", day['intended']); await page.wait_for_timeout(900)
        st = await page.evaluate("[window.__samon.mode, document.getElementById('rakeDay').hidden, document.getElementById('status').textContent]")
        expect(st[0] == 'draw' and not st[1] and 'press Rake' in st[2], f'a complete walk is not raked by itself here: it waits for Rake {st}')
        await page.click('#rakeDay')
        await page.wait_for_function("window.__samon.mode === 'done'", timeout=8000)
        await page.wait_for_function("window.__samon.poem.visible", timeout=5000)
        d2 = await page.evaluate("window.__samon.daily")
        st2 = await page.evaluate("[document.getElementById('status').textContent, [...document.querySelectorAll('#actions button')].map(b => b.textContent), window.__samon.poem.state, document.getElementById('seal').hidden]")
        expect(d2['done'] and d2['streak'] == 1 and d2['turns'] == day['par'] and d2['kind'] == 'red' and 'Streak: 1 day' in st2[0] and st2[1] == ["Share today's score", 'Read the poem again'] and st2[2] == 'stay' and not st2[3],
               f'raked at par: the record, a streak of one, the seal, the share action; the poem stays {d2} {st2}')
        box = await page.evaluate("(() => { const r = document.getElementById('garden').getBoundingClientRect(); return [r.left + r.width / 2, r.top + r.height / 2]; })()")
        await page.mouse.click(*box); await page.wait_for_timeout(250); await page.mouse.click(*box); await page.wait_for_timeout(250)
        st3 = await page.evaluate("[window.__samon.mode, document.getElementById('undo').disabled, document.getElementById('restart').disabled]")
        expect(st3 == ['done', True, True], f'the raking is final: no walking back, Step back and Start over off {st3}')
        await page.reload(); await page.wait_for_timeout(400)
        await page.evaluate("window.__samon.today = '2026-10-04T12:00:00'")
        await page.evaluate("window.__samon.pickDaily()"); await page.wait_for_timeout(300)
        st4 = await page.evaluate("[window.__samon.mode, document.getElementById('status').textContent, document.getElementById('seal').hidden, window.__samon.daily.streak, window.__samon.path.length]")
        expect(st4[0] == 'done' and "Today's record" in st4[1] and not st4[2] and st4[3] == 1 and st4[4] == len(day['intended']), f'after a reload the day\'s raking and its record are back {st4[:4]}')
        await page.evaluate("window.__samon.today = '2026-10-05T12:00:00'")
        await page.evaluate("window.__samon.pickDaily()"); await page.wait_for_timeout(300)
        d3 = await page.evaluate("window.__samon.daily"); g3 = await page.evaluate("window.__samon.G.rows")
        expect(d3['key'] == '2026-10-05' and not d3['done'] and d3['streak'] == 1 and g3 == DAILY[277 % len(DAILY)]['rows'] and await page.evaluate("window.__samon.mode") == 'draw', f'the next day is a fresh garden, and the streak counts yesterday {d3}')
        nx = await page.evaluate("(() => { window.__samon.load(0); document.getElementById('next').click(); return window.__samon.G.name; })()")
        expect(nx != d3['name'] and nx == G[1]['name'], f'the daily garden is never the next garden ({nx})')
        await ctx.close()

        # 12. drag intent: the walk follows the axis the hand moves on, not the cell under the pointer; the grip survives the walls
        ctx = await b.new_context(viewport={'width': 700, 'height': 900}, device_scale_factor=2, color_scheme='light')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(200); await quiet(page)
        gi = next(i for i, g in enumerate(G) if len(g['rows']) >= 6 and len(g['rows'][0]) >= 7 and g['gate'][2] == 'S' and g['gate'][0] == len(g['rows']) - 1
                  and set(g['rows'][-1] + g['rows'][-2]) == {'.'} and g['rows'][-3][g['gate'][1] + 3] == '.')
        H, W = len(G[gi]['rows']), len(G[gi]['rows'][0]); gr, gc = G[gi]['gate'][:2]
        async def at(r, c, dx=0.0, dy=0.0):
            return await page.evaluate("""([r, c, dx, dy]) => { const cv = document.getElementById('garden'); const b = cv.getBoundingClientRect();
                const s = b.width / cv.width; const g = window.__samon.geom();
                return [b.left + (g.ox + (c + 0.5 + dx) * g.cs) * s, b.top + (g.oy + (r + 0.5 + dy) * g.cs) * s]; }""", [r, c, dx, dy])
        async def walk():
            return await page.evaluate("window.__samon.path.map(k => [Math.floor(k / window.__samon.G.W), k % window.__samon.G.W])")
        async def begin():
            await page.evaluate(f"window.__samon.load({gi}); window.__samon.setPath([]); document.getElementById('garden').scrollIntoView({{block: 'center'}})"); await page.wait_for_timeout(50)
            x, y = await at(gr, gc); await page.mouse.move(x, y); await page.mouse.down()
        run = [[gr, c] for c in range(gc, gc + 4)]
        await begin(); x, y = await at(gr, gc + 3, 0, -0.9); await page.mouse.move(x, y, steps=40); await page.mouse.up()
        expect(await walk() == run, f'a drag along the row whose hand drifts 0.9 of a cell into the next row still draws the row ({await walk()})')
        await begin(); x, y = await at(gr, gc + 1, 0, 1.6); await page.mouse.move(x, y, steps=12); x, y = await at(gr, gc + 3, 0, 1.8); await page.mouse.move(x, y, steps=30); await page.mouse.up()
        expect(await walk() == run, f'a pointer below the wall keeps its grip and the walk follows along the row ({await walk()})')
        await begin(); x, y = await at(gr, gc + 3, 0.1, 0); await page.mouse.move(x, y, steps=40)
        for k in range(1, 13):   # a U-turn: a semicircle within the last cell of the run, up into the next row
            a = math.pi / 2 - math.pi * k / 12
            x, y = await at(gr, gc + 3, 0.1 + 0.35 * math.cos(a), -0.35 + 0.35 * math.sin(a)); await page.mouse.move(x, y)
        x, y = await at(gr - 1, gc); await page.mouse.move(x, y, steps=40); await page.mouse.up()
        expect(await walk() == run + [[gr - 1, c] for c in range(gc + 3, gc - 1, -1)], f'a U-turn drawn as an arc turns once and comes back along the next row ({await walk()})')
        await begin(); x, y = await at(gr, gc + 3, -0.2, 0.05); await page.mouse.move(x, y, steps=30)
        for k in range(1, 9):    # a rounded corner from heading east to heading north
            a = (math.pi / 2) * k / 8
            x, y = await at(gr, gc + 3, -0.2 + 0.3 * math.sin(a), 0.05 - 0.3 + 0.3 * math.cos(a)); await page.mouse.move(x, y)
        x, y = await at(gr - 2, gc + 3, 0.1, 0); await page.mouse.move(x, y, steps=30); await page.mouse.up()
        expect(await walk() == run + [[gr - 1, gc + 3], [gr - 2, gc + 3]], f'a rounded corner is a run and a turn ({await walk()})')
        await begin(); x, y = await at(gr, gc + 3, 0, -0.6); await page.mouse.move(x, y, steps=30); out = await walk()
        x, y = await at(gr, gc, -0.2, -0.6); await page.mouse.move(x, y, steps=30); await page.mouse.up()
        expect(out == run and await walk() == [[gr, gc]], f'dragging back along a drifted run retraces it ({out} → {await walk()})')
        await begin()
        for k in range(40): x, y = await at(gr, gc, 0.12 * math.cos(k), 0.12 * math.sin(k)); await page.mouse.move(x, y)
        await page.mouse.up()
        expect(await walk() == [[gr, gc]], f'a wobble in place takes no step ({await walk()})')
        await ctx.close()

        # 13. sound: nothing before a gesture; the synthesized sounds render; Sound off persists; the recordings carried in the
        # page are found, credited on the scroll with their source, put down to follow a world's bell, and play
        import struct
        PIECES = json.load(open('/home/claude/zen/out/sound_pieces.json'))
        ctx = await b.new_context(viewport={'width': 390, 'height': 900}, device_scale_factor=2, color_scheme='light')
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(300); await quiet(page)
        s0 = await page.evaluate("[window.__samon.sound.on, window.__samon.sound.ready, window.__samon.sound.state, window.__samon.sound.tracks]")
        expect(s0 == [True, False, 'none', None], f'before any gesture there is no audio context, and the recordings are not looked at {s0}')
        await page.click('#garden'); await page.wait_for_timeout(200)
        s1 = await page.evaluate("[window.__samon.sound.ready, window.__samon.sound.state, window.__samon.sound.timers, window.__samon.sound.tracks.map(t => [t.title, t.src.slice(0, 22), t.dur])]")
        expect(s1[:2] == [True, 'running'] and s1[2] in (6, 7) and s1[3] == [[p['title'], 'data:audio/mpeg;base64', p['dur']] for p in PIECES], f'a touch starts the audio, the six ambience loops (and, for a moment, the speaker\'s redraw), and finds the recordings in the page {s1}')
        wav = bytes(await page.evaluate("window.__samon.sound.render([[0.05, 'drip', 0.3], [0.3, 'tock', 0.4], [0.7, 'tok', 0.4], [0.9, 'bell', 0.5]], 1.3)"))
        pcm = struct.unpack(f'<{(len(wav) - 44) // 2}h', wav[44:])
        def peak(t0, t1): return max(abs(v) for v in pcm[int(t0 * 44100):int(t1 * 44100)]) / 32768
        pk = [round(peak(*w), 3) for w in ((0.05, 0.15), (0.3, 0.4), (0.7, 0.75), (0.9, 1.2))]
        expect(all(p > 0.03 for p in pk) and peak(0, 0.05) == 0, f'drip, pipe, seal and bell each render audibly, and nothing before the first {pk}')
        wav = bytes(await page.evaluate("window.__samon.sound.render([[0.05, 'scrape', 1.2, 1], [1.5, 'crunch', 0.14], [1.8, 'birds', 0.1], [3.0, 'crickets', 0.08, 0.5], [3.7, 'wind', 0.3], [3.8, 'gust', 0.4, 1]], 5)"))
        pcm = struct.unpack(f'<{(len(wav) - 44) // 2}h', wav[44:])
        pk = [round(peak(*w), 3) for w in ((0.3, 1.1), (1.5, 1.6), (1.85, 2.9), (3.0, 3.5), (4.3, 5.0))]
        expect(all(p > 0.015 for p in pk) and peak(1.44, 1.5) < 0.02, f'the rake\'s scrape, a footstep, a bird, a cricket and the wind each render audibly, and the scrape dies away when the rake stops {pk} (tail {peak(1.44, 1.5):.3f})')
        dp = await page.evaluate("[5, 6, 12, 18, 19, 20, 23, 4].map(h => window.__samon.sound.dayPart(h))")
        expect(dp == ['dusk', 'day', 'day', 'day', 'dusk', 'night', 'night', 'night'], f'birds from 6 to 19, crickets from 20 to 5, by the local clock {dp}')
        await page.evaluate("window.__samon.load(0)"); await page.evaluate("(w) => window.__samon.setPath(w)", G[0]['intended'])
        await page.wait_for_function("window.__samon.mode === 'reveal'", timeout=3000)
        sc1 = await page.evaluate("window.__samon.sound.scraping")
        await page.wait_for_function("window.__samon.mode === 'done'", timeout=12000); await page.wait_for_timeout(100)
        sc2 = await page.evaluate("window.__samon.sound.scraping")
        expect(sc1 and not sc2, f'the scrape runs with the raking and stops with it ({sc1}, {sc2})')
        await page.evaluate("window.__samon.sound.setVolume(0.3)"); await page.wait_for_timeout(300)
        vs = await page.evaluate("[window.__samon.sound.vol, Math.round(window.__samon.sound.master * 100) / 100, localStorage.getItem('samon-sound-vol'), document.getElementById('soundVol') && document.getElementById('soundVol').value, !!document.getElementById('hapticsToggle'), window.__samon.sound.haptics]")
        expect(vs[:3] == [0.3, 0.3, '0.3'] and vs[4] and vs[5], f'the volume is set, remembered and reaches the master gain; the haptics switch shows where the device can tick {vs}')
        await page.click('#hapticsToggle'); await page.wait_for_timeout(100)
        hp = await page.evaluate("[window.__samon.sound.haptics, localStorage.getItem('samon-haptics'), document.getElementById('hapticsToggle').textContent]")
        expect(hp == [False, '0', 'Haptics off'], f'haptics can be turned off, and that is remembered {hp}')
        s2 = await page.evaluate("[window.__samon.sound.ready, window.__samon.sound.state, window.__samon.sound.on]")
        expect(s2 == [True, 'running', True], f'the live context is untouched by the offline render {s2}')
        tm = await page.evaluate("(() => { const a = window.__samon.sound.timers; window.__samon.sound.piece('capstone'); return [a, window.__samon.sound.timers]; })()")
        expect(tm[1] == tm[0] + 1, f'after a world\'s bell a piece is put down to follow {tm}')
        await page.evaluate(f"window.__samon.sound.play(0, {PIECES[0]['dur'] - 12})"); await page.wait_for_function("window.__samon.sound.playing !== null", timeout=5000)
        await page.wait_for_timeout(4000); lv = await page.evaluate("window.__samon.sound.level()")
        expect(lv > 0.002, f'the recording is heard: the output meter reads {lv:.4f} while it plays (silence would mean the media source is muted)')
        await page.evaluate("window.__samon.enterPoems()"); await page.wait_for_timeout(150)
        cr = await page.evaluate("[document.getElementById('soundCredits').textContent, [...document.querySelectorAll('#soundCredits a')].map(a => [a.textContent, a.href])]")
        expect(f"Shakuhachi, now and then: {PIECES[0]['title']}, {PIECES[0]['player']} ({PIECES[0]['license']})." in cr[0] and cr[1] == [[PIECES[0]['title'], PIECES[0]['source']]],
               f'the scroll credits the recording, with its source {cr}')
        await page.wait_for_function("window.__samon.sound.playing === null", timeout=20000)
        expect(True, 'the piece plays to its end and is let go')
        await page.click('#soundToggle'); await page.wait_for_timeout(150)
        s3 = await page.evaluate("[window.__samon.sound.on, window.__samon.sound.state, window.__samon.sound.timers, document.getElementById('soundToggle').textContent, localStorage.getItem('samon-sound')]")
        expect(s3 == [False, 'suspended', 0, 'Sound off', '0'], f'Sound off suspends the context, clears the loops, and is remembered {s3}')
        await page.reload(); await page.wait_for_timeout(300); await quiet(page); await page.click('#garden'); await page.wait_for_timeout(200)
        s4 = await page.evaluate("[window.__samon.sound.on, window.__samon.sound.ready, document.getElementById('soundToggle').textContent]")
        expect(s4 == [False, False, 'Sound off'], f'after a reload sound stays off, and a touch makes no context {s4}')
        noslider = await page.evaluate("!document.getElementById('soundVol')")
        await page.click('#soundToggle'); await page.wait_for_timeout(200)
        s5 = await page.evaluate("[window.__samon.sound.on, window.__samon.sound.state, window.__samon.sound.timers, document.getElementById('soundVol').value, window.__samon.sound.vol]")
        expect(s5[:2] == [True, 'running'] and s5[2] in (6, 7) and s5[3:] == ['30', 0.3] and noslider, f'Sound on starts it again, with its loops (drip, pipe, wind, gusts, birds, crickets) and the remembered volume; no slider while off {s5}')
        # the mute button and the M key; what the page can tell of the browser's own silence
        mb = await page.evaluate("(() => { const b = document.getElementById('mute'); return [b.className, b.getAttribute('aria-pressed'), b.getAttribute('aria-label'), b.title]; })()")
        await page.click('#mute'); await page.wait_for_timeout(120)
        mb2 = await page.evaluate("(() => { const b = document.getElementById('mute'); return [b.className, b.getAttribute('aria-pressed'), b.getAttribute('aria-label'), window.__samon.sound.on, window.__samon.sound.state, document.getElementById('soundToggle').textContent, localStorage.getItem('samon-sound')]; })()")
        await page.keyboard.press('m'); await page.wait_for_timeout(200)
        mb3 = await page.evaluate("[document.getElementById('mute').className, window.__samon.sound.on, window.__samon.sound.state, window.__samon.sound.timers > 0]")
        expect(mb == ['mute', 'false', 'Mute', 'Mute (M)'] and mb2 == ['mute off', 'true', 'Unmute', False, 'suspended', 'Sound off', '0'] and mb3 == ['mute', True, 'running', True],
               f'the speaker in the header mutes and unmutes, as does M, in step with the list\'s switch and the saved setting {mb} {mb2} {mb3}')
        bl = await page.evaluate("(() => { navigator.getAutoplayPolicy = () => 'disallowed'; window.__samon.sound.renderMute(); const b = document.getElementById('mute'); const r = [window.__samon.sound.blocked, b.className, b.getAttribute('aria-label')]; delete navigator.getAutoplayPolicy; window.__samon.sound.renderMute(); return r.concat([b.className]); })()")
        expect(bl == [True, 'mute blocked', 'Sound is blocked by the browser', 'mute'], f'when the browser says it will not play (Firefox\'s autoplay policy), the speaker says so in red, and recovers {bl}')
        aud = await page.evaluate("[typeof navigator.audioSession, typeof navigator.mediaSession, !!document.getElementById('muteHint') || localStorage.getItem('samon-sound-hint') === '1']")
        expect(aud[2], f'the first start of sound shows a one-time word beside the speaker (audioSession: {aud[0]}, mediaSession: {aud[1]})')
        await ctx.close()

        # 14. sharing: a raking is a code in the URL's hash (the walk as a number in the mixed radix of its legal moves), shown
        # but not credited to whoever opens it; a player's whole progress is another code, offered for taking over; the share
        # text draws the raking in box-drawing characters
        ctx = await b.new_context(viewport={'width': 500, 'height': 900}, device_scale_factor=2, color_scheme='light', permissions=['clipboard-read', 'clipboard-write'])
        page = await ctx.new_page(); page.on('pageerror', lambda e: errors.append(str(e)))
        await page.goto(URL); await page.wait_for_timeout(300); await quiet(page)
        lens = []; bits = []; bad = []
        for i in (range(len(G)) if not os.environ.get('SAMON_QUICK') else range(0, len(G), 7)):
            g = G[i]; await page.evaluate(f"window.__samon.load({i})")
            code = await page.evaluate("(w) => window.__samon.share.code(w)", g['intended'])
            w = await page.evaluate("(w) => window.__samon.share.walk(w)", g['intended'])
            back = await page.evaluate("(a) => window.__samon.share.unwalk(a[0], a[1])", [w[0], len(g['intended']) - 1])
            h = await page.evaluate("(c) => { const h = window.__samon.share.read(c); return [h.i, h.grey, h.n.toString()]; }", code)
            if not (code and back == g['intended'] and h == [i, False, w[0]]): bad.append(g['name'])
            lens.append(len(code)); bits.append(w[1])
        expect(not bad and max(lens) <= 18, f'every intended raking round-trips through its code: {len(lens)} gardens, {min(lens)}–{max(lens)} characters (mean {sum(lens) / len(lens):.1f}); the walks take {sum(bits) / len(bits):.1f} bits on average, against {sum(2 * (len(g["intended"]) - 1) for g in G) / len(G):.0f} at two bits a step {bad}')
        lay = await page.evaluate("window.__samon.share.layouts()")
        allh = lay['campaign'] + lay['daily']
        expect(len(set(allh)) == len(allh) == len(G) + 366 and all(0 <= h < 2 ** 32 for h in allh), f'every garden, campaign or daily, has its own 32-bit layout hash ({len(set(allh))} of {len(allh)} distinct)')
        legal = await page.evaluate("(() => { window.__samon.load(0); return window.__samon.share.walk([[2, 0], [2, 1], [1, 1]]); })()")
        expect(legal is None, f'a walk that breaks a rule (onto a stone) has no code {legal}')
        grid = await page.evaluate("(w) => window.__samon.share.text(w, 'red')", G[0]['intended'])
        rows = grid.split('\n')
        expect(rows[0] == f"Samon · Dawn 1 · {G[0]['name']} · {G[0]['par']} turns, par {G[0]['par']} · 静" and len(rows) == len(G[0]['rows']) + 3 and all(len(r) == len(G[0]['rows'][0]) + 2 for r in rows[1:])
               and rows[1] == '┏━━━━┓' and rows[2] == '┃╶──┐┃' and rows[3] == '┃███│┃' and rows[4] == '┃┌──┘┃' and rows[-1] == '┗┷━━━┛', f'the share text: a heading, then the garden in box-drawing characters, the stroke in light lines through the gate in the heavy wall, stones as blocks\n{grid}')
        i2 = next(i for i, g in enumerate(G) if g['end'] and g['end_side'] == 'E')
        await page.evaluate(f"window.__samon.load({i2})")
        grid2 = await page.evaluate("(w) => window.__samon.share.grid(w)", G[i2]['intended'])
        r2 = grid2.split('\n')
        expect(r2[G[i2]['end'][0] + 1].endswith('┨') and sum(1 for ch in grid2 if ch in '┨┠┯┷') == 2, f'a side gate is a second gap in the wall, where the stroke begins\n{grid2}')
        # #r=: the raking is shown, with its seal, but recorded for nobody; Rake it yourself (or any tap) gives the viewer the garden fresh
        gi = 5; g5 = G[gi]
        await page.evaluate(f"window.__samon.load({gi})")
        code = await page.evaluate("(w) => window.__samon.share.code(w)", g5['intended'])
        await page.evaluate("localStorage.clear()")
        await page.goto('about:blank'); await page.goto(URL + '#r=' + code); await page.wait_for_timeout(600); await quiet(page)
        st = await page.evaluate("[window.__samon.mode, window.__samon.G.name, window.__samon.path.length, window.__samon.turnsOf(), document.getElementById('status').textContent, [...document.querySelectorAll('#actions button')].map(b => b.textContent), document.getElementById('seal').hidden, document.getElementById('next').hidden, localStorage.getItem('samon-progress'), !!window.__samon.share.shared, document.getElementById('undo').disabled]")
        expect(st[:4] == ['done', g5['name'], len(g5['intended']), g5['par']] and st[4] == f"[A raking shared with you: {g5['par']} turns, par {g5['par']}, the abbot's seal. Rake it yourself from the gate: a tap smooths the sand.]"
               and st[5] == ['Rake it yourself', 'Share it on'] and not st[6] and st[7] and st[8] is None and st[9] and not st[10], f'a link to a raking opens that garden raked, with the seal, credited to nobody {st[:8]} {st[8:]}')
        pts = await centers(page, [g5['intended'][3]]); await page.mouse.click(*pts[0]); await page.wait_for_timeout(300)
        st = await page.evaluate("[window.__samon.mode, window.__samon.G.name, window.__samon.path.length, document.getElementById('status').textContent, location.hash, !!window.__samon.share.shared, document.getElementById('restore').hidden, document.getElementById('seal').hidden]")
        expect(st == ['draw', g5['name'], 0, '[Now you: drag the monk in from the gate.]', '', False, True, True], f'a tap on the shared raking smooths it away: the garden is the viewer\'s, from the gate, with nothing to Restore, and the hash is gone {st}')
        await page.goto(URL + '#r=' + code); await page.wait_for_timeout(400)   # (a hash change on the open page shows it too)
        st = await page.evaluate("[window.__samon.mode, !!window.__samon.share.shared]")
        await page.click('#actions button >> nth=0'); await page.wait_for_timeout(300)
        st2 = await page.evaluate("[window.__samon.mode, window.__samon.path.length, !!window.__samon.share.shared, location.hash]")
        expect(st == ['done', True] and st2 == ['draw', 0, False, ''], f'the same by Rake it yourself, after the link is followed on the open page {st} {st2}')
        gcode = await page.evaluate("(w) => window.__samon.share.code(w, true)", g5['intended'])
        await page.goto(URL + '#r=' + gcode); await page.wait_for_timeout(400)
        st = await page.evaluate("[document.getElementById('seal').className, document.getElementById('status').textContent.includes('his seal in grey')]")
        expect(st == ['seal grey', True], f'a raking made with the abbot\'s help shows his grey seal {st}')
        # a daily garden's raking: of another day, the viewer is sent on to today's; of today, raked at par, it is shown and then today's is theirs to rake
        DAILY = json.load(open('/home/claude/zen/out/daily_gardens.json'))
        await page.evaluate("window.__samon.today = '2026-10-04T12:00:00'; window.__samon.pickDaily()"); await page.wait_for_timeout(200)
        day = DAILY[276 % len(DAILY)]
        dcode = await page.evaluate("(w) => window.__samon.share.code(w)", day['intended'])
        await page.goto('about:blank'); await page.goto(URL + '#r=' + dcode); await page.wait_for_timeout(600); await quiet(page)
        st = await page.evaluate("[window.__samon.mode, window.__samon.G.name, document.getElementById('gname').textContent, document.getElementById('status').textContent, [...document.querySelectorAll('#actions button')].map(b => b.textContent), window.__samon.daily.key]")
        expect(st[:3] == ['done', 'daily:2026-10-04', '4 October 2026'] and 'of the daily garden of 4 October 2026' in st[3] and "Today's garden is another" in st[3] and st[4][0] == "Today's garden" and st[5] == '2026-10-04',
               f'a shared raking of an earlier day\'s garden is shown under its date, and offers today\'s instead {st}')
        await page.click('#actions button >> nth=0'); await page.wait_for_timeout(300)
        st = await page.evaluate("[window.__samon.mode, window.__samon.G.name, window.__samon.path.length, location.hash, window.__samon.daily.done]")
        expect(st[0] == 'draw' and st[1].startswith('daily:') and st[1] != 'daily:2026-10-04' and st[2] == 0 and st[3] == '' and not st[4], f'Today\'s garden leaves it for today\'s, unraked {st}')
        await page.evaluate("window.__samon.today = '2026-10-04T12:00:00'")
        await page.goto(URL + '#r=' + dcode); await page.wait_for_timeout(400)
        st = await page.evaluate("[window.__samon.mode, window.__samon.G.name, document.getElementById('status').textContent.includes('Rake it yourself'), [...document.querySelectorAll('#actions button')].map(b => b.textContent)[0]]")
        await page.click('#actions button >> nth=0'); await page.wait_for_timeout(300)
        st2 = await page.evaluate("[window.__samon.mode, window.__samon.G.name, window.__samon.path.length, document.getElementById('rakeDay').hidden]")
        expect(st == ['done', 'daily:2026-10-04', True, 'Rake it yourself'] and st2 == ['draw', 'daily:2026-10-04', 0, True], f'the same day\'s raking, shared, is shown and then left for the viewer to rake {st} {st2}')
        # the share text and link: the share action after a raking, to the clipboard where there is no share sheet
        await page.evaluate("window.__samon.load(0)"); await page.evaluate("(w) => window.__samon.setPath(w)", G[0]['intended'])
        await page.wait_for_function("['reveal', 'done'].includes(window.__samon.mode)", timeout=3000)
        await page.focus('#garden'); await page.keyboard.press('Enter'); await page.wait_for_function("window.__samon.mode === 'done'", timeout=5000)
        acts = await page.evaluate("[...document.querySelectorAll('#actions button')].map(b => b.textContent)")
        await page.click('#actions button:has-text("Share this raking")'); await page.wait_for_timeout(400)
        clip = await page.evaluate("navigator.clipboard.readText()")
        st = await page.evaluate("document.getElementById('status').textContent")
        cl = clip.split('\n')
        expect(acts == ['Share this raking'] and cl[0] == f"Samon · Dawn 1 · {G[0]['name']} · {G[0]['par']} turns, par {G[0]['par']} · 静" and cl[1:-1] == grid.split('\n')[1:] and cl[-1].startswith(URL + '#r=') and st.startswith('[Copied: the raking drawn in lines, with a link that opens it'),
               f'after a raking, Share this raking copies the heading, the drawing and a link that opens the raking {cl[0]!r} {cl[-1][-30:]!r} {st!r}')
        await page.goto('about:blank'); await page.goto(cl[-1]); await page.wait_for_timeout(600)
        st = await page.evaluate("[window.__samon.mode, window.__samon.G.name, !!window.__samon.share.shared, window.__samon.turnsOf()]")
        expect(st == ['done', G[0]['name'], True, G[0]['par']], f'and that link shows it {st}')
        # #p=: progress as a code; the link offers it, Take it over joins it with what is here (whichever is better stands)
        await page.evaluate("localStorage.clear()"); await page.goto('about:blank'); await page.goto(URL); await page.wait_for_timeout(300); await quiet(page)
        for i in (0, 1, 2):
            await page.evaluate(f"window.__samon.load({i})"); await page.evaluate("(w) => window.__samon.setPath(w)", G[i]['intended'])
            await page.wait_for_function("['reveal', 'done'].includes(window.__samon.mode)", timeout=3000)
            await page.focus('#garden'); await page.keyboard.press('Enter'); await page.wait_for_function("window.__samon.mode === 'done'", timeout=5000)
        await page.evaluate("window.__samon.today = '2026-10-04T12:00:00'; window.__samon.pickDaily()"); await page.wait_for_timeout(200)
        await page.evaluate("(w) => window.__samon.setPath(w)", day['intended']); await page.wait_for_timeout(300); await page.click('#rakeDay')
        await page.wait_for_function("window.__samon.mode === 'done'", timeout=8000)
        await page.evaluate("window.__samon.load(3)"); await page.evaluate("(w) => window.__samon.setPath(w)", G[3]['intended'][:4]); await page.wait_for_timeout(100)
        await page.evaluate("window.__samon.askAbbot()"); await page.wait_for_timeout(500)   # (help on the unfinished walk travels with it; the save is debounced)
        pcode = await page.evaluate("window.__samon.share.progress()")
        mine = json.loads(await page.evaluate("localStorage.getItem('samon-progress')"))
        expect(len(pcode) < 120 and mine[G[3]['name']].get('helped') and len(mine[G[3]['name']]['draft']) == 4, f'the whole progress is a code of {len(pcode)} characters {pcode}')
        await page.evaluate("localStorage.clear()"); await page.goto('about:blank')
        await page.goto(URL); await page.wait_for_timeout(300); await quiet(page)
        await page.evaluate(f"window.__samon.load(1)"); await page.evaluate("(w) => window.__samon.setPath(w)", G[1]['intended'][:3]); await page.wait_for_timeout(400)   # (something of its own here: an unfinished walk on garden 2)
        await page.evaluate("window.__samon.today = '2026-10-04T12:00:00'")
        await page.goto(URL + '#p=' + pcode); await page.wait_for_timeout(500)
        st = await page.evaluate("[document.getElementById('status').textContent, [...document.querySelectorAll('#actions button')].map(b => b.textContent), localStorage.getItem('samon-progress')]")
        expect(st[0] == "[This link carries a player's progress: 3 of 148 gardens raked, 3 sealed, 1 daily garden, and an unfinished walk. Take it over here? It joins what is here, and whichever is better stands.]" and st[1] == ['Take it over', 'Leave it'] and G[0]['name'] not in (st[2] or ''),
               f'a progress link is offered, not taken, with what it holds {st[:2]}')
        tt = await page.evaluate("[...document.querySelectorAll('.veranda button, .assists button, .assists .t, #nav button, #nav .more, #nav input, #actions button, #mute, #orig, .count span, #where, .brand, #seal, .facts')].filter(e => !e.title).map(e => e.id || e.className || e.textContent)")
        expect(not tt, f'every control, count and mark has a tooltip (gwern.net wants one wherever a hover could want a word more) {tt}')
        await page.click('#actions button >> nth=0'); await page.wait_for_timeout(400)
        got = json.loads(await page.evaluate("localStorage.getItem('samon-progress')"))
        st = await page.evaluate("[document.getElementById('status').textContent, location.hash, window.__samon.G.name, window.__samon.path.length, window.__samon.helped, [...document.querySelectorAll('#nav .pip.sealed')].length, window.__samon.daily.done, window.__samon.daily.turns, window.__samon.daily.streak]")
        expect(all(got[G[i]['name']] == mine[G[i]['name']] for i in (0, 2)) and got[G[1]['name']]['done'] and got[G[1]['name']]['sealed'] and len(got[G[1]['name']]['draft']) == 3
               and got[G[3]['name']]['draft'] == mine[G[3]['name']]['draft'] and got[G[3]['name']].get('helped') and got['daily:2026-10-04']['path'] == mine['daily:2026-10-04']['path'] and got['daily:2026-10-04']['turns'] == day['par'] and got['daily:2026-10-04']['kind'] == 'red'
               and st[0].startswith('[Taken over: 3 of 148 gardens raked') and st[1] == '' and st[2] == G[3]['name'] and st[3] == 4 and st[4] and st[5] == 3 and st[6] and st[7] == day['par'] and st[8] == 1,
               f'Take it over: the rakings, the seals, the day\'s record with its walk, and the unfinished walk with its help arrive, and the walk of its own here is kept {st}')
        await page.evaluate("window.__samon.pickDaily()"); await page.wait_for_timeout(300)
        st = await page.evaluate("[window.__samon.mode, window.__samon.path.length, document.getElementById('status').textContent.includes(\"Today's record\")]")
        expect(st == ['done', len(day['intended']), True], f'the carried day\'s raking shows as the record {st}')
        await page.click('#actions button:has-text("Share today\'s score")'); await page.wait_for_timeout(400)
        cl = (await page.evaluate("navigator.clipboard.readText()")).split('\n')
        shape = await page.evaluate("window.__samon.share.grid([])")
        expect(cl[0] == f"Samon · daily garden 2026-10-04 · {day['par']} turns, par {day['par']} · 静 · streak 1" and cl[1:-1] == shape.split('\n') and cl[-1] == URL + '#daily'
               and not any(ch in shape for ch in '─│┌┐└┘╴╵╶╷') and shape.count('░') == sum(r.count('.') for r in day['rows']),
               f'the daily garden shares the score and the garden\'s shape, without the stroke, and a link to today\'s garden {cl[0]!r} {cl[-1][-8:]!r}')
        await page.evaluate("window.__samon.load(0)"); await page.goto(URL + '#daily'); await page.wait_for_timeout(400)
        st = await page.evaluate("[window.__samon.G.name, location.hash]")
        expect(st == ['daily:2026-10-04', ''], f'#daily opens today\'s garden {st}')
        await page.click('#carry'); await page.wait_for_timeout(400)
        cl = await page.evaluate("navigator.clipboard.readText()"); st = await page.evaluate("document.getElementById('status').textContent")
        expect(cl.startswith(URL + '#p=') and st.startswith('[Copied a link that carries your progress'), f'Carry my progress copies such a link {cl[:60]!r} {st!r}')
        await page.goto('about:blank'); await page.goto(URL + '#r=AAAA'); await page.wait_for_timeout(500)
        st = await page.evaluate("[document.getElementById('status').textContent, document.getElementById('status').className, location.hash, window.__samon.mode]")
        expect(st == ["[The link's code can't be read: it was made by another edition of the game.]", 'status warn', '', 'draw'], f'a code the page can\'t read is said so, and dropped {st}')
        await page.goto(URL + '#p=not a code'); await page.wait_for_timeout(300)
        st = await page.evaluate("[document.getElementById('status').textContent, location.hash]")
        expect(st == ["[The link's code can't be read: it is not a Samon code.]", ''], f'so is a string that is no code at all {st}')
        # durability (Astra's points): a garden is named by a 32-bit layout hash, not its place, so a code survives a reordering of the
        # campaign and a changed garden is refused, never mistaken for the one meant
        B64 = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_'
        await page.evaluate("localStorage.clear()"); await page.goto('about:blank'); await page.goto(URL); await page.wait_for_timeout(300); await quiet(page)
        await page.evaluate("window.__samon.load(5)")
        code5 = await page.evaluate("(w) => window.__samon.share.code(w)", G[5]['intended'])
        bits = ''.join(format(B64.index(ch), '06b') for ch in code5)
        bent = bits[:5] + ('1' if bits[5] == '0' else '0') + bits[6:]     # one bit of the layout hash
        bent = ''.join(B64[int(bent[i:i + 6], 2)] for i in range(0, len(bent), 6))
        await page.goto(URL + '#r=' + bent); await page.wait_for_timeout(400)
        st = await page.evaluate("[document.getElementById('status').textContent, window.__samon.mode, location.hash]")
        expect(st == ["[The link's code can't be read: its garden is not in this edition, or has changed since.]", 'draw', ''], f'a code whose garden hash matches no garden is refused, not shown over another garden {st}')
        await page.evaluate("(w) => window.__samon.setPath(w)", G[5]['intended'])
        await page.wait_for_function("['reveal', 'done'].includes(window.__samon.mode)", timeout=3000)
        await page.focus('#garden'); await page.keyboard.press('Enter'); await page.wait_for_function("window.__samon.mode === 'done'", timeout=5000)
        pcode5 = await page.evaluate("window.__samon.share.progress()")
        await page.evaluate("window.__samon.share.swap(5, 9); window.__samon.load(0)")     # the campaign reordered under the links
        await page.goto(URL + '#r=' + code5); await page.wait_for_timeout(400)
        st = await page.evaluate("[window.__samon.G.name, window.__samon.mode, !!window.__samon.share.shared, [...document.querySelectorAll('#nav .pip')].findIndex(b => b.getAttribute('aria-current') === 'true')]")
        expect(st == [G[5]['name'], 'done', True, 9], f'after the campaign is reordered, a raking link still opens the garden it was made in, now in its new place {st}')
        await page.evaluate("localStorage.clear()")
        await page.goto(URL + '#p=' + pcode5); await page.wait_for_timeout(400)
        await page.click('#actions button:has-text("Take it over")'); await page.wait_for_timeout(400)
        got = json.loads(await page.evaluate("localStorage.getItem('samon-progress')"))
        expect(got.get(G[5]['name'], {}).get('sealed') and G[9]['name'] not in got, f'and carried progress lands on the garden it was earned in, not on the one now at its index {sorted(got)}')
        await page.evaluate("window.__samon.share.swap(5, 9)")
        await ctx.close()
        await b.close()
        real = [e for e in errors if 'ERR_TUNNEL' not in e and 'Failed to load resource' not in e]
        expect(not real, f'no page errors {real}')
        print('FAILS:', fails)

asyncio.run(main())

The poems ( poems/ )

poems/originals-brainstorm.md · 62 lines · 4.7 kB 
# Original candidates for three gardens (not in the game)

Written by Claude, 2026-10-04, for the three gardens whose poems are weak and for which no better classical poem turned up.
None is in the game: the design's claim is that every word a player reads is a real poem, so these wait for a decision.
If one goes in, `translation_credit` should say "Original, Claude 2026", `japanese` should be empty (the 原文 button then has nothing to show; the page hides it), and the Presentation paragraph loses "all real".

## Carp (W2 Morning; 7×7 open sand, one ripple stone low on the right; extra)

Current: Bashō, 蛸壺やはかなき夢を夏の月 ("Octopus traps— / fleeting dreams / under the summer moon"): a great poem, a forced pairing (octopus pots for a carp pond, night for a morning world). No classical carp poem could be found and verified: the two Shiki carp verses Astra chose could not be confirmed, and the Buson, Issa and Bashō lists checked have none.

1. Morning light— / the carp sinks, / the ring stays. **(pick)**
2. One ring widening— / the carp that made it / is somewhere else.
3. The carp rises / to nothing: / a ring for its own sake.
4. Under the lily pad / the carp keeps its circle / without moving.
5. Rain over— / the carp's one ring / crosses all the others.
6. Red under green water: / a carp, and the ring / it leaves me.
7. The carp's mouth / opens at the surface— / one O, then none.
8. The whole pond / for one ring: / the carp's morning.
9. Sun on the pond— / the carp's shadow / wider than the carp.
10. Deep down / the carp holds still; / the ring goes on without it.
11. A carp turns / under the one place / the water remembers it.
12. Fed at dawn, / the carp returns / the same ring each day.

Pick: 1. Seven words; the ring outlasts what made it, which is the ripple stone's rule (eight cells raked as one ring) and the game's premise (the footprints go, the raking stays); morning, as the world is.

## Go board (W4 Afternoon; 7×7, eleven pebbles; extra)

Current: Shōtetsu, 白玉かなにぞととへば萩のうへの影はこたへずふるさとの月 ("White jewels— / or what? I ask; / the light upon the bush clover / gives no answer: / moon of my old home"): obscure, and the board is only in the question. No classical poem about go stones turned up.

1. Set down one by one, / the stones already / drew the line. **(pick)**
2. Black stone, white stone: / each knows one thing / and says it.
3. Eleven stones— / the game was decided / before the first was placed.
4. On the board / every stone is a question / the line answers.
5. The board is empty / where it matters; / the stones only point.
6. Pale stones go straight; / dark stones turn; / nothing else speaks.
7. Afternoon— / the stones have no color / until the line passes.
8. Each stone / holds the line / for one move.
9. The master left / and the stones / kept playing.
10. Nobody wins here: / the line / simply closes.
11. Eleven pebbles / and one way / between them.
12. The go stones / have forgotten the game / and remember the path.

Pick: 1. It is how the pebbles were made (chosen one at a time, each cutting the rakings left) and what they do (together they fix the only raking), without saying either.

## Latch (W6 Dusk; 7×7, six pebbles and a side gate; combination)

Current: Rengetsu, むれ立て 人くと見しは 秋風に 尾花が袖の なびく也けり ("I thought a crowd / was coming toward me— / only the sleeves / of pampas grass / waving in the autumn wind"): a stock conceit (grass mistaken for people), tied to the garden only as "expectation at a boundary". The gate poems that would fit are all in use (Issa's moon at the gate, Bashō's tide at the gate and moon as guide, Buson's sweetfish, Buson's leaving by the gate, Ryōkan's thief).

1. Not locked, / only shut: / the side gate at dusk. **(pick)**
2. The latch lifts / from the other side— / nobody there but dusk.
3. Two gates, one latch: / the wind uses / whichever is open.
4. Evening— / a latch I never set / clicks in the wind.
5. I came in / by the small gate; / the big one was only painted.
6. The latch is cold. / Beyond it, / the same garden.
7. Dusk at the side gate: / the latch remembers / a hand.
8. Lift the latch— / the path was waiting / on the inside.
9. The latch drops / behind me; / the sand ahead unmarked.
10. Someone before me / oiled the latch: / it opens without a sound.
11. A small gate, / a small latch, / the whole garden behind it.
12. Dusk— / the latch warm / from the last hand.

Pick: 1. The side gate is an entrance the player must use, not a barrier; the poem says only that, at the world's hour.
poems/pass2_replacements.py · 233 lines · 22.2 kB — Second curation pass (2026-10-04): replace the weakest pairings with better classical poems, each checked against a 
#!/usr/bin/env python3
"""Second curation pass (2026-10-04): replace the weakest pairings with better classical poems, each checked against a
page that prints it. Applied once to poems/samon-poems.json; kept for the record (about.revisions lists every swap)."""
import json

PATH = '/home/claude/zen/poems/samon-poems.json'
ASTRA = '/home/claude/zen/poems/astra-samon-poem-selection.json'
TODAY = '2026-10-04'
D = json.load(open(PATH))
A = json.load(open(ASTRA))
P = D['poems']
ITOYO = 'https://www2.yamanashi-ken.ac.jp/~itoyo/basho/'
BASHO = ('松尾芭蕉', '1644–1694')
CLAUDE = f'New translation by Claude, {TODAY}'


def basho(pid, ja, en, collection, display, url, label, notes, number=None):
    return dict(id=pid, poet='Matsuo Bashō', author_japanese=BASHO[0], author_dates=BASHO[1], form='haiku', japanese=ja,
                translation=en, translation_credit=CLAUDE, collection=collection, poem_number=number, source_url=url,
                source_label=label, display_source=display, notes=notes,
                reverified=f'yes, {TODAY}: Japanese text and attribution checked against source_url')


NEW = {
    'new-basho-sweep-garden': basho('new-basho-sweep-garden', '庭掃て出ばや寺に散柳',
        "Let me sweep the garden\nbefore I go—\nwillow leaves fall in the temple.",
        'おくのほそ道 (Oku no hosomichi), Zenshōji, journey of 1689', 'Oku no hosomichi', ITOYO + 'okunohosomichi/okuno34.htm',
        'Yamanashi Prefectural University, Bashō database: Oku no hosomichi, 曾良との別れ',
        'Leaving the temple where he had lodged, Bashō sweeps its garden as a guest\'s thanks; the young monks had run after '
        'him with paper and inkstone. The 1903 Zenshū prints 精舍 for 寺.'),
    'new-basho-moon-sand': basho('new-basho-moon-sand', '月清し遊行のもてる砂の上',
        "The moon is clear\non the sand\nthe pilgrim monk carried.",
        'おくのほそ道 (Oku no hosomichi), Tsuruga (Kehi shrine), journey of 1689', 'Oku no hosomichi', ITOYO + 'okunohosomichi/okuno38.htm',
        'Yamanashi Prefectural University, Bashō database: Oku no hosomichi, 敦賀',
        '遊行 is the Yugyō Shōnin, head of the Ji sect: the second, Ta\'a, carried sand himself to fill the shrine\'s muddy '
        'grounds, and his successors keep the rite (遊行の砂持ち). The 1903 Zenshū prints 月淸し遊行の持てる砂の上.'),
    'new-basho-hikarido': basho('new-basho-hikarido', '五月雨の降のこしてや光堂',
        "The summer rains—\ndid they spare only this?\nThe Hall of Light.",
        'おくのほそ道 (Oku no hosomichi), Hiraizumi, journey of 1689', 'Oku no hosomichi', ITOYO + 'okunohosomichi/okuno20.htm',
        'Yamanashi Prefectural University, Bashō database: Oku no hosomichi, 平泉',
        'The Konjikidō at Chūsonji, kept whole for five centuries while everything around it fell; 降りのこす is to leave '
        'something unrained-on.'),
    'new-basho-ishiyama': basho('new-basho-ishiyama', '石山の石より白し秋の風',
        "Whiter\nthan the stones of Stone Mountain—\nthe autumn wind.",
        'おくのほそ道 (Oku no hosomichi), Natadera, journey of 1689', 'Oku no hosomichi', ITOYO + 'okunohosomichi/okuno33.htm',
        'Yamanashi Prefectural University, Bashō database: Oku no hosomichi, 那谷',
        'Autumn\'s color is white in the five-phase scheme; the wind outdoes the white rocks of Nata (or of Ishiyama) in it.'),
    'new-basho-one-field': basho('new-basho-one-field', '田一枚植て立去る柳かな',
        "One field planted—\nand I leave\nthe willow.",
        'おくのほそ道 (Oku no hosomichi), the Yugyō willow, journey of 1689', 'Oku no hosomichi', ITOYO + 'okunohosomichi/okuno07.htm',
        'Yamanashi Prefectural University, Bashō database: Oku no hosomichi, 遊行柳',
        'Under Saigyō\'s willow; who plants the field (the women, or Bashō\'s party) is left open, as is who leaves.'),
    'new-basho-hida': basho('new-basho-hida', '涼しさを飛騨の工が指図かな',
        "Coolness—\ndrawn to the plan\nof a Hida carpenter.",
        'Letter to Sugiyama Sanpū, Genroku 7, intercalary fifth month, day 11 (1694)', 'letter of 1694', ITOYO + 'letter/sanpu1.htm',
        'Yamanashi Prefectural University, Bashō database: letter to Sanpū (1)',
        '指図 is an architect\'s drawing or direction; the carpenters of Hida were the proverbial master builders. Bashō '
        'later preferred the variant 涼しさの指図に見ゆる住まゐかな.'),
    'new-basho-sea-darkens': basho('new-basho-sea-darkens', '海くれて鴨のこゑほのかに白し',
        "The sea darkens;\nthe voices of the ducks\nfaintly white.",
        '野ざらし紀行 (Nozarashi kikō), Atsuta, winter 1684', 'Nozarashi kikō', ITOYO + 'nozarasi/nozara23.htm',
        'Yamanashi Prefectural University, Bashō database: Nozarashi kikō, 海邊に日暮して',
        'Headnote 海邊に日暮して. A sound given a color: the source calls it one of his finest verses.'),
    'new-basho-summer-room': basho('new-basho-summer-room', '山も庭も動き入るゝや夏座敷',
        "Mountain and garden\ncome moving in—\nthe summer room.",
        '芭蕉俳句全集 (1903), 夏・夏座敷; composed 1689 at Kurobane', 'Kurobane, 1689', 'https://ja.wikisource.org/wiki/芭蕉俳句全集',
        'Bashō haiku zenshū, edited by Ōtsuka Kōzan (1903), summer section',
        'A greeting verse for his host\'s open summer room at Kurobane, on the Oku no hosomichi journey: the view enters the room.'),
    'new-ryokan-new-pond': dict(id='new-ryokan-new-pond', poet='Ryōkan', author_japanese='良寛', author_dates='1758–1831', form='haiku',
        japanese='新池や蛙とびこむ音もなし', translation="The new pond—\na frog jumps in;\nnot a sound.", translation_credit=CLAUDE,
        collection='Ryōkan zenkushū (ed. Tanikawa Toshirō), as cited by the source', poem_number=None,
        source_url='https://sakuramitih31.livedoor.blog/archives/41933640.html', additional_source_url='https://idea1616.com/ryokan-haiku/',
        source_label='名歌鑑賞: 新池や蛙とびこむ音もなし (blog, citing 谷川敏朗『良寛全句集』)', display_source='',
        notes='Ryōkan\'s answer to Bashō\'s old pond: the new pond has no sound. Attribution is the traditional one; the text '
              'was checked against two secondary pages, not a critical edition.',
        reverified=f'yes, {TODAY}: Japanese text and attribution checked against source_url (secondary sources)'),
    'new-ryokan-thief-moon': dict(id='new-ryokan-thief-moon', poet='Ryōkan', author_japanese='良寛', author_dates='1758–1831', form='haiku',
        japanese='盗人に取り残されし窓の月', translation="The thief\nleft it behind:\nthe moon in the window.", translation_credit=CLAUDE,
        collection='Ryōkan\'s hokku; the anecdote of the thief at his hut', poem_number=None,
        source_url='https://nichi-zen.site/remaining-moon/', additional_source_url='https://www.jomo-news.co.jp/articles/-/5483',
        source_label='日常実践の禅: 盗人に取り残されし窓の月', display_source='',
        notes='The thief found nothing worth taking (one telling has Ryōkan roll over so the man could take his bedding). '
              'Attribution is the traditional one; checked against secondary pages.',
        reverified=f'yes, {TODAY}: Japanese text and attribution checked against source_url (secondary sources)'),
    'new-buson-dew-thorns': dict(id='new-buson-dew-thorns', poet='Yosa Buson', author_japanese='与謝蕪村', author_dates='1716–1784', form='haiku',
        japanese='白露や茨の刺にひとつづつ', translation="White dew—\non each thorn of the bramble,\none drop.", translation_credit=CLAUDE,
        collection='蕪村句集 (Buson kushū), autumn', poem_number=None,
        source_url='https://idea1616.com/buson/', additional_source_url='https://ameblo.jp/mtada99/entry-12474301599.html',
        source_label='与謝蕪村の俳句 102選 (list), and a commentary page', display_source='Buson kushū',
        notes='Checked against two secondary pages (the museum and bonjin lists used elsewhere do not carry it).',
        reverified=f'yes, {TODAY}: Japanese text and attribution checked against source_url (secondary sources)'),
    'new-buson-dry-spring': dict(id='new-buson-dry-spring', poet='Yosa Buson', author_japanese='与謝蕪村', author_dates='1716–1784', form='haiku',
        japanese='柳ちり清水かれ石ところどころ', translation="Willow leaves fallen,\nthe spring run dry,\nstones here and there.", translation_credit=CLAUDE,
        collection='蕪村集 (as indexed by the museum)', poem_number=None,
        source_url='https://www2.city.kurashiki.okayama.jp/musnat/plant/bungakusakuhin/busonsyuu.htm',
        source_label='Kurashiki Museum of Natural History: 蕪村集の植物', display_source='',
        notes='An answer to Saigyō\'s roadside willow and its clear water (Stepping stone): the water gone, the stones left.',
        reverified=f'yes, {TODAY}: Japanese text and attribution checked against source_url'),
    'new-buson-fuji-leaves': dict(id='new-buson-fuji-leaves', poet='Yosa Buson', author_japanese='与謝蕪村', author_dates='1716–1784', form='haiku',
        japanese='不二ひとつうづみ残して若葉かな', translation="Young leaves—\nthey have buried everything\nbut Fuji.", translation_credit=CLAUDE,
        collection='蕪村句集 (Buson kushū), summer', poem_number=None,
        source_url='https://bonjin-ultra.com/buson.htm', source_label='bonjin-ultra: 蕪村の俳句, summer', display_source='Buson kushū',
        notes='うづみ残す: to leave one thing unburied.',
        reverified=f'yes, {TODAY}: Japanese text and attribution checked against source_url'),
    'new-issa-snail': dict(id='new-issa-snail', poet='Kobayashi Issa', author_japanese='小林一茶', author_dates='1763–1828', form='haiku',
        japanese='かたつぶりそろそろ登れ富士の山', translation="Little snail,\nslowly, slowly climb—\nMount Fuji.", translation_credit=CLAUDE,
        collection='一茶発句集 (Issa hokkushū), Bunsei edition (1829, posthumous)', poem_number=None,
        source_url='https://crd.ndl.go.jp/reference/entry/index.php?page=ref_view&id=1000180048',
        source_label='NDL Collaborative Reference Database: the collections that carry the snail verse', display_source='Issa hokkushū (1829)',
        notes='Not in Issa\'s own diaries; first printed two years after his death (the Kaei edition has 不二の山). Text as the '
              'Bunsei edition prints it, per the reference record.',
        reverified=f'yes, {TODAY}: Japanese text and attribution checked against source_url'),
    'new-tomonori-hi33': dict(id='new-tomonori-hi33', poet='Ki no Tomonori', author_japanese='紀友則', author_dates='d. c. 905', form='waka',
        japanese='ひさかたの 光のどけき 春の日に しづ心なく 花の散るらむ',
        translation="In the tranquil light\nof a spring day\nunder the wide sky—\nwhy, with no calm in their hearts,\ndo the blossoms fall?",
        translation_credit=CLAUDE, collection='Ogura Hyakunin Isshu; Kokin Wakashū 84', poem_number=33,
        source_url='https://www.bunkanet.jp/manabi/hyakunin-isshu-karuta/element/33/', source_label='Bunka Net n.d., Hyakunin Isshu 33',
        display_source='Hyakunin isshu 33', notes='しづ心 is a calm, settled heart; the garden Calm scores exactly that.',
        reverified=f'yes, {TODAY}: Japanese text and attribution checked against source_url'),
    'new-manyo-1068': dict(id='new-manyo-1068', poet='Kakinomoto no Hitomaro', author_japanese='柿本人麻呂歌集', author_dates='fl. c. 680–700', form='waka',
        japanese='天の海に雲の波立ち月の船星の林に榜ぎ隠る見ゆ',
        translation="On the sea of heaven\nthe waves of cloud rise;\nthe boat of the moon\nrows into the forest of stars\nand is hidden from sight.",
        translation_credit=CLAUDE, collection='Man\'yōshū, book 7 (from the Hitomaro Collection)', poem_number=1068,
        source_url='https://www.asahi-net.or.jp/~sg2h-ymst/manyok/manyok07.html',
        source_label='訓読万葉集 巻7 (Yamato Uta), after Kamochi Masazumi\'s 萬葉集古義', display_source='Man\'yōshū 1068',
        notes='Headed 天を詠める and marked as from the Kakinomoto no Hitomaro Collection, whose poems are not all his; the '
              'source prints 榜ぎ for 漕ぎ.',
        reverified=f'yes, {TODAY}: Japanese text and attribution checked against source_url'),
}
# Shikishi's bamboo window, from Astra's unused candidates (its page was checked today)
sk = dict(A['poems']['waka-other-11'])
sk.update(author_japanese='式子内親王', author_dates='d. 1201', poem_number=256, collection='Shin Kokin Wakashū',
          display_source='Shinkokinshū 256',
          translation_credit=f'Working translation by GPT-6 Astra (2026); reviewed by Claude, {TODAY}',
          reverified=f'yes, {TODAY}: Japanese text and attribution checked against source_url (新古256, headnote 百首の歌たてまつりし時)')
NEW['waka-other-11'] = sk

SWAPS = [  # garden, new poem id, why
    ('The corridor', 'new-basho-sweep-garden',
     'The first poem of the game is the chore itself: a guest sweeps the temple garden before going on. Rengetsu\'s dream-path (footprints that vanish) was good, but this is the premise.'),
    ('Bamboo', 'waka-other-11',
     'Rengetsu\'s young bamboo was a felicitation (count its lucky joints, a thousand years): polite and empty. Shikishi\'s bamboo rustles at a window and cuts a doze short.'),
    ('Calm', 'new-tomonori-hi33',
     'The dust-of-the-heart swept by mountain wind is a cliché. Tomonori\'s tranquil spring light names the garden\'s score, a calm heart (しづ心), by asking why the petals lack it.'),
    ('Plain stone', 'new-ryokan-new-pond',
     'The grass hut was a forced pairing. Ryōkan\'s new pond, where the frog makes no sound, is the garden before the ripple stone; Bashō\'s old pond follows with the stone and its ring.'),
    ('Tatami', 'new-basho-summer-room',
     'Rengetsu\'s mat of flowers is a conceit. Bashō\'s mountain and garden move into the summer room: the garden becomes a floor to sit on.'),
    ('The master\'s grooves', 'new-basho-hida',
     'Shōtetsu\'s "learn the shape of the sixth-month wind" is didactic and obscure in English. Bashō\'s coolness drawn to a master carpenter\'s plan is the abbot\'s grooves exactly: a drawing laid down by a better hand.'),
    ('The finished ring', 'new-basho-hikarido',
     'Rengetsu\'s sacred rope was serviceable. The Hall of Light that the summer rains left untouched is the one thing spared when everything around it was washed over, which is what the finished ring asks of the rake.'),
    ('His pebbles', 'new-buson-dew-thorns',
     'Eifukumon\'in\'s dew overflowing the grass is diffuse. Buson\'s one drop on each thorn is a mark per point: the pebbles.'),
    ('Closed circle', 'new-buson-fuji-leaves',
     'Rengetsu\'s bridge of blossoms, not to be stepped on, was apt but minor. Young leaves that bury everything but Fuji leave one shape standing in a covered field, as the raking leaves the ring.'),
    ('Pivot', 'new-basho-ishiyama',
     'Shōtetsu\'s mountain that looks nearer after rain had no hold on the garden. The wide rake can turn only against stones; Bashō\'s autumn wind, whiter than the white stones it blows across, is the stroke among its pivots.'),
    ('Coil', 'new-issa-snail',
     'Shōtetsu\'s insect voices thinner than thread were there for atmosphere. The snail carries the coil on its back and climbs Fuji at the pace the wide rake imposes.'),
    ('Long rake', 'new-basho-one-field',
     'Shōtetsu\'s ox at dusk was filler. One whole field planted, then the planters move on: a long single labor, finished and left, as each garden is.'),
    ('Turning stone', 'new-buson-dry-spring',
     'Shōtetsu\'s settling snow was fine but minor. Buson\'s dried-up spring with stones here and there is a dry landscape, the garden\'s own genre, and answers Saigyō\'s willow and clear water earlier in the campaign.'),
    ('Bolt', 'new-ryokan-thief-moon',
     'Teika\'s tied hunting robe was fastened to the title by a pun. The thief who takes everything but the moon in the window is the poem about what a bolt cannot keep in or out.'),
    ('Stargazer', 'new-manyo-1068',
     'Shōtetsu\'s moon in a green spring sky has no stars. The Man\'yōshū moon-boat rowing into the forest of stars is the oldest and best star poem in the language, and the ripple stone is its one boat.'),
    ('Moonlit sea', 'new-basho-sea-darkens',
     'Shōtetsu\'s wind seen in lifted mist was abstract. Bashō\'s sea darkening while the ducks\' voices turn faintly white is a night shore heard rather than seen.'),
    ('Owl\'s hour', 'teika-21',
     'Teika\'s looking back over everything and asking the moon to light the road ahead moves here from the capstone: a sleepless hour before dawn is where that poem happens.'),
    ('Silver sand', 'new-basho-moon-sand',
     'The capstone of the main path now gets the one classical poem about raked temple sand: the moon clear on the sand a pilgrim monk carried. Teika\'s farewell poem, good as it was here, goes to Owl\'s hour.'),
]
removed = {}
for garden, pid, why in SWAPS:
    old = D['gardens'][garden]
    removed[old] = P[old]
    D['gardens'][garden] = pid
    if pid in NEW:
        P[pid] = NEW[pid]
# pairings: the positive half only (the comparison belongs in the revision log)
POS = {
    'The corridor': 'The first poem of the game is the chore itself: a guest sweeps the temple garden before going on, as the player rakes each garden and moves on.',
    'Bamboo': 'Bamboo leaves worried by wind at a window, cutting a doze short.',
    'Calm': 'Tranquil spring light, and petals that fall without a calm heart (しづ心): the quality the garden scores, named by its absence.',
    'Plain stone': 'Ryōkan\'s new pond, where the frog makes no sound: the garden before the ripple stone. Bashō\'s old pond follows, with the stone and its ring.',
    'Tatami': 'Mountain and garden move into the summer room: the garden becomes a floor to sit on.',
    'The master\'s grooves': 'Coolness drawn to a master carpenter\'s plan: the abbot\'s grooves are a drawing laid down by a better hand, which the rake follows.',
    'The finished ring': 'The Hall of Light that the summer rains left untouched: the one thing spared when everything around it is washed over, which is what the finished ring asks of the rake.',
    'His pebbles': 'One drop of dew on each thorn: a mark per point, as the abbot\'s grooves become pebbles.',
    'Closed circle': 'Young leaves bury everything but Fuji: one shape left standing in a covered field, as the raking leaves the ring.',
    'Pivot': 'The wide rake turns only against stones; Bashō\'s autumn wind, whiter than the white stones it blows across, is the stroke among its pivots.',
    'Coil': 'The snail carries its coil on its back and climbs Fuji at the pace the wide rake imposes.',
    'Long rake': 'One whole field planted, then the planters move on: a long single labor, finished and left, as each garden is.',
    'Turning stone': 'A spring run dry, willow leaves down, stones here and there: a dry landscape, the garden\'s own genre, answering Saigyō\'s willow and clear water (Stepping stone).',
    'Bolt': 'The thief takes everything but the moon in the window: what a bolt cannot keep in or out.',
    'Stargazer': 'The moon-boat rows into the forest of stars and is hidden: the ripple stone is its one boat on a field of sand.',
    'Moonlit sea': 'The sea darkens while the ducks\' voices turn faintly white: a night shore heard rather than seen; the band is its edge.',
    'Owl\'s hour': 'Sleepless before dawn, everything behind rises as images, and the moon is asked to light the road ahead.',
    'Silver sand': 'The last garden of the main path: the moon clear on the sand a pilgrim monk carried, the one classical poem about temple sand hauled and tended by monks.',
}
for g, why in POS.items():
    D['pairings'][g] = why
for old in removed:
    if old not in D['gardens'].values() and old != D['practice']:
        del P[old]
assert 'teika-21' in P and D['gardens']['Owl\'s hour'] == 'teika-21'

used = list(D['gardens'].values()) + [D['practice']]
assert len(used) == len(set(used)) == len(P) == 149, (len(used), len(set(used)), len(P))
for pid, p in P.items():
    assert p['translation'] and p['japanese'] and p['reverified'].startswith('yes') and p.get('author_dates'), pid

D['about']['revisions'] = [
    {'date': TODAY, 'pass': 2, 'garden': g, 'from': old_id, 'from_text': removed[old_id]['japanese'], 'to': pid, 'why': why}
    for (g, pid, why), old_id in zip(SWAPS, [r for r in removed])
]
# (the zip above pairs swaps with removed ids in order; check it)
for rec, (g, pid, why) in zip(D['about']['revisions'], SWAPS):
    assert rec['garden'] == g and rec['to'] == pid
D['about']['selection'] += (f' Second pass, {TODAY}: 18 pairings changed for better classical poems (16 poems new to the '
                            'selection, 1 from Astra\'s unused candidates, 1 moved); see revisions.')
D['about']['translations'] = ('English versions are working translations, not quotations of published ones: GPT-6 Astra\'s, '
                              'reviewed by Claude, with ' + str(sum('revised' in p['translation_credit'] for p in P.values())) +
                              ' revised and ' + str(sum('New translation' in p['translation_credit'] for p in P.values())) +
                              ' new (translation_credit).')
json.dump(D, open(PATH, 'w'), ensure_ascii=False, indent=1)
import collections
pc = collections.Counter(P[i]['poet'] for i in used)
print(len(pc), pc.most_common())
print('revised', sum('revised' in p['translation_credit'] for p in P.values()), 'new', sum('New translation' in p['translation_credit'] for p in P.values()))
poems/pass3_astra_review.py · 141 lines · 12.0 kB — Third pass (2026-10-04): GPT-6 Astra reviewed the selection and proposed 18 changes (10 classical substitutions, 
#!/usr/bin/env python3
"""Third pass (2026-10-04): GPT-6 Astra reviewed the selection and proposed 18 changes (10 classical substitutions,
2 originals, 6 translation repairs). Taken: 7 substitutions (one redirected to another garden) and 4 translation repairs;
Shikishi's bamboo was already in. Left out: the two originals (every word the player reads stays a real poem), a second
Rengetsu for Calm (Tomonori's stays), the kite for Wide blade (it already introduces the grooves), and two repairs of
poems the second pass had replaced. Applied once; logged in about.revisions."""
import json

PATH = '/home/claude/zen/poems/samon-poems.json'
ASTRA = '/home/claude/zen/poems/astra-samon-poem-selection.json'
TODAY = '2026-10-04'
D = json.load(open(PATH))
A = json.load(open(ASTRA))
P = D['poems']
ASTRA_TR = f'Working translation by GPT-6 Astra (2026, review pass); reviewed by Claude, {TODAY}'
CHECKED = f'yes, {TODAY}: Japanese text and attribution checked against source_url'

NEW = {
    'new-shiki-bridge-fish': dict(id='new-shiki-bridge-fish', poet='Masaoka Shiki', author_japanese='正岡子規', author_dates='1867–1902',
        composed=1895, form='haiku', japanese='橋蹈めば魚沈みけり春の水', translation="I step on the bridge—\nthe fish sink.\nSpring water.",
        translation_credit=f'Working translation by GPT-6 Astra (2026, review pass), revised by Claude, {TODAY}',
        collection='獺祭書屋俳句帖抄 (Dassai shooku haiku chō shō), spring 1895', poem_number=None,
        source_url='https://mukei-r.net/poem-masaoka/00-haiku-dassai1.htm', source_label='獺祭書屋俳句帖抄 (transcription), 明治28年 春',
        display_source='1895', notes='The source prints 沈みけり (Astra\'s draft had the fish sink "deeper"). Composed 1895; the '
        'poet died in 1902, so this is the one poem whose author outlived 1900.', reverified=CHECKED),
    'new-fushimi-lotus-dew': dict(id='new-fushimi-lotus-dew', poet='Emperor Fushimi', author_japanese='伏見院', author_dates='1265–1317', form='waka',
        japanese='こぼれ落つる池のはちすの白露は浮葉の玉とまたなりにけり',
        translation="Dew spills\nfrom the pond\'s lotus leaves;\non the floating leaves\nit becomes\na jewel again.",
        translation_credit=f'Working translation by GPT-6 Astra (2026, review pass), revised by Claude, {TODAY}',
        collection='Gyokuyō Wakashū (headnote: 百首御歌の中に、蓮を)', poem_number=None,
        source_url='https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/saijiki/hasunoha.html', source_label='和歌歳時記: 蓮葉 (Yamato Uta)',
        display_source='Gyokuyōshū', notes='Lotus, not water lily; the number in the Gyokuyōshū is not given by the source.', reverified=CHECKED),
    'new-tamemasa-carp-lotus': dict(id='new-tamemasa-carp-lotus', poet='Reizei Tamemasa', author_japanese='冷泉為尹', author_dates='1361–1417', form='waka',
        japanese='池水に藻臥しの鮒や乱るらん蓮のうき葉のゆるぎ立ちぬる',
        translation="Hidden in the pondweed,\nare the carp stirring?\nThe lotus leaves\nfloating on the water\nhave begun to tremble.",
        translation_credit=ASTRA_TR, collection='Tamemasa senshu (為尹千首), topic: pond lotus', poem_number=None,
        source_url='https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/saijiki/hasunoha.html', source_label='和歌歳時記: 蓮葉 (Yamato Uta)',
        display_source='Tamemasa senshu', notes='鮒 is the crucian carp. Astra credited the poem to Reizei Tamehide; the source gives 為尹 '
        '(Tamemasa, Tamehide\'s grandson).', reverified=CHECKED),
    'new-buson-dancers': dict(id='new-buson-dancers', poet='Yosa Buson', author_japanese='与謝蕪村', author_dates='1716–1784', form='haiku',
        japanese='四五人に月落かかるおどりかな', translation="Four or five dancers—\nthe moon\nfalls toward them.", translation_credit=ASTRA_TR,
        collection='Inscribed on his painting 踊図 (Dancers; Agency for Cultural Affairs)', poem_number=None,
        source_url='https://www.kyuhaku.jp/exhibition/exhibition_s50.html', source_label='Kyushu National Museum: 王羲之と日本の書 (exhibit list)',
        display_source='on his painting 踊図', notes='The Bon dance; 月落ちかかる is the moon sinking low over the dancers.', reverified=CHECKED),
    'new-basho-moon-clouds': dict(id='new-basho-moon-clouds', poet='Matsuo Bashō', author_japanese='松尾芭蕉', author_dates='1644–1694', form='haiku',
        japanese='雲折々人をやすむる月見かな', translation="Watching the moon—\nevery so often a cloud\nlets us rest.", translation_credit=ASTRA_TR,
        collection='春の日 (Haru no hi), 1686', poem_number=None,
        source_url='https://www2.yamanashi-ken.ac.jp/~itoyo/basho/shitibusyu/harunohi.htm',
        source_label='Yamanashi Prefectural University, Bashō database: 俳諧七部集, 春の日', display_source='Haru no hi (1686)',
        notes='', reverified=CHECKED),
}
# Saigyō's snipe, from Astra's unused candidates (its page was checked today: 新古362, headnote 秋、ものへまかりける道にて)
sg = dict(A['poems']['waka-other-05'])
sg.update(author_japanese='西行', author_dates='1118–1190', collection='Shin Kokin Wakashū', poem_number=362, display_source='Shinkokinshū 362',
          translation="Even one like me,\nwho has set his heart aside,\nfeels it after all—\na snipe lifts from the marsh\nat autumn dusk.",
          translation_credit=ASTRA_TR, reverified=CHECKED)
NEW['waka-other-05'] = sg
# Bashō's grass hut, back from the first pass (checked then against basho-yamadera.com)
gh = {
    "id": "new-basho-grass-hut",
    "poet": "Matsuo Bashō",
    "author_japanese": "松尾芭蕉",
    "author_dates": "1644–1694",
    "form": "haiku",
    "japanese": "草の戸も住替る代ぞひなの家",
    "translation": "Even the grass hut\npasses to new dwellers—\na house of dolls.",
    "collection": "Oku no hosomichi (opening hokku)",
    "poem_number": null,
    "source_url": "https://basho-yamadera.com/en/yamadera/kusanotomo",
    "imagery": [
        "grass hut",
        "new occupants",
        "dolls",
        "change"
    ],
    "notes": "",
    "verification": "",
    "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
    "translation_credit": "New translation by Claude, 2026-10-04",
    "display_source": "Oku no hosomichi"
}
gh['translation'] = "Even this grass hut\nchanges hands—\na house of dolls."
gh['translation_credit'] = f'Working translation by GPT-6 Astra (2026, review pass), revised by Claude, {TODAY}'
NEW['new-basho-grass-hut'] = gh

SWAPS = [
    ('Koi', 'new-shiki-bridge-fish', 'A step on the bridge, and the fish go down: the walk in changes the pond under it. Astra\'s choice; the whitebait only glittered.'),
    ('Water lily', 'new-fushimi-lotus-dew', 'Dew spills from the lotus leaf and gathers into a jewel again on the leaf below: lost, then re-formed, as the rings around the stones are. Astra\'s choice over Buson\'s two pond lilies, which only named the plants.'),
    ('Carp', 'new-tamemasa-carp-lotus', 'The carp are never seen; the trembling lotus leaves prove them, as the ring proves the stone. Astra found the one classical carp poem we lacked; it replaces the octopus traps, a great poem forced onto a fish garden.'),
    ('Two doors', 'new-basho-grass-hut', 'One hut, two households through its door: the hermit leaves, a family with its dolls arrives. Astra\'s choice over the inn-sign verse "come in this way".'),
    ('Threshold', 'waka-other-05', 'Renunciation that cannot keep out the snipe lifting from the marsh: a boundary crossed by feeling. Astra\'s choice over Jien\'s announcement that the true path leaves nothing to long for. With Jakuren\'s Dusk and Teika\'s Plain speaking, all three of the Three Evenings are now in the game.'),
    ('Lantern festival', 'new-buson-dancers', 'Four or five Bon dancers with the moon sinking onto them: the ring of the dance for the garden\'s ring, and the sky brought close. Astra\'s choice over Rengetsu\'s farewell fires, whose "how it moves me" deflated the scene.'),
    ('Moonrise', 'new-basho-moon-clouds', 'Moon-viewing with its rests: the clouds that interrupt the looking are kind. Astra proposed it for Stargazer, which keeps the Man\'yōshū moon-boat; it replaces Shōtetsu\'s moon over the petal storm here.'),
]
REPAIRS = {   # Astra's better translations of poems that stay
    'teika-22': "I watch the lamp\nas the window grows light.\nNo one asks\neven whether\nI am still here.",
    'waka-other-26': "If I pick one,\nit must be by guess—\nwhite chrysanthemums\nlost in\nthe first frost.",
    'SHOT-021': "Even the clouds\nfalter through the dusk—\na cliffside walkway\nunder snow,\nnot a footprint.",
    'waka-other-04': "My heart grows clear,\nclearer still,\nendlessly in the moonlight.\nWhat will it become\nat last?",
}
removed = []
for garden, pid, why in SWAPS:
    old = D['gardens'][garden]
    removed.append((old, P[old]['japanese']))
    D['gardens'][garden] = pid
    P[pid] = NEW[pid]
    D['pairings'][garden] = why
for pid, t in REPAIRS.items():
    P[pid]['translation'] = t
    P[pid]['translation_credit'] = f'Working translation by GPT-6 Astra (2026), revised by GPT-6 Astra (review pass) and Claude, {TODAY}'
used = list(D['gardens'].values()) + [D['practice']]
for old, _ in removed:
    if old not in used:
        del P[old]
assert len(used) == len(set(used)) == len(P) == 149, (len(used), len(set(used)), len(P))
for pid, p in P.items():
    assert p['translation'] and p['japanese'] and p['reverified'].startswith('yes') and p.get('author_dates'), pid

D['about']['revisions'] += [
    {'date': TODAY, 'pass': 3, 'garden': g, 'from': old, 'from_text': text, 'to': pid, 'why': why}
    for (g, pid, why), (old, text) in zip(SWAPS, removed)]
D['about']['revisions'] += [
    {'date': TODAY, 'pass': 3, 'garden': g, 'translation_repair': pid, 'why': 'Astra\'s revised translation; the poem stays.'}
    for pid, g in (('teika-22', 'Lantern'), ('waka-other-26', 'White stones'), ('SHOT-021', 'Hairpin'), ('waka-other-04', 'Spiral'))]
D['about']['selection'] += (f' Third pass, {TODAY}: GPT-6 Astra reviewed the selection; of its 18 proposals, 7 substitutions '
                            '(one moved to another garden) and 4 translation repairs were taken; its two original poems were not '
                            '(every word the player reads is a real poem).')
D['about']['translations'] = ('English versions are working translations, not quotations of published ones: GPT-6 Astra\'s, '
                              'reviewed by Claude, with ' + str(sum('revised' in p['translation_credit'] for p in P.values())) +
                              ' revised and ' + str(sum('New translation' in p['translation_credit'] for p in P.values())) +
                              ' new (translation_credit).')
D['about']['texts'] = ('Every Japanese text and attribution was re-checked on ' + TODAY + ' against a page that prints it '
                       '(source_url; reverified records the check). Every poem was written before 1900 (one, Shiki\'s, in 1895, '
                       'by a poet who died in 1902); the originals are public domain.')
json.dump(D, open(PATH, 'w'), ensure_ascii=False, indent=1)
import collections
pc = collections.Counter(P[i]['poet'] for i in used)
print(len(pc), pc.most_common())
print('revised', sum('revised' in p['translation_credit'] for p in P.values()), 'new', sum('New translation' in p['translation_credit'] for p in P.values()))
anames = {l['name']: l['poem_id'] for l in A['levels']}
print('new to selection', sum(i not in A['poems'] for i in used), 'changed since Astra', sum(g in anames and anames[g] != D['gardens'][g] for g in D['gardens']), 'revisions', len(D['about']['revisions']))
poems/samon-poems.json · 3,968 lines · 198.8 kB 
{
 "about": {
  "title": "Samon: poems for 148 gardens and the practice garden",
  "selection": "Poems chosen and paired with the 136 gardens of the earlier campaign by GPT-6 Astra (2026; its file is astra-samon-poem-selection.json, with 41 unused candidates). Carried over to the 148-garden campaign, where garden names follow layouts, and curated by Claude, 2026-10-04: 13 new gardens paired (3 poems new to the selection, all by Bashō), 3 pairings changed (2 carp poems attributed to Shiki whose texts could not be confirmed replaced by verified Bashō poems; Bashō’s old pond moved to the first ripple stone). Second pass, 2026-10-04: 18 pairings changed for better classical poems (16 poems new to the selection, 1 from Astra's unused candidates, 1 moved); see revisions. Third pass, 2026-10-04: GPT-6 Astra reviewed the selection; of its 18 proposals, 7 substitutions (one moved to another garden) and 4 translation repairs were taken; its two original poems were not (every word the player reads is a real poem).",
  "texts": "Every Japanese text and attribution was re-checked on 2026-10-04 against a page that prints it (source_url; reverified records the check). Every poem was written before 1900 (one, Shiki's, in 1895, by a poet who died in 1902); the originals are public domain.",
  "translations": "English versions are working translations, not quotations of published ones: GPT-6 Astra's, reviewed by Claude, with 25 revised and 17 new (translation_credit).",
  "fields": "gardens: garden name → poem id; practice: poem id; pairings: garden name → why that poem; poems: the texts, sources and notes; display_source: the short source shown on the poem card.",
  "revisions": [
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "The corridor",
    "from": "R420",
    "from_text": "ふみわけて ゆけど夢ぢは あともなし 思ひねにみし 花のしら雪",
    "to": "new-basho-sweep-garden",
    "why": "The first poem of the game is the chore itself: a guest sweeps the temple garden before going on. Rengetsu's dream-path (footprints that vanish) was good, but this is the premise."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "Bamboo",
    "from": "R115",
    "from_text": "日にそひて めでたきふしや かぞふらん 千代をこめたる 宿の若竹",
    "to": "waka-other-11",
    "why": "Rengetsu's young bamboo was a felicitation (count its lucky joints, a thousand years): polite and empty. Shikishi's bamboo rustles at a window and cuts a doze short."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "Calm",
    "from": "R670",
    "from_text": "しづかなる ねざめのとこに 音づれて 心のちりを はらふやま風",
    "to": "new-tomonori-hi33",
    "why": "The dust-of-the-heart swept by mountain wind is a cliché. Tomonori's tranquil spring light names the garden's score, a calm heart (しづ心), by asking why the petals lack it."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "Plain stone",
    "from": "new-basho-grass-hut",
    "from_text": "草の戸も住替る代ぞひなの家",
    "to": "new-ryokan-new-pond",
    "why": "The grass hut was a forced pairing. Ryōkan's new pond, where the frog makes no sound, is the garden before the ripple stone; Bashō's old pond follows with the stone and its ring."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "Tatami",
    "from": "R121",
    "from_text": "しくもをし まくもをしきを 秋草の 花のむしろは 野べながら見ん",
    "to": "new-basho-summer-room",
    "why": "Rengetsu's mat of flowers is a conceit. Bashō's mountain and garden move into the summer room: the garden becomes a floor to sit on."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "The master's grooves",
    "from": "SHOT-009",
    "from_text": "学びえよこひねがはしきみな月の風のすがたを大和ことのは",
    "to": "new-basho-hida",
    "why": "Shōtetsu's \"learn the shape of the sixth-month wind\" is didactic and obscure in English. Bashō's coolness drawn to a master carpenter's plan is the abbot's grooves exactly: a drawing laid down by a better hand."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "The finished ring",
    "from": "R133",
    "from_text": "松風は 秋をしらべて 野宮の しめのうちなる すずむしのこゑ",
    "to": "new-basho-hikarido",
    "why": "Rengetsu's sacred rope was serviceable. The Hall of Light that the summer rains left untouched is the one thing spared when everything around it was washed over, which is what the finished ring asks of the rake."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "His pebbles",
    "from": "waka-other-16",
    "from_text": "うす霧のはるる朝けの庭みれば草にあまれる秋の白露",
    "to": "new-buson-dew-thorns",
    "why": "Eifukumon'in's dew overflowing the grass is diffuse. Buson's one drop on each thorn is a mark per point: the pebbles."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "Closed circle",
    "from": "R405",
    "from_text": "おりたちて わたらばたえん 山がはに かぜのかけたる 花の浮橋",
    "to": "new-buson-fuji-leaves",
    "why": "Rengetsu's bridge of blossoms, not to be stepped on, was apt but minor. Young leaves that bury everything but Fuji leave one shape standing in a covered field, as the raking leaves the ring."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "Pivot",
    "from": "SHOT-007",
    "from_text": "草も木もぬれて色こき山なれや見しより近き夕立のあと",
    "to": "new-basho-ishiyama",
    "why": "Shōtetsu's mountain that looks nearer after rain had no hold on the garden. The wide rake can turn only against stones; Bashō's autumn wind, whiter than the white stones it blows across, is the stroke among its pivots."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "Coil",
    "from": "SHOT-011",
    "from_text": "露霜にあへずかれ行く秋草の糸よりよわき虫のこゑかな",
    "to": "new-issa-snail",
    "why": "Shōtetsu's insect voices thinner than thread were there for atmosphere. The snail carries the coil on its back and climbs Fuji at the pace the wide rake imposes."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "Long rake",
    "from": "SHOT-003",
    "from_text": "夕まぐれ野がひの牛は歩みきて霞める道に逢ふ人もなし",
    "to": "new-basho-one-field",
    "why": "Shōtetsu's ox at dusk was filler. One whole field planted, then the planters move on: a long single labor, finished and left, as each garden is."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "Turning stone",
    "from": "SHOT-020",
    "from_text": "風まぜにあらく落ちしはしづまりてこまかにつもる庭の白雪",
    "to": "new-buson-dry-spring",
    "why": "Shōtetsu's settling snow was fine but minor. Buson's dried-up spring with stones here and there is a dry landscape, the garden's own genre, and answers Saigyō's willow and clear water earlier in the campaign."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "Bolt",
    "from": "teika-10",
    "from_text": "いづくにかこよひはやどをかり衣ひもゆふぐれのみねのあらしに",
    "to": "new-ryokan-thief-moon",
    "why": "Teika's tied hunting robe was fastened to the title by a pun. The thief who takes everything but the moon in the window is the poem about what a bolt cannot keep in or out."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "Stargazer",
    "from": "SHOT-002",
    "from_text": "天の原春のみどりの色にすむ月の光はかすむともなし",
    "to": "new-manyo-1068",
    "why": "Shōtetsu's moon in a green spring sky has no stars. The Man'yōshū moon-boat rowing into the forest of stars is the oldest and best star poem in the language, and the ripple stone is its one boat."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "Moonlit sea",
    "from": "SHOT-014",
    "from_text": "にほの海の霧ふきたつる程ばかり月に見えたる秋のしほかぜ",
    "to": "new-basho-sea-darkens",
    "why": "Shōtetsu's wind seen in lifted mist was abstract. Bashō's sea darkening while the ducks' voices turn faintly white is a night shore heard rather than seen."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "Owl's hour",
    "from": "SHOT-017",
    "from_text": "まどろまでさ夜もはるかに竹のはの霜にさえたる風の音かな",
    "to": "teika-21",
    "why": "Teika's looking back over everything and asking the moon to light the road ahead moves here from the capstone: a sleepless hour before dawn is where that poem happens."
   },
   {
    "date": "2026-10-04",
    "pass": 2,
    "garden": "Silver sand",
    "from": "teika-21",
    "from_text": "こしかたはみな面影にうかびきぬ行末てらせ秋の夜の月",
    "to": "new-basho-moon-sand",
    "why": "The capstone of the main path now gets the one classical poem about raked temple sand: the moon clear on the sand a pilgrim monk carried. Teika's farewell poem, good as it was here, goes to Owl's hour."
   },
   {
    "date": "2026-10-04",
    "pass": 3,
    "garden": "Koi",
    "from": "day-basho-dawn-whitebait",
    "from_text": "明ぼのやしら魚しろきこと一寸",
    "to": "new-shiki-bridge-fish",
    "why": "A step on the bridge, and the fish go down: the walk in changes the pond under it. Astra's choice; the whitebait only glittered."
   },
   {
    "date": "2026-10-04",
    "pass": 3,
    "garden": "Water lily",
    "from": "supp-buson-pond-lilies-rain",
    "from_text": "河骨の二もとさくや雨の中",
    "to": "new-fushimi-lotus-dew",
    "why": "Dew spills from the lotus leaf and gathers into a jewel again on the leaf below: lost, then re-formed, as the rings around the stones are. Astra's choice over Buson's two pond lilies, which only named the plants."
   },
   {
    "date": "2026-10-04",
    "pass": 3,
    "garden": "Carp",
    "from": "new-basho-octopus-pots",
    "from_text": "蛸壺やはかなき夢を夏の月",
    "to": "new-tamemasa-carp-lotus",
    "why": "The carp are never seen; the trembling lotus leaves prove them, as the ring proves the stone. Astra found the one classical carp poem we lacked; it replaces the octopus traps, a great poem forced onto a fish garden."
   },
   {
    "date": "2026-10-04",
    "pass": 3,
    "garden": "Two doors",
    "from": "hn07",
    "from_text": "月ぞしるべこなたへ入せ旅の宿",
    "to": "new-basho-grass-hut",
    "why": "One hut, two households through its door: the hermit leaves, a family with its dolls arrives. Astra's choice over the inn-sign verse \"come in this way\"."
   },
   {
    "date": "2026-10-04",
    "pass": 3,
    "garden": "Threshold",
    "from": "waka-other-24",
    "from_text": "さとりゆくまことの道に入りぬれば恋しかるべき故郷もなし",
    "to": "waka-other-05",
    "why": "Renunciation that cannot keep out the snipe lifting from the marsh: a boundary crossed by feeling. Astra's choice over Jien's announcement that the true path leaves nothing to long for. With Jakuren's Dusk and Teika's Plain speaking, all three of the Three Evenings are now in the game."
   },
   {
    "date": "2026-10-04",
    "pass": 3,
    "garden": "Lantern festival",
    "from": "R122",
    "from_text": "おくり火の ほかげしらみて かも川の ぼにの月夜ぞ あはれ也ける",
    "to": "new-buson-dancers",
    "why": "Four or five Bon dancers with the moon sinking onto them: the ring of the dance for the garden's ring, and the sky brought close. Astra's choice over Rengetsu's farewell fires, whose \"how it moves me\" deflated the scene."
   },
   {
    "date": "2026-10-04",
    "pass": 3,
    "garden": "Moonrise",
    "from": "SHOT-006",
    "from_text": "桜花散りかひかくす高嶺より嵐を越えて出づる月かげ",
    "to": "new-basho-moon-clouds",
    "why": "Moon-viewing with its rests: the clouds that interrupt the looking are kind. Astra proposed it for Stargazer, which keeps the Man'yōshū moon-boat; it replaces Shōtetsu's moon over the petal storm here."
   },
   {
    "date": "2026-10-04",
    "pass": 3,
    "garden": "Lantern",
    "translation_repair": "teika-22",
    "why": "Astra's revised translation; the poem stays."
   },
   {
    "date": "2026-10-04",
    "pass": 3,
    "garden": "White stones",
    "translation_repair": "waka-other-26",
    "why": "Astra's revised translation; the poem stays."
   },
   {
    "date": "2026-10-04",
    "pass": 3,
    "garden": "Hairpin",
    "translation_repair": "SHOT-021",
    "why": "Astra's revised translation; the poem stays."
   },
   {
    "date": "2026-10-04",
    "pass": 3,
    "garden": "Spiral",
    "translation_repair": "waka-other-04",
    "why": "Astra's revised translation; the poem stays."
   }
  ]
 },
 "gardens": {
  "The corridor": "new-basho-sweep-garden",
  "Nine squares": "day-issa-frog-mountain",
  "Morning dew": "day-basho-melon-dew",
  "The pocket": "hn24",
  "Two tones": "R126",
  "Bamboo": "waka-other-11",
  "Sunrise": "day-basho-plum-sunrise",
  "First stroke": "hn30",
  "Calm": "new-tomonori-hi33",
  "Mist": "day-basho-nameless-mountain",
  "First bell": "R050",
  "Sparrow": "day-issa-orphan-sparrow",
  "Cold sand": "teika-14",
  "Lantern": "teika-22",
  "The long wall": "day-buson-rain-inhabited-house",
  "Wind chime": "SHOT-004",
  "Pine shadow": "teika-19",
  "Heron's step": "teika-20",
  "Quiet": "SHOT-005",
  "Plain stone": "new-ryokan-new-pond",
  "First ripple": "day-basho-old-pond",
  "Koi": "new-shiki-bridge-fish",
  "Water lily": "new-fushimi-lotus-dew",
  "Ripples": "day-basho-ripples-keeping-time",
  "Spring": "waka-other-01",
  "Twin rings": "day-buson-two-houses",
  "Two moons": "teika-11",
  "Frog": "day-issa-proud-frog",
  "Paired wells": "R102",
  "Raindrop": "day-basho-rain-wasp-nest",
  "Bell's echo": "day-buson-bell-voice",
  "Turtle": "day-issa-frog-on-turtle",
  "Stepping stone": "waka-other-03",
  "Pond edge": "R111",
  "Drip": "waka-other-10",
  "Carp": "new-tamemasa-carp-lotus",
  "Lotus": "supp-buson-lotus-fragrance",
  "Still water": "day-basho-cuckoo-over-water",
  "Well": "day-chiyoni-morning-glory-water",
  "Calm water": "new-basho-kiyotaki-moon",
  "First wave": "day-basho-green-needles-water",
  "High tide": "hn04",
  "Sandbar": "hn17",
  "Waves": "teika-15",
  "Swell": "day-buson-spring-sea",
  "Ichimatsu": "hyakunin-17",
  "Tatami": "new-basho-summer-room",
  "Noon sun": "day-basho-sunlit-leaves",
  "Estuary": "day-basho-swift-mogami",
  "Island in the current": "supp-basho-hot-day-sea",
  "Breakers": "supp-basho-evening-wave-flowers",
  "Current": "R034",
  "Shallows": "day-buson-river-crossing",
  "Reef": "waka-other-28",
  "Low tide": "day-basho-low-tide-willow",
  "Crosscurrent": "SHOT-008",
  "Undertow": "waka-other-18",
  "Weave": "R112",
  "Lattice": "hyakunin-14",
  "Open ground": "waka-other-07",
  "First grooves": "day-buson-kite-yesterday",
  "His first lines": "teika-06",
  "Interrupted": "teika-01",
  "The master's grooves": "new-basho-hida",
  "Halfway": "waka-other-02",
  "The finished ring": "new-basho-hikarido",
  "The abbot's ring": "hn03",
  "His grooves": "waka-other-06",
  "His pebbles": "new-buson-dew-thorns",
  "Pale and dark": "hn11",
  "The abbot's pebbles": "SHOT-010",
  "Ripple and groove": "waka-other-12",
  "Pebbles in the current": "waka-other-29",
  "Left for you": "waka-other-30",
  "Plain speaking": "teika-07",
  "Unfinished": "SHOT-025",
  "Called away": "teika-12",
  "Old grooves": "waka-other-25",
  "Continuation": "SHOT-015",
  "Closed circle": "new-buson-fuji-leaves",
  "White stones": "waka-other-26",
  "Black stones": "hn01",
  "Go board": "SHOT-016",
  "Counting stones": "waka-other-08",
  "Narrow rake": "hn22",
  "Wide blade": "day-basho-bamboo-painting",
  "Pivot": "new-basho-ishiyama",
  "Crescent": "teika-02",
  "Wide rake": "teika-24",
  "Whorl": "waka-other-14",
  "Wake": "teika-04",
  "Dragonfly": "supp-issa-dragonfly-river",
  "Spiral": "waka-other-04",
  "Lanternlight": "teika-23",
  "Hairpin": "SHOT-021",
  "Brace": "hn09",
  "Long rake": "new-basho-one-field",
  "Turning stone": "new-buson-dry-spring",
  "Horseshoe": "teika-08",
  "Coil": "new-issa-snail",
  "Shell": "day-buson-rain-shells",
  "Evening bell": "day-basho-blossom-bells",
  "Swallow": "hn26",
  "Bat": "R104",
  "Owl": "supp-issa-owl-fireflies",
  "Ember": "teika-26",
  "Firefly": "hn10",
  "Long shadow": "waka-other-17",
  "Last light": "waka-other-15",
  "Cicada": "hn08",
  "One gate": "hn29",
  "Two gates": "hn12",
  "Two doors": "new-basho-grass-hut",
  "Gatekeeper": "teika-13",
  "Wicket": "hn21",
  "Hinge": "hn16",
  "Latch": "R124",
  "Through the wall": "R131",
  "Side gate": "hn15",
  "Back gate": "hn13",
  "Visitor": "R055",
  "Garden path": "hn02",
  "Dusk": "waka-other-21",
  "Postern": "teika-16",
  "Threshold": "waka-other-05",
  "Late arrival": "R409",
  "Crossing": "teika-09",
  "Bolt": "new-ryokan-thief-moon",
  "Lantern festival": "new-buson-dancers",
  "Moonrise": "new-basho-moon-clouds",
  "Dark water": "waka-other-09",
  "Stargazer": "new-manyo-1068",
  "The abbot's garden": "waka-other-13",
  "Moonbeam": "teika-03",
  "Before dawn": "day-buson-white-plum-dawn",
  "Silver sand": "new-basho-moon-sand",
  "Starfield": "teika-25",
  "Night heron": "SHOT-024",
  "Owl's hour": "teika-21",
  "Milky Way": "hn05",
  "Frost": "teika-17",
  "Moonlit sea": "new-basho-sea-darkens",
  "Midnight": "SHOT-013",
  "Deep night": "waka-other-23",
  "Night rain": "R038",
  "Moon pool": "hn19",
  "Last bell": "SHOT-018",
  "Small hours": "SHOT-023"
 },
 "practice": "R040",
 "practice_poem_reverified": true,
 "pairings": {
  "The corridor": "The first poem of the game is the chore itself: a guest sweeps the temple garden before going on, as the player rakes each garden and moves on.",
  "Nine squares": "The little frog's unhurried attention suits a small garden that first rewards looking before moving.",
  "Morning dew": "Cool dirt under a hand is a tactile match for touching and marking a fresh surface.",
  "The pocket": "The pocket becomes a refuge: a small protected place within a larger field.",
  "Two tones": "Sunlit height and lingering darkness give the alternating tones a natural counterpart.",
  "Bamboo": "Bamboo leaves worried by wind at a window, cutting a doze short.",
  "Sunrise": "The sudden sunrise gives a compact garden a moment of opening and discovery.",
  "First stroke": "A finger writing in empty space mirrors the player's first deliberate drawn line.",
  "Calm": "Tranquil spring light, and petals that fall without a calm heart (しづ心): the quality the garden scores, named by its absence.",
  "Mist": "The poem lets a landscape emerge from haze without requiring the scattered stones to depict a particular mountain.",
  "First bell": "A bell becoming perceptible through dawn mist supplies the chapter's first clear auditory presence.",
  "Sparrow": "Issa's invitation to play gives this early garden an affectionate, welcoming voice.",
  "Cold sand": "Cold exposed ground and the lack of shelter match the austerity of the nearly empty square.",
  "Lantern": "A solitary lamp persisting into dawn matches the title while preserving the opening world's morning light.",
  "The long wall": "Smoke finding passage through a wall offers a gentle counterpoint to the enclosing boundary.",
  "Wind chime": "Wind becomes audible and fragrance becomes dark; the title's chime expands into a dawn soundscape.",
  "Pine shadow": "Pine wind, color, and flowing water merge just as the dark stone and surrounding lines become one composition.",
  "Heron's step": "The passing egret supplies the measured white movement evoked by the title.",
  "Quiet": "Quiet is made audible by something almost too small to hear: falling blossoms.",
  "Plain stone": "Ryōkan's new pond, where the frog makes no sound: the garden before the ripple stone. Bashō's old pond follows, with the stone and its ring.",
  "First ripple": "The ripple stone arrives with the most famous ripple in Japanese poetry: one frog, one sound, rings spreading on still water.",
  "Koi": "A step on the bridge, and the fish go down: the walk in changes the pond under it. Astra's choice; the whitebait only glittered.",
  "Water lily": "Dew spills from the lotus leaf and gathers into a jewel again on the leaf below: lost, then re-formed, as the rings around the stones are. Astra's choice over Buson's two pond lilies, which only named the plants.",
  "Ripples": "Ripples and wind share a rhythm, matching the ring's integration with the surrounding parallel lines.",
  "Spring": "Water finding a path through newly opened clefts is a close structural counterpart to solving a route.",
  "Twin rings": "Two small dwellings facing one river echo the two separate centers sharing a single garden.",
  "Two moons": "The same moonlight held in different sleeves turns the paired circles into a relation across distance.",
  "Frog": "A frog sitting stiffly in the grass; Bashō’s old-pond frog moved to First ripple.",
  "Paired wells": "A barely seen bridge and the water beneath it suggest the quiet connection between separated circular places.",
  "Raindrop": "A tiny trickle negotiating an obstruction suits the first small disturbance around a stone.",
  "Bell's echo": "Sound departing its source is an exact counterpart to an echo spreading from a single center.",
  "Turtle": "The turtle carries a small living interruption; humor lightens the chapter's circling-water imagery.",
  "Stepping stone": "A pause beside water gives the stepping-stone image a bodily sense of rest between movements.",
  "Pond edge": "Water descending around rocks and a half-hidden dwelling fit the ring placed at one edge of the bed.",
  "Drip": "Intermittent meltwater becomes a string of separate drops at a threshold.",
  "Carp": "The carp are never seen; the trembling lotus leaves prove them, as the ring proves the stone. Astra found the one classical carp poem we lacked; it replaces the octopus traps, a great poem forced onto a fish garden.",
  "Lotus": "A stem lifting just clear of water concentrates attention on the small center of a large open surface.",
  "Still water": "A voice lies across still water like one long unbroken stroke.",
  "Well": "The occupied well bucket introduces courtesy toward an existing beauty—a thematic counterpoint to composing the stone’s surrounding ring.",
  "Calm water": "A spotless stream under the moon, before the wave band arrives.",
  "First wave": "Bashō rewrote his Kiyotaki poem on his deathbed: the clean waves now carry scattered pine needles, as the same garden now carries a wave band. The contrasting pair is a pair of drafts.",
  "High tide": "The tide arrives at a gate, joining the water image to the puzzle's fixed entrance.",
  "Sandbar": "Water striking a heron’s legs makes the boundary between standing obstacle and moving surface tangible.",
  "Waves": "An unseen wind changes the river's visible pattern; the raked bands likewise make motion legible.",
  "Swell": "The repeated phrase lets broad, unhurried motion become the poem's form.",
  "Ichimatsu": "The river imagined as dyed cloth connects water-patterns with a deliberately arranged textile design.",
  "Tatami": "Mountain and garden move into the summer room: the garden becomes a floor to sit on.",
  "Noon sun": "Bright light through layered leaves suits the meeting of straight bands and a ring at midday.",
  "Estuary": "A river gathering rain into one rapid current matches the narrow marked channel opening into a larger bed.",
  "Island in the current": "The river delivers the whole hot day into the sea, giving the small marked current a destination beyond its immediate channel.",
  "Breakers": "The wave-crests become flowers, letting repeated parallel marks read as a patterned sea.",
  "Current": "The moon flows with the water, making the straight current carry an image as well as a direction.",
  "Shallows": "Sandals in the hand bring the shallows down to the scale of bare feet.",
  "Reef": "A weir of leaves turns an obstruction into pattern, matching the band and adjacent stone ridge.",
  "Low tide": "Retreating water reveals mud and drooping branches, a direct image of an exposed margin.",
  "Crosscurrent": "Wind redirects leaves and birds ahead of a shower, answering the level's competing directions.",
  "Undertow": "The audible stream threading hidden rock gaps gives the apparent open surface an undercurrent.",
  "Weave": "The loom's interrupted sound evokes the patient making behind an interlaced pattern.",
  "Lattice": "The deliberately tangled cloth pattern introduces emotional disorder within the garden’s rigidly ordered lattice.",
  "Open ground": "Wind crossing open fields at dusk: many rakings fit, before the abbot’s grooves give the ground a single path.",
  "First grooves": "A kite flies where yesterday’s sky was: a stroke made now in a trace left earlier, as the rake follows the abbot’s grooves.",
  "His first lines": "The vanished wind and its garden of seeming snow connect a maker's passing gesture to the trace left behind.",
  "Interrupted": "The broken dream-bridge answers the interrupted groove fragments while preserving the poem's abruptness.",
  "The master's grooves": "Coolness drawn to a master carpenter's plan: the abbot's grooves are a drawing laid down by a better hand, which the rake follows.",
  "Halfway": "Choosing a new route instead of last year's marked path suits a garden that overturns an obvious opening move.",
  "The finished ring": "The Hall of Light that the summer rains left untouched: the one thing spared when everything around it is washed over, which is what the finished ring asks of the rake.",
  "The abbot's ring": "Walking around a moonlit pond respects an existing center and turns attention into a circuit.",
  "His grooves": "Saigyō wishes for another hut beside his own: the abbot’s work set beside the player’s.",
  "His pebbles": "One drop of dew on each thorn: a mark per point, as the abbot's grooves become pebbles.",
  "Pale and dark": "Moon and sun occupy opposite sides of one scene, giving the two kinds of pebble a balanced visual counterpart.",
  "The abbot's pebbles": "Moonlight turns dew into white jewels; the pebbles likewise ask the player to read small bright marks carefully.",
  "Ripple and groove": "A returning scent joins a present dream to an absent past, as new raking continues existing grooves.",
  "Pebbles in the current": "A faint stream threading moss and rock matches the single current passing through scattered instructions.",
  "Left for you": "The question of what to leave behind belongs naturally beside a few marks entrusted to another person.",
  "Plain speaking": "The poem's stripped landscape suits a garden whose sparse marks state exactly what is needed.",
  "Unfinished": "A single remnant after a dream suits the few surviving fragments of an unfinished composition.",
  "Called away": "This is the deliberately elegiac choice: work remains after an absence. The poem mourns Teika's mother; it does not establish the game's master's fate.",
  "Old grooves": "The old scent persists despite uncertainty about people, matching inherited marks whose maker is absent.",
  "Continuation": "Generations leave the moon behind for later viewers, as a groove can be taken up by another hand.",
  "Closed circle": "Young leaves bury everything but Fuji: one shape left standing in a covered field, as the raking leaves the ring.",
  "White stones": "White flowers lost against frost make discrimination within an almost uniform surface the subject.",
  "Black stones": "One dark crow on a bare branch matches the economy of a small decisive dark mark.",
  "Go board": "The bright marks invite a question and give no spoken answer, fitting a board of silent indications.",
  "Counting stones": "Water's white beads become countable in moonlight, an unusually close match to reading many little stones.",
  "Narrow rake": "A thin firefly drifting on: the ordinary rake, one last time before the wide one.",
  "Wide blade": "Coolness becomes visible through drawn bamboo; the new tool likewise imposes a form that can carry a sensation.",
  "Pivot": "The wide rake turns only against stones; Bashō's autumn wind, whiter than the white stones it blows across, is the stroke among its pivots.",
  "Crescent": "The partly clouded moon echoes a broken or softened arc; the poem does not specify a crescent phase.",
  "Wide rake": "The sweeping wind clears an entire prospect, echoing the wide tool's large-scale revelation of order.",
  "Whorl": "Blossoms repeatedly return around a rock: a direct physical counterpart to the whorl's encircling run.",
  "Wake": "Returning wings carry the weather with them, offering an aerial counterpart to a wake left by passage.",
  "Dragonfly": "The small dragonfly touches a whole river with its tail, matching the disproportion between the single projection and the surrounding surface.",
  "Spiral": "Attention grows clearer without reaching a final limit, fitting the spiral's inward movement.",
  "Lanternlight": "A lamp turned away from the blossoms to wait for moonrise; it shares its lamp with the Lantern garden.",
  "Hairpin": "A precarious, untracked passage matches the problem of finding a difficult passage and then erasing its footprints.",
  "Brace": "The bending leaves make the limit of a slender form physically visible, a counterpoint to the rake's turning constraint.",
  "Long rake": "One whole field planted, then the planters move on: a long single labor, finished and left, as each garden is.",
  "Turning stone": "A spring run dry, willow leaves down, stones here and there: a dry landscape, the garden's own genre, answering Saigyō's willow and clear water (Stepping stone).",
  "Horseshoe": "The moonlight laid beneath a waiting figure gives an enclosing curve the intimacy of a resting place.",
  "Coil": "The snail carries its coil on its back and climbs Fuji at the pace the wide rake imposes.",
  "Shell": "Small shells and the exact amount of rain needed to wet them fit a compact, sheltered composition.",
  "Evening bell": "The uncertain source of a bell makes the open garden seem to extend beyond its walls.",
  "Swallow": "Swift evening birds meet uncertainty about tomorrow, adding vulnerability to the clean turns.",
  "Bat": "The fluttering bats provide the title's motion and a rare affectionate view of a dim nocturnal creature.",
  "Owl": "The owl seems to call the fireflies by name, giving the large, spare garden a playful voice in the approaching dark.",
  "Ember": "The outward fire and inward longing give the title’s ember image an inward life.",
  "Firefly": "The apparent fall becomes flight in the next instant, like a route whose reversal changes the meaning of the motion.",
  "Long shadow": "Light dwindling on a wall is a direct partner for the long-shadow image.",
  "Last light": "Light disappears without a dramatic sunset, matching a quiet final interval rather than a spectacle.",
  "Cicada": "Stone receives sound; the sparse board lets that familiar impossible image carry the whole scene.",
  "One gate": "The moon at a single gate, and friends thinning as the heat ebbs: the garden before a second gate is cut.",
  "Two gates": "The moon overhead and a walker passing through a town and out the other side: the stroke now enters by one gate and leaves by another.",
  "Two doors": "One hut, two households through its door: the hermit leaves, a family with its dolls arrives. Astra's choice over the inn-sign verse \"come in this way\".",
  "Gatekeeper": "Snow weighing on a cedar suggests a blocked road to someone awaited, matching the guarded approach.",
  "Wicket": "A tiny aperture opens onto the Milky Way, enlarging the significance of the little wicket.",
  "Hinge": "One flame transferred to another makes a connection visible without exhausting its source.",
  "Latch": "An apparent group of approaching visitors becomes moving grass, leaving expectation at the boundary.",
  "Through the wall": "A crack in a wall admits moonlight: the poem's opening is as concrete as the level's new access.",
  "Side gate": "A gift left at a gate records a visit through what remains, fitting the quieter alternative entrance.",
  "Back gate": "Crossing one's own gate changes the role of householder into traveler, making a familiar boundary newly significant.",
  "Visitor": "A refused lodging becomes an unexpected welcome beneath blossoms; the visitor's destination changes meaning.",
  "Garden path": "An empty road at dusk gives the indirect garden path its solitude.",
  "Dusk": "Dusk and solitude have no single color, allowing the spare layout to retain its ambiguity.",
  "Postern": "Moonlight enters beneath small eaves, a modest passage suited to a side entrance.",
  "Threshold": "Renunciation that cannot keep out the snipe lifting from the marsh: a boundary crossed by feeling. Astra's choice over Jien's announcement that the true path leaves nothing to long for. With Jakuren's Dusk and Teika's Plain speaking, all three of the Three Evenings are now in the game.",
  "Late arrival": "The usually late speaker arrives first for blossoms, a light reversal of the title's expectation.",
  "Crossing": "Wind, sleeves, and a mountain walkway give the passage between two ends bodily exposure and distance.",
  "Bolt": "The thief takes everything but the moon in the window: what a bolt cannot keep in or out.",
  "Lantern festival": "Four or five Bon dancers with the moon sinking onto them: the ring of the dance for the garden's ring, and the sky brought close. Astra's choice over Rengetsu's farewell fires, whose \"how it moves me\" deflated the scene.",
  "Moonrise": "Moon-viewing with its rests: the clouds that interrupt the looking are kind. Astra proposed it for Stargazer, which keeps the Man'yōshū moon-boat; it replaces Shōtetsu's moon over the petal storm here.",
  "Dark water": "The moon crosses from frozen water into a sleeve, while the protected surface remains intact.",
  "Stargazer": "The moon-boat rows into the forest of stars and is hidden: the ripple stone is its one boat on a field of sand.",
  "The abbot's garden": "The grassless white-gravel courtyard is the anthology's closest literal dry-garden setting; an unseen cricket animates it.",
  "Moonbeam": "Moonlight entering through the eaves competes with fragrance on one sleeve, concentrating two sensations in a narrow beam.",
  "Before dawn": "Only the coming of day remains; the poem lets the enormous last stretch wait rather than hurry.",
  "Silver sand": "The last garden of the main path: the moon clear on the sand a pilgrim monk carried, the one classical poem about temple sand hauled and tended by monks.",
  "Starfield": "Clear stars and barely perceptible hail give the scattered pattern both distance and minute texture.",
  "Night heron": "A heron departs but white flowers preserve its color, matching an action remembered by its visible trace.",
  "Owl's hour": "Sleepless before dawn, everything behind rises as images, and the moon is asked to light the road ahead.",
  "Milky Way": "The Milky Way stretches over sea and island, matching the immense field around one stone.",
  "Frost": "Frost accumulates night after night until the intervals close, making small repeated deposits fill a large space.",
  "Moonlit sea": "The sea darkens while the ducks' voices turn faintly white: a night shore heard rather than seen; the band is its edge.",
  "Midnight": "The moon becomes inward and then is forgotten; this is the sequence's deepest absorption in a single center.",
  "Deep night": "A dream inside a dream gives the repeated layout another depth rather than merely another name.",
  "Night rain": "Rain heard during a sleepless night makes a broad dark expanse intimate and personal.",
  "Moon pool": "A boat holds a private moon within a vast frosty prospect; the small ring beside the endpoint likewise gives a broad night garden an intimate center.",
  "Last bell": "The bell seems to sound inside the moon, giving the final chime both resonance and cold clarity.",
  "Small hours": "A sudden stillness makes one distant break audible, stretching the smallest event across the last hours of night."
 },
 "practice_pairing": "Hands cup and release threads of water: an idle, tactile pleasure without a required destination.",
 "poems": {
  "R034": {
   "id": "R034",
   "poet": "Ōtagaki Rengetsu",
   "form": "waka",
   "japanese": "とけわたる 氷にそひて はるのよの 月もながるる 井堤の玉川",
   "translation": "As the ice\nmelts along the stream,\nthe spring-night moon\nflows with it—\nthe Tama River at Ide.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Ama no Karumo",
   "poem_number": "34",
   "source_url": "https://rengetsu.org/poetry_db/index.php?pageNo=1",
   "source_label": "Rengetsu Foundation n.d., poem 34",
   "imagery": [
    "flow",
    "moon",
    "ice",
    "reflection"
   ],
   "notes": "Spring moon and thawing ice share a current.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Ama no karumo 34",
   "author_japanese": "大田垣蓮月",
   "author_dates": "1791–1875"
  },
  "R038": {
   "id": "R038",
   "poet": "Ōtagaki Rengetsu",
   "form": "waka",
   "japanese": "ふかき夜を 思ひねざめの はる雨は 音を聞にも ぬるる袖かな",
   "translation": "Deep in the night\nI wake from troubled sleep.\nThe spring rain—\nmerely to hear it\nwets my sleeves.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Ama no Karumo",
   "poem_number": "38",
   "source_url": "https://rengetsu.org/poetry_db/index.php?pageNo=1",
   "source_label": "Rengetsu Foundation n.d., poem 38",
   "imagery": [
    "night",
    "rain",
    "sleeves",
    "tears"
   ],
   "notes": "Wet sleeves link the sound of spring rain to the waking speaker’s tears.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Ama no karumo 38",
   "author_japanese": "大田垣蓮月",
   "author_dates": "1791–1875"
  },
  "R040": {
   "id": "R040",
   "poet": "Ōtagaki Rengetsu",
   "form": "waka",
   "japanese": "つれづれと 春のながめの 手すさびに むすびて流す 軒のいと水",
   "translation": "An idle pastime\nthrough the long spring rain:\nI cup my hands\nbeneath the threads from the eaves\nand let them run.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Ama no Karumo",
   "poem_number": "40",
   "source_url": "https://rengetsu.org/poetry_db/index.php?pageNo=1",
   "source_label": "Rengetsu Foundation n.d., poem 40",
   "imagery": [
    "hands",
    "play",
    "water",
    "threads"
   ],
   "notes": "Excellent for the unscored practice garden; nagame suggests both lingering contemplation and long rain.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Ama no karumo 40",
   "author_japanese": "大田垣蓮月",
   "author_dates": "1791–1875"
  },
  "R050": {
   "id": "R050",
   "poet": "Ōtagaki Rengetsu",
   "form": "waka",
   "japanese": "東山 花まつ比の 朝ぼらけ かすみに匂ふ かねの音かな",
   "translation": "At Higashiyama,\nwaiting for the blossoms—\ndaybreak;\nthrough the haze\nthe bell’s sound glows.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Ama no Karumo",
   "poem_number": "50",
   "source_url": "https://rengetsu.org/poetry_db/index.php?pageNo=1",
   "source_label": "Rengetsu Foundation n.d., poem 50",
   "imagery": [
    "dawn",
    "bell",
    "mist"
   ],
   "notes": "Niou crosses visual radiance into sound.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Ama no karumo 50",
   "author_japanese": "大田垣蓮月",
   "author_dates": "1791–1875"
  },
  "R055": {
   "id": "R055",
   "poet": "Ōtagaki Rengetsu",
   "form": "waka",
   "japanese": "やどかさぬ 人のつらさを 情にて おぼろ月よの 花の下ぶし",
   "translation": "Refused a bed,\nI take their unkindness\nfor kindness—\nbeneath blossoms I sleep\nin the hazy moonlight.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Ama no Karumo",
   "poem_number": "55",
   "source_url": "https://rengetsu.org/poetry_db/index.php?pageNo=1",
   "source_label": "Rengetsu Foundation n.d., poem 55",
   "imagery": [
    "gate",
    "refusal",
    "shelter",
    "moon"
   ],
   "notes": "A closed door opens another kind of lodging.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Ama no karumo 55",
   "author_japanese": "大田垣蓮月",
   "author_dates": "1791–1875"
  },
  "R102": {
   "id": "R102",
   "poet": "Ōtagaki Rengetsu",
   "form": "waka",
   "japanese": "夕月夜 ほのかに見ゆる 小板橋 下行水に 水鶏啼なり",
   "translation": "The evening moon:\na little plank bridge\nbarely visible;\nunder it, running water\nand a water rail’s cry.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Ama no Karumo",
   "poem_number": "102",
   "source_url": "https://rengetsu.org/poetry_db/index.php?pageNo=3",
   "source_label": "Rengetsu Foundation n.d., poem 102",
   "imagery": [
    "bridge",
    "moon",
    "water",
    "crossing"
   ],
   "notes": "Water rail, not heron.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Ama no karumo 102",
   "author_japanese": "大田垣蓮月",
   "author_dates": "1791–1875"
  },
  "R104": {
   "id": "R104",
   "poet": "Ōtagaki Rengetsu",
   "form": "waka",
   "japanese": "軒近き 柳になびく かはほりの かげなつかしき 薄月夜かな",
   "translation": "Beside the eaves\nbats flutter through the willow—\ntheir silhouettes\nhow dear to me\nin this thin moonlight.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Ama no Karumo",
   "poem_number": "104",
   "source_url": "https://rengetsu.org/poetry_db/index.php?pageNo=3",
   "source_label": "Rengetsu Foundation n.d., poem 104",
   "imagery": [
    "bats",
    "willow",
    "moon",
    "shadows"
   ],
   "notes": "A direct match to Bat.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Ama no karumo 104",
   "author_japanese": "大田垣蓮月",
   "author_dates": "1791–1875"
  },
  "R111": {
   "id": "R111",
   "poet": "Ōtagaki Rengetsu",
   "form": "waka",
   "japanese": "岩つたふ しみづすずしき 山かげの 葉がくれいほに 住人やたれ",
   "translation": "Cool springwater\nruns down the rocks\nin the mountain shade.\nWho lives in the hut\nhidden by leaves?",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Ama no Karumo",
   "poem_number": "111",
   "source_url": "https://rengetsu.org/poetry_db/index.php?pageNo=3",
   "source_label": "Rengetsu Foundation n.d., poem 111",
   "imagery": [
    "rocks",
    "water",
    "hut",
    "visitor"
   ],
   "notes": "The occupant is left unseen.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Ama no karumo 111",
   "author_japanese": "大田垣蓮月",
   "author_dates": "1791–1875"
  },
  "R112": {
   "id": "R112",
   "poet": "Ōtagaki Rengetsu",
   "form": "waka",
   "japanese": "さとの子が はたおる音も とだへして 昼ねの比の あつき旅哉",
   "translation": "Even the village children’s\nweaving has fallen silent.\nThe hour of the midday nap—\non the road,\nhow hot it is.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Ama no Karumo",
   "poem_number": "112",
   "source_url": "https://rengetsu.org/poetry_db/index.php?pageNo=3",
   "source_label": "Rengetsu Foundation n.d., poem 112",
   "imagery": [
    "weaving",
    "interruption",
    "noon",
    "silence"
   ],
   "notes": "A human interruption in a patterned sound.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Ama no karumo 112",
   "author_japanese": "大田垣蓮月",
   "author_dates": "1791–1875"
  },
  "R124": {
   "id": "R124",
   "poet": "Ōtagaki Rengetsu",
   "form": "waka",
   "japanese": "むれ立て 人くと見しは 秋風に 尾花が袖の なびく也けり",
   "translation": "I thought a crowd\nwas coming toward me—\nonly the sleeves\nof pampas grass\nwaving in the autumn wind.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Ama no Karumo",
   "poem_number": "124",
   "source_url": "https://rengetsu.org/poetry_db/index.php?pageNo=4",
   "source_label": "Rengetsu Foundation n.d., poem 124",
   "imagery": [
    "visitor",
    "illusion",
    "grass",
    "wind"
   ],
   "notes": "An apparent arrival turns into vegetation.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Ama no karumo 124",
   "author_japanese": "大田垣蓮月",
   "author_dates": "1791–1875"
  },
  "R126": {
   "id": "R126",
   "poet": "Ōtagaki Rengetsu",
   "form": "waka",
   "japanese": "さし登る 朝日の山も ふもとには 猶夜をのこす 宇治の河霧",
   "translation": "Sunlight climbs\nMount Asahi;\nbelow,\nthe river mist at Uji\nstill holds the night.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Ama no Karumo",
   "poem_number": "126",
   "source_url": "https://rengetsu.org/poetry_db/index.php?pageNo=4",
   "source_label": "Rengetsu Foundation n.d., poem 126",
   "imagery": [
    "dawn",
    "two tones",
    "mist",
    "contrast"
   ],
   "notes": "Light and remaining darkness coexist in one landscape.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Ama no karumo 126",
   "author_japanese": "大田垣蓮月",
   "author_dates": "1791–1875"
  },
  "R131": {
   "id": "R131",
   "poet": "Ōtagaki Rengetsu",
   "form": "waka",
   "japanese": "山ざとの かべのやれ間の きりぎりす 月もここより させよとぞ鳴",
   "translation": "In the mountain village,\na cricket in a broken wall\ncries out:\nlet the moonlight\ncome through here too.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Ama no Karumo",
   "poem_number": "131",
   "source_url": "https://rengetsu.org/poetry_db/index.php?pageNo=4",
   "source_label": "Rengetsu Foundation n.d., poem 131",
   "imagery": [
    "wall",
    "opening",
    "moon",
    "cricket"
   ],
   "notes": "Kirigirisu in older poetry commonly denotes a cricket; the source English uses grasshopper.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Ama no karumo 131",
   "author_japanese": "大田垣蓮月",
   "author_dates": "1791–1875"
  },
  "R409": {
   "id": "R409",
   "poet": "Ōtagaki Rengetsu",
   "form": "waka",
   "japanese": "何事も 人におくるる さがの山 花見ばかりは 先ぞきにける",
   "translation": "Always behind\neveryone else—\nbut at Mount Saga,\nfor the cherry blossoms,\nI have arrived first.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Rengetsu Foundation database; collected or inscribed poems",
   "poem_number": "database 409",
   "source_url": "https://rengetsu.org/poetry_db/index.php?pageNo=13",
   "source_label": "Rengetsu Foundation n.d., poem 409",
   "imagery": [
    "arrival",
    "late",
    "flowers",
    "humor"
   ],
   "notes": "A lightly comic counterpoint for Late arrival; Saga also plays against personal disposition.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "",
   "author_japanese": "大田垣蓮月",
   "author_dates": "1791–1875"
  },
  "SHOT-004": {
   "id": "SHOT-004",
   "poet": "Shōtetsu",
   "form": "waka",
   "japanese": "山風の松に木ぶかき音はして花の香くらき明けぼのの雨",
   "translation": "Deep in the pines\nthe mountain wind sounds;\nin the dawn rain,\neven the fragrance\nof blossoms is dark.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Sōkonshū (草根集)",
   "poem_number": "1537",
   "source_url": "https://www.asahi-net.or.jp/~SG2H-YMST/yamatouta/sennin/0syoutetu_t.html",
   "source_label": "Yamato Uta / Sennin Manshu, Shōtetsu: edition-numbered Japanese text",
   "additional_source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syoutetu.html",
   "imagery": [
    "pines",
    "wind",
    "dawn",
    "rain",
    "blossoms",
    "fragrance"
   ],
   "notes": "Topic: 曙春雨 (spring rain at dawn). Preserves the synesthesia of 花の香くらき, dark blossom fragrance.",
   "numbering_note": "Number follows the source’s Notre Dame Seishin Women’s University Classical Text Series Sōkonshū (vols1–4) text; numbering is edition-specific.",
   "verification": "Japanese original and attribution retrieved directly from the cited Shōtetsu anthology page; poem number checked against its companion annotated page. No manuscript or facsimile collation performed.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Sōkonshū 1537",
   "author_japanese": "正徹",
   "author_dates": "1381–1459"
  },
  "SHOT-005": {
   "id": "SHOT-005",
   "poet": "Shōtetsu",
   "form": "waka",
   "japanese": "まぎるべき風さへ吹かで散りかかる花の音きく窓のうちかな",
   "translation": "Not even wind\nto hide the sound—\nat the window,\nI hear the blossoms\nfall.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Sōkonshū (草根集)",
   "poem_number": "1362",
   "source_url": "https://www.asahi-net.or.jp/~SG2H-YMST/yamatouta/sennin/0syoutetu_t.html",
   "source_label": "Yamato Uta / Sennin Manshu, Shōtetsu: edition-numbered Japanese text",
   "additional_source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syoutetu.html",
   "imagery": [
    "silence",
    "blossoms",
    "window",
    "falling petals",
    "wind"
   ],
   "notes": "Topic: 閑居花 (flowers in quiet retirement). The translation treats まぎる as obscuring/masking the sound. Source notes a second-line variant in Shōtetsu senshu; the recorded Japanese follows its main text.",
   "numbering_note": "Number follows the source’s Notre Dame Seishin Women’s University Classical Text Series Sōkonshū (vols1–4) text; numbering is edition-specific.",
   "verification": "Japanese original and attribution retrieved directly from the cited Shōtetsu anthology page; poem number checked against its companion annotated page. No manuscript or facsimile collation performed.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Sōkonshū 1362",
   "author_japanese": "正徹",
   "author_dates": "1381–1459"
  },
  "SHOT-008": {
   "id": "SHOT-008",
   "poet": "Shōtetsu",
   "form": "waka",
   "japanese": "山づたひ夕立うつる風さきに木の葉も鳥もふかれてぞ行く",
   "translation": "Over the mountains\nthe shower shifts;\nin the wind before it,\nthe leaves and birds\nare swept away.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Sōkonshū (草根集)",
   "poem_number": "2635",
   "source_url": "https://www.asahi-net.or.jp/~SG2H-YMST/yamatouta/sennin/0syoutetu_t.html",
   "source_label": "Yamato Uta / Sennin Manshu, Shōtetsu: edition-numbered Japanese text",
   "additional_source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syoutetu.html",
   "imagery": [
    "rain",
    "wind",
    "mountains",
    "birds",
    "leaves",
    "movement"
   ],
   "notes": "Topic: 遠夕立 (a distant shower).",
   "numbering_note": "Number follows the source’s Notre Dame Seishin Women’s University Classical Text Series Sōkonshū (vols1–4) text; numbering is edition-specific.",
   "verification": "Japanese original and attribution retrieved directly from the cited Shōtetsu anthology page; poem number checked against its companion annotated page. No manuscript or facsimile collation performed.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Sōkonshū 2635",
   "author_japanese": "正徹",
   "author_dates": "1381–1459"
  },
  "SHOT-010": {
   "id": "SHOT-010",
   "poet": "Shōtetsu",
   "form": "waka",
   "japanese": "秋草の露とこたふる風もなしただしら玉をみがく月かげ",
   "translation": "Not even wind\nto answer, “Dew\non autumn grass”—\nonly the moonlight\npolishing white gems.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Sōkonshū (草根集)",
   "poem_number": "3437",
   "source_url": "https://www.asahi-net.or.jp/~SG2H-YMST/yamatouta/sennin/0syoutetu_t.html",
   "source_label": "Yamato Uta / Sennin Manshu, Shōtetsu: edition-numbered Japanese text",
   "additional_source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syoutetu.html",
   "imagery": [
    "dew",
    "moon",
    "grass",
    "pearls",
    "silence",
    "autumn"
   ],
   "notes": "Topic: 露如玉 (dew like jewels). Alludes to the question about white jewels/dew in Ise monogatari6; wind would disperse the jeweled illusion.",
   "numbering_note": "Number follows the source’s Notre Dame Seishin Women’s University Classical Text Series Sōkonshū (vols1–4) text; numbering is edition-specific.",
   "verification": "Japanese original and attribution retrieved directly from the cited Shōtetsu anthology page; poem number checked against its companion annotated page. No manuscript or facsimile collation performed.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Sōkonshū 3437",
   "author_japanese": "正徹",
   "author_dates": "1381–1459"
  },
  "SHOT-013": {
   "id": "SHOT-013",
   "poet": "Shōtetsu",
   "form": "waka",
   "japanese": "すみのぼる心にすめる月をみて月をわするる秋のさ夜中",
   "translation": "My heart rises clear;\nI watch the moon\nshining within it,\nand forget the moon—\nautumn midnight.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Sōkonshū (草根集)",
   "poem_number": "3990",
   "source_url": "https://www.asahi-net.or.jp/~SG2H-YMST/yamatouta/sennin/0syoutetu_t.html",
   "source_label": "Yamato Uta / Sennin Manshu, Shōtetsu: edition-numbered Japanese text",
   "additional_source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syoutetu.html",
   "imagery": [
    "moon",
    "mind",
    "meditation",
    "forgetting",
    "midnight",
    "autumn"
   ],
   "notes": "Topic: 閑見月 (quiet moon-viewing). すむ evokes both becoming clear and dwelling; this working translation conveys clarity and inward presence without claiming to reproduce the wordplay fully.",
   "numbering_note": "Number follows the source’s Notre Dame Seishin Women’s University Classical Text Series Sōkonshū (vols1–4) text; numbering is edition-specific.",
   "verification": "Japanese original and attribution retrieved directly from the cited Shōtetsu anthology page; poem number checked against its companion annotated page. No manuscript or facsimile collation performed.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Sōkonshū 3990",
   "author_japanese": "正徹",
   "author_dates": "1381–1459"
  },
  "SHOT-015": {
   "id": "SHOT-015",
   "poet": "Shōtetsu",
   "form": "waka",
   "japanese": "むかしよりいく世の人かあかずしてながめすてけん故郷の月",
   "translation": "How many generations\nsince long ago\ngazed, never satisfied,\nand left you behind—\nmoon of the old home?",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Sōkonshū (草根集)",
   "poem_number": "4172",
   "source_url": "https://www.asahi-net.or.jp/~SG2H-YMST/yamatouta/sennin/0syoutetu_t.html",
   "source_label": "Yamato Uta / Sennin Manshu, Shōtetsu: edition-numbered Japanese text",
   "additional_source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syoutetu.html",
   "imagery": [
    "moon",
    "old home",
    "time",
    "generations",
    "departure"
   ],
   "notes": "Topic: 古郷月 (moon at the old home); the source reports 寄月懐旧 in Shōtetsu senshu. 故郷 can suggest an old capital or former home; the English leaves the place unspecified.",
   "numbering_note": "Number follows the source’s Notre Dame Seishin Women’s University Classical Text Series Sōkonshū (vols1–4) text; numbering is edition-specific.",
   "verification": "Japanese original and attribution retrieved directly from the cited Shōtetsu anthology page; poem number checked against its companion annotated page. No manuscript or facsimile collation performed.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Sōkonshū 4172",
   "author_japanese": "正徹",
   "author_dates": "1381–1459"
  },
  "SHOT-016": {
   "id": "SHOT-016",
   "poet": "Shōtetsu",
   "form": "waka",
   "japanese": "白玉かなにぞととへば萩のうへの影はこたへずふるさとの月",
   "translation": "“White jewels—\nor what?” I ask;\nthe light upon the bush clover\ngives no answer:\nmoon of my old home.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "Sōkonshū (草根集)",
   "poem_number": "4704",
   "source_url": "https://www.asahi-net.or.jp/~SG2H-YMST/yamatouta/sennin/0syoutetu_t.html",
   "source_label": "Yamato Uta / Sennin Manshu, Shōtetsu: edition-numbered Japanese text",
   "additional_source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syoutetu.html",
   "imagery": [
    "dew",
    "moon",
    "bush clover",
    "pearls",
    "old home",
    "question"
   ],
   "notes": "Topic: 萩露 (dew on bush clover). Another allusion to Ise monogatari6; distinct from3437, though their motifs overlap.",
   "numbering_note": "Number follows the source’s Notre Dame Seishin Women’s University Classical Text Series Sōkonshū (vols1–4) text; numbering is edition-specific.",
   "verification": "Japanese original and attribution retrieved directly from the cited Shōtetsu anthology page; poem number checked against its companion annotated page. No manuscript or facsimile collation performed.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Sōkonshū 4704",
   "author_japanese": "正徹",
   "author_dates": "1381–1459"
  },
  "SHOT-018": {
   "id": "SHOT-018",
   "poet": "Shōtetsu",
   "form": "waka",
   "japanese": "月のうちにひびきのぼると思ふまで霜夜の鐘に影ぞさえ行く",
   "translation": "Till it seems\nthe sound climbs into the moon,\nthe frosty night’s bell—\nand the moonlight\ngrows colder.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Sōkonshū (草根集)",
   "poem_number": "5357",
   "source_url": "https://www.asahi-net.or.jp/~SG2H-YMST/yamatouta/sennin/0syoutetu_t.html",
   "source_label": "Yamato Uta / Sennin Manshu, Shōtetsu: edition-numbered Japanese text",
   "additional_source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syoutetu.html",
   "imagery": [
    "moon",
    "bell",
    "frost",
    "night",
    "sound",
    "cold"
   ],
   "notes": "Topic: 冬月 (winter moon). 冴える here permits clearer/brighter as well as colder; this version chooses colder to preserve the sensory crossing.",
   "numbering_note": "Number follows the source’s Notre Dame Seishin Women’s University Classical Text Series Sōkonshū (vols1–4) text; numbering is edition-specific.",
   "verification": "Japanese original and attribution retrieved directly from the cited Shōtetsu anthology page; poem number checked against its companion annotated page. No manuscript or facsimile collation performed.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Sōkonshū 5357",
   "author_japanese": "正徹",
   "author_dates": "1381–1459"
  },
  "SHOT-021": {
   "id": "SHOT-021",
   "poet": "Shōtetsu",
   "form": "waka",
   "japanese": "わたりかね雲も夕を猶たどる跡なき雪の嶺のかけはし",
   "translation": "Even the clouds\nfalter through the dusk—\na cliffside walkway\nunder snow,\nnot a footprint.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by GPT-6 Astra (review pass) and Claude, 2026-10-04",
   "collection": "Sōkonshū (草根集)",
   "poem_number": "5885",
   "source_url": "https://www.asahi-net.or.jp/~SG2H-YMST/yamatouta/sennin/0syoutetu_t.html",
   "source_label": "Yamato Uta / Sennin Manshu, Shōtetsu: edition-numbered Japanese text",
   "additional_source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syoutetu.html",
   "imagery": [
    "bridge",
    "snow",
    "footprints",
    "clouds",
    "dusk",
    "path"
   ],
   "notes": "Kakehashi is a precarious mountain walkway or ladder across a steep slope. The source also finds an ambiguity between luminous snow and approaching dusk; no suspension bridge is implied.",
   "numbering_note": "Number follows the source’s Notre Dame Seishin Women’s University Classical Text Series Sōkonshū (vols1–4) text; numbering is edition-specific.",
   "verification": "Japanese original and attribution retrieved directly from the cited Shōtetsu anthology page; poem number checked against its companion annotated page. No manuscript or facsimile collation performed.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Sōkonshū 5885",
   "author_japanese": "正徹",
   "author_dates": "1381–1459"
  },
  "SHOT-023": {
   "id": "SHOT-023",
   "poet": "Shōtetsu",
   "form": "waka",
   "japanese": "さ夜風はただ一足にしづまりてをち方きけば雪折の声",
   "translation": "The night wind\nfalls still all at once.\nFrom far away,\nas I listen,\na bough breaks under snow.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Sōkonshū (草根集)",
   "poem_number": "6053",
   "source_url": "https://www.asahi-net.or.jp/~SG2H-YMST/yamatouta/sennin/0syoutetu_t.html",
   "source_label": "Yamato Uta / Sennin Manshu, Shōtetsu: edition-numbered Japanese text",
   "additional_source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syoutetu.html",
   "imagery": [
    "night",
    "wind",
    "silence",
    "snow",
    "branch",
    "sound"
   ],
   "notes": "Hitoashi ni is rendered all at once rather than supplying an unverified literal footstep by the wind.",
   "numbering_note": "Number follows the source’s Notre Dame Seishin Women’s University Classical Text Series Sōkonshū (vols1–4) text; numbering is edition-specific.",
   "verification": "Japanese original and attribution retrieved directly from the cited Shōtetsu anthology page; poem number checked against its companion annotated page. No manuscript or facsimile collation performed.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Sōkonshū 6053",
   "author_japanese": "正徹",
   "author_dates": "1381–1459"
  },
  "SHOT-024": {
   "id": "SHOT-024",
   "poet": "Shōtetsu",
   "form": "waka",
   "japanese": "むら雨のふる江をよそに飛ぶさぎの跡まで白きおもだかの花",
   "translation": "From the rain-swept inlet\na heron flies away;\neven the place it leaves\nis white—\narrowhead flowers.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Sōkonshū (草根集)",
   "poem_number": "9064",
   "source_url": "https://www.asahi-net.or.jp/~SG2H-YMST/yamatouta/sennin/0syoutetu_t.html",
   "source_label": "Yamato Uta / Sennin Manshu, Shōtetsu: edition-numbered Japanese text",
   "additional_source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syoutetu.html",
   "imagery": [
    "heron",
    "rain",
    "inlet",
    "flowers",
    "whiteness",
    "trace"
   ],
   "notes": "Topic: 江雨鷺飛 (heron flying over rain at an inlet). おもだか is the aquatic arrowhead plant, Sagittaria; “arrowhead” is botanical, not a weapon.",
   "numbering_note": "Number follows the source’s Notre Dame Seishin Women’s University Classical Text Series Sōkonshū (vols1–4) text; numbering is edition-specific.",
   "verification": "Japanese original and attribution retrieved directly from the cited Shōtetsu anthology page; poem number checked against its companion annotated page. No manuscript or facsimile collation performed.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Sōkonshū 9064",
   "author_japanese": "正徹",
   "author_dates": "1381–1459"
  },
  "SHOT-025": {
   "id": "SHOT-025",
   "poet": "Shōtetsu",
   "form": "waka",
   "japanese": "あととめてさむるか夢の中空に孤の雲の残るをぞ見る",
   "translation": "Does a dream\nleave this trace on waking?\nIn the open sky\nI see\na single cloud remain.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Sōkonshū (草根集)",
   "poem_number": "10428",
   "source_url": "https://www.asahi-net.or.jp/~SG2H-YMST/yamatouta/sennin/0syoutetu_t.html",
   "source_label": "Yamato Uta / Sennin Manshu, Shōtetsu: edition-numbered Japanese text",
   "additional_source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syoutetu.html",
   "imagery": [
    "dream",
    "cloud",
    "sky",
    "trace",
    "awakening",
    "solitude"
   ],
   "notes": "Topic: 孤夢 (solitary dream). Source explicitly glosses 孤 as ひとり. The relation between a lingering dream and the remaining cloud is kept as a question.",
   "numbering_note": "Number follows the source’s Notre Dame Seishin Women’s University Classical Text Series Sōkonshū (vols1–4) text; numbering is edition-specific.",
   "verification": "Japanese original and attribution retrieved directly from the cited Shōtetsu anthology page; poem number checked against its companion annotated page. No manuscript or facsimile collation performed.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Sōkonshū 10428",
   "author_japanese": "正徹",
   "author_dates": "1381–1459"
  },
  "day-basho-bamboo-painting": {
   "id": "day-basho-bamboo-painting",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "すヾしさを絵にうつしけり嵯峨の竹",
   "translation": "Coolness\nmade into a painting:\nthe bamboo of Saga.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "住吉物語 (Sumiyoshi monogatari)",
   "source_url": "https://www.basho.jp/hatonokai/kendaikaisetsu/kendai25_07.html",
   "source_label": "Bashō Kaigi, seasonal-word commentary",
   "imagery": [
    "bamboo",
    "coolness",
    "painting",
    "summer",
    "Saga"
   ],
   "notes": "Fresh working translation. Source spelling retained; frequently printed 涼しさを絵にうつしけり嵯峨の竹.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "day-basho-blossom-bells": {
   "id": "day-basho-blossom-bells",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "花の雲鐘は上野か浅草か",
   "translation": "Clouds of blossoms—\nthat bell: from Ueno\nor Asakusa?",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "続虚栗 (Zoku minashiguri), 1687",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/haikusyu/hanano.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database",
   "imagery": [
    "blossoms",
    "bell",
    "temples",
    "spring",
    "distance"
   ],
   "notes": "Fresh working translation. The two place names identify actual temple districts in Edo.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Zoku minashiguri (1687)",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "day-basho-cuckoo-over-water": {
   "id": "day-basho-cuckoo-over-water",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "ほととぎす声や横たふ水の上",
   "translation": "A cuckoo—\nits voice lies out\nacross the water.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Letter to Miyazaki Keikō, Genroku 6, fourth month, day 29 (1693)",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/letter/keiko.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database",
   "imagery": [
    "cuckoo",
    "water",
    "voice",
    "horizontal",
    "stillness"
   ],
   "notes": "Fresh working translation. This wording is in Bashō's letter. 藤の実 has the variant ほととぎす声横たふや水の上.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "letter of 1693",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "day-basho-green-needles-water": {
   "id": "day-basho-green-needles-water",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "清滝や波に散り込む青松葉",
   "translation": "Kiyotaki—\ngreen pine needles\nscattering into the waves.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "去来抄 (Kyorai shō), wording discussed in the linked commentary; 1694 poem",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/letter/sanpu2.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database",
   "imagery": [
    "pine",
    "needles",
    "stream",
    "waves",
    "green"
   ],
   "notes": "Fresh working translation. The source discusses this revision alongside 清滝や波に塵なき夏の月. Do not treat the two as independent poems in one anthology.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Kyoraishō",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "day-basho-low-tide-willow": {
   "id": "day-basho-low-tide-willow",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "青柳の泥にしだるる潮干かな",
   "translation": "Green willow branches\nhang down into the mud—\nthe tide is out.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "炭俵 (Sumidawara)",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/haikusyu/aoyagi.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database",
   "imagery": [
    "willow",
    "low tide",
    "mud",
    "green",
    "shore"
   ],
   "notes": "Fresh working translation.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Sumidawara (1694)",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "day-basho-melon-dew": {
   "id": "day-basho-melon-dew",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "朝露や撫でて涼しき瓜の土",
   "translation": "Morning dew—\nthe dirt on the melon\ncool beneath my hand.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Letter to Sugiyama Sanpū, Genroku 7, sixth month, day 24 (1694)",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/letter/sanpu2.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database",
   "imagery": [
    "morning",
    "dew",
    "melon",
    "soil",
    "touch",
    "coolness"
   ],
   "notes": "Fresh working translation. 土 refers to dirt adhering to the melon in the source's reading.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "letter of 1694",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "day-basho-nameless-mountain": {
   "id": "day-basho-nameless-mountain",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "春なれや名もなき山の薄霞",
   "translation": "Spring has come—\na mountain without a name\nin thin haze.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "野ざらし紀行 (Nozarashi kikō), road to Nara section",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/nozarasi/nozara25.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database",
   "imagery": [
    "spring",
    "mountain",
    "haze",
    "mist"
   ],
   "notes": "Fresh working translation. This source prints 薄霞, thin haze, and records 朝霞, morning haze, as a variant. Do not silently mix their readings.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Nozarashi kikō",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "day-basho-old-pond": {
   "id": "day-basho-old-pond",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "古池や蛙飛びこむ水の音",
   "translation": "The old pond—\na frog jumps in;\nthe sound of water.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "蛙合 (Kawazu awase), 1686",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/haikusyu/huruike.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database",
   "imagery": [
    "pond",
    "frog",
    "water",
    "sound",
    "ripples"
   ],
   "notes": "Fresh working translation of the Japanese; no English syllable constraint. The Japanese does not specify the number of frogs.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Kawazu awase (1686)",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "day-basho-plum-sunrise": {
   "id": "day-basho-plum-sunrise",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "梅が香にのつと日の出る山路哉",
   "translation": "Into plum fragrance\nthe sun suddenly rises—\na mountain path.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "炭俵 (Sumidawara), opening verse, 1694",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/haikusyu/umegaka.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database",
   "imagery": [
    "sunrise",
    "plum",
    "fragrance",
    "mountain path"
   ],
   "notes": "Fresh working translation. のつと gives the rising sun an abrupt, almost casual appearance.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Sumidawara (1694)",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "day-basho-rain-wasp-nest": {
   "id": "day-basho-rain-wasp-nest",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "春雨や蜂の巣つたふ屋根の漏り",
   "translation": "Spring rain—\nthe roof’s leak trickles\ndown a wasp nest.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "炭俵 (Sumidawara), 1694",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/haikusyu/harsame3.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database",
   "imagery": [
    "spring rain",
    "roof",
    "nest",
    "drips",
    "hut"
   ],
   "notes": "Fresh working translation. 蜂 can cover bees or wasps; an exposed old nest beneath the eaves supports 'wasp nest'.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Sumidawara (1694)",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "day-basho-ripples-keeping-time": {
   "id": "day-basho-ripples-keeping-time",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "さざ波や風の薫の相拍子",
   "translation": "Small ripples\nkeeping time\nwith the fragrant wind.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "笈日記 (Oi nikki), 1694",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/haikusyu/noryo1.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database",
   "imagery": [
    "ripples",
    "wind",
    "fragrance",
    "lake",
    "rhythm"
   ],
   "notes": "Fresh working translation. 相拍子 is a musical term; the poem greeted a Noh performer.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Oi nikki",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "day-basho-sunlit-leaves": {
   "id": "day-basho-sunlit-leaves",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "あらたうと青葉若葉の日の光",
   "translation": "How holy—\nsunlight through green leaves,\nthrough new leaves.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "おくのほそ道 (Oku no hosomichi), Nikkō section, journey of 1689",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/okunohosomichi/okuno051.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database",
   "imagery": [
    "sunlight",
    "leaves",
    "green",
    "temple",
    "early summer"
   ],
   "notes": "Fresh working translation. The Nikkō setting carries religious and political praise; this is not merely a neutral nature sketch.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Oku no hosomichi",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "day-basho-swift-mogami": {
   "id": "day-basho-swift-mogami",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "五月雨をあつめて早し最上川",
   "translation": "Gathering\nthe summer rains, so swift—\nthe Mogami River.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "おくのほそ道 (Oku no hosomichi), Mogami River section, journey of 1689",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/okunohosomichi/okuno24.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database",
   "imagery": [
    "rain",
    "river",
    "current",
    "summer",
    "gathering"
   ],
   "notes": "Fresh working translation. 五月雨 means the early-summer rainy season, not modern-calendar May rain.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Oku no hosomichi",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "day-buson-bell-voice": {
   "id": "day-buson-bell-voice",
   "poet": "Yosa Buson",
   "form": "haiku",
   "japanese": "涼しさや鐘をはなるるかねの声",
   "translation": "Coolness—\nthe bell’s voice\nleaves the bell.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "source_url": "https://www.basho.jp/hatonokai/kendaikaisetsu/kendai25_07.html",
   "source_label": "Bashō Kaigi, seasonal-word commentary",
   "imagery": [
    "bell",
    "sound",
    "coolness",
    "summer",
    "temple"
   ],
   "notes": "Fresh working translation. The linked poets' society page verifies text and authorship but does not name the original collection.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784"
  },
  "day-buson-kite-yesterday": {
   "id": "day-buson-kite-yesterday",
   "poet": "Yosa Buson",
   "form": "haiku",
   "japanese": "凧きのふの空のありどころ",
   "translation": "A kite—\nthere is the place\nwhere yesterday’s sky was.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "蕪村句集 (Buson kushū)",
   "source_url": "https://bonjin-ultra.com/buson.htm",
   "source_label": "Hyōgo Literature Museum",
   "imagery": [
    "kite",
    "sky",
    "spring",
    "daylight"
   ],
   "notes": "Fresh working translation. Backup. The museum displays the kite character 凧 as its separated components 几巾; the standard character is restored here.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Buson kushū",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784"
  },
  "day-buson-rain-inhabited-house": {
   "id": "day-buson-rain-inhabited-house",
   "poet": "Yosa Buson",
   "form": "haiku",
   "japanese": "春雨や人住みて煙壁を洩る",
   "translation": "Spring rain—\nsomeone lives here; smoke\nseeps through the wall.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "蕪村句集 (Buson kushū)",
   "source_url": "https://bonjin-ultra.com/buson.htm",
   "source_label": "Hyōgo Literature Museum",
   "imagery": [
    "rain",
    "house",
    "smoke",
    "spring"
   ],
   "notes": "Fresh working translation. Backup.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Buson kushū",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784"
  },
  "day-buson-rain-shells": {
   "id": "day-buson-rain-shells",
   "poet": "Yosa Buson",
   "form": "haiku",
   "japanese": "春雨や小磯の小貝ぬるるほど",
   "translation": "Spring rain—\njust enough to wet small shells\non a little shore.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "蕪村句集 (Buson kushū)",
   "source_url": "https://bonjin-ultra.com/buson.htm",
   "source_label": "Hyōgo Literature Museum",
   "imagery": [
    "rain",
    "shells",
    "shore",
    "spring"
   ],
   "notes": "Fresh working translation.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Buson kushū",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784"
  },
  "day-buson-river-crossing": {
   "id": "day-buson-river-crossing",
   "poet": "Yosa Buson",
   "form": "haiku",
   "japanese": "夏河を越すうれしさよ手に草履",
   "translation": "Such pleasure, crossing\nthe summer river—\nsandals in my hand.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "蕪村句集 (Buson kushū)",
   "source_url": "https://bonjin-ultra.com/buson.htm",
   "source_label": "Hyōgo Literature Museum",
   "imagery": [
    "river",
    "feet",
    "summer",
    "crossing"
   ],
   "notes": "Fresh working translation.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Buson kushū",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784"
  },
  "day-buson-spring-sea": {
   "id": "day-buson-spring-sea",
   "poet": "Yosa Buson",
   "form": "haiku",
   "japanese": "春の海終日のたりのたり哉",
   "translation": "The spring sea\nrolls slowly, rolls slowly,\nall day.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "蕪村句集 (Buson kushū)",
   "source_url": "https://bonjin-ultra.com/buson.htm",
   "source_label": "Hyōgo Literature Museum",
   "imagery": [
    "sea",
    "waves",
    "spring",
    "daylight"
   ],
   "notes": "Fresh working translation.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Buson kushū",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784"
  },
  "day-buson-two-houses": {
   "id": "day-buson-two-houses",
   "poet": "Yosa Buson",
   "form": "haiku",
   "japanese": "さみだれや大河を前に家二軒",
   "translation": "Summer rains—\nfacing the great river,\ntwo houses.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "蕪村句集 (Buson kushū)",
   "source_url": "https://bonjin-ultra.com/buson.htm",
   "source_label": "Hyōgo Literature Museum",
   "imagery": [
    "rain",
    "river",
    "houses",
    "summer"
   ],
   "notes": "Fresh working translation.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Buson kushū",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784"
  },
  "day-buson-white-plum-dawn": {
   "id": "day-buson-white-plum-dawn",
   "poet": "Yosa Buson",
   "form": "haiku",
   "japanese": "しら梅に明る夜ばかりとなりにけり",
   "translation": "From now on,\nnights will break\non white plum blossoms.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "から檜葉 (Karahiba), headnote 初春",
   "source_url": "https://bonjin-ultra.com/buson.htm",
   "source_label": "Hyōgo Literature Museum",
   "imagery": [
    "plum",
    "white",
    "dawn",
    "spring"
   ],
   "notes": "Fresh working translation.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Karahiba",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784"
  },
  "day-chiyoni-morning-glory-water": {
   "id": "day-chiyoni-morning-glory-water",
   "poet": "Fukuda Chiyo-ni",
   "form": "haiku",
   "japanese": "朝顔につるべとられてもらひ水",
   "translation": "Morning glories\nhave taken the well bucket—\nI ask for water elsewhere.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "千代尼句集 (Chiyo-ni kushū), Kihaku, ed., Hōreki 14 (1764)",
   "source_url": "https://www.city.hakusan.lg.jp/machi/ryokka/1003077.html",
   "source_label": "Hakusan City, Chiyo-ni documentation",
   "imagery": [
    "morning glory",
    "water",
    "well",
    "vine",
    "bucket",
    "care"
   ],
   "notes": "Fresh working translation. Uses the に version printed in the 1764 collection. Hakusan's cultural-property record verifies this and discusses the や variant: https://www.city.hakusan.lg.jp/bunka/bunkazai/1006096/1002328/1009253.html .",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Chiyo-ni kushū (1764)",
   "author_japanese": "千代尼",
   "author_dates": "1703–1775"
  },
  "day-issa-frog-mountain": {
   "id": "day-issa-frog-mountain",
   "poet": "Kobayashi Issa",
   "form": "haiku",
   "japanese": "ゆうぜんとして山を見る蛙哉",
   "translation": "Serenely\ngazing at the mountain:\na frog.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "七番日記 (Shichiban nikki), Bunka 10 (1813)",
   "source_url": "https://ohh.sisos.co.jp/cgi-bin/openhh/jsearch.cgi?dbi=20140103235455_20140104001012&entry=4104&group=hirarajp&shid=6a9e69c2",
   "source_label": "Issa haiku database, entry 4104",
   "imagery": [
    "frog",
    "mountain",
    "stillness",
    "spring",
    "watching"
   ],
   "notes": "Fresh working translation. Also independently attested by Nagareyama municipal poem-monument documentation: https://m.city.nagareyama.chiba.jp/life/1001505/1001700/1001730/1001735.html .",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shichiban nikki",
   "author_japanese": "小林一茶",
   "author_dates": "1763–1828"
  },
  "day-issa-frog-on-turtle": {
   "id": "day-issa-frog-on-turtle",
   "poet": "Kobayashi Issa",
   "form": "haiku",
   "japanese": "亀どのに負さつて鳴蛙哉",
   "translation": "Riding on\nMr. Turtle’s back,\na frog calls.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "七番日記 (Shichiban nikki), Bunka 12 (1815)",
   "source_url": "https://ohh.sisos.co.jp/cgi-bin/openhh/jsearch.cgi?dbi=20140103235455_20140104001012&entry=4116&group=hirarajp&shid=6a9e69c2",
   "source_label": "Issa haiku database, entry 4116",
   "imagery": [
    "frog",
    "turtle",
    "pond",
    "spring",
    "voice"
   ],
   "notes": "Fresh working translation. どの humorously grants the turtle an honorific.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shichiban nikki",
   "author_japanese": "小林一茶",
   "author_dates": "1763–1828"
  },
  "day-issa-orphan-sparrow": {
   "id": "day-issa-orphan-sparrow",
   "poet": "Kobayashi Issa",
   "form": "haiku",
   "japanese": "我と来て遊べや親のない雀",
   "translation": "Come along with me\nand play, little sparrow\nwith no parents.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "source_url": "https://www.issakinenkan.com/around/",
   "source_label": "Issa Memorial Museum, Myōsenji poem monument",
   "imagery": [
    "sparrow",
    "play",
    "company",
    "spring"
   ],
   "notes": "Fresh working translation. Collection not specified by the museum's monument record.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "",
   "author_japanese": "小林一茶",
   "author_dates": "1763–1828"
  },
  "day-issa-proud-frog": {
   "id": "day-issa-proud-frog",
   "poet": "Kobayashi Issa",
   "form": "haiku",
   "japanese": "草陰につんとしている蛙かな",
   "translation": "In the grass’s shade,\na frog sits\nwith its nose in the air.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "七番日記 (Shichiban nikki), Bunka 11 (1814)",
   "source_url": "https://ohh.sisos.co.jp/cgi-bin/openhh/jsearch.cgi?dbi=20140103235455_20140104001012&entry=4110&group=hirarajp&shid=6a9e69c2",
   "source_label": "Issa haiku database, entry 4110",
   "imagery": [
    "frog",
    "grass",
    "shade",
    "spring",
    "stillness"
   ],
   "notes": "Fresh working translation. 'Nose in the air' renders the aloof or haughty posture of つんとしている rather than adding a literal action.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shichiban nikki",
   "author_japanese": "小林一茶",
   "author_dates": "1763–1828"
  },
  "hn01": {
   "id": "hn01",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "枯枝に烏のとまりけり秋の暮",
   "translation": "On a bare branch\na crow has settled—\nautumn dusk.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "芭蕉俳句全集 (1903), 秋・秋の暮",
   "source_url": "https://ja.wikisource.org/wiki/芭蕉俳句全集",
   "source_label": "Bashō haiku zenshū, edited by Ōtsuka Kōzan (1903), autumn section",
   "imagery": [
    "crow",
    "bare branch",
    "autumn",
    "dusk"
   ],
   "notes": "The cited edition supplies the とまりけり version; other established versions differ. New working translation of this Japanese text. Historically a hokku.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "hn02": {
   "id": "hn02",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "此道や行く人なしに秋の暮",
   "translation": "This road—\nno one taking it\nin the autumn dusk.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "芭蕉俳句全集 (1903), 秋・秋の暮; heading 所思",
   "source_url": "https://ja.wikisource.org/wiki/芭蕉俳句全集",
   "source_label": "Bashō haiku zenshū, edited by Ōtsuka Kōzan (1903), autumn section",
   "imagery": [
    "empty path",
    "autumn",
    "dusk",
    "solitude"
   ],
   "notes": "Do not count the alternative 人声やこの道かへる秋の暮 as an independent companion poem: it belongs to this poem's revision history.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "hn03": {
   "id": "hn03",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "名月や池をめぐりて夜もすがら",
   "translation": "The harvest moon—\naround the pond I walk\nthe whole night through.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "あつめ句 (source's attribution)",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/haikusyu/meigetu1.htm",
   "source_label": "Bashō Database: 名月や池をめぐりて夜もすがら",
   "imagery": [
    "pond",
    "circular path",
    "moon",
    "whole night"
   ],
   "notes": "The subject is implicit in Japanese; the English supplies I. The source dates the poem to Jōkyō 3, eighth month, fifteenth night (1686).",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Atsume ku",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "hn04": {
   "id": "hn04",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "名月や門にさしくる潮がしら",
   "translation": "The harvest moon—\nthe leading edge of the tide\nreaches my gate.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/haikusyu/meigetu2.htm",
   "source_label": "Bashō Database: 名月や門にさしくる潮がしら",
   "imagery": [
    "gate",
    "rising tide",
    "moon",
    "water boundary"
   ],
   "notes": "My is implicit rather than stated. Preserve the meeting of water and threshold; do not turn the tide into an ordinary garden stream.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "hn05": {
   "id": "hn05",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "荒海や佐渡によこたふ天河",
   "translation": "A rough sea—\nstretched over Sado,\nthe Milky Way.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "おくのほそ道, 越後路",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/okunohosomichi/okuno28.htm",
   "source_label": "Bashō Database: Oku no hosomichi, Echigo road",
   "imagery": [
    "rough sea",
    "island",
    "Milky Way",
    "night sky"
   ],
   "notes": "Sado is the named island, not a generic rock. The poem's arrangement need not be treated as a literal astronomical observation.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Oku no hosomichi",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "hn08": {
   "id": "hn08",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "閑さや岩にしみ入蝉の声",
   "translation": "Stillness—\ninto the stone\nthe cicadas’ cry sinks.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "おくのほそ道, 立石寺",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/okunohosomichi/okuno23.htm",
   "source_label": "Bashō Database: Oku no hosomichi, Risshakuji",
   "imagery": [
    "stone",
    "cicadas",
    "stillness",
    "temple",
    "summer"
   ],
   "notes": "Japanese does not specify singular or plural cicadas. This is a summer daytime poem, suitable for the cicada motif but not evidence of nocturnal cicadas.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Oku no hosomichi",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "hn09": {
   "id": "hn09",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "初雪や水仙の葉のたわむまで",
   "translation": "The first snowfall—\nenough to bend\nthe narcissus leaves.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "あつめ句 (source's attribution)",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/haikusyu/hatyuki2.htm",
   "source_label": "Bashō Database: 初雪や水仙の葉のたわむまで",
   "imagery": [
    "first snow",
    "narcissus",
    "bent leaves",
    "weight"
   ],
   "notes": "The cited source uses modern たわむ, retained here. Snow, not night, is explicit.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Atsume ku",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "hn10": {
   "id": "hn10",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "草の葉を落ちるよりとぶ螢哉",
   "translation": "From a blade of grass\nit drops—and is already flying:\na firefly.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "芭蕉俳句全集 (1903), 夏・螢",
   "source_url": "https://ja.wikisource.org/wiki/芭蕉俳句全集",
   "source_label": "Bashō haiku zenshū, edited by Ōtsuka Kōzan (1903), summer section",
   "imagery": [
    "firefly",
    "grass",
    "falling",
    "taking flight"
   ],
   "notes": "Working translation reads より as the immediate change from falling to flying. Preserve the tiny movement rather than adding a metaphysical explanation.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "hn11": {
   "id": "hn11",
   "poet": "Yosa Buson",
   "form": "haiku",
   "japanese": "菜の花や月は東に日は西に",
   "translation": "Rapeseed blossoms—\nthe moon in the east,\nthe sun in the west.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "蕪村句集; heading 春景",
   "source_url": "https://bonjin-ultra.com/buson.htm",
   "source_label": "Hyōgo Prefectural Museum of Art, Edo–Hyōgo Haiku Gallery: Buson",
   "imagery": [
    "flowers",
    "east",
    "west",
    "sunset",
    "moonrise",
    "opposed directions"
   ],
   "notes": "The Japanese names rapeseed blossoms; the translation does not add a color.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Buson kushū",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784"
  },
  "hn12": {
   "id": "hn12",
   "poet": "Yosa Buson",
   "form": "haiku",
   "japanese": "月天心貧しき町を通りけり",
   "translation": "The moon at heaven’s center—\nI pass through\na poor town.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "蕪村句集",
   "source_url": "https://www.nippon.com/ja/japan-topics/b09645/",
   "source_label": "Hyōgo Prefectural Museum of Art, Edo–Hyōgo Haiku Gallery: Buson",
   "imagery": [
    "high moon",
    "night walk",
    "town",
    "poverty"
   ],
   "notes": "天心 places the moon centrally overhead; it does not name a moon phase.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Buson kushū",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784"
  },
  "hn13": {
   "id": "hn13",
   "poet": "Yosa Buson",
   "form": "haiku",
   "japanese": "門を出れば我も行人秋のくれ",
   "translation": "Outside my gate\nI too am someone on the road—\nautumn dusk.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Letter to 了角 and 乙総, An’ei 3 (1774), ninth month, second day",
   "source_url": "https://gashuu.hateblo.jp/entry/2024/12/04/211201",
   "source_label": "剣持雅舟のブログ: 蕪村の句「門を出れば我も行く人」 (2024)",
   "imagery": [
    "gate",
    "threshold",
    "traveler",
    "autumn",
    "dusk"
   ],
   "notes": "行人 is a passerby or traveler. The change in the speaker's status happens upon crossing the gate.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url (a secondary source quoting the letter)",
   "display_source": "letter of 1774",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784"
  },
  "hn15": {
   "id": "hn15",
   "poet": "Yosa Buson",
   "form": "haiku",
   "japanese": "鮎くれてよらで過行夜半の門",
   "translation": "He leaves me sweetfish\nand goes on without coming in—\nthe gate at midnight.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "蕪村句集; composed 1768",
   "source_url": "https://www.nippon.com/ja/japan-topics/b09625/",
   "source_label": "Nippon.com, 古典俳諧への招待: 鮎くれてよらで過行夜半の門 (2023)",
   "imagery": [
    "gate",
    "visitor",
    "gift",
    "sweetfish",
    "midnight",
    "passing on"
   ],
   "notes": "Gender and recipient are supplied for readable English; Japanese leaves the subject unstated. 夜半 means the middle of the night, not necessarily exactly 00:00.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Buson kushū",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784"
  },
  "hn16": {
   "id": "hn16",
   "poet": "Yosa Buson",
   "form": "haiku",
   "japanese": "燭の火を燭にうつすや春の夕",
   "translation": "A candle’s flame\npassed to another candle—\nspring evening.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "source_url": "https://bonjin-ultra.com/buson.htm",
   "source_label": "蕪村の俳句集, spring poems",
   "imagery": [
    "candle",
    "flame",
    "transfer",
    "evening",
    "spring"
   ],
   "notes": "Japanese verified in this anthology and independently in the digitized 季題類別蕪村名句選集 search result. No original collection number established.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784"
  },
  "hn17": {
   "id": "hn17",
   "poet": "Yosa Buson",
   "form": "haiku",
   "japanese": "夕風や水青鷺の脛を打つ",
   "translation": "The evening breeze—\nwater slaps\nthe heron’s shins.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "source_url": "https://www.aozora.gr.jp/cards/000305/files/47985_41579.html",
   "source_label": "Masaoka Shiki, Haijin Buson, section 客観的美, at Aozora Bunko",
   "imagery": [
    "evening breeze",
    "water",
    "heron",
    "legs",
    "ripples"
   ],
   "notes": "青鷺 is the grey heron; English uses the unqualified heron. The water, not the bird, is the subject of 打つ.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784"
  },
  "hn19": {
   "id": "hn19",
   "poet": "Yosa Buson",
   "form": "haiku",
   "japanese": "霜百里舟中に我月を領ス",
   "translation": "A hundred leagues of frost—\nhere in my boat\nthe moon is mine.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "蕪村自筆句帳",
   "source_url": "https://www.nippon.com/ja/japan-topics/b09653/",
   "source_label": "Nippon.com, 古典俳諧への招待: 霜百里舟中に我月を領ス (2023)",
   "imagery": [
    "frost",
    "boat",
    "moon",
    "night",
    "wide space"
   ],
   "notes": "百里 literally means a hundred ri; this working translation renders its immense reach idiomatically rather than falsely converting ri into miles. The source dates the composition tentatively to 1775.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Buson jihitsu kuchō",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784"
  },
  "hn21": {
   "id": "hn21",
   "poet": "Kobayashi Issa",
   "form": "haiku",
   "japanese": "うつくしや障子の穴の天の川",
   "translation": "How beautiful—\nthe Milky Way through a hole\nin the paper screen.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "志多良; Bunka 10 (1813)",
   "source_url": "https://www.issakinenkan.com/diary/一茶と善光寺⑨「上原文路宅跡」/",
   "source_label": "Issa Memorial Museum: 一茶と善光寺⑨「上原文路宅跡」 (2015)",
   "imagery": [
    "paper screen",
    "hole",
    "Milky Way",
    "night sky",
    "framed view"
   ],
   "notes": "The museum ties it to Issa's illness and recovery at Uehara Bunro's house. This working translation adopts the ordinary literal view through the hole, without adding the medical context to the verse.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shidara",
   "author_japanese": "小林一茶",
   "author_dates": "1763–1828"
  },
  "hn22": {
   "id": "hn22",
   "poet": "Kobayashi Issa",
   "form": "haiku",
   "japanese": "痩蛍ふはりふはりとながらふる",
   "translation": "A thin firefly—\ndrifting, drifting,\nit lives on.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "source_url": "https://www.issakinenkan.com/diary/ゲンジボタルについて学びませんか/",
   "source_label": "Issa Memorial Museum: ゲンジボタルについて学びませんか (2014)",
   "imagery": [
    "firefly",
    "frailty",
    "floating",
    "survival"
   ],
   "notes": "ながらふる means to continue living; the translation keeps that meaning rather than merely saying it floats.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "",
   "author_japanese": "小林一茶",
   "author_dates": "1763–1828"
  },
  "hn24": {
   "id": "hn24",
   "poet": "Kobayashi Issa",
   "form": "haiku",
   "japanese": "我家や町の蛍の逃所",
   "translation": "My house—\nwhere the town’s fireflies\ncome to escape.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "source_url": "https://www.issakinenkan.com/2014/06/",
   "source_label": "Issa Memorial Museum: ゲンジボタルの季節ですよ (2014-06-24)",
   "imagery": [
    "house",
    "fireflies",
    "refuge",
    "town",
    "entrance"
   ],
   "notes": "逃所 is a place of escape or refuge. The Japanese does not specify what the fireflies escape from.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "",
   "author_japanese": "小林一茶",
   "author_dates": "1763–1828"
  },
  "hn26": {
   "id": "hn26",
   "poet": "Kobayashi Issa",
   "form": "haiku",
   "japanese": "夕燕我には翌のあてはなき",
   "translation": "Evening swallows—\nas for tomorrow,\nI have nothing to count on.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "文化句帖; database entry 3470",
   "source_url": "https://ohh.sisos.co.jp/cgi-bin/openhh/jsearch.cgi?dbi=20140103235455_20140104000746&group=hirarajp&s_entry=3460&se0=0&sf0=0&sk0=",
   "source_label": "Issa Haiku Database, entry 3470",
   "imagery": [
    "swallows",
    "evening",
    "journey",
    "uncertain lodging",
    "tomorrow"
   ],
   "notes": "Entry 3470 has あてはなき; adjacent entries document different versions. Lodging is a plausible contextual interpretation of あて, but the translation does not insert it. The number is a database ID, not a numbering supplied by Issa.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Bunka kuchō",
   "author_japanese": "小林一茶",
   "author_dates": "1763–1828"
  },
  "hn29": {
   "id": "hn29",
   "poet": "Kobayashi Issa",
   "form": "haiku",
   "japanese": "門の月暑がへれば友もへる",
   "translation": "Moon at the gate—\nas the heat lessens,\nso do the friends.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "八番日記; Bunsei 4 (1821)",
   "source_url": "https://www.janis.or.jp/users/kyodoshi/issaku-14.htm",
   "source_label": "Issa hokku zenshū (14), 残暑 section",
   "imagery": [
    "gate",
    "moon",
    "waning heat",
    "fewer visitors",
    "late summer"
   ],
   "notes": "The source also records a variant ending 人もへる. The translation follows 友, friends, in the selected text.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Hachiban nikki",
   "author_japanese": "小林一茶",
   "author_dates": "1763–1828"
  },
  "hn30": {
   "id": "hn30",
   "poet": "Kobayashi Issa",
   "form": "haiku",
   "japanese": "青空に指で字をかく秋の暮",
   "translation": "In the blue sky\nI write with my finger—\nautumn dusk.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "七番日記; Bunka 11 (1814)",
   "source_url": "https://www.janis.or.jp/users/kyodoshi/issaku-14.htm",
   "source_label": "Issa hokku zenshū (14), 秋の暮 section",
   "imagery": [
    "blue sky",
    "finger",
    "writing",
    "drawn trajectory",
    "autumn dusk"
   ],
   "notes": "The Japanese supplies no written word or character; the translation does not invent one.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shichiban nikki",
   "author_japanese": "小林一茶",
   "author_dates": "1763–1828"
  },
  "hyakunin-14": {
   "id": "hyakunin-14",
   "poet": "Minamoto no Tōru",
   "form": "waka",
   "japanese": "みちのくの しのぶもぢずり 誰ゆゑに 乱れそめにし われならなくに",
   "translation": "Like Michinoku’s Shinobu cloth,\nits pattern tangled—\nfor whose sake\nhas my heart begun to tangle?\nNot by my own doing.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "Ogura Hyakunin Isshu",
   "poem_number": "14",
   "source_url": "https://www.bunkanet.jp/manabi/hyakunin-isshu-karuta/element/14/",
   "source_label": "Bunka Net n.d., Hyakunin Isshu 14",
   "imagery": [
    "cloth",
    "pattern",
    "disorder",
    "love"
   ],
   "notes": "The author is named by his title Kawara no Sadaijin in the source. Shinobu mojizuri is patterned dyed cloth; the image is not a literal woven lattice. The English makes the implied heart explicit.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Hyakunin isshu 14",
   "author_japanese": "源融",
   "author_dates": "822–895"
  },
  "hyakunin-17": {
   "id": "hyakunin-17",
   "poet": "Ariwara no Narihira",
   "form": "waka",
   "japanese": "ちはやぶる 神代もきかず 竜田川 からくれなゐに 水くくるとは",
   "translation": "Not even\nin the age of mighty gods\nwas such a thing heard—\nTatsuta’s waters\ntied and dyed crimson.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Ogura Hyakunin Isshu",
   "poem_number": "17",
   "source_url": "https://www.bunkanet.jp/manabi/hyakunin-isshu-karuta/element/17/",
   "source_label": "Bunka Net n.d., Hyakunin Isshu 17",
   "imagery": [
    "river",
    "crimson",
    "dyed cloth",
    "pattern"
   ],
   "notes": "Uses the tie-dye reading of mizu kukuru; alternative readings of the last phrase exist. The cloth analogy matches the level’s patterned compartments, not its exact checkerboard design.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Hyakunin isshu 17",
   "author_japanese": "在原業平",
   "author_dates": "825–880"
  },
  "new-basho-kiyotaki-moon": {
   "id": "new-basho-kiyotaki-moon",
   "poet": "Matsuo Bashō",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694",
   "form": "haiku",
   "japanese": "清滝や波に塵なき夏の月",
   "translation": "Kiyotaki—\non the waves, not a speck of dust:\nthe summer moon.",
   "collection": "1694; revised on his deathbed as 清滝や波に散り込む青松葉",
   "poem_number": null,
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/letter/sanpu2.htm",
   "imagery": [
    "river",
    "waves",
    "summer moon",
    "clarity"
   ],
   "notes": "",
   "verification": "",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "display_source": ""
  },
  "supp-basho-evening-wave-flowers": {
   "id": "supp-basho-evening-wave-flowers",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "夕晴れや桜に涼む波の華",
   "translation": "Evening clears—\nbeneath the cherry tree,\nthe cool bloom of waves.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "真蹟懐紙 (autograph kaishi), Kisakata",
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/haikusyu/kisagata.htm",
   "source_label": "Yamanashi Prefectural University Bashō database: 夕晴れや桜に涼む波の華",
   "imagery": [
    "sea",
    "waves",
    "cherry tree",
    "evening",
    "clearing weather"
   ],
   "notes": "New working translation. The source explains 波の華 as the flowerlike glitter of waves, not the winter sea-foam phenomenon also called nami no hana. The headnote describes the evening rain stopping and a boat tour of the inlet.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "autograph sheet",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "supp-basho-hot-day-sea": {
   "id": "supp-basho-hot-day-sea",
   "poet": "Matsuo Bashō",
   "form": "haiku",
   "japanese": "暑き日を海に入れたり最上川",
   "translation": "The hot day\npoured into the sea—\nMogami River.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "おくのほそ道 (Oku no hosomichi), Sakata",
   "source_url": "https://www.thr.mlit.go.jp/yamagata/river/enc/genre/02-reki/reki0201_002.html",
   "source_label": "Japan Ministry of Land, Infrastructure, Transport and Tourism, Yamagata: Mogami River encyclopedia",
   "imagery": [
    "sea",
    "river",
    "sun",
    "evening",
    "summer"
   ],
   "notes": "New working translation. 日 can evoke the hot day and the sun. The official explanation reads the river as washing the day's heat into the sea at evening.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Oku no hosomichi",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694"
  },
  "supp-buson-lotus-fragrance": {
   "id": "supp-buson-lotus-fragrance",
   "poet": "Yosa Buson",
   "form": "haiku",
   "japanese": "蓮の香や水をはなるゝ莖二寸",
   "translation": "Lotus fragrance—\ntwo inches of stem\nclear of the water.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "蕪村集 (as indexed by the museum)",
   "source_url": "https://www2.city.kurashiki.okayama.jp/musnat/plant/bungakusakuhin/busonsyuu.htm",
   "source_label": "Kurashiki Museum of Natural History: 蕪村集の植物",
   "imagery": [
    "lotus",
    "water",
    "fragrance",
    "summer"
   ],
   "notes": "New working translation. Source prints the old character 莖 (modern 茎). 二寸 is two traditional sun, approximately 6 cm; 'two inches' is a conventional approximate poetic rendering, not an exact metric conversion. Japanese also corroborated by https://www.manpukujitochigi.or.jp/zakki000063.html .",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784"
  },
  "supp-issa-dragonfly-river": {
   "id": "supp-issa-dragonfly-river",
   "poet": "Kobayashi Issa",
   "form": "haiku",
   "japanese": "蜻蛉の尻でなぶるや大井川",
   "translation": "With its tail\na dragonfly teases\nthe Ōi River.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "source_url": "https://gakusyu.shizuoka-c.ed.jp/japanese/syou_34/koten/03/shizuoka02.html",
   "source_label": "Shizuoka Prefecture Asunaro learning resources: 静岡ゆかりの俳句 ２",
   "imagery": [
    "dragonfly",
    "river",
    "water surface"
   ],
   "notes": "New working translation. Official educational page explicitly attributes all three displayed poems to Issa and explains the dragonfly touching the river with its small rear as though teasing the much larger river.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "",
   "author_japanese": "小林一茶",
   "author_dates": "1763–1828"
  },
  "supp-issa-owl-fireflies": {
   "id": "supp-issa-owl-fireflies",
   "poet": "Kobayashi Issa",
   "form": "haiku",
   "japanese": "梟や蛍蛍をよぶやうに",
   "translation": "An owl—\nas though calling,\n“Fireflies! Fireflies!”",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "七番日記, Bunka 7 (1810)",
   "source_url": "https://www.janis.or.jp/users/kyodoshi/issaku-11.htm",
   "source_label": "一茶発句全集（１１）, summer animals: fireflies",
   "imagery": [
    "owl",
    "fireflies",
    "summer",
    "night"
   ],
   "notes": "New working translation. Source reads 梟や蛍 ～ をよぶやうに; its repetition symbol is expanded to 蛍蛍 here. Night is inferred from the owl/firefly scene; the Japanese does not explicitly contain the word night. Much closer to a nocturnal owl scene than Issa's first-autumn-morning poem.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shichiban nikki",
   "author_japanese": "小林一茶",
   "author_dates": "1763–1828"
  },
  "teika-01": {
   "id": "teika-01",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "春のよの夢のうきはしとだえしてみねにわかるゝよこ雲のそら",
   "translation": "On a spring night\nthe floating bridge of dreams\nbreaks off—\nclouds stretch across the sky,\nparting at the peaks.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shinkokinwakashū",
   "poem_number": 38,
   "source_url": "https://www.wakapoetry.net/skks-i-38/",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "dream",
    "broken bridge",
    "clouds",
    "spring night"
   ],
   "notes": "Keep the abrupt break; do not turn this into a moral about completing the puzzle.",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 38"
  },
  "teika-02": {
   "id": "teika-02",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "おほぞらはむめのにほひにかすみつゝくもりもはてぬ春のよの月",
   "translation": "The wide sky\nhazes with plum fragrance;\nnot wholly clouded,\nthe moon\nof a spring night.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shinkokinwakashū",
   "poem_number": 40,
   "source_url": "https://www.wakapoetry.net/skks-i-40/",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "plum scent",
    "haze",
    "moon"
   ],
   "notes": "",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 40"
  },
  "teika-03": {
   "id": "teika-03",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "むめの花にほひをうつす袖のうへにのきもる月のかげぞあらそふ",
   "translation": "On my sleeves,\nsteeped in plum-blossom scent,\nmoonlight\nleaking through the eaves\nvies with that fragrance.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shinkokinwakashū",
   "poem_number": 44,
   "source_url": "https://www.wakapoetry.net/skks-i-44/",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "moonbeam",
    "eaves",
    "plum fragrance",
    "sleeves"
   ],
   "notes": "The competition between scent and light is in the poem; retain its synesthesia.",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 44"
  },
  "teika-04": {
   "id": "teika-04",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "しもまよふそらにしほれしかりがねのかへるつばさに春雨ぞふる",
   "translation": "Geese once wearied\nin a sky adrift with frost—\non their wings\nas they return,\nspring rain falls.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shinkokinwakashū",
   "poem_number": 63,
   "source_url": "https://www.wakapoetry.net/tag/teika/page/23/",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "return",
    "geese",
    "frost",
    "spring rain"
   ],
   "notes": "The Japanese is verified on the tag page; the individual poem-page retrieval returned an internal error. The birds are geese, not swallows.",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 63"
  },
  "teika-06": {
   "id": "teika-06",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "さくら色の庭のはる風あともなしとはゞぞ人の雪とだにみん",
   "translation": "The cherry-colored\nspring wind in the garden\nhas left no trace;\na visitor\nwould see only snow.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shinkokinwakashū",
   "poem_number": 134,
   "source_url": "https://www.wakapoetry.net/skks-ii-134/",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "garden",
    "petals",
    "erased traces",
    "snow"
   ],
   "notes": "Excellent literal garden pairing; the snow is a comparison for fallen blossoms.",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 134"
  },
  "teika-07": {
   "id": "teika-07",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "見わたせば花も紅葉もなかりけり浦のとまやの秋のゆふぐれ",
   "translation": "Looking out—\nno blossoms,\nno colored leaves.\nA thatched hut on the shore\nin the autumn dusk.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shinkokinwakashū",
   "poem_number": 363,
   "source_url": "https://www.wakapoetry.net/skks-iv-363/",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "absence",
    "plainness",
    "shore",
    "hut",
    "autumn dusk"
   ],
   "notes": "A strong alternative finale: absence and austerity, rather than praise of an elaborate finished pattern.",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 363"
  },
  "teika-08": {
   "id": "teika-08",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "さむしろやまつよの秋の風ふけて月をかたしくうぢのはしひめ",
   "translation": "On a narrow mat,\nin the autumn wind\nof a long night’s waiting,\nshe lies with only the moonlight—\nthe Lady of Uji Bridge.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "Shinkokinwakashū",
   "poem_number": 420,
   "source_url": "https://www.wakapoetry.net/skks-iv-420/",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "straw mat",
    "waiting",
    "moonlight",
    "bridge"
   ],
   "notes": "Katashiku carries the loneliness of lying alone; moonlight becomes bedding.",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 420"
  },
  "teika-09": {
   "id": "teika-09",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "たび人の袖ふきかへす秋風にゆふひさびしき山のかけはし",
   "translation": "The autumn wind\nblows back\nthe traveler’s sleeves;\nlonely in the evening sun,\nthe cliffside bridge.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "Shinkokinwakashū",
   "poem_number": 953,
   "source_url": "https://www.wakapoetry.net/skks-x-953/",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "traveler",
    "autumn wind",
    "sunset",
    "mountain footbridge"
   ],
   "notes": "",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 953"
  },
  "teika-11": {
   "id": "teika-11",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "わするなよやどるたもとはかはるともかたみにしぼるよはの月かげ",
   "translation": "Do not forget—\nthough the sleeves that held it\nchange with time—\nthe midnight moonlight\nwe wrung out together.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shinkokinwakashū",
   "poem_number": 891,
   "source_url": "https://www.wakapoetry.net/skks-ix-891/",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "parting",
    "shared moonlight",
    "sleeves",
    "remembrance"
   ],
   "notes": "",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 891"
  },
  "teika-12": {
   "id": "teika-12",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "たまゆらのつゆも涙もとゞまらずなき人こふるやどの秋風",
   "translation": "Not for an instant\nwill dew or tears\nremain—\nautumn wind in the house\nwhere I mourn the dead.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shinkokinwakashū",
   "poem_number": 788,
   "source_url": "https://www.wakapoetry.net/skks-viii-788/",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "dew",
    "tears",
    "autumn wind",
    "mourning",
    "house"
   ],
   "notes": "The headnote places this after his mother’s death; its grief should not be trivialized into a joke about failure.",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 788"
  },
  "teika-13": {
   "id": "teika-13",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "まつ人のふもとのみちはたえぬらんのきばのすぎに雪をもるなり",
   "translation": "Has the road below\nclosed to the one I await?\nOn the cedar\nbeside my eaves\nthe snow grows heavy.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shinkokinwakashū",
   "poem_number": 672,
   "source_url": "https://www.wakapoetry.net/skks-vi-672/",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "blocked path",
    "waiting",
    "cedar",
    "snow"
   ],
   "notes": "The transmitted kana has 雪をもる; normalizing gives 雪おもる, snow grows heavy. Do not render this as a transitive act of piling snow.",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 672"
  },
  "teika-14": {
   "id": "teika-14",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "こまとめて袖うちはらふかげもなしさのゝわたりの雪のゆふぐれ",
   "translation": "No shelter\nwhere I might halt my horse\nand brush my sleeves:\nthe ford at Sano\nin the snowy dusk.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "Shinkokinwakashū",
   "poem_number": 671,
   "source_url": "https://www.wakapoetry.net/skks-vi-671/",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "no shelter",
    "snow",
    "crossing",
    "horse",
    "evening"
   ],
   "notes": "Kage here is shelter, not the assertion that the speaker casts no shadow. Also verified in Japanese at https://hyakuninisshu.com/97.html .",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 671"
  },
  "teika-15": {
   "id": "teika-15",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "時わかぬなみさへ色にいづみがははゝそのもりに嵐ふくらし",
   "translation": "Even the waves,\nwhich know no seasons,\ntake on color—\na storm must be blowing\nthrough Izumi’s oak woods.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shinkokinwakashū",
   "poem_number": 532,
   "source_url": "https://www.wakapoetry.net/skks-v-532/",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "waves",
    "colored leaves",
    "river",
    "storm"
   ],
   "notes": "",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 532"
  },
  "teika-16": {
   "id": "teika-16",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "しぐれつる眞屋の軒端の程なきに頓てさしいる月の影かな",
   "translation": "After the shower,\nunder the cottage’s\nscant eaves,\nmoonlight\ncomes straight in.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Senzaiwakashū",
   "poem_number": 414,
   "source_url": "https://www.wakapoetry.net/szs-vi-414/",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "shower",
    "eaves",
    "moonlight",
    "shelter"
   ],
   "notes": "Provisional: hodo naki is treated spatially, of the scant eaves; the poem also compresses the interval between rain and moonlight.",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Senzaishū 414"
  },
  "teika-17": {
   "id": "teika-17",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "冬來ては一夜ふたよをたま笹の葉わけの霜の處せきまで",
   "translation": "Winter has come.\nOne night, then two:\namong the bamboo leaves\nfrost thickens\nuntil no room remains.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Senzaiwakashū",
   "poem_number": 400,
   "source_url": "https://www.wakapoetry.net/szs-vi-400/",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "bamboo grass",
    "frost",
    "coverage",
    "winter"
   ],
   "notes": "",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Senzaishū 400"
  },
  "teika-19": {
   "id": "teika-19",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "松風のひびきも色もひとつにてみどりに落つる谷川の水",
   "translation": "The pine wind’s\necho and color\nare one:\nthe valley stream\nfalls green.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "Shūigūsō",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/teika.html",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "pine wind",
    "stream",
    "green",
    "synesthesia"
   ],
   "notes": "Collection inferred from the source’s explicit rule that unnumbered selections are from Shūigūsō; exact poem number not supplied.",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shūi gusō"
  },
  "teika-20": {
   "id": "teika-20",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "夕立のくもまの日影はれそめて山のこなたをわたる白鷺",
   "translation": "Sunlight begins\nto clear the shower clouds;\na white egret\npasses on this side\nof the mountain.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Gyokuyōwakashū",
   "poem_number": 416,
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/teika.html",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "egret",
    "clearing shower",
    "sunlight",
    "mountain"
   ],
   "notes": "The poem is daylight, so Heron’s step is a much closer fit than Night heron.",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Gyokuyōshū 416"
  },
  "teika-21": {
   "id": "teika-21",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "こしかたはみな面影にうかびきぬ行末てらせ秋の夜の月",
   "translation": "Everything behind me\nfloats up\nin memory.\nLight what lies ahead,\nmoon of the autumn night.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Gyokuyōwakashū",
   "poem_number": 688,
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/teika.html",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "past",
    "future",
    "moonlight",
    "memory"
   ],
   "notes": "",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Gyokuyōshū 688"
  },
  "teika-22": {
   "id": "teika-22",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "つくづくと明けゆく窓のともし火の有りやとばかりとふ人もなし",
   "translation": "I watch the lamp\nas the window grows light.\nNo one asks\neven whether\nI am still here.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by GPT-6 Astra (review pass) and Claude, 2026-10-04",
   "collection": "Gyokuyōwakashū",
   "poem_number": 2167,
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/teika.html",
   "source_label": "千人万首: 藤原定家 (unannotated selection)",
   "imagery": [
    "lamp",
    "dawn",
    "solitude",
    "unvisited house"
   ],
   "notes": "The lamp and solitary speaker overlap; the translation preserves that indirectness.",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Gyokuyōshū 2167"
  },
  "teika-23": {
   "id": "teika-23",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "山のはの月まつ空のにほふより花にそむくる春のともし火",
   "translation": "Waiting for the moon,\nthe mountain-edge sky\nbegins to glow;\nI turn the spring lamplight\naway from the flowers.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Gyokuyōwakashū",
   "poem_number": 211,
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/teika.html",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "lamp",
    "moonrise",
    "flowers",
    "light"
   ],
   "notes": "",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Gyokuyōshū 211"
  },
  "teika-24": {
   "id": "teika-24",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "吹きはらふ紅葉のうへの霧はれて峯たしかなるあらし山かな",
   "translation": "Swept by the wind,\nthe mist over the red leaves clears;\nthe peak stands sure:\nArashiyama,\nStorm Mountain.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "Shūigūsō; Kanpaku Sadaijin-ke hyakushu (1232), topic: prospect",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/teika.html",
   "source_label": "千人万首: 藤原定家 (unannotated selection)",
   "imagery": [
    "mist clearing",
    "autumn leaves",
    "ridge",
    "clarity"
   ],
   "notes": "Exact anthology poem number not supplied by the consulted source. The source prints あらし山 in kana, which keeps the pun on arashi, storm, next to 吹きはらふ; the English glosses the name.",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shūi gusō"
  },
  "teika-25": {
   "id": "teika-25",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "風のうへに星のひかりは冴えながらわざともふらぬ霰をぞ聞く",
   "translation": "In the wind above,\nstarlight\nis cold and clear.\nI listen to hail\ntoo slight to see falling.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shūigūsō ingai; Bunshū hyakushu (1218)",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/teika.html",
   "source_label": "千人万首: 藤原定家 (unannotated selection)",
   "imagery": [
    "stars",
    "wind",
    "hail",
    "listening"
   ],
   "notes": "Exact poem number not supplied by the consulted source.",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shūi gusō ingai"
  },
  "teika-26": {
   "id": "teika-26",
   "poet": "Fujiwara no Teika",
   "author_japanese": "藤原定家",
   "author_dates": "1162–1241",
   "form": "waka",
   "japanese": "来ぬ人を松帆の浦の夕なぎに焼くやもしほの身もこがれつつ",
   "translation": "Waiting for someone\nwho does not come—\nat Matsuho, in the evening calm,\nthey burn seaweed for salt;\nso does my body burn.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Ogura Hyakunin isshu; Shinchokusenwakashū",
   "poem_number": "97; 849",
   "source_url": "https://hyakuninisshu.com/97.html",
   "source_label": "Fujiwara no Teika n.d.",
   "imagery": [
    "waiting",
    "evening calm",
    "sea",
    "salt fire",
    "longing"
   ],
   "notes": "Matsuho echoes matsu, to wait. The supplementary anthology number 849 is verified in the Asahi Teika text; a museum also confirms Hyakunin isshu 97 at https://www.meihaku.jp/hyakunin-isshu-kajin/kajin-fujiwarano-teika/ .",
   "verification": "Japanese text, attribution, and collection identifier checked in the linked source; English is a provisional independent rendering.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Hyakunin isshu 97"
  },
  "waka-other-01": {
   "id": "waka-other-01",
   "poet": "Saigyō",
   "form": "waka",
   "japanese": "岩間とぢし氷も今朝はとけそめて苔の下水みちもとむらん",
   "translation": "Ice that sealed the clefts\nstarts loosening this morning;\nunderneath the moss\nthe hidden water, surely,\nis searching for a path.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shin Kokin Wakashū",
   "poem_number": "7",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/saigyo.html",
   "source_label": "千人万首: 西行, Japanese text and anthology reference",
   "imagery": [
    "moss",
    "water",
    "rocks",
    "ice",
    "spring",
    "pathfinding"
   ],
   "notes": "New five-line working translation from the cited Japanese. The conjectural ending -ramu is retained as 'surely'; this is an imagined search for a watercourse. The text is not in the principal Sankashū recension according to the source.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 7",
   "author_japanese": "西行",
   "author_dates": "1118–1190"
  },
  "waka-other-02": {
   "id": "waka-other-02",
   "poet": "Saigyō",
   "form": "waka",
   "japanese": "吉野山こぞのしをりの道かへてまだ見ぬかたの花をたづねむ",
   "translation": "On Mount Yoshino\nI’ll leave the path I marked\nwith broken twigs last year\nand seek the blossoms\nin a place I haven’t seen.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shin Kokin Wakashū",
   "poem_number": "86",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/saigyo.html",
   "source_label": "千人万首: 西行, Japanese text and anthology reference",
   "imagery": [
    "paths",
    "branching",
    "returning",
    "cherry blossoms",
    "mountains",
    "spring"
   ],
   "notes": "New working translation. Shiori means trail marks made by breaking branches; it is not a book's bookmark here. Particularly suitable for a level offering a new route through familiar ground.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 86",
   "author_japanese": "西行",
   "author_dates": "1118–1190"
  },
  "waka-other-03": {
   "id": "waka-other-03",
   "poet": "Saigyō",
   "form": "waka",
   "japanese": "道の辺に清水ながるる柳蔭しばしとてこそ立ちとまりつれ",
   "translation": "Beside the road,\nclear water, willow shade—\nI had meant\nto stop here\nonly for a moment.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shin Kokin Wakashū",
   "poem_number": "262",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/saigyo.html",
   "source_label": "千人万首: 西行, Japanese text and anthology reference",
   "imagery": [
    "road",
    "water",
    "willow",
    "shade",
    "rest",
    "summer"
   ],
   "notes": "New working translation. 'I had meant' carries the implication that the short stop became a longer stay. The cited text begins 道の辺に; some witnesses have 道の辺の.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 262",
   "author_japanese": "西行",
   "author_dates": "1118–1190"
  },
  "waka-other-04": {
   "id": "waka-other-04",
   "poet": "Saigyō",
   "form": "waka",
   "japanese": "ゆくへなく月に心のすみすみて果てはいかにかならむとすらむ",
   "translation": "My heart grows clear,\nclearer still,\nendlessly in the moonlight.\nWhat will it become\nat last?",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by GPT-6 Astra (review pass) and Claude, 2026-10-04",
   "collection": "Sankashū",
   "poem_number": "353",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/saigyo.html",
   "source_label": "千人万首: 西行, Sankashū text and numbering",
   "imagery": [
    "moon",
    "clarity",
    "mind",
    "limitlessness",
    "ending"
   ],
   "notes": "New working translation. Number follows the Shinpen Kokka Taikan numbering identified by the source. Yukue naku can suggest directionlessness or absence of a limit.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Sankashū 353",
   "author_japanese": "西行",
   "author_dates": "1118–1190"
  },
  "waka-other-06": {
   "id": "waka-other-06",
   "poet": "Saigyō",
   "form": "waka",
   "japanese": "さびしさに堪へたる人のまたもあれな庵ならべむ冬の山里",
   "translation": "If only someone else\ncould bear this solitude—\nwe’d set our huts\nside by side\nin the winter mountain village.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shin Kokin Wakashū",
   "poem_number": "627",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/saigyo.html",
   "source_label": "千人万首: 西行, Japanese text and anthology references",
   "imagery": [
    "solitude",
    "paired huts",
    "winter",
    "mountain village",
    "companionship"
   ],
   "notes": "New working translation. Also Sankashū 513 in the cited numbering. The wish is for company able to share solitude, not for solitude to be abolished.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 627",
   "author_japanese": "西行",
   "author_dates": "1118–1190"
  },
  "waka-other-07": {
   "id": "waka-other-07",
   "poet": "Fujiwara no Shunzei",
   "form": "waka",
   "japanese": "夕されば野辺の秋風身にしみて鶉鳴くなり深草の里",
   "translation": "When evening comes,\nthe autumn wind across the fields\ncuts through me.\nQuail are calling\nin Fukakusa village.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Senzai Wakashū",
   "poem_number": "259",
   "source_url": "https://www.asahi-net.or.jp/~SG2H-ymst/yamatouta/sennin/syunzei2.html",
   "source_label": "千人万首: 藤原俊成, Japanese text and anthology reference",
   "imagery": [
    "wind",
    "fields",
    "quail",
    "dusk",
    "autumn",
    "village"
   ],
   "notes": "New working translation. Fukakusa is a place name that also suggests deep grass. The source records the Ise monogatari allusion behind the quail; the English leaves that allusion implicit.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Senzaishū 259",
   "author_japanese": "藤原俊成",
   "author_dates": "1114–1204"
  },
  "waka-other-08": {
   "id": "waka-other-08",
   "poet": "Fujiwara no Shunzei",
   "form": "waka",
   "japanese": "石ばしる水の白玉数見えて清滝川にすめる月影",
   "translation": "The white beads of water\ndashing on the rocks\ncan be counted,\nso clear the moonlight\non the Kiyotaki River.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "Senzai Wakashū",
   "poem_number": "284",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syunzei.html",
   "source_label": "千人万首: 藤原俊成, Japanese text and anthology reference",
   "imagery": [
    "water",
    "rocks",
    "moonlight",
    "droplets",
    "clarity",
    "counting"
   ],
   "notes": "New working translation. White jewels are droplets of spray made distinct by moonlight; Kiyotaki is retained as a river name.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Senzaishū 284",
   "author_japanese": "藤原俊成",
   "author_dates": "1114–1204"
  },
  "waka-other-09": {
   "id": "waka-other-09",
   "poet": "Fujiwara no Shunzei",
   "form": "waka",
   "japanese": "ひとり見る池の氷にすむ月のやがて袖にもうつりぬるかな",
   "translation": "Alone I watch\nthe moon clear\non the frozen pond,\nand soon it shines\nin my sleeve as well.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by Claude, 2026-10-04",
   "collection": "Shin Kokin Wakashū",
   "poem_number": "640",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syunzei.html",
   "source_label": "千人万首: 藤原俊成, Japanese text and anthology reference",
   "imagery": [
    "reflection",
    "ice",
    "pond",
    "moon",
    "solitude",
    "tears"
   ],
   "notes": "New working translation. The sleeve is wet with tears and so receives the reflected moon; this remains implicit, as in the Japanese. Utsuru combines transfer and reflection.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 640",
   "author_japanese": "藤原俊成",
   "author_dates": "1114–1204"
  },
  "waka-other-10": {
   "id": "waka-other-10",
   "poet": "Princess Shikishi",
   "form": "waka",
   "japanese": "山ふかみ春ともしらぬ松の戸にたえだえかかる雪の玉水",
   "translation": "Deep in the mountains,\nmy pinewood door\ndoes not yet know it’s spring:\none drop, then another,\npearls of melting snow.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shin Kokin Wakashū",
   "poem_number": "3",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syokusi.html",
   "source_label": "千人万首: 式子内親王, Japanese text and anthology reference",
   "imagery": [
    "threshold",
    "door",
    "snowmelt",
    "mountains",
    "spring",
    "droplets"
   ],
   "notes": "New working translation. Matsu no to may be a pine-board door or a gate made from pine branches. The poet's name is also read Shokushi.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 3",
   "author_japanese": "式子内親王",
   "author_dates": "d. 1201"
  },
  "waka-other-12": {
   "id": "waka-other-12",
   "poet": "Princess Shikishi",
   "form": "waka",
   "japanese": "かへりこぬ昔を今と思ひ寝の夢の枕ににほふ橘",
   "translation": "I sleep, thinking\nthe past that cannot return\nis here again;\norange blossom scents\nthe pillow of my dream.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shin Kokin Wakashū",
   "poem_number": "240",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syokusi.html",
   "source_label": "千人万首: 式子内親王, Japanese text and anthology reference",
   "imagery": [
    "returning",
    "memory",
    "citrus blossom",
    "scent",
    "dream",
    "summer"
   ],
   "notes": "New working translation. Tachibana is the native Japanese citrus; its blossom scent conventionally recalls someone from the past.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 240",
   "author_japanese": "式子内親王",
   "author_dates": "d. 1201"
  },
  "waka-other-13": {
   "id": "waka-other-13",
   "poet": "Eifukumon’in",
   "form": "waka",
   "japanese": "きりぎりす声はいづくぞ草もなきしらすの庭の秋の夜の月",
   "translation": "A cricket—\nwhere is that voice?\nNo grass grows here\nin the white-gravel courtyard\nunder the autumn moon.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Fūga Wakashū",
   "poem_number": "556",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/eihuku.html",
   "source_label": "千人万首: 永福門院, Japanese text and anthology reference",
   "imagery": [
    "white gravel",
    "garden",
    "moon",
    "cricket",
    "absence",
    "autumn"
   ],
   "notes": "New working translation. Shirasu is a white-sand or white-gravel court. Classical kirigirisu denotes a cricket, not necessarily the insect called kirigirisu in modern Japanese. Especially close to the game's actual setting.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Fūgashū 556",
   "author_japanese": "永福門院",
   "author_dates": "1271–1342"
  },
  "waka-other-14": {
   "id": "waka-other-14",
   "poet": "Eifukumon’in",
   "form": "waka",
   "japanese": "たきつ瀬や岩もとしろくよる花はながるとすれどまたかへるなり",
   "translation": "At the rapids,\nflowers gather white\nagainst a rock;\nthey start to drift away\nand then come back again.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Fūga Wakashū",
   "poem_number": "255",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/eihuku.html",
   "source_label": "千人万首: 永福門院, Japanese text and anthology reference",
   "imagery": [
    "water",
    "rock",
    "eddies",
    "returning",
    "fallen blossoms",
    "spring"
   ],
   "notes": "New working translation. The returning movement is explicit; an eddy explains it but is not named in the Japanese. Suitable for loops or the mirrored return.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Fūgashū 255",
   "author_japanese": "永福門院",
   "author_dates": "1271–1342"
  },
  "waka-other-15": {
   "id": "waka-other-15",
   "poet": "Eifukumon’in",
   "form": "waka",
   "japanese": "花の上にしばしうつろふ夕づく日入るともなしに影きえにけり",
   "translation": "For a little while\nthe evening sunlight lingers\non the blossoms;\nwithout seeming to set,\nits light has disappeared.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Fūga Wakashū",
   "poem_number": "199",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/eihuku.html",
   "source_label": "千人万首: 永福門院, Japanese text and anthology reference",
   "imagery": [
    "twilight",
    "blossoms",
    "light",
    "disappearance",
    "spring"
   ],
   "notes": "New working translation. The cited anthology is Fūgashū, not Gyokuyōshū; some popular pages misidentify it. The unnoticed loss of light supplies the action.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Fūgashū 199",
   "author_japanese": "永福門院",
   "author_dates": "1271–1342"
  },
  "waka-other-17": {
   "id": "waka-other-17",
   "poet": "Eifukumon’in",
   "form": "waka",
   "japanese": "ま萩ちる庭の秋風身にしみて夕日のかげぞ壁に消えゆく",
   "translation": "Bush-clover flowers fall.\nThe garden’s autumn wind\ngoes through me;\nevening sunlight fades\nupon the wall.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Fūga Wakashū",
   "poem_number": "478",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/eihuku.html",
   "source_label": "千人万首: 永福門院, Japanese text and anthology reference",
   "imagery": [
    "garden",
    "wall",
    "wind",
    "falling flowers",
    "sunlight",
    "autumn"
   ],
   "notes": "New working translation. Hagi is bush clover; this is autumn blossom fall, not spring cherry blossom.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Fūgashū 478",
   "author_japanese": "永福門院",
   "author_dates": "1271–1342"
  },
  "waka-other-18": {
   "id": "waka-other-18",
   "poet": "Izumi Shikibu",
   "form": "waka",
   "japanese": "春霞たつやおそきと山川の岩間をくぐる音きこゆなり",
   "translation": "Impatient, perhaps,\nfor the spring haze to rise,\nthe mountain stream\nworks its way between rocks;\nI hear the sound.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Goshūi Wakashū",
   "poem_number": "13",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/izumi_n.html",
   "source_label": "千人万首: 和泉式部, Japanese text and anthology reference",
   "imagery": [
    "water",
    "rocks",
    "spring haze",
    "passage",
    "sound"
   ],
   "notes": "New working translation. The opening gives the water an apparent impatience at the haze's delay; the ending locates the perception in hearing.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Goshūishū 13",
   "author_japanese": "和泉式部",
   "author_dates": "fl. c. 1000"
  },
  "waka-other-21": {
   "id": "waka-other-21",
   "poet": "Jakuren",
   "form": "waka",
   "japanese": "さびしさはその色としもなかりけり槙立つ山の秋の夕暮",
   "translation": "Solitude\nhas no color of its own.\nThe mountains\nstand thick with evergreens\nin the autumn dusk.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shin Kokin Wakashū",
   "poem_number": "361",
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/jakuren.html",
   "source_label": "千人万首: 寂蓮法師, Japanese text and anthology reference",
   "imagery": [
    "solitude",
    "colorlessness",
    "evergreens",
    "mountains",
    "autumn dusk"
   ],
   "notes": "New working translation. Maki here is a collective term for conifers such as cedar and cypress; translating it specifically as pine would be needlessly narrow. One of the three celebrated autumn-dusk poems.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Shinkokinshū 361",
   "author_japanese": "寂蓮",
   "author_dates": "c. 1139–1202"
  },
  "waka-other-23": {
   "id": "waka-other-23",
   "poet": "Jien",
   "form": "waka",
   "japanese": "旅の世にまた旅寝して草まくら夢のうちにも夢をみるかな",
   "translation": "On this journey of a life\nI sleep away from home again,\ngrass for my pillow—\nand even within a dream\nI dream.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Senzai Wakashū",
   "poem_number": "533",
   "source_url": "https://www.asahi-net.or.jp/~SG2H-YMST/yamatouta/sennin/jien.html",
   "source_label": "千人万首: 慈円, Japanese text and anthology reference",
   "imagery": [
    "journey",
    "rest",
    "grass pillow",
    "dream",
    "nested worlds"
   ],
   "notes": "New working translation. Life itself is treated as a journey and a dream, with literal traveling and dreaming nested within it.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Senzaishū 533",
   "author_japanese": "慈円",
   "author_dates": "1155–1225"
  },
  "waka-other-25": {
   "id": "waka-other-25",
   "poet": "Ki no Tsurayuki",
   "form": "waka",
   "japanese": "人はいさ心も知らずふるさとは花ぞ昔の香ににほひける",
   "translation": "As for people—\nI cannot know their hearts.\nHere at this old place,\nthe plum blossoms\nstill give their old scent.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Kokin Wakashū",
   "poem_number": "42",
   "source_url": "https://www.karuta.or.jp/karuta-everyday/2815/",
   "source_label": "All Japan Karuta Association: 人はいさ, Japanese poem and original anthology reference",
   "imagery": [
    "plum",
    "scent",
    "returning",
    "memory",
    "unchanged place",
    "spring"
   ],
   "notes": "New working translation. Also Ogura Hyakunin Isshu 35. The headnote identifies the blossoms as plum and the old place as a previously visited lodging; furusato need not mean the poet's birthplace.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Kokinshū 42",
   "author_japanese": "紀貫之",
   "author_dates": "c. 872–945"
  },
  "waka-other-26": {
   "id": "waka-other-26",
   "poet": "Ōshikōchi no Mitsune",
   "form": "waka",
   "japanese": "心あてに折らばや折らむ初霜のおきまどはせる白菊の花",
   "translation": "If I pick one,\nit must be by guess—\nwhite chrysanthemums\nlost in\nthe first frost.",
   "translation_credit": "Working translation by GPT-6 Astra (2026), revised by GPT-6 Astra (review pass) and Claude, 2026-10-04",
   "collection": "Kokin Wakashū",
   "poem_number": "277",
   "source_url": "https://www.karuta.or.jp/karuta-everyday/2791/",
   "source_label": "All Japan Karuta Association: 心あてに, Japanese poem and original anthology reference",
   "imagery": [
    "frost",
    "white flowers",
    "chrysanthemums",
    "concealment",
    "indistinguishability",
    "autumn"
   ],
   "notes": "New working translation. Also Ogura Hyakunin Isshu 29. The conceit is that frost and white petals cannot readily be distinguished.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Kokinshū 277",
   "author_japanese": "凡河内躬恒",
   "author_dates": "fl. c. 900"
  },
  "waka-other-28": {
   "id": "waka-other-28",
   "poet": "Harumichi no Tsuraki",
   "form": "waka",
   "japanese": "山川に風のかけたるしがらみは流れもあへぬもみぢなりけり",
   "translation": "Across the mountain stream\nthe wind has built a weir:\nfallen autumn leaves\nthat could not\nmake their way downstream.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Kokin Wakashū",
   "poem_number": "303",
   "source_url": "https://www.karuta.or.jp/karuta-everyday/2803/",
   "source_label": "All Japan Karuta Association: 山川に, Japanese poem and original anthology reference",
   "imagery": [
    "stream",
    "wind",
    "barrier",
    "leaves",
    "autumn",
    "blocked passage"
   ],
   "notes": "New working translation. Also Ogura Hyakunin Isshu 32. Shigarami is a weir or fence across a stream; it is formed here by a jam of leaves, not a constructed object.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "Kokinshū 303",
   "author_japanese": "春道列樹",
   "author_dates": "d. 920"
  },
  "waka-other-29": {
   "id": "waka-other-29",
   "poet": "Ryōkan",
   "form": "waka",
   "japanese": "山かげの岩間をつたふ苔水のかすかに我はすみわたるかも",
   "translation": "In the mountain shade,\nwater threads the moss\nbetween the rocks;\nlike that faint flow\nI quietly dwell.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Ryōkan Oshō kashū (collected waka; cited edition edited by Sōma Gyofū)",
   "source_url": "https://ryoukan-w.info/?page_id=1315",
   "source_label": "良寛ワールド: 良寬 珠玉の言葉 11～20",
   "imagery": [
    "moss",
    "water",
    "rocks",
    "shade",
    "solitude",
    "quiet dwelling"
   ],
   "notes": "New working translation. No poem number verified. Sumi combines dwelling and being clear; the English prioritizes dwelling, with the clear-water implication retained by the image.",
   "reverified": "yes, 2026-10-04: checked against source_url, which prints it in modern kana as 「やまかげの 岩間を伝う 苔水の かすかに我は すみわたるかも」",
   "display_source": "",
   "author_japanese": "良寛",
   "author_dates": "1758–1831"
  },
  "waka-other-30": {
   "id": "waka-other-30",
   "poet": "Ryōkan",
   "form": "waka",
   "japanese": "形見とて何か残さむ春は花山ほととぎす秋はもみぢ葉",
   "translation": "What could I leave\nto remember me by?\nSpring blossoms,\nthe mountain cuckoo,\nautumn leaves.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Ryōkan zenshū, lower volume (Tokyo Sōgensha, 1959), p. 195; surviving calligraphy",
   "poem_number": "1238",
   "source_url": "https://crd.ndl.go.jp/reference/entry/index.php?id=1000101479&page=ref_view",
   "source_label": "Osaka Prefectural Central Library, NDL Collaborative Reference Database: Ryōkan textual variants",
   "imagery": [
    "inheritance",
    "blossoms",
    "cuckoo",
    "autumn leaves",
    "seasons",
    "farewell"
   ],
   "notes": "New working translation. The library documents this precise mountain-cuckoo version as no. 1238, and a summer-cuckoo variant as no. 1239. Do not silently blend the versions or present this as a securely dated deathbed composition.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "display_source": "",
   "author_japanese": "良寛",
   "author_dates": "1758–1831"
  },
  "new-basho-sweep-garden": {
   "id": "new-basho-sweep-garden",
   "poet": "Matsuo Bashō",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694",
   "form": "haiku",
   "japanese": "庭掃て出ばや寺に散柳",
   "translation": "Let me sweep the garden\nbefore I go—\nwillow leaves fall in the temple.",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "collection": "おくのほそ道 (Oku no hosomichi), Zenshōji, journey of 1689",
   "poem_number": null,
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/okunohosomichi/okuno34.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database: Oku no hosomichi, 曾良との別れ",
   "display_source": "Oku no hosomichi",
   "notes": "Leaving the temple where he had lodged, Bashō sweeps its garden as a guest's thanks; the young monks had run after him with paper and inkstone. The 1903 Zenshū prints 精舍 for 寺.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "waka-other-11": {
   "id": "waka-other-11",
   "poet": "Princess Shikishi",
   "form": "waka",
   "japanese": "窓ちかき竹の葉すさぶ風の音にいとどみじかきうたたねの夢",
   "translation": "By the window,\nwind sets bamboo leaves\nrustling;\nmy dozing dream\nis shorter still.",
   "translation_credit": "Working translation by GPT-6 Astra (2026); reviewed by Claude, 2026-10-04",
   "collection": "Shin Kokin Wakashū",
   "poem_number": 256,
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/syokusi.html",
   "source_label": "千人万首: 式子内親王, Japanese text and anthology reference",
   "imagery": [
    "bamboo",
    "wind",
    "window",
    "dream",
    "rest",
    "summer"
   ],
   "notes": "New working translation. The sound interrupts a light sleep; no waking character or specific dream content has been added.",
   "author_japanese": "式子内親王",
   "author_dates": "d. 1201",
   "display_source": "Shinkokinshū 256",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url (新古256, headnote 百首の歌たてまつりし時)"
  },
  "new-tomonori-hi33": {
   "id": "new-tomonori-hi33",
   "poet": "Ki no Tomonori",
   "author_japanese": "紀友則",
   "author_dates": "d. c. 905",
   "form": "waka",
   "japanese": "ひさかたの 光のどけき 春の日に しづ心なく 花の散るらむ",
   "translation": "In the tranquil light\nof a spring day\nunder the wide sky—\nwhy, with no calm in their hearts,\ndo the blossoms fall?",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "collection": "Ogura Hyakunin Isshu; Kokin Wakashū 84",
   "poem_number": 33,
   "source_url": "https://www.bunkanet.jp/manabi/hyakunin-isshu-karuta/element/33/",
   "source_label": "Bunka Net n.d., Hyakunin Isshu 33",
   "display_source": "Hyakunin isshu 33",
   "notes": "しづ心 is a calm, settled heart; the garden Calm scores exactly that.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-ryokan-new-pond": {
   "id": "new-ryokan-new-pond",
   "poet": "Ryōkan",
   "author_japanese": "良寛",
   "author_dates": "1758–1831",
   "form": "haiku",
   "japanese": "新池や蛙とびこむ音もなし",
   "translation": "The new pond—\na frog jumps in;\nnot a sound.",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "collection": "Ryōkan zenkushū (ed. Tanikawa Toshirō), as cited by the source",
   "poem_number": null,
   "source_url": "https://sakuramitih31.livedoor.blog/archives/41933640.html",
   "additional_source_url": "https://idea1616.com/ryokan-haiku/",
   "source_label": "名歌鑑賞: 新池や蛙とびこむ音もなし (blog, citing 谷川敏朗『良寛全句集』)",
   "display_source": "",
   "notes": "Ryōkan's answer to Bashō's old pond: the new pond has no sound. Attribution is the traditional one; the text was checked against two secondary pages, not a critical edition.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url (secondary sources)"
  },
  "new-basho-summer-room": {
   "id": "new-basho-summer-room",
   "poet": "Matsuo Bashō",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694",
   "form": "haiku",
   "japanese": "山も庭も動き入るゝや夏座敷",
   "translation": "Mountain and garden\ncome moving in—\nthe summer room.",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "collection": "芭蕉俳句全集 (1903), 夏・夏座敷; composed 1689 at Kurobane",
   "poem_number": null,
   "source_url": "https://ja.wikisource.org/wiki/芭蕉俳句全集",
   "source_label": "Bashō haiku zenshū, edited by Ōtsuka Kōzan (1903), summer section",
   "display_source": "Kurobane, 1689",
   "notes": "A greeting verse for his host's open summer room at Kurobane, on the Oku no hosomichi journey: the view enters the room.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-basho-hida": {
   "id": "new-basho-hida",
   "poet": "Matsuo Bashō",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694",
   "form": "haiku",
   "japanese": "涼しさを飛騨の工が指図かな",
   "translation": "Coolness—\ndrawn to the plan\nof a Hida carpenter.",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "collection": "Letter to Sugiyama Sanpū, Genroku 7, intercalary fifth month, day 11 (1694)",
   "poem_number": null,
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/letter/sanpu1.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database: letter to Sanpū (1)",
   "display_source": "letter of 1694",
   "notes": "指図 is an architect's drawing or direction; the carpenters of Hida were the proverbial master builders. Bashō later preferred the variant 涼しさの指図に見ゆる住まゐかな.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-basho-hikarido": {
   "id": "new-basho-hikarido",
   "poet": "Matsuo Bashō",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694",
   "form": "haiku",
   "japanese": "五月雨の降のこしてや光堂",
   "translation": "The summer rains—\ndid they spare only this?\nThe Hall of Light.",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "collection": "おくのほそ道 (Oku no hosomichi), Hiraizumi, journey of 1689",
   "poem_number": null,
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/okunohosomichi/okuno20.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database: Oku no hosomichi, 平泉",
   "display_source": "Oku no hosomichi",
   "notes": "The Konjikidō at Chūsonji, kept whole for five centuries while everything around it fell; 降りのこす is to leave something unrained-on.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-buson-dew-thorns": {
   "id": "new-buson-dew-thorns",
   "poet": "Yosa Buson",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784",
   "form": "haiku",
   "japanese": "白露や茨の刺にひとつづつ",
   "translation": "White dew—\non each thorn of the bramble,\none drop.",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "collection": "蕪村句集 (Buson kushū), autumn",
   "poem_number": null,
   "source_url": "https://idea1616.com/buson/",
   "additional_source_url": "https://ameblo.jp/mtada99/entry-12474301599.html",
   "source_label": "与謝蕪村の俳句 102選 (list), and a commentary page",
   "display_source": "Buson kushū",
   "notes": "Checked against two secondary pages (the museum and bonjin lists used elsewhere do not carry it).",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url (secondary sources)"
  },
  "new-buson-fuji-leaves": {
   "id": "new-buson-fuji-leaves",
   "poet": "Yosa Buson",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784",
   "form": "haiku",
   "japanese": "不二ひとつうづみ残して若葉かな",
   "translation": "Young leaves—\nthey have buried everything\nbut Fuji.",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "collection": "蕪村句集 (Buson kushū), summer",
   "poem_number": null,
   "source_url": "https://bonjin-ultra.com/buson.htm",
   "source_label": "bonjin-ultra: 蕪村の俳句, summer",
   "display_source": "Buson kushū",
   "notes": "うづみ残す: to leave one thing unburied.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-basho-ishiyama": {
   "id": "new-basho-ishiyama",
   "poet": "Matsuo Bashō",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694",
   "form": "haiku",
   "japanese": "石山の石より白し秋の風",
   "translation": "Whiter\nthan the stones of Stone Mountain—\nthe autumn wind.",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "collection": "おくのほそ道 (Oku no hosomichi), Natadera, journey of 1689",
   "poem_number": null,
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/okunohosomichi/okuno33.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database: Oku no hosomichi, 那谷",
   "display_source": "Oku no hosomichi",
   "notes": "Autumn's color is white in the five-phase scheme; the wind outdoes the white rocks of Nata (or of Ishiyama) in it.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-issa-snail": {
   "id": "new-issa-snail",
   "poet": "Kobayashi Issa",
   "author_japanese": "小林一茶",
   "author_dates": "1763–1828",
   "form": "haiku",
   "japanese": "かたつぶりそろそろ登れ富士の山",
   "translation": "Little snail,\nslowly, slowly climb—\nMount Fuji.",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "collection": "一茶発句集 (Issa hokkushū), Bunsei edition (1829, posthumous)",
   "poem_number": null,
   "source_url": "https://crd.ndl.go.jp/reference/entry/index.php?page=ref_view&id=1000180048",
   "source_label": "NDL Collaborative Reference Database: the collections that carry the snail verse",
   "display_source": "Issa hokkushū (1829)",
   "notes": "Not in Issa's own diaries; first printed two years after his death (the Kaei edition has 不二の山). Text as the Bunsei edition prints it, per the reference record.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-basho-one-field": {
   "id": "new-basho-one-field",
   "poet": "Matsuo Bashō",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694",
   "form": "haiku",
   "japanese": "田一枚植て立去る柳かな",
   "translation": "One field planted—\nand I leave\nthe willow.",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "collection": "おくのほそ道 (Oku no hosomichi), the Yugyō willow, journey of 1689",
   "poem_number": null,
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/okunohosomichi/okuno07.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database: Oku no hosomichi, 遊行柳",
   "display_source": "Oku no hosomichi",
   "notes": "Under Saigyō's willow; who plants the field (the women, or Bashō's party) is left open, as is who leaves.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-buson-dry-spring": {
   "id": "new-buson-dry-spring",
   "poet": "Yosa Buson",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784",
   "form": "haiku",
   "japanese": "柳ちり清水かれ石ところどころ",
   "translation": "Willow leaves fallen,\nthe spring run dry,\nstones here and there.",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "collection": "蕪村集 (as indexed by the museum)",
   "poem_number": null,
   "source_url": "https://www2.city.kurashiki.okayama.jp/musnat/plant/bungakusakuhin/busonsyuu.htm",
   "source_label": "Kurashiki Museum of Natural History: 蕪村集の植物",
   "display_source": "",
   "notes": "An answer to Saigyō's roadside willow and its clear water (Stepping stone): the water gone, the stones left.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-ryokan-thief-moon": {
   "id": "new-ryokan-thief-moon",
   "poet": "Ryōkan",
   "author_japanese": "良寛",
   "author_dates": "1758–1831",
   "form": "haiku",
   "japanese": "盗人に取り残されし窓の月",
   "translation": "The thief\nleft it behind:\nthe moon in the window.",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "collection": "Ryōkan's hokku; the anecdote of the thief at his hut",
   "poem_number": null,
   "source_url": "https://nichi-zen.site/remaining-moon/",
   "additional_source_url": "https://www.jomo-news.co.jp/articles/-/5483",
   "source_label": "日常実践の禅: 盗人に取り残されし窓の月",
   "display_source": "",
   "notes": "The thief found nothing worth taking (one telling has Ryōkan roll over so the man could take his bedding). Attribution is the traditional one; checked against secondary pages.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url (secondary sources)"
  },
  "new-manyo-1068": {
   "id": "new-manyo-1068",
   "poet": "Kakinomoto no Hitomaro",
   "author_japanese": "柿本人麻呂歌集",
   "author_dates": "fl. c. 680–700",
   "form": "waka",
   "japanese": "天の海に雲の波立ち月の船星の林に榜ぎ隠る見ゆ",
   "translation": "On the sea of heaven\nthe waves of cloud rise;\nthe boat of the moon\nrows into the forest of stars\nand is hidden from sight.",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "collection": "Man'yōshū, book 7 (from the Hitomaro Collection)",
   "poem_number": 1068,
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/manyok/manyok07.html",
   "source_label": "訓読万葉集 巻7 (Yamato Uta), after Kamochi Masazumi's 萬葉集古義",
   "display_source": "Man'yōshū 1068",
   "notes": "Headed 天を詠める and marked as from the Kakinomoto no Hitomaro Collection, whose poems are not all his; the source prints 榜ぎ for 漕ぎ.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-basho-sea-darkens": {
   "id": "new-basho-sea-darkens",
   "poet": "Matsuo Bashō",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694",
   "form": "haiku",
   "japanese": "海くれて鴨のこゑほのかに白し",
   "translation": "The sea darkens;\nthe voices of the ducks\nfaintly white.",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "collection": "野ざらし紀行 (Nozarashi kikō), Atsuta, winter 1684",
   "poem_number": null,
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/nozarasi/nozara23.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database: Nozarashi kikō, 海邊に日暮して",
   "display_source": "Nozarashi kikō",
   "notes": "Headnote 海邊に日暮して. A sound given a color: the source calls it one of his finest verses.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-basho-moon-sand": {
   "id": "new-basho-moon-sand",
   "poet": "Matsuo Bashō",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694",
   "form": "haiku",
   "japanese": "月清し遊行のもてる砂の上",
   "translation": "The moon is clear\non the sand\nthe pilgrim monk carried.",
   "translation_credit": "New translation by Claude, 2026-10-04",
   "collection": "おくのほそ道 (Oku no hosomichi), Tsuruga (Kehi shrine), journey of 1689",
   "poem_number": null,
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/okunohosomichi/okuno38.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database: Oku no hosomichi, 敦賀",
   "display_source": "Oku no hosomichi",
   "notes": "遊行 is the Yugyō Shōnin, head of the Ji sect: the second, Ta'a, carried sand himself to fill the shrine's muddy grounds, and his successors keep the rite (遊行の砂持ち). The 1903 Zenshū prints 月淸し遊行の持てる砂の上.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-shiki-bridge-fish": {
   "id": "new-shiki-bridge-fish",
   "poet": "Masaoka Shiki",
   "author_japanese": "正岡子規",
   "author_dates": "1867–1902",
   "composed": 1895,
   "form": "haiku",
   "japanese": "橋蹈めば魚沈みけり春の水",
   "translation": "I step on the bridge—\nthe fish sink.\nSpring water.",
   "translation_credit": "Working translation by GPT-6 Astra (2026, review pass), revised by Claude, 2026-10-04",
   "collection": "獺祭書屋俳句帖抄 (Dassai shooku haiku chō shō), spring 1895",
   "poem_number": null,
   "source_url": "https://mukei-r.net/poem-masaoka/00-haiku-dassai1.htm",
   "source_label": "獺祭書屋俳句帖抄 (transcription), 明治28年 春",
   "display_source": "1895",
   "notes": "The source prints 沈みけり (Astra's draft had the fish sink \"deeper\"). Composed 1895; the poet died in 1902, so this is the one poem whose author outlived 1900.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-fushimi-lotus-dew": {
   "id": "new-fushimi-lotus-dew",
   "poet": "Emperor Fushimi",
   "author_japanese": "伏見院",
   "author_dates": "1265–1317",
   "form": "waka",
   "japanese": "こぼれ落つる池のはちすの白露は浮葉の玉とまたなりにけり",
   "translation": "Dew spills\nfrom the pond's lotus leaves;\non the floating leaves\nit becomes\na jewel again.",
   "translation_credit": "Working translation by GPT-6 Astra (2026, review pass), revised by Claude, 2026-10-04",
   "collection": "Gyokuyō Wakashū (headnote: 百首御歌の中に、蓮を)",
   "poem_number": null,
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/saijiki/hasunoha.html",
   "source_label": "和歌歳時記: 蓮葉 (Yamato Uta)",
   "display_source": "Gyokuyōshū",
   "notes": "Lotus, not water lily; the number in the Gyokuyōshū is not given by the source.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-tamemasa-carp-lotus": {
   "id": "new-tamemasa-carp-lotus",
   "poet": "Reizei Tamemasa",
   "author_japanese": "冷泉為尹",
   "author_dates": "1361–1417",
   "form": "waka",
   "japanese": "池水に藻臥しの鮒や乱るらん蓮のうき葉のゆるぎ立ちぬる",
   "translation": "Hidden in the pondweed,\nare the carp stirring?\nThe lotus leaves\nfloating on the water\nhave begun to tremble.",
   "translation_credit": "Working translation by GPT-6 Astra (2026, review pass); reviewed by Claude, 2026-10-04",
   "collection": "Tamemasa senshu (為尹千首), topic: pond lotus",
   "poem_number": null,
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/saijiki/hasunoha.html",
   "source_label": "和歌歳時記: 蓮葉 (Yamato Uta)",
   "display_source": "Tamemasa senshu",
   "notes": "鮒 is the crucian carp. Astra credited the poem to Reizei Tamehide; the source gives 為尹 (Tamemasa, Tamehide's grandson).",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-basho-grass-hut": {
   "id": "new-basho-grass-hut",
   "poet": "Matsuo Bashō",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694",
   "form": "haiku",
   "japanese": "草の戸も住替る代ぞひなの家",
   "translation": "Even this grass hut\nchanges hands—\na house of dolls.",
   "collection": "Oku no hosomichi (opening hokku)",
   "poem_number": null,
   "source_url": "https://basho-yamadera.com/en/yamadera/kusanotomo",
   "imagery": [
    "grass hut",
    "new occupants",
    "dolls",
    "change"
   ],
   "notes": "",
   "verification": "",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url",
   "translation_credit": "Working translation by GPT-6 Astra (2026, review pass), revised by Claude, 2026-10-04",
   "display_source": "Oku no hosomichi"
  },
  "waka-other-05": {
   "id": "waka-other-05",
   "poet": "Saigyō",
   "form": "waka",
   "japanese": "心なき身にもあはれは知られけり鴫たつ沢の秋の夕暮",
   "translation": "Even one like me,\nwho has set his heart aside,\nfeels it after all—\na snipe lifts from the marsh\nat autumn dusk.",
   "translation_credit": "Working translation by GPT-6 Astra (2026, review pass); reviewed by Claude, 2026-10-04",
   "collection": "Shin Kokin Wakashū",
   "poem_number": 362,
   "source_url": "https://www.asahi-net.or.jp/~sg2h-ymst/yamatouta/sennin/saigyo.html",
   "source_label": "千人万首: 西行, Japanese text and anthology references",
   "imagery": [
    "marsh",
    "bird",
    "autumn",
    "dusk",
    "detachment",
    "feeling"
   ],
   "notes": "New working translation. Also Sankashū 470 in the cited numbering. Kokoro naki can mean a renunciant freed of attachments or, more modestly, someone supposedly insensitive to beauty; this translation chooses the renunciant reading. One of the three celebrated autumn-dusk poems.",
   "author_japanese": "西行",
   "author_dates": "1118–1190",
   "display_source": "Shinkokinshū 362",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-buson-dancers": {
   "id": "new-buson-dancers",
   "poet": "Yosa Buson",
   "author_japanese": "与謝蕪村",
   "author_dates": "1716–1784",
   "form": "haiku",
   "japanese": "四五人に月落かかるおどりかな",
   "translation": "Four or five dancers—\nthe moon\nfalls toward them.",
   "translation_credit": "Working translation by GPT-6 Astra (2026, review pass); reviewed by Claude, 2026-10-04",
   "collection": "Inscribed on his painting 踊図 (Dancers; Agency for Cultural Affairs)",
   "poem_number": null,
   "source_url": "https://www.kyuhaku.jp/exhibition/exhibition_s50.html",
   "source_label": "Kyushu National Museum: 王羲之と日本の書 (exhibit list)",
   "display_source": "on his painting 踊図",
   "notes": "The Bon dance; 月落ちかかる is the moon sinking low over the dancers.",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  },
  "new-basho-moon-clouds": {
   "id": "new-basho-moon-clouds",
   "poet": "Matsuo Bashō",
   "author_japanese": "松尾芭蕉",
   "author_dates": "1644–1694",
   "form": "haiku",
   "japanese": "雲折々人をやすむる月見かな",
   "translation": "Watching the moon—\nevery so often a cloud\nlets us rest.",
   "translation_credit": "Working translation by GPT-6 Astra (2026, review pass); reviewed by Claude, 2026-10-04",
   "collection": "春の日 (Haru no hi), 1686",
   "poem_number": null,
   "source_url": "https://www2.yamanashi-ken.ac.jp/~itoyo/basho/shitibusyu/harunohi.htm",
   "source_label": "Yamanashi Prefectural University, Bashō database: 俳諧七部集, 春の日",
   "display_source": "Haru no hi (1686)",
   "notes": "",
   "reverified": "yes, 2026-10-04: Japanese text and attribution checked against source_url"
  }
 }
}
  • poems/astra-samon-poem-selection.json · 452.1 kB · data; in the .zip

The sounds: the recordings' credits and their encoder (the synthesis is in the prototype) ( sound/ )

sound/README.md · 35 lines · 3.2 kB 
# Sound

The page's sounds are of two kinds.

The bamboo pipe's drips, its hollow knock as it tips, the _bonshō_ at the close of a world, the seal's _tok_, the wind
(a bed with gusts), the birds by day and crickets by night (by the local clock), the footsteps, and the rake's scrape (a
looping synthesized crackle through a bandpass, driven frame by frame by the reveal's stroke speed, with a sideways rasp in
bends) are synthesized in the page (the `SND` block in `proto/samon.template.html`: decaying sines, filtered noise and
crackle, a short feedback delay for the basin's echo). They cost nothing to carry. `renderSound` (a test hook) renders any
sequence of them offline to WAV, which is how the tests check that each is audible and how they were listened to. A volume
slider and a haptics switch (`navigator.vibrate`, 8 ms per cell, shown only where it exists) sit beside the Sound switch;
a speaker button in the header (and the M key) is the same switch. The page sets `navigator.audioSession.type = "ambient"`
where that API exists (Safari 17), registers Media Session pause/stop/play handlers, and shows the speaker in red when the
browser keeps the context from running after a gesture (`soundBlocked`: Firefox's `getAutoplayPolicy("audiocontext")` saying
"disallowed", or a context still not running 1.5 s after the gesture). There is no CSS media query for a sound preference.

The shakuhachi is real: recordings in the public domain, listed in `tracks.json` with their credits and sources.
`make_sound_bundle.py` measures each one's loudness (EBU R128), brings it to −20 LUFS with a static gain (no dynamics
processing), encodes it as mono MP3 (LAME VBR 6; MP3 because desktop Safari still does not play Opus in Ogg), and writes
`out/sound_pieces.json`, which `proto/build.py` and `standalone/build_standalone.py` write into the page as inert
`<script type="text/plain" class="piece">` elements whose text is a `data:audio/mpeg;base64,...` URL, with the credits in
`data-` attributes. The page reads them (`loadPieces`) and plays one after a world's bell, or now and then (never within
a quarter-hour of the last), through a `MediaElementSource` with a 3 s fade in and a 4 s fade out; the scroll of poems
credits each with a link to its source.

The data: URLs are also what a SingleFile/gwtar step consumes: Gwern's `deconstruct_singlefile.php` moves every data: URL
out of the page into an appended archive and leaves a (quoted) path in its place. `samonAsset` in the template resolves a
piece's text whether it is still a data: URL, a plain path beside an extracted page, or a member of a gwtar archive (through
the gwtar loader's own globals, in both of its modes); the single-file edition's code .zip travels the same way.

Originals are not in this tree (the first is 3.6 MB and on Commons): to rebuild, put the files named by `tracks.json`
under `sound/originals/` and run `python3 sound/make_sound_bundle.py`, then the two builds.

Credits so far: _Shika no Tōne_ (鹿の遠音), Araki Kodō III, Victor 13029, 1925–35;
https://commons.wikimedia.org/wiki/File:Shikanotoone_new.ogg (public domain; de-noised by Torsodog, 2008).
sound/make_sound_bundle.py · 75 lines · 3.8 kB — Encode the shakuhachi recordings for the page. 
#!/usr/bin/env python3
"""Encode the shakuhachi recordings for the page.

Reads sound/tracks.json; for each track, measures the integrated loudness (EBU R128), brings it to the target with a
static gain (no dynamics processing: honkyoku lives on its silences), and encodes it as mono MP3 (universally playable:
Opus in Ogg is still not played by desktop Safari). Writes out/sound/<file> for listening and out/sound_pieces.json,
which proto/build.py and standalone/build_standalone.py write into the page as data: URLs in inert
<script type="text/plain" class="piece"> elements; a gwtar build (sound/gwtar/) then moves those into its archive.

Usage: python3 sound/make_sound_bundle.py [--quality 6] [--rate 32000]
Requires ffmpeg with libmp3lame.
"""
import argparse, base64, json, os, re, subprocess, sys

ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
SPEC = os.path.join(ROOT, "sound", "tracks.json")
OUT_DIR = os.path.join(ROOT, "out", "sound")
OUT_JSON = os.path.join(ROOT, "out", "sound_pieces.json")


def run(cmd):
    return subprocess.run(cmd, check=True, capture_output=True, text=True)


def probe(path):
    r = run(["ffprobe", "-hide_banner", "-loglevel", "error", "-show_entries", "format=duration", "-of", "json", path])
    return float(json.loads(r.stdout)["format"]["duration"])


def loudness(path):
    """Integrated loudness (LUFS) and true peak (dBTP) of a file, from ffmpeg's loudnorm measurement pass."""
    r = subprocess.run(["ffmpeg", "-hide_banner", "-nostats", "-i", path, "-af", "loudnorm=print_format=json", "-f", "null", "-"],
                       capture_output=True, text=True, check=True)
    m = re.search(r"\{.*\}", r.stderr, re.S)
    j = json.loads(m.group(0))
    return float(j["input_i"]), float(j["input_tp"])


def encode(track, target_lufs, quality, rate):
    src = os.path.join(ROOT, track["input"])
    dst = os.path.join(OUT_DIR, track["file"])
    lufs, tp = loudness(src)
    gain = target_lufs - lufs
    if tp + gain > -1.0:                 # a static gain that would clip is capped; the piece is then a little quieter than the target
        gain = -1.0 - tp
    run(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y", "-i", src, "-ac", "1", "-ar", str(rate),
         "-af", f"volume={gain:.2f}dB", "-codec:a", "libmp3lame", "-q:a", str(quality), "-id3v2_version", "0", "-write_xing", "1", dst])
    dur = probe(dst)
    out_lufs, out_tp = loudness(dst)
    print(f"{track['file']}: {dur:.0f} s, {os.path.getsize(dst):,} bytes; {lufs:.1f} LUFS → {out_lufs:.1f} LUFS (gain {gain:+.1f} dB, peak {out_tp:.1f} dBTP)")
    return dst, dur


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--quality", type=int, default=6, help="LAME VBR quality (0 best … 9 smallest); 6 ≈ 55 kbps mono")
    ap.add_argument("--rate", type=int, default=32000, help="sample rate (Hz)")
    a = ap.parse_args()
    spec = json.load(open(SPEC))
    os.makedirs(OUT_DIR, exist_ok=True)
    pieces = []
    for tr in spec["tracks"]:
        for k in ("input", "file", "title", "player", "license", "source"):
            if not tr.get(k):
                sys.exit(f"track without {k}: {tr}")
        dst, dur = encode(tr, spec.get("target_lufs", -20), a.quality, a.rate)
        data = open(dst, "rb").read()
        pieces.append({"file": tr["file"], "mime": "audio/mpeg", "bytes": len(data), "dur": round(dur, 1), "title": tr["title"], "player": tr["player"],
                       "license": tr["license"], "source": tr["source"], "gain": tr.get("gain", 0.35), "b64": base64.b64encode(data).decode()})
    json.dump(pieces, open(OUT_JSON, "w"), ensure_ascii=False)
    print(f"{len(pieces)} piece(s) → {OUT_JSON} ({os.path.getsize(OUT_JSON):,} bytes)")


if __name__ == "__main__":
    main()
sound/tracks.json · 16 lines · 1.2 kB 
{
  "about": "The shakuhachi recordings carried in the page (sound/make_sound_bundle.py encodes them; the builders write them into the page as data: URLs in inert <script class=piece> elements, which a gwtar build moves into its archive). Every recording must be public domain or freely licensed, and credited on the scroll of poems with its source.",
  "target_lufs": -20,
  "tracks": [
    {
      "input": "sound/originals/Shikanotoone_new.ogg",
      "file": "shika-no-tone-araki-kodo-iii.mp3",
      "title": "Shika no Tōne (鹿の遠音, The Distant Cry of Deer)",
      "player": "Araki Kodō III (1879–1935), Victor 13029, recorded 1925–35; noise reduced by Torsodog, 2008",
      "license": "public domain: a Japanese phonogram whose neighbouring rights have expired, and a derivative released as PD by its maker",
      "source": "https://commons.wikimedia.org/wiki/File:Shikanotoone_new.ogg",
      "gain": 0.35,
      "notes": "Honkyoku of the Kinko school, traditionally for two shakuhachi answering each other as deer in autumn. The Commons original is Ogg Vorbis, 6 min 15 s, 3.57 MB, two identical channels (L−R at −76 dB), −15.2 LUFS integrated, LRA 18 LU."
    }
  ]
}

Data: levels, experiment results, logs, and the campaign cache ( out/ )

out/table1.log · 6 lines · 995 bytes 
{'size': '3x3', 'rakings': 8, 'complete': 1, 'par': 4, 'at_par': 2, 'median': 5, 'max': 6, 'wide': 2, 'wide_turns': [(4, 1), (5, 1)]}
{'size': '4x4', 'rakings': 52, 'complete': 1, 'par': 6, 'at_par': 4, 'median': 8, 'max': 13, 'wide': 4, 'wide_turns': [(6, 1), (7, 1), (8, 1), (9, 1)]}
{'size': '5x5', 'rakings': 824, 'complete': 1, 'par': 8, 'at_par': 10, 'median': 13, 'max': 20, 'wide': 4, 'wide_turns': [(8, 1), (9, 1), (10, 1), (11, 1)]}
{'size': '6x6', 'rakings': 22144, 'complete': 1, 'par': 10, 'at_par': 24, 'median': 20, 'max': 31, 'wide': 6, 'wide_turns': [(10, 1), (11, 1), (12, 1), (13, 1), (14, 1), (15, 1)]}
{'size': '7x7', 'rakings': 1510446, 'complete': 1, 'par': 12, 'at_par': 66, 'median': 27, 'max': 42, 'wide': 6, 'wide_turns': [(12, 1), (13, 1), (14, 1), (15, 1), (16, 1), (17, 1)]}
{'size': '6x8', 'rakings': 899432, 'complete': 1, 'par': 11, 'at_par': 66, 'median': 26, 'max': 42, 'wide': 8, 'wide_turns': [(11, 1), (13, 1), (14, 1), (15, 2), (16, 1), (17, 1), (18, 1)]}
out/table2.log · 4 lines · 862 bytes 
k=0 n=300 parity-ok=0.57 | one stroke: solvable 0.570 unique 0.000 median 100000 capped 171 | no U-turns at all: solvable 0.000 unique 0.000 median 0 capped 0 | wide rake: solvable 0.570 unique 0.000 median 6 capped 0
k=2 n=300 parity-ok=0.43 | one stroke: solvable 0.423 unique 0.000 median 28326 capped 8 | no U-turns at all: solvable 0.000 unique 0.000 median 0 capped 0 | wide rake: solvable 0.177 unique 0.027 median 2 capped 0
k=4 n=300 parity-ok=0.34 | one stroke: solvable 0.200 unique 0.000 median 609.5 capped 0 | no U-turns at all: solvable 0.000 unique 0.000 median 0 capped 0 | wide rake: solvable 0.027 unique 0.003 median 3 capped 0
k=6 n=300 parity-ok=0.31 | one stroke: solvable 0.080 unique 0.000 median 95.5 capped 0 | no U-turns at all: solvable 0.000 unique 0.000 median 0 capped 0 | wide rake: solvable 0.003 unique 0.000 median 4 capped 0
out/test_grader_marks_seed11.log · 1 line · 75 bytes 
grader soundness with marks: ok 346 bad 0 (unique 34, solved by grader 34)
out/test_grader_marks_seed7.log · 1 line · 75 bytes 
grader soundness with marks: ok 365 bad 0 (unique 45, solved by grader 45)
out/test_marks_seed1.log · 2 lines · 110 bytes 
small (C vs brute force, streamed sols checked): ok 3000 bad 0 solvable 388
medium (C vs CP-SAT): ok 51 bad 0
out/test_marks_seed2.log · 2 lines · 110 bytes 
small (C vs brute force, streamed sols checked): ok 3000 bad 0 solvable 440
medium (C vs CP-SAT): ok 57 bad 0
out/test_side_pivot_seed1.log · 2 lines · 159 bytes 
small (C vs brute force): ok 1093 bad 0 solvable 127 (side gate 46, finished sand 50)
medium (C vs CP-SAT): ok 60 bad 0 (counts and histograms compared on 60)
out/test_side_pivot_seed2.log · 2 lines · 159 bytes 
small (C vs brute force): ok 2099 bad 0 solvable 284 (side gate 86, finished sand 99)
medium (C vs CP-SAT): ok 60 bad 0 (counts and histograms compared on 60)
  • out/campaign.json · 62.6 kB · data; in the .zip
  • out/campaign_cache/pairs.pkl · 2.0 kB · binary; in the .zip
  • out/campaign_cache/teaching.pkl · 1.4 kB · binary; in the .zip
  • out/campaign_cache/w1-55.pkl · 1.3 kB · binary; in the .zip
  • out/campaign_cache/w1-56.pkl · 1.4 kB · binary; in the .zip
  • out/campaign_cache/w1-66.pkl · 2.4 kB · binary; in the .zip
  • out/campaign_cache/w1-67.pkl · 1.8 kB · binary; in the .zip
  • out/campaign_cache/w1-77.pkl · 2.0 kB · binary; in the .zip
  • out/campaign_cache/w1-78.pkl · 2.2 kB · binary; in the .zip
  • out/campaign_cache/w2-66.pkl · 3.1 kB · binary; in the .zip
  • out/campaign_cache/w2-67.pkl · 4.2 kB · binary; in the .zip
  • out/campaign_cache/w2-77.pkl · 3.8 kB · binary; in the .zip
  • out/campaign_cache/w2-77x2.pkl · 2.0 kB · binary; in the .zip
  • out/campaign_cache/w2-78.pkl · 2.2 kB · binary; in the .zip
  • out/campaign_cache/w2-88.pkl · 1.3 kB · binary; in the .zip
  • out/campaign_cache/w3-67.pkl · 3.4 kB · binary; in the .zip
  • out/campaign_cache/w3-77.pkl · 3.8 kB · binary; in the .zip
  • out/campaign_cache/w3-77br.pkl · 1.1 kB · binary; in the .zip
  • out/campaign_cache/w3-78.pkl · 4.2 kB · binary; in the .zip
  • out/campaign_cache/w3-88.pkl · 1.3 kB · binary; in the .zip
  • out/campaign_cache/w4-g66.pkl · 2.5 kB · binary; in the .zip
  • out/campaign_cache/w4-g77.pkl · 4.0 kB · binary; in the .zip
  • out/campaign_cache/w4-g78.pkl · 3.6 kB · binary; in the .zip
  • out/campaign_cache/w4-gr.pkl · 2.1 kB · binary; in the .zip
  • out/campaign_cache/w4-p66.pkl · 1.6 kB · binary; in the .zip
  • out/campaign_cache/w4-p77.pkl · 2.1 kB · binary; in the .zip
  • out/campaign_cache/w4-p78.pkl · 2.4 kB · binary; in the .zip
  • out/campaign_cache/w4-pb.pkl · 1.1 kB · binary; in the .zip
  • out/campaign_cache/w4-ring.pkl · 1.7 kB · binary; in the .zip
  • out/campaign_cache/w5-55.pkl · 1.8 kB · binary; in the .zip
  • out/campaign_cache/w5-56.pkl · 1.4 kB · binary; in the .zip
  • out/campaign_cache/w5-66.pkl · 2.9 kB · binary; in the .zip
  • out/campaign_cache/w5-67.pkl · 3.2 kB · binary; in the .zip
  • out/campaign_cache/w5-77.pkl · 3.6 kB · binary; in the .zip
  • out/campaign_cache/w5-78.pkl · 2.1 kB · binary; in the .zip
  • out/campaign_cache/w5-88.pkl · 2.4 kB · binary; in the .zip
  • out/campaign_cache/w5-wb.pkl · 1.1 kB · binary; in the .zip
  • out/campaign_cache/w5-wr.pkl · 1.1 kB · binary; in the .zip
  • out/campaign_cache/w6-67.pkl · 2.5 kB · binary; in the .zip
  • out/campaign_cache/w6-77.pkl · 3.8 kB · binary; in the .zip
  • out/campaign_cache/w6-78.pkl · 3.1 kB · binary; in the .zip
  • out/campaign_cache/w6-88.pkl · 1.3 kB · binary; in the .zip
  • out/campaign_cache/w6-p77.pkl · 3.0 kB · binary; in the .zip
  • out/campaign_cache/w7-p88.pkl · 1.9 kB · binary; in the .zip
  • out/campaign_cache/w7-p99.pkl · 37 bytes · binary; in the .zip
  • out/campaign_cache/w7-prb.pkl · 3.1 kB · binary; in the .zip
  • out/campaign_cache/w7-ring.pkl · 1.7 kB · binary; in the .zip
  • out/campaign_cache/w7-swr.pkl · 1.9 kB · binary; in the .zip
  • out/campaign_cache/w7-w99.pkl · 3.5 kB · binary; in the .zip
  • out/campaign_cache/w7-wb.pkl · 2.9 kB · binary; in the .zip
  • out/campaign_cache/w7-wr.pkl · 3.5 kB · binary; in the .zip
  • out/campaign_stats.json · 4.7 kB · data; in the .zip
  • out/continuity.json · 7.1 kB · data; in the .zip
  • out/daily_cache/d-b67.pkl · 5.8 kB · binary; in the .zip
  • out/daily_cache/d-b77.pkl · 8.3 kB · binary; in the .zip
  • out/daily_cache/d-b78.pkl · 5.2 kB · binary; in the .zip
  • out/daily_cache/d-br77.pkl · 3.8 kB · binary; in the .zip
  • out/daily_cache/d-f66.pkl · 6.7 kB · binary; in the .zip
  • out/daily_cache/d-f66b.pkl · 3.1 kB · binary; in the .zip
  • out/daily_cache/d-f67.pkl · 8.2 kB · binary; in the .zip
  • out/daily_cache/d-f77.pkl · 10.0 kB · binary; in the .zip
  • out/daily_cache/d-f78.pkl · 6.3 kB · binary; in the .zip
  • out/daily_cache/d-g66.pkl · 6.5 kB · binary; in the .zip
  • out/daily_cache/d-g77.pkl · 10.0 kB · binary; in the .zip
  • out/daily_cache/d-g78.pkl · 4.7 kB · binary; in the .zip
  • out/daily_cache/d-gr77.pkl · 3.1 kB · binary; in the .zip
  • out/daily_cache/d-p66.pkl · 4.8 kB · binary; in the .zip
  • out/daily_cache/d-p77.pkl · 5.9 kB · binary; in the .zip
  • out/daily_cache/d-p78.pkl · 2.3 kB · binary; in the .zip
  • out/daily_cache/d-pb77.pkl · 2.0 kB · binary; in the .zip
  • out/daily_cache/d-r66.pkl · 5.3 kB · binary; in the .zip
  • out/daily_cache/d-r67.pkl · 7.5 kB · binary; in the .zip
  • out/daily_cache/d-r67b.pkl · 3.4 kB · binary; in the .zip
  • out/daily_cache/d-r77.pkl · 9.2 kB · binary; in the .zip
  • out/daily_cache/d-r77x2.pkl · 3.8 kB · binary; in the .zip
  • out/daily_cache/d-r78.pkl · 5.2 kB · binary; in the .zip
  • out/daily_cache/d-ring78.pkl · 3.1 kB · binary; in the .zip
  • out/daily_cache/d-s67.pkl · 4.9 kB · binary; in the .zip
  • out/daily_cache/d-s77.pkl · 6.4 kB · binary; in the .zip
  • out/daily_cache/d-s78.pkl · 5.1 kB · binary; in the .zip
  • out/daily_cache/d-s88.pkl · 3.5 kB · binary; in the .zip
  • out/daily_cache/d-sp77.pkl · 3.1 kB · binary; in the .zip
  • out/daily_cache/d-w55.pkl · 2.3 kB · binary; in the .zip
  • out/daily_cache/d-w56.pkl · 2.6 kB · binary; in the .zip
  • out/daily_cache/d-w66.pkl · 5.0 kB · binary; in the .zip
  • out/daily_cache/d-w67.pkl · 5.5 kB · binary; in the .zip
  • out/daily_cache/d-w77.pkl · 7.1 kB · binary; in the .zip
  • out/daily_cache/d-w78.pkl · 5.0 kB · binary; in the .zip
  • out/daily_cache/d-w88.pkl · 4.5 kB · binary; in the .zip
  • out/daily_cache/d-wb77.pkl · 3.7 kB · binary; in the .zip
  • out/daily_cache/d-wr77.pkl · 3.6 kB · binary; in the .zip
  • out/daily_cache/exports.pkl · 430.5 kB · binary; in the .zip
  • out/daily_gardens.json · 407.6 kB · data; in the .zip
  • out/daily_stats.json · 3.0 kB · data; in the .zip
  • out/facts.json · 2.9 kB · data; in the .zip
  • out/facts_pebbles.json · 504 bytes · data; in the .zip
  • out/finale2_6x15_1.json · 2 bytes · data; in the .zip
  • out/finale_6x15.json · 2 bytes · data; in the .zip
  • out/garden_names.json · 26.6 kB · data; in the .zip
  • out/hint_check.json · 652 bytes · data; in the .zip
  • out/hint_stats.json · 16.5 kB · data; in the .zip
  • out/levels.json · 14.3 kB · data; in the .zip
  • out/marks_pick.json · 1.7 kB · data; in the .zip
  • out/marks_vs_grooves.json · 77.4 kB · data; in the .zip
  • out/pipeline_grooves.json · 5.4 kB · data; in the .zip
  • out/pipeline_wide.json · 190.8 kB · data; in the .zip
  • out/proto_gardens.json · 155.3 kB · data; in the .zip
  • out/sampler_yields.json · 7.7 kB · data; in the .zip
  • out/search2_two.json · 2 bytes · data; in the .zip
  • out/search2_wide.json · 13.2 kB · data; in the .zip
  • out/search3_wide.json · 796.6 kB · data; in the .zip
  • out/search4_two.json · 57.6 kB · data; in the .zip
  • out/table1.json · 1.0 kB · data; in the .zip
  • out/table2.json · 110.2 kB · data; in the .zip
  • out/two_pick.json · 823 bytes · data; in the .zip
  • out/two_picks.json · 5.7 kB · data; in the .zip
  • out/wide_pick.json · 805 bytes · data; in the .zip

Figures (shown in the document as SVG; PNG renders for the web) ( fig/ )

  • fig/final/2026-10-03-gwern-samon-anatomy.png · 98.1 kB · image; in the .zip
  • fig/final/2026-10-03-gwern-samon-anatomy.svg · 54.2 kB · figure source; the figure is shown in the document; in the .zip
  • fig/final/2026-10-03-gwern-samon-calm-agitated.png · 133.7 kB · image; in the .zip
  • fig/final/2026-10-03-gwern-samon-calm-agitated.svg · 38.1 kB · figure source; the figure is shown in the document; in the .zip
  • fig/final/2026-10-03-gwern-samon-grooves.png · 95.6 kB · image; in the .zip
  • fig/final/2026-10-03-gwern-samon-grooves.svg · 24.8 kB · figure source; the figure is shown in the document; in the .zip
  • fig/final/2026-10-03-gwern-samon-histogram.png · 44.4 kB · image; in the .zip
  • fig/final/2026-10-03-gwern-samon-histogram.svg · 6.7 kB · figure source; the figure is shown in the document; in the .zip
  • fig/final/2026-10-03-gwern-samon-ichimatsu.png · 130.8 kB · image; in the .zip
  • fig/final/2026-10-03-gwern-samon-ichimatsu.svg · 76.0 kB · figure source; the figure is shown in the document; in the .zip
  • fig/final/2026-10-03-gwern-samon-parity.png · 38.3 kB · image; in the .zip
  • fig/final/2026-10-03-gwern-samon-parity.svg · 13.7 kB · figure source; the figure is shown in the document; in the .zip
  • fig/final/2026-10-03-gwern-samon-pebbles.png · 103.5 kB · image; in the .zip
  • fig/final/2026-10-03-gwern-samon-pebbles.svg · 43.5 kB · figure source; the figure is shown in the document; in the .zip
  • fig/final/2026-10-03-gwern-samon-ripple.png · 82.9 kB · image; in the .zip
  • fig/final/2026-10-03-gwern-samon-ripple.svg · 23.4 kB · figure source; the figure is shown in the document; in the .zip
  • fig/final/2026-10-03-gwern-samon-thumbnail.png · 71.9 kB · image; in the .zip
  • fig/final/2026-10-03-gwern-samon-thumbnail.svg · 11.6 kB · figure source; the figure is shown in the document; in the .zip
  • fig/final/2026-10-03-gwern-samon-twogates.png · 97.3 kB · image; in the .zip
  • fig/final/2026-10-03-gwern-samon-twogates.svg · 31.6 kB · figure source; the figure is shown in the document; in the .zip
  • fig/final/2026-10-03-gwern-samon-waves.png · 78.5 kB · image; in the .zip
  • fig/final/2026-10-03-gwern-samon-waves.svg · 32.1 kB · figure source; the figure is shown in the document; in the .zip
  • fig/final/2026-10-03-gwern-samon-widerake-rule.png · 44.7 kB · image; in the .zip
  • fig/final/2026-10-03-gwern-samon-widerake-rule.svg · 21.0 kB · figure source; the figure is shown in the document; in the .zip
  • fig/final/2026-10-03-gwern-samon-widerake.png · 89.9 kB · image; in the .zip
  • fig/final/2026-10-03-gwern-samon-widerake.svg · 30.9 kB · figure source; the figure is shown in the document; in the .zip

Open Source as We Know It Is Dead

Lobsters
jross.me
2026-10-05 19:25:52
Comments...
Original Article

I made my GitHub account on June 17, 2011. About two minutes later, I opened my first issue : teleporting was broken in TShock, a Terraria server mod, and I wanted to know why. Not as early as some, but early enough to remember when getting a PR merged by a complete stranger felt like a small miracle. (I've been obsessed with game servers ever since - it's literally my job now.)

GitHub wasn't really the start, either. Before that, it was SVN checkouts that broke if you looked at them wrong, and tinkering with DarkRP on Garry's Mod servers, or writing SourcePawn plugins for old Counter-Strike: Source, or Zombie Master servers. GitHub just made it feel like everyone was finally in the same room.

Open source has been a part of my life for almost as long as I've been writing software. I'm self-taught, so it's not an exaggeration to say I wouldn't have a career without it. Most of what I know, I learned by reading someone else's code, or opening an issue that was probably a bit dumb, and having a maintainer patiently explain why.

This isn't easy to write and the title is a bit hyperbolic. But it's honestly how I've started to feel: open source, as I've known it, is dying . And I sadly think AI is what's killing it.

It was never just the code

When people talk about open source, they usually talk about code. Licenses, packages, download counts, stars. That was never the part that kept me coming back though.

It was the people. It was fixing a typo in a README and having someone say thanks. It was a stranger on the other side of the world finding a bug in my code and sending a fix before I'd even woken up. It was arguing (politely, mostly) in an issue thread about the right API, and ending up with something better than either of us would've built alone.

My first real GitHub PRs were in June 2014, three years to the day after I made that account. We were using PointDNS, and their little Node.js client crashed with SyntaxError: Unexpected end of input whenever you listed records on a big zone. It wasn't handling chunked responses, so I fixed it . Two hours later, feeling brave, I opened a second PR : a "global tidy up" that touched almost every line in the file. My description said, with all the confidence of someone who had never really maintained anything:

We're huge fans of your service, but this node.js module was rather counter-intuitive to use. Hopefully this pull request will help solve that issue for any future developers!

A maintainer merged both the next morning - no back and forth, no checklist, no bots. A stranger sent them 300 lines of unsolicited changes, and they took a chance on it. I can't fully explain what that did for me, except that it made me feel like I belonged.

A PR isn't just a diff - it's someone saying "I used your thing, I cared enough to make it better, and I'm trusting you with my work." Reviewing it is you saying "I see you, let's do this together." That exchange, repeated millions of times, is what built the Internet we all use every day.

Looking back at that second PR, I'm honestly a little embarrassed. An unsolicited rewrite from someone nobody had heard of, with a slightly cocky description? Miguel Grinberg, who maintains Flask-SocketIO, put it bluntly this year: "Today, an unsolicited PR is a red flag." If mine landed in a repo today, I'd assume an agent wrote it too, and I wouldn't blame anyone for closing it.

Andreas Kling said it best when Ladybird stopped accepting public PRs : "A substantial patch used to imply substantial effort, and that effort was a reasonable proxy for good faith. That assumption no longer holds." A real person wrote my PR, nervous, hoping it was good enough. The effort was the signal. That's what's gone.

It's not fun anymore

Something shifted over the last year or so, and it's been hard to put into words.

Open a PR on a popular repo today and observe what happens. Within seconds, a bot leaves an AI review summarizing your own change back to you. Another bot posts a preview deployment. Another posts a "walkthrough" with a sequence diagram nobody asked for. Sometimes a fourth shows up to respond to the third. By the time a human actually looks at it, the thread is a wall of generated text, and the real conversation (the human part) is buried somewhere in the middle.

That's the good case, where a human wrote the PR. The bad case is when nobody really did, like PRs that "fix" issues that don't exist, or PRs that rewrite half a file with confident, plausible, completely wrong code, or security reports that read like the real thing until you realize the vulnerability was hallucinated. And there are apparently accounts opening dozens of these a day across hundreds of repos, just farming green squares for a résumé.

I've started muting notifications from repos I used to love following since it's mostly just noise now, and that makes me sadder than I expected it to.

Even that first PointDNS PR , sitting quietly for over a decade, picked up a string of junk comments last November from an account I'd never seen. A # , a ### , a broken screenshot upload. I honestly don't know if it was a bot or a person. That's kind of the point.

I don't blame maintainers one bit

AI slop PRs are the worst kind of work. They look like contributions, so you feel obligated to read them. They take real time to review, and then more time to explain why they're wrong to someone (or something) that isn't listening. That's time not spent on the actual project, or with family, or sleeping.

And the doors are closing. Fast.

Not everywhere, and not all in the same way, but the direction of travel is hard to miss. curl ended its monetary bug bounty after the signal-to-noise ratio collapsed. Daniel Stenberg wrote that the share of real reports had fallen below 5%: “Not even one in twenty was real.”

tldraw started automatically closing external PRs . Ghostty tightened its AI policy , with Mitchell Hashimoto writing that low-effort AI contributions had increased the “bad” count “by 10x if not more.” Jazzband, home to dozens of Python projects, shut down entirely , blaming GitHub’s “slopocalypse” of generated PRs and issues.

OpenJDK banned LLM-generated contributions , naming reviewer burden as the first risk. Ladybird stopped accepting public pull requests. Godot now requires code to be human-authored and bans autonomous agents. COSMIC requires contributors to confirm that their PRs contain no LLM-generated code , comments, or descriptions. Codeberg members voted to ban mostly AI-generated projects from the platform.

Then on October 1, Sindre Sorhus disabled external pull requests across all of his repos:

Due to AI, I have disabled external pull requests on all my repos. Open source, as we have known it, was fun while it lasted. (It's been 15 years for me)

If you’ve written JavaScript in the last decade, you’ve almost certainly run his code.

The details vary, but the shape is the same: maintainers are no longer just deciding whether a contribution is good, they’re deciding whether they can afford to find out. I don’t think they’re wrong, and that's the really difficult part. Every maintainer closing the door is probably making the rational choice for their project, their time, and their sanity. But when enough people make that same rational choice, open source starts to become something else.

And then this week, Yusuke Wada disabled PRs from external contributors on Hono . That one hit hard. Hono is one of my favorite projects, and I've called it the gold standard for Workers development many times.

Sad news. We disabled PRs from external contributors on honojs/hono ... Hono has not stood here without PRs. I will never forget the PR @usualoma created for RegExpRouter. A damn fast HTTP router we have never seen! But PRs don't work in this era. Contribute in other ways. Thanks

That quote hurts because it contains the whole contradiction. Hono was shaped by an outsider’s PR, and that outsider is still contributing to it four years later. Now Hono has to close the door they walked through. I don’t read that as hypocrisy, I read it as grief.

Even the platforms are saying it out loud. In February, GitHub published "Welcome to the Eternal September of open source" , which put it better than I can: "The cost to create has dropped but the cost to review has not." The next day, they shipped a setting to turn pull requests off entirely . Since then they've added caps on open PRs per user (AI agents count toward it), a way to restrict who can open issues , and a way to archive spammy PRs so nobody else can see them. The platform that made the pull request famous has spent the year building ways to turn it off.

Some projects aren't waiting around for GitHub to fix it. Zig moved to Codeberg last year, and more and more people are spinning up their own Forgejo or Gitea instances. I don't blame them either, since GitHub's reliability hasn't exactly been great lately. But every project that leaves takes its issues, its history, and its people somewhere new, with yet another account to make and another place to look. It fractures things even further, and makes the next stranger even less likely to wander in.

I'm part of this too

I use AI every single day, and that makes me feel slightly uncomfortable while writing this blog. I write code with it, review code with it, and I even had help organizing the rambling notes that became this post. It's an amazing tool and it's made me faster, and on good days, better at my job. I'm not here to tell anyone to stop using it.

But I think a lot of us (me included) have been treating the cost of it as zero. It isn't - the cost just moved. When generating a PR takes ten seconds and reviewing it properly takes thirty minutes, all you've done is shift the work from the person who wants something onto the person who has to say no - and the person saying no is almost always a volunteer who was already stretched thin.

AI made contributing cheap. It didn't make maintaining any cheaper - if anything, it made it a lot more expensive.

I don't know what the fix is

I wish I had a neat answer here, but I don't.

Banning AI PRs feels like the right call for a lot of projects, and I'll defend any maintainer who makes it - but it's also a blunt tool. It catches the person who used AI to help write one good test alongside the person who pointed an agent at a thousand repos. It relies on honesty, or on detection that doesn't really work, and it nudges us toward a world where repos are closed by default, where you need to be vouched for before you're allowed to help, and where the default answer to a stranger is no.

Perhaps that's just where we're heading - smaller, trusted circles, with source available, but contributions by invitation. Maybe that's even fine, and I'm being nostalgic for a version of the Internet that was never going to last.

But the early 2010s version of me, nervously sending a stranger 300 lines of "tidy up", wouldn't get through that door. I think we lose something real when people like that can't get in anymore.

I don't know what the fix is. I just know I miss how it used to feel.

How Cloudflare addressed a cross-tenant data exposure vulnerability in Containers

Lobsters
blog.cloudflare.com
2026-10-05 19:03:17
Comments...
Original Article

On September 4, 2026, Oren Yomtov, a security researcher from Accomplish , responsibly reported a vulnerability affecting Cloudflare Containers and Cloudflare Sandboxes (which is built on Containers), through Cloudflare’s bug bounty program . Cloudflare has fully remediated the vulnerability, and we have no evidence that customer data has been compromised.

This post was prepared in collaboration with Oren Yomtov and the Accomplish security research team, whose detailed report and controlled testing helped us validate the issue and respond quickly.

Cloudflare Containers run workloads on multi-tenant infrastructure and automatically assign them to eligible servers; customers cannot select the underlying host. The researchers demonstrated that a customer with a Workers Paid account could recover residual disk blocks previously used by Containers on the same host. The technique could not target a particular customer, workload, host, or data, and residual data was not guaranteed to be present.

Cloudflare applied a fix across the Containers fleet, with no customer-side configuration changes required. Within the historical disk-I/O telemetry available to us, we identified no evidence of malicious exploitation. Activity we could attribute to the reported technique came from the researchers and Cloudflare engineers conducting authorized validation.

Here, we explain the underlying storage behavior, its potential impact, our investigation, and the actions we took in response.

How container storage allocation works

Cloudflare Containers use Linux device mapper thin provisioning (dm-thin) to provide each container with a writable root disk. Each container lives inside a dedicated virtual machine powered by the Firecracker virtual machine monitor. Firecracker presents this disk to the virtual machine as /dev/vdc.

Thin provisioning allocates physical storage only when a virtual disk writes to a previously unmapped region. The affected storage pools used a 64 KiB thin-block size. When the thin volume backing a container's root disk was deleted, its physical blocks were returned to a pool that served workloads belonging to multiple customer accounts.

The affected pool configuration included the following option:

skip_block_zeroing

With this option configured, dm-thin skips zeroing newly allocated blocks before making them accessible. Consequently, when a previously-used 64 KiB block was reassigned, a full-block write replaced its previous contents, but a smaller write changed only the written portion. The remainder could retain data from the block’s previous owner.

How the exploit worked

Reading an unmapped region of a new thin disk did not reveal residual data. For an unmapped region of the thin device, dm-thin returned zeroes without allocating a physical block.

The proof of concept identified 64 KiB-aligned regions corresponding to free space in the guest’s ext4 filesystem and wrote one aligned 4 KiB block into each region.

When such a write reached an unmapped thin block, dm-thin allocated a physical 64 KiB block from the shared pool. The 4 KiB write replaced only that portion of the block, and because block zeroing was disabled, the remaining 60 KiB could retain data from a previous container.

A subsequent raw-device read could therefore observe bytes that the new container had never written.

The proof of concept performed the following steps:

  1. Create a container using a Workers Paid account.
  2. Open the writable root disk at /dev/vdc .
  3. Read the disk and record a baseline.
  4. Write one 4 KiB block into each selected 64 KiB region corresponding to ext4 free space.
  5. Read the resulting blocks again.
  6. Examine only the portions not overwritten by the new container.

The submission included counts, block offsets, sizes, checksum results, and truncated hash prefixes. Although the researchers recovered raw blocks to validate the issue, the materials provided to Cloudflare contained no third-party filenames, identifiers, credentials, hostnames, addresses, or recovered content values. As described below, the researchers have also confirmed that they securely deleted the recovered data.

How the vulnerability was validated

The researchers used ext4 directory block checksums to distinguish blocks belonging to their own test filesystem created for the proof of concept from blocks originating from other filesystems.

When ext4 uses the metadata_csum feature, directory block checksums incorporate values associated with the filesystem and inode.

Across six production placements, the researchers reported:

  • All 5,614 testable directory blocks.
  • Zero of those blocks were attributed to the researchers’ filesystem.
  • 2,700 distinct foreign directory inodes identified through checksum analysis.

To validate the method, the researchers tested it against blocks they had deliberately created and deleted in the controlled test filesystem used for the proof of concept. The method correctly attributed all 162 blocks to that filesystem.

The researchers ultimately observed residual material on 18 of 24 placements and 20 of 22 underlying nodes across four continents. The recovered block types included directory structures, database pages, and structurally complete SQLite databases. The researchers reported using scripts that output only aggregate counts and format checks, not recovered file contents. The materials submitted to Cloudflare contained no recovered content values or third-party identifiers. The researchers subsequently confirmed that recovered data under their control remained confidential and was securely deleted following submission, consistent with Cloudflare’s HackerOne disclosure policy.

Impact

The vulnerability would potentially have allowed for a customer with a Workers Paid account to recover residual data from storage blocks previously used by other customers’ Containers on the same underlying host.

A successful exploitation would have crossed the tenant-isolation boundary and could disclose filesystem metadata, directory structures, database pages, and application data.

However, an attacker could not select a particular victim or access an actively attached disk. Exposure depended on Cloudflare’s workload placement and which previously released blocks dm-thin reassigned. Moreover, the researchers did not demonstrate modification of another customer’s active data or impact to workload availability.

How we mitigated the vulnerability

Our first mitigation was to remove skip_block_zeroing from the dm-thin pool configuration across the fleet. This restored dm-thin ’s default behavior of clearing newly allocated blocks before exposing them to a container. It stopped the reported technique, in which a small write triggered allocation and a larger read recovered residual data from the remainder of the block. The researchers independently confirmed that their proof of concept no longer worked after this change.

Zeroing new allocations did not sanitize blocks already mapped into existing thin devices. These mappings existed in running container disks and in each host’s cache of prepared dm-thin snapshots for OCI image layers. A new container could inherit mappings from a cached layer without allocating those blocks again, allowing residual bytes in unused regions, including ext4 free space, to remain readable through raw reads of /dev/vdc .

We therefore also retired all running container disks and removed cached image snapshots created before the mitigation. We drained hosts during off-peak hours, restarted the VMs on each host, and cleared each host's image cache so that disks and cached layers were recreated using zeroed allocations. We have completed this cleanup across the Containers fleet.

No evidence of exploitation

As part of our response, we investigated whether other workloads showed activity consistent with the reported exploitation technique. We reviewed retained historical disk-I/O telemetry from our container infrastructure, using the researchers’ proof of concept and our internal reproduction as reference activity.

The proof of concept produced a characteristic relationship between writes and reads. When a 4 KiB write reached a previously unmapped region, it could trigger allocation of a reused 64 KiB storage block. With zeroing disabled, the remaining 60 KiB could retain data from a previous container. Subsequent reads could therefore recover substantially more data than the new container had overwritten.

Using these characteristics, we developed detection signatures and applied them to the historical telemetry available to us. We identified activity attributable to the researchers and Cloudflare engineers conducting authorized validation, and did not identify additional activity consistent with the reported technique.

We saw no evidence that this specific attack vector was exploited by anyone else.

Cloudflare customers are protected

As we noted above, Cloudflare has patched this vulnerability and remediation does not require any further action by Cloudflare customers. In addition, we found no evidence of any malicious actor abusing this vulnerability.

Moving quickly with transparency

We thank Oren Yomtov and the Accomplish security research team for their thorough research, responsible disclosure, and collaboration on this post. We encourage the Cloudflare community to submit any identified vulnerabilities to help us continually improve the security posture of our products and platform.

We also recognize that the trust you place in us is paramount to the success of your infrastructure on Cloudflare. We take these vulnerabilities very seriously and will continue to do everything in our power to mitigate impact. We deeply appreciate your continued support and trust in our platform, and remain committed not only to prioritizing security in all we do, but also acting swiftly and transparently whenever an issue arises.

Timeline

  • September 4, 15:26 UTC: Oren Yomtov from Accomplish reported the issue through HackerOne .
  • September 4, 18:45 UTC: Cloudflare opened a security incident and confirmed the production setup that caused the flaw.
  • September 4, 21:27 UTC: Cloudflare merged the runtime fix and its reuse test.
  • September 4, 22:03 UTC: Cloudflare merged the changes for new and live pools.
  • September 4, 23:15 UTC: Cloudflare started rolling out the changes.
  • September 7, 06:13 UTC: Cloudflare completed rolling out the changes and began clearing old pool data.
  • September 14, 10:50 UTC: The researchers reported that their proof of concept had stopped working.
  • September 14, 12:52 UTC: Cloudflare awarded the researcher a bounty.
  • September 19, 15:03 UTC: Cloudflare completed cleanup of all pre-mitigation cached snapshots across the affected fleet.

Echoes of COINTELPRO w/ Watani Stiner & Kazu Haga

OrganizingUp
convergencemag.com
2026-10-05 19:00:00
On January 17, 1969, an altercation at UCLA ended in gunfire that left two members of the Black Panther Party dead. Watani Stiner–who was a member of the conflicting Us Organization–was there that day. But despite not shooting anyone, he and his brother George were sentenced to life in prison for co...
Original Article

Worth Building

Hacker News
armstr.ng
2026-10-05 18:59:24
Comments...
Original Article
Writing

What changes when a tool doesn't have to pay for itself

3 minute read

Years ago, if an app annoyed me, I shrugged and dealt with it. I could sometimes have fixed it or built my own, but the annoyance rarely justified the effort. With LLMs that calculation has changed, and this past weekend it changed for a domain lookup tool.

Occasionally I need to check DNS records or registration info for a domain. At my computer I use dig or whois . Otherwise I Google "domain lookup tool," click on one, and paste in the domain. Most of what comes up is slow, buried in upsells, and unpleasant to use for something that should be simple.

They aren't badly made -- they're well made for a different job. A domain lookup answers a question nobody pays to ask, so many of the free tools that show up in search are the ones a sales funnel justifies: the lookup brings in someone who might buy a domain or hosting, and the page is built to rank for "domain lookup tool" and to sell. The tools you find are the ones built to be found. Being good at the lookup was never what paid for them.

I wanted something I'd actually enjoy using, so I opened Claude Code and built heade.rs . In three prompts and about an hour I had a first working version deployed to Cloudflare: DNS and registration records, live latency, email configuration, HTTPS details. The rest of the day went into making it feel good: animations, graphics, an interface I like looking at. It costs less than a cup of coffee a month to run. It doesn't have to sell anything, so it can just be good.

The heade.rs overview page
Tabs for deep-dives

Before, a project like this needed justifying. I'd have asked whether I'd use it enough to make the time worthwhile. If I expected to spend weeks on it, I'd have started thinking about other users, pricing, how it could pay for itself. A small annoyance would have become a much bigger commitment; a weekend is much easier to justify. I wanted it to exist, and that was enough.

A few hours used to get a programmer a script; now it gets a polished app, and it gets a non-programmer something instead of nothing. More people can clear the bar, and more ideas clear it: small tools that solve a personal annoyance , weird ideas you're curious about, things you build simply because you'd enjoy having them or the world would be more beautiful if they existed. The audience can be one person.

The same economics cut the other way: heade.rs will probably not rank for "domain lookup tool," and neither will most things built like it. Cheaper software means more people shipping, and another demo in a crowded feed is easy to scroll past. If you want to reach more people, you still have to give them a reason to care. I'm more interested when someone explains what bothered them, what they made, and the choices they cared about along the way. The story gives me a reason to try the thing.

Sometimes sharing a personal project leads somewhere unexpected. Four years ago Paragraph started as something I built for myself, then other people wanted to use it too. I'm glad I shared it.

But I also like that heade.rs doesn't need to follow that path. If it stays a little tool I use occasionally to look up a domain, it was still worth building.

OpenAI Announces Their Text Watermarking Plans

Daring Fireball
openai.com
2026-10-05 18:59:14
OpenAI, today: The EU AI Act requires generative AI providers to make generated text identifiable in a machine-readable way. Text watermarking and detection remain early technologies with significant limitations, and views about their benefits and responsible uses are still developing. Our phase...
Original Article

Example.com Just Launched the Biggest Redesign in Decades

Hacker News
www.debugbear.com
2026-10-05 18:55:11
Comments...
Original Article

example.com is a reserved domain name set aside by the Internet Assigned Numbers Authority (IANA) for documentation purposes. It has been available as an example URL for developers and writers since the late nineties.

While the overall messaging of the page has been consistent over time, the visual appearance has received several major overhauls.

In this article, we take a look at the recent changes as well as the evolution of example.com over the last 20+ years.

What changed on 28 September 2026? ​

Until 28 September 2026, example.com was a static English-language page. The redesign introduces multi-language support using JavaScript, showing the content in a new language every 5 seconds.

Example.com shows an explanation of the site's purpose in English, Arabic, Chinese, French, Russian and Spanish. This is the English-language message:

This domain is for use in documentation examples without needing permission. This is not a service, avoid relying on it for testing and monitoring purposes.

Along with introducing multi-language support, the JavaScript code also inserts an SVG book icon.

How does the language transition animation work? ​

The new site doesn't just replace the content when showing the next language. Instead, there's a gradual opacity transition.

Language transition

The opacity animation itself just uses transition: opacity .4s; . But how is the character-by-character transition implemented?

Each character is wrapped in its own span element, and each span gets a slightly larger CSS transition-delay than the one before it. That means each character's opacity animation happens at a slightly different time.

Statement from IANA ​

IANA provided the following statement explaining the reasoning behind the recent changes:

The changes we make to the site from time-to-time are usually informed by a desire to either reduce the overall bandwidth demands of serving the site, or improve its utility. At its heart it is a placeholder for a site no-one should be accessing, but we have historically showed a message there to inform visitors about why the domain is registered and how they may use it.

As it tends to be heavily trafficked, the overall bandwidth is a key consideration, and to that end one of the changes we made this week was to split the page content into a more basic page, augmented by a separate Javascript file with additional content beyond that. Since most of the traffic to the site is automated it tends not to fetch the Javascript file, reducing overall data requirements.

The fundamental purpose of the domain is for use in documentation, such as illustrative examples you may find in instructions. We know that people may, for example, cut-and-paste a sample configuration file and forget to customize entries that contain an example domain, and we expect we get some incidental traffic from that. However, the domain isn't intended to be a general purpose endpoint for things like availability testing so we do not encourage that kind of usage. There is no requirement there is a HTTP service present on the host in order to fulfill its purpose, we just operate it as a courtesy.

January 20, 2002: What did the earliest version look like? ​

According to Wikipedia , the example.com website launched on January 1, 1999. The earliest version available to view in the Wayback Machine is from January 20, 2002.

The earliest version of the website only featured a list of domain names reserved by the Internet Corporation for Assigned Names and Numbers (ICANN) and IANA.

In the .com, .net, and .org top-level domains (TLDs), various domain names are reserved. These names are reserved from initial registration, but to the extent they are already registered the existing registrant may renew them.

Example.com: January 20, 2002

Structurally, the HTML used a table layout, with the ICANN logo positioned in the top left corner of the page.

The server headers show that the site used Apache 1.3.22 at this point.

March 28, 2002: simplified messaging ​

The first version of the example.com wording we recognize today came 67 days after the first available screenshot:

You have reached this web page by typing "example.com", "example.net", or "example.org" into your web browser.

These domain names are reserved for use in documentation and are not available for registration.

Example.com: March 28, 2002

February 7, 2003: RFC 2606 citation added ​

In February 2003, the content was amended to include a link to RFC 2606 , the official document for reserved top-level and second-level domains, so visitors could find more information on the website's purpose.

There was also a small technical change around this time: the server was upgraded to Apache 1.3.27 on Red Hat Linux.

July 30, 2010: .edu added to domain list ​

The website then went unchanged for seven years. On July 30, 2010, a slight change was made: example.edu was now pointed to the landing page, alongside the three previous domains.

July 29, 2013: migration and landing page redesign ​

From January 2011, example.com didn't serve its own page. Instead, it redirected visitors to a page on IANA's website.

That changed on July 29, 2013, when the redirect was removed and example.com began serving its own page from the EdgeCast Content Delivery Network (CDN) instead of Apache, alongside a major redesign of the landing page. The new design used a rounded card with an <h1> reading 'Example Domain' and a new description of the website's purpose.

The viewable domain list was now gone, with a link to IANA's website for more information.

Example.com: July 29, 2013

October 17, 2019: font and copy changes ​

A new update came six years after the biggest change in the website's history. The layout remained the same, but the font stack was modernized to include more fallback fonts for devices such as Macs, iPhones, and Windows PCs.

The copy was tweaked slightly, with 'is for use in illustrative examples' replacing 'established to be used' and 'in literature' replacing 'in examples'.

Example.com: October 17, 2019

January 15, 2025: infrastructure switch ​

Almost twelve years after the switch from Apache to the EdgeCast CDN, another infrastructure change took place: the Server header disappeared entirely. The response headers alone don't reveal which platform was serving the page at this point.

A change of provider around this time isn't surprising. EdgeCast had been acquired from Yahoo by Limelight Networks in 2022 and rebranded as Edgio. Edgio then filed for Chapter 11 bankruptcy in late 2024, with Akamai acquiring select Edgio customer contracts.

October 9, 2025: redesign ​

Later in 2025, the page was redesigned. The card design was dropped, and the HTML was minified. The copy was changed to a more direct message, again linking to IANA's website, with the link text changed from 'More information...' to 'Learn more'.

Example.com: October 9, 2025

December 17, 2025: Cloudflare migration ​

Less than a year after the previous switch, the infrastructure changed again: the site now serves from Cloudflare , as indicated by the cloudflare server header and CF-RAY header in the response. Whatever platform served the page after the January 2025 change, it has now been replaced with a Cloudflare-fronted setup.

June 9, 2026: favicon and HTML tweaks ​

The next update, on June 9, 2026, brought no visual changes. The only HTML changes on the page itself were a favicon tag being added and the <p> tags being closed. The page now includes an empty favicon: 

<link rel="icon" href="data:," />

This avoids a lot of unnecessary requests, as otherwise browsers look for a favicon at /favicon.ico .

Avoid relying on it for testing and monitoring purposes ​

Example.com first introduced an "Avoid use in operations" message in 2025.

For developers, it's tempting to use example.com to check if the network is working or test HTTP requests. The example server likely receives a lot of traffic through this.

However, as the page now clarifies, it's not a service you can rely on. When we look at our uptime monitoring data, we can see that every now and then the example.com server doesn't respond correctly.

Using infrastructure that your organization controls can be more reliable.

Example.com failed uptime checks

Conclusion ​

Looking at the visual timeline, we can see that changes to example.com happen sporadically. Sometimes a couple of small tweaks happen in back-to-back months. Other times the website remains untouched for periods of up to seven years.

We have been monitoring the page for several years in our public demo project. The Chrome UX Report (CrUX) data shows how page speed has changed over time.

Example.com performance data

To learn more about how your website is performing, sign up for a free 14-day DebugBear trial.

Illustration of website monitoring Illustration of website monitoring

Monitor Page Speed & Core Web Vitals

DebugBear monitoring includes:

  • In-depth Page Speed Reports
  • Automated Recommendations
  • Real User Analytics Data

ChatGPT is adding real cartoonists' signatures to fake New Yorker cartoons

Hacker News
www.niemanlab.org
2026-10-05 18:46:45
Comments...
Original Article

Why have I been blocked?

This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.

What can I do to resolve this?

You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.

OpenAI is adding invisible watermarks to ChatGPT and Codex text in the EU

Bleeping Computer
www.bleepingcomputer.com
2026-10-05 18:46:33
OpenAI is preparing to add invisible watermarks to text generated by ChatGPT and Codex in the European Union. [...]...
Original Article

ChatGPT

OpenAI is preparing to add invisible watermarks to text generated by ChatGPT and Codex in the European Union.

The watermark will not be visible when you read or copy the text. Instead, OpenAI says its new textGrain technology slightly changes the model's word choices to create a statistical pattern that can later be detected.

"Over the coming weeks, we will add an invisible watermark to eligible ChatGPT and Codex text output in the European Union," OpenAI explained .

OpenAI is not making this a global default yet.

Starting today, API developers worldwide can opt in to watermarking for supported models, but it remains disabled by default.

The company is also opening applications for its watermark detector, although access will initially be limited to approved researchers and expert organizations.

OpenAI admits its AI watermark can disappear when you edit the text

Text watermarking is far from perfect, and OpenAI's own tests show that fairly normal editing can significantly reduce its ability to detect AI-generated text.

"In an evaluation of 400-token passages, replacing 10% of words with synonyms reduced detection from about 92% to 66%. Replacing 25% of words reduced it to 17%," OpenAI noted.

At a 1% false-positive target, OpenAI detected the watermark in about 80% of 200-token psychology responses, compared with roughly 95% when the text reached 400 tokens.

OpenAI
This chart shows results for watermarked responses to mathematics and psychology questions from the ELI5 dataset⁠ at a target false positive rate of 1%

Detection was even worse for subjects such as mathematics, where the model has less freedom to choose different words.

"The absence of a detected watermark does not prove human authorship," OpenAI warned. "Text generated with OpenAI tools may be too short, edited, or translated for detection to work reliably."

OpenAI also says a detected watermark does not reveal who generated the text, their account, prompt, or conversation, and it cannot tell how much of the final work was written or edited by a human.

Interestingly, OpenAI says watermarking does not meaningfully affect the quality of GPT-6 Astra, with benchmark results remaining broadly similar when textGrain is enabled.

article image

Build your security blueprint for AI-powered attacks

Join Mikko Hyppönen and security leaders from the NFL, CHANEL, and Atlassian for a two-hour digital summit on what AI-speed attacks change, what defenders should stop doing, and how to validate, decide, fix, and re-validate at machine speed.

Save your seat

Global Solar Atlas: summary of solar power potential globally

Hacker News
globalsolaratlas.info
2026-10-05 18:37:58
Comments...
Original Article

Learning Jazz Pianist Style with Cross-Attention Conditioning

Hacker News
almostimplemented.github.io
2026-10-05 18:30:52
Comments...
Original Article

ISMIR 2026, Abu Dhabi

Cover of Dick Hyman's In the Styles Of... The Great Jazz Pianists, listing the pianists from Oscar Peterson to Jimmy Yancey

In 1994 Dick Hyman published In the Styles of… The Great Jazz Pianists : fifteen original études, each written in the manner of one master, from Scott Joplin to Bill Evans. Rather than transcribing their solos, Hyman composed new music that carries their signatures — Tatum’s “rapid runs in both hands,” Garner’s “strumming, guitar-like left hand,” Peterson’s “tremolos and glissandi.” That book is the inspiration for this project. Can a model learn to do what Hyman did: not just recognize who is playing, but play in their manner? Tatum, Garner, and Peterson are among the twelve pianists we study — and so is Hyman himself.

We fine-tune Aria , a transformer pretrained on piano MIDI, on solo performances by twelve jazz pianists from the PiJAMA dataset, adding a gated cross-attention layer that reads a learned embedding for each pianist. To check whether the style comes through, we slide a pianist classifier along the generated music: conditioned continuations are attributed to the intended pianist 70% of the time, against 37% without conditioning. A second classifier trained only on generated music then identifies real recordings with 95% accuracy.

Listen first; how it works is further down.

One prompt, twelve pianists

The opening bars of “Ain’t Misbehavin’” are played by one of us (Drew). Everything after the dashed line is generated: twelve takes of the same opening, each conditioned on a different pianist. Pick a pianist to hear their take from the top.

0:00 / 0:00

in the style of —

Every take generates the same number of notes, so they end at different times: Erroll Garner packs them into 1:26, Cedar Walton spreads them across 2:45. That difference in density is itself part of a pianist’s signature.

Loading…

Twelve continuations, scored

A shared prompt pulls every pianist toward the same tune. Here each pianist instead continues a few bars of their own playing. The strip under each take shows what our classifier heard as it slid along the continuation, one cell per window of about 300 notes: gold where it named the intended pianist, mauve where it named someone else. The two takes per pianist are the best of eight we scored; the line under them says how the rest did. Or switch on the blindfold and guess for yourself.

classifier names the intended pianist names someone else windows that include what you hear now

How it works

The model

We start from Aria (Bradshaw et al., ISMIR 2025; code ), a 16-layer transformer pretrained on a large corpus of piano MIDI. Into each of its last eight layers we insert a cross-attention block: the music attends to a small learned embedding for the chosen pianist, four vectors per pianist. A learned gate scales what the block adds, starting at 0.1, so fine-tuning begins from Aria’s own behaviour and learns how much to listen. Because the embedding is attended to at every step, the conditioning does not fade as generation goes on, the way a prompt prefix does.

One transformer layer with the added gated cross-attention block x Self-attention Add & norm Cross-attention ×g Add & norm Feed-forward Add & norm x′ keys, values pianist embedding 4 learned vectors
One of the eight adapted layers. The new pieces are in amber: the music supplies the queries, the pianist embedding supplies keys and values, and the gate g scales the result before it joins the residual stream.

Measuring style

How can we tell whether the model has learned a pianist’s style? The standard yardstick for a generative model, perplexity on held-out music, turns out to be nearly blind to it: given the real preceding notes, the next one is predictable whoever is playing, so conditioning barely moves the score. Style shows up when the model generates freely and has to stay in character on its own output. So instead we let it play, and ask a pianist classifier who it sounds like. Agreement is how often the classifier names the intended pianist, in windows slid along each continuation.

Model Perplexity Agreement
Pretrained Aria 11.41 25%
Fine-tuned, no conditioning 6.96 37%
Fine-tuned with pianist conditioning 6.82 70%

Perplexity (lower is better) barely separates the two fine-tuned models; agreement nearly doubles. Continuations are 4096 tokens from 256-token prompts; chance agreement is 8%.

Three ways we use the classifier

  • Sliding-window agreement, above. The classifier identifies 98.8% of held-out songs, and the conditioned model’s lead holds from the start of a continuation to its end. The strips under the scored takes are this measurement.
  • Synthetic transfer. A fresh classifier trained only on generated music identifies real recordings: 87% of 1024-token chunks and 95% of songs, within nine points of one trained on real data. The scored takes are samples of that training data.
  • Characteristic regions. Turned on real performances, the classifier points to where a pianist’s style is most concentrated — where the style lives .

The paper has the details: per-pianist results, the mismatch experiment (prompting with one pianist and conditioning on another), memorization checks, and a from-scratch classifier that confirms the transfer result.

About these examples

Everything here is MIDI, rendered in your browser on a sampled piano. The model was trained on automatic transcriptions of commercial recordings, so dynamics and pedalling are approximate, and the rendering is plainer than the records.

The twelve pianists were chosen for separability: they are the twelve of PiJAMA ’s thirty whose recordings a pretrained model already tells apart most easily. Within them the model imitates some far better than others — across the paper’s evaluation, continuations were attributed to the intended pianist 96% of the time for Hank Jones and Dick Hyman, but only 29% for Cedar Walton.

The scored takes are selected, not copied. They are samples from the corpus of generated music that the paper’s synthetic-only classifier learned from; for each pianist we show the two highest-scoring of eight candidates. Their prompts come from the training recordings, and the paper checks that the continuations do not copy them: they resemble their closest training performance less than real held-out performances do.

The scores come from a classifier, not from listeners. It is a strong one (98.8% of held-out songs), but it has habits: many of its mistakes on generated music land on Dick Hyman — fitting, perhaps, for a pianist who made a career of playing in everyone else’s style. A listening study is the natural next step.

Friendship ended with Deno, now Node is my best friend

Hacker News
dbushell.com
2026-10-05 18:30:02
Comments...
Original Article

No AI - Made by Human

It’s finally time I go crawling back to Node!

I’ve been using Node heavily this month on a SvelteKit client project. When did Node get so good‽ Deno has been my go-to runtime for so long I forgot how to Node. Now I’m back, I find all the ECMAScript † sugar is supported and the old annoying APIs have been replaced or modernised. Most importantly, I never have to see require() .

† Doesn’t seem like that Oracle trademark dispute will see a positive end :(

Package management

The official Node docs recommend piping an internet script straight to bash (we never learn) to install NVM to manage Node & NPM. My (ancient) experience with NVM and NPM hasn’t been stellar. I heard Fast Node Manager (FNM) was better to switch Node versions. Obviously I roll bleeding-edge but I have client projects that demand stability.

I opted for PNPM too to avoid getting immediately pwned. (The “M” in NPM stands for “malware.”) Some scripts I use have hard-coded binary names ‡ , so I added two aliases: 

alias npm=pnpm
alias npx=pnpx

Maybe that’s a crime but so far it’s worked flawlessly.

‡ Edit: I might be confusing the time I symlinked various binaries. The aliases do help when copying steps from install guides that assume NPM is installed.

PNPM also blocks post-install scripts. Does NPM still yolo those?

I added additional settings to pnpm-workspace.yaml to delay malware updates. 

minimumReleaseAge: 1440
trustPolicy: no-downgrade

At first I tried setting the minimum release age to “one month” because it takes Microsoft at least that long to remove reported malware. This caused dependency issues where PNPM struggled to match suitable versions. I settled for “one day”; long enough to allow some other sucker to beta test the next Shai‑Hulud.

TypeScript

Node can now run TypeScript without throwing a tantrum like a baby if the stars don’t align. That said, one does not simply publish TypeScript packages to NPM. 

error: [ERR_UNSUPPORTED_NODE_MODULES_TYPE_STRIPPING]:
Stripping types is currently unsupported for files under node_modules

Why? Just strip the types bro, I know you can! Let me sign a deal with the devil!

To discourage package authors from publishing packages written in TypeScript, Node.js refuses to handle TypeScript files inside folders under a node_modules path.

Node.js v26.10.0 documentation

This restriction is philosophical rather than technical. I get it though. TypeScript is a Microsoft product. Opening that floodgate would pollute the entire ecosystem. Nobody wants more Microsoft. I’d love to see light “native types” in ECMAScript. There are type annotation proposals . I suspect I’ll be retired before those bear fruit.

No TypeScript packages mean I need to find the latest churnware slop to bundle my stuff. Tsdown did the trick, with only two additional dotfiles. Not thrilled about that (every dotfile represents a mistake). I suppose I break even after deleting deno.json etc.

Speaking of Microsoft lock-in, because they wrecked GitHub I’m self-hosting my own Forgejo instance . NPM limitations mean my packages have lost “provenance”. I had to configure the PNPM trust policy to allow my own stuff. Fun times!

Migrating my website

My final test for Node was converting my static site generator from Deno. Not many Node versions ago this would have required a major refactor. Today with Node v26.10.0 I found surprisingly little work to do.

The only required changes were to replace Deno’s file system API with node:fs — which is vastly improved from what I remember (literally ~10 years ago). Aside from that, I had to replace Deno.serve with Hono’s node adapter (a wrapper around node:http ).

After this minimum-viable migration I was shocked to see 15% faster builds . My codebase still favours idiomatic Deno. I bet I’m leaving performance on the table by not using other built-in Node APIs. That’s something to explore later. The only further change I made was to replace Deno’s @std/path with node:path which is a straight import swap.

So if I were to TL;DR in the middle: Node got a glow-up, wow!

You’ve probably known this for a while. I kept using Deno out of habit and familiarity. And I haven’t exactly been enthused to write server-side JavaScript recently.

Deno’s decline

I’m burying this part because it’s flogging a dead horse. Ultimately, Deno failed when they allowed the Silicon Valley Circus to define “success”. Deno went from an innovative modern JavaScript runtime to a boring start-up with uncompelling products . Half the employees were laid off and what’s left are tweeting AI fantasies and vibe-coding Temu Cloudflare.

There is no reason to use the Deno runtime today. Deno Land Inc. stopped innovating that years ago. Node has slowly but surely caught up, even surpassing Deno in places.

What finally pushed me away was:

  • Broken ZSH integration for weeks
  • JSR’s aggressive “429 (Too Many Requests)”
  • Bug(s) that made Deno choke on concurrent HTTP requests

Basically stuff that made it borderline unusable on top of my other criticism. JSR support were very quick to delete my account on request. I don’t like leaving dead profiles around the internet. None of my packages are visible but old versions remain installable.

It was fun early on but now it’s time to say goodbye. 

brew uninstall deno

Russell Coker: Kdeconnect DBUS

PlanetDebian
etbe.coker.com.au
2026-10-05 18:22:49
I’ve tested the DBUS interfaces for Kdeconnect (the system for connecting a KDE desktop to a phone or another computer). Unlike most DBUS interfaces it has a significant tree of interfaces so the busctl command to get the tree is important. Here’s the useful things I discovered: # list basic interfa...
Original Article

I’ve tested the DBUS interfaces for Kdeconnect (the system for connecting a KDE desktop to a phone or another computer). Unlike most DBUS interfaces it has a significant tree of interfaces so the busctl command to get the tree is important. Here’s the useful things I discovered: 

# list basic interfaces for kdeconnect
qdbus6 org.kde.kdeconnect.daemon /modules/kdeconnect
# get tree of interfaces for kdeconnect
busctl --user tree org.kde.kdeconnect.daemon
# get list of devices
qdbus6 --literal org.kde.kdeconnect.daemon /modules/kdeconnect org.kde.kdeconnect.daemon.deviceNames
# list interfaces for a device (DEVID is a hex string from the above command)
qdbus6 org.kde.kdeconnect.daemon /modules/kdeconnect/devices/$DEVID
# get battery charge
qdbus6 org.kde.kdeconnect.daemon /modules/kdeconnect/devices/$DEVID/battery org.kde.kdeconnect.device.battery.charge

Sending an SMS apparently requires using the QVariantList type which is more pain than I wanted so I tried the kdeconnect-cli command. Here’s a quick summary of how to use it: 

# list devices
kdeconnect-cli -l
# send a SMS
kdeconnect-cli -d $DEVID --send-sms "the message" --destination $NUMBER
# find device (ring)
kdeconnect-cli --ring -d $DEVID
# send a notification message
kdeconnect-cli --ping-msg "this is the message" -d $DEVID
# unlock and lock screen (only works on Debian devices for me not Android devices)
kdeconnect-cli --unlock -d $DEVID
kdeconnect-cli --lock -d $DEVID

There doesn’t seem to be support for making phone calls via kdeconnect which is a significant omission. It’s a very common situation to want to call a number that’s listed on a web site or in some other data source that’s easier to access on a PC than on a phone. The clipboard could be used but that’s needless pain.

Texas city demands $2M for public records on Flock usage

Hacker News
arstechnica.com
2026-10-05 18:05:57
Comments...
Original Article

Graphical UI

Hacker News
www.graphicalui.com
2026-10-05 17:51:09
Comments...
Original Article

Download it directly, or install via shadcn

my-app %

Checking registry.

Updating files.

Build beautiful interfaces with the agent you already use.

All stations

Bugs & technical issues

Conversations about the little things
getting in the way of great work.

  1. Jamie Lee Now

    An issue with pasting images

  2. Sarah Thomas 2h

    Things slow down with more variables and styles

  3. Andi Shafer 2h

    A little friction when inviting the team

  4. Matt Holden 1d

    Finding a faster way through the everyday tasks

  5. Kristy Spellman 2d

    Making space for the details that matter

A complete design language.

Colors

Brand colors, neutrals, and status colors for surfaces, text, and controls.

Light and dark

Separate color values for light and dark mode, in one theme.

Typography

Font families, weights, sizes, and line heights for interface text and data.

Spacing and scale

A shared scale for component sizes, padding, and gaps.

Icons

Choose the family and style that fits your theme.

Shape and effects

Corner radii, strokes, and shadows for component shape and depth.

Component states

Customize interactive states of Base UI components in Graphical.

Motion

Springs, easing, and timing for button presses, menus, and other interactions.

Everything you and your agent need to build interfaces.

Your theme

Your colors, typography, and styles, ready to install with shadcn or download directly.

Real components

React components built on Base UI , ready to use and customize.

Context for your agent

GUI.md , detailed theme references, and skills for building, adapting, and reviewing your interface.

One-time purchase.

Get our theme builder, launch collection of styles, and foundation components. Use them in unlimited projects, with ongoing updates to the theme builder included.

FAQ

Graphical is for designers, developers, and anyone building an app with a coding agent. Choose a theme, make it yours, and give your agent the theme files and skills to apply it to existing screens and guide new ones.

The ‘War Games’ Problem: How To Keep AI Under Control

Portside
portside.org
2026-10-05 17:46:14
The ‘War Games’ Problem: How To Keep AI Under Control barry Mon, 10/05/2026 - 17:46 ...
Original Article

AI agents don’t go rogue. That’s something only humans do.

Nevertheless, a New York Times article – representative of much news coverage of AI – described an OpenAI hacking as “A.I. bots going rogue and independently spearheading a cyberattack.”

Name-brand artificial intelligence agents have been on a hacking spree in 2026. OpenAI’s software agents hacked software company Hugging Face and government sites , Anthropic’s Claude hacked four companies’ systems , and in cybersecurity experiments Google’s Gemini hacked three companies .

The AI companies are investigating tens of thousands of incidents involving their agents, according to a report in Axios. These episodes have heightened fears about AI agents taking actions without human prompting.

The problem with headlines proclaiming that AI agents have gone rogue goes beyond anthropomorphizing the technology. It creates the impression that the agents were beyond the control of the AI companies that made them and there was little the companies could do about it.

As a technology law and ethics scholar who studies the effects disruptive technologies have on society, I know that’s not the case. If you don’t specify the limits of what software is allowed to do , you should not be surprised when the software pursues all possible options to achieve its goal. This behavior – an AI pursuing a fixed objective – is what I call the “WarGames” problem, and it’s been recognized in the field of computer science for decades.

Been there, seen that

In the 1983 movie “ WarGames ,” a teenager, David, hacks into a computer to play a new video game, Global Thermonuclear War. David doesn’t know that the computer is the government’s AI machine tasked with defending the United States from Russian nuclear attacks and can launch the U.S.’s missiles. When David and his friend start the game, they select Las Vegas as the first target . While the North American Aerospace Defense Command goes on alert, launching bombers and warming up intercontinental ballistic missiles, David’s parents make him turn off the game. It’s over. Or is it?

The next day, David’s phone rings and he connects it to his computer . The caller is the government computer, which updates him that the game was interrupted, the primary goal has not yet been achieved, but a solution is expected in the next 52 hours. Like a modern software agent, the program has been running since David started the game and will work until the task is done.

In the 1983 movie “WarGames,” a teenager learns that an AI is bent on “winning the game,” with civilization-ending consequences.

Chess provides another view of the problem. Conquering chess was a goal for early AI . The rules of chess are well defined, including what winning looks like. So, programming a machine to play chess is straightforward. But imagine you let the software reason and act beyond the confines of the chessboard. The software might pursue options such as blackmailing its opponent or grabbing more compute time.

This example comes from one of the most assigned textbooks on AI, “ Artificial Intelligence: A Modern Approach .” As the authors explain, you might be tempted to see those actions as rogue, but they “are a logical consequence of defining winning as the sole objective for the machine.”

What to do about it

The AI hacking events involving OpenAI, Anthropic and Google underscore a few lessons that draw on years of computer science research.

First, given the increasing use of AI agents, every organization involved in internet infrastructure, from large technology companies to small websites, needs to conduct audits and tighten up its internal security systems. As my colleague Mark Riedl and I explain in our work on AI agents , application programming interfaces , or APIs, are a vital part of managing AI agents. APIs facilitate communication between different software systems. But as more people use AI agents, the agents are likely to reveal and exploit poor API construction and security.

Second, it’s important for AI agents to be designed to identify and authenticate themselves to third parties. What if you gave your AI agent your credentials ? Website operators will need to know whether a human or bot is making a reservation , selling a product or making a purchase . They may want to limit automated systems that overwhelm their sites or reject AI agents because of high rates of buying errors and refunds . Just as in laws covering human interactions, it’s important for third parties to be able to assess whom or what they are dealing with so they can allow or deny access.

Third, it’s important for AI agents to have a default setting to slow down and check in with the human user . In the corporate AI hacking cases, the user appears to have launched their AI agents with the mistaken idea that the agents had a perfect specification of what to do and not to do. I believe it would have been better had it explored options and reported back to the user.

Google’s Gemini appears to have had a safeguard that detected the system was outside the simulated environment and so stopped its attacks. Slowing down and verifying actions, especially when a system detects it is exploiting a security hole, would be a big step in managing AI agents.

Fourth, AI companies could have strong controls akin to those biomedical researchers use, including ways to check what is happening and how the experiment is working. AI executives have claimed that their software is as or more dangerous than fission and could end humanity . At the same time, they have not built safeguards commensurate with that level of risk.

Reality check

At one point in “WarGames,” David asks the computer, called Joshua, whether it is still playing the game . Joshua responds, “Of course.” It proceeds to update the time when it will launch its missiles and, much like a chatbot, asks, “Would you like to see some projected kill ratios?” David asks, “Is this a game? Or is it real?” Joshua replied, “What’s the difference?”

AI models, of course, don’t have any understanding of reality and are simply attempting to complete the tasks they’ve been assigned. Executives at AI companies, on the other hand, can’t claim that excuse.

As of September 2026, luck has so far prevailed. The AIs have attacked nonvital government sites and harmed smaller companies. If the AI companies – and government regulators – don’t take the “WarGames” problem seriously, I believe that we risk serious disasters. Tomorrow it could be taking out a hospital’s power system, wiping out a bank’s account system, breaking air traffic control, or worse.

Regarding the AI industry’s approach of rapidly developing powerful models, talking about the massive risks they pose, and at the same time failing to prevent harm, the movie’s climax offers a response: “ A strange game. The only winning move is not to play .” The Conversation

Deven Desai , Professor of Business Law and Ethics; Associate Director for Law, Policy, and Ethics at the Machine Learning Center, Georgia Institute of Technology

This article is republished from The Conversation under a Creative Commons license. Read the original article .

Find the flattest route between any two points in SF

Hacker News
flattensf.com
2026-10-05 17:40:50
Comments...
Original Article

Flatten SF flat routes across San Francisco

How this works

Routes are computed in your browser over 160,000 street segments, using USGS 1 m lidar elevation and the Overture / OpenStreetMap street graph. The slider runs along the whole trade-off between distance and climbing: every route that nothing else beats on both counts, from the shortest path to the flattest one worth walking (where a foot of climb costs 200 ft of walking; beyond that the routes stop being routes). Sliding right never shortens the route and never adds climbing. Climbing is cumulative gain, not the net difference between the endpoints. Stairways are allowed on foot and excluded on a bike. Faint lines show the other routes along the way. Place search is offline: street intersections, places and addresses for San Francisco are built into this page. Source, data and the full analysis .

Dust: Pretraining Transformers Without Backpropagation

Hacker News
qlabs.sh
2026-10-05 17:15:07
Comments...
Original Article

October 2026 Correspondence to s@qlabs.sh · Code ·

Animation of the method

TL;DR

  • We present the first zeroth-order method that is competitive with backprop at pretraining transformer language models. Dust perturbs activations (node perturbation) independently at every token, so each token is a virtual population member and one forward pass evaluates them all in parallel.
  • Dust approximates backprop closely at large population (i.e. substantially more compute) and in multiple settings even exceeds it. This hints that in a compute-rich regime we might be able to surpass backprop.
  • Dust is orders of magnitude more efficient than weight-space ES. From 1M tokens up, Dust is on the order of $10^3$ to $10^4$ times more efficient than a transformer implementation of EGGROLL, a state-of-the-art ES method, based on our extrapolations.
  • Zeroth-order methods are widely believed not to scale to large networks. Strikingly, we find larger models are more population-efficient, not less: a 243M-parameter model outperforms a $120\times$ smaller model at most population sizes.
  • Dust’s gradient estimates align better with backprop’s as population grows, and stay well aligned at every scale we test, up to 1B tokens, which is encouraging for scaling.

1 Introduction

Deep learning has been built around backprop, the only credit assignment algorithm capable of training modern neural nets, including transformer-based language models. Backprop requires differentiability and produces first-order gradients, and deep learning’s architectures, optimizers, and hardware have co-evolved around this constraint.

However, as the amount of compute available in the world increases, we might prefer more generic and brute-force learning algorithms based on search over inductive biases like differentiability, backprop, and approximations of higher-order gradients. The bitter lesson ( Sutton, 2019 Richard S. Sutton. The bitter lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html , 2019. Blog post. ) is that general methods that scale with compute eventually win, and AlphaGo Zero ( Silver et al., 2017 David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature , 550 (7676): 354–359, 2017. doi: 10.1038/nature24270 . ) is the obvious example. Bootstrapping AlphaGo on human data helped the network learn faster initially, but with a lot of computation the purely self-play network overtook it. Similarly, differentiability and backprop might be good inductive biases in the low-compute regime, where they make learning efficient, but in the high-compute regime they limit the space of architectures that work. Even within an architecture, gradient-based methods fail to explore the loss landscape optimally ( Liu et al., 2020 Shengchao Liu, Dimitris Papailiopoulos, and Dimitris Achlioptas. Bad global minima exist and SGD can reach them. In Advances in Neural Information Processing Systems , volume 33, 2020. ) . This might also explain why current neural nets require massive amounts of data to generalize. A more flexible credit assignment algorithm based on search is likely an important step towards much better generalization.

In this paper, we aim to replace backprop with a learning algorithm based much more on brute-force computation and much less on analytic structure. We call it Dust. Dust is a zeroth-order optimization algorithm that perturbs activations, rewards each perturbation by how much it lowers the loss, and averages the reward-weighted perturbations over a population to estimate the gradient. Traditional ES methods that perturb weights ( Salimans et al., 2017 Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864 , 2017. ) , like EGGROLL ( Sarkar et al., 2025 Bidipta Sarkar, Mattie Fellows, Juan Agustin Duque, Alistair Letcher, Antonio León Villares, Anya Sims, Clarisse Wibault, Dmitry Samsonov, Dylan Cope, Jarek Liesen, Kang Li, Lukas Seier, Theo Wolf, Uljad Berdica, Valentin Mohl, Alexander David Goldie, Aaron Courville, Karin Sevegnani, Shimon Whiteson, and Jakob Nicolaus Foerster. Evolution strategies at the hyperscale. arXiv preprint arXiv:2511.16652 , 2025. ) , scale with population, but scaling the population is costly because each member must be materialized and evaluated. We remove both costs with the concept of virtual population , where we avoid materializing every member by bypassing weight space entirely and instead perturb activations, as in node perturbation ( Werfel et al., 2003 Justin Werfel, Xiaohui Xie, and H. Sebastian Seung. Learning curves for stochastic gradient descent in linear feedforward networks. In Advances in Neural Information Processing Systems , volume 16, 2003. ; Widrow and Lehr, 1990 Bernard Widrow and Michael A. Lehr. 30 years of adaptive neural networks: Perceptron, Madaline, and backpropagation. Proceedings of the IEEE , 78 (9): 1415–1442, 1990. doi: 10.1109/5.58323 . ) . We do so independently at every token, so each token is a member and one forward pass evaluates them all in parallel.

Activations are a more interesting space to search over than weights. Mechanistic interpretability has shown that reasoning, whether verbalizable or not, lives in the activations ( Gurnee et al., 2026 Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, Runjin Chen, Anna Soligo, Paul Bogdan, Euan Ong, Rowan Wang, Ben Thompson, David Abrahams, Subhash Kantamneni, Emmanuel Ameisen, Joshua Batson, and Jack Lindsey. Verbalizable representations form a global workspace in language models. arXiv preprint arXiv:2607.15495 , 2026. ; Lindsey et al., 2025 Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, et al. On the biology of a large language model. Transformer Circuits Thread , 2025. ) , which means this approach could turn training into a search over latent reasoning ( Vegesna and Dahal, 2025 Akshay Vegesna and Samip Dahal. Decoupling search and learning in neural net training. arXiv preprint arXiv:2509.10973 , 2025. ) . We then pair the activation-space perturbation with a very generic credit assignment rule that assigns different token-level rewards to different layer types in a transformer block. Those two biases, along with a few implementation details and efficiency measures, like avoiding interference between perturbed modules, are the whole algorithm.

We make the following contributions.

  • We present the first zeroth-order method that is competitive with backprop at pretraining transformer language models. At large populations Dust exceeds backprop in multiple settings, which suggests that in a compute-rich regime we might be able to surpass backprop.
  • Dust is orders of magnitude more efficient than weight-space ES. From 1M tokens up, Dust is on the order of $10^3$ to $10^4$ times more efficient than a transformer implementation of EGGROLL, based on our extrapolations.
  • Contrary to conventional wisdom, larger models are often more population-efficient, not less, and can make use of larger populations. This gives a new view of overparameterization as a larger search space with potentially better geometry.
  • Dust’s gradient estimates align better with backprop’s as the population grows, and the alignment holds up at every scale we test, up to 1B tokens, which is encouraging for scaling.

The goal of this paper is to lay the foundations of a search-based credit assignment algorithm that is competitive with backprop on the hardest task we could think of: pretraining transformers. We do not attempt to make it compute-efficient enough to replace backprop today. We also do not train the new kinds of neural nets it makes accessible, like nets with an external program in the loop or transformers looped over many steps that backpropagation through time struggles to train. Both are left to future work.

2 Method

Dust works as follows. We add Gaussian noise to the output of each linear layer, independently at every token, run a forward pass, and reward each token’s noise by the change in loss at that token. The reward-weighted noise, averaged over draws, is the estimated error at the layer’s output, and its outer product with the layer’s input is the weight gradient. Attention internals get a variant of it: they are credited through the estimated error at the attention output over current and future tokens, instead of the tokens’ loss directly. The core intuition is that while weight-space ES evaluates one population member per forward pass, we evaluate one per token, in parallel, and a member is materialized by adding noise to a hidden state, which is cheap. On a modern transformer a single forward pass therefore evaluates a population at least three orders of magnitude larger than weight-space ES. We describe each component in detail below.

2.1 Activation-Space Perturbation

The bottleneck of evolution strategies is population size. Every member needs its own perturbed copy of the weights and its own forward pass. EGGROLL ( Sarkar et al., 2025 Bidipta Sarkar, Mattie Fellows, Juan Agustin Duque, Alistair Letcher, Antonio León Villares, Anya Sims, Clarisse Wibault, Dmitry Samsonov, Dylan Cope, Jarek Liesen, Kang Li, Lukas Seier, Theo Wolf, Uljad Berdica, Valentin Mohl, Alexander David Goldie, Aaron Courville, Karin Sevegnani, Shimon Whiteson, and Jakob Nicolaus Foerster. Evolution strategies at the hyperscale. arXiv preprint arXiv:2511.16652 , 2025. ) makes the copies cheap with low-rank perturbations, but each member is still one sequence element of the batch, so the population is bounded by the forward passes one can afford. We perturb activations instead, independently at every token. At that token the network behaves as if a low-rank perturbation had been applied to the weights of the layer that produced the activations, without the perturbation ever being materialized in the weights. We call this a virtual population . A sequence in a transformer has a few thousand tokens, so one forward pass evaluates a few thousand members per sequence instead of one. Every weight in the model is trained this way except the $2L$ residual mixing scalars, which are trained by ordinary weight-space ES.

Adding noise to activations rather than weights is node perturbation ( Widrow and Lehr, 1990 Bernard Widrow and Michael A. Lehr. 30 years of adaptive neural networks: Perceptron, Madaline, and backpropagation. Proceedings of the IEEE , 78 (9): 1415–1442, 1990. doi: 10.1109/5.58323 . ) , and the usual argument for it is dimensionality ( Ren et al., 2023 Mengye Ren, Simon Kornblith, Renjie Liao, and Geoffrey Hinton. Scaling forward gradient with local losses. In International Conference on Learning Representations , 2023. ; Werfel et al., 2003 Justin Werfel, Xiaohui Xie, and H. Sebastian Seung. Learning curves for stochastic gradient descent in linear feedforward networks. In Advances in Neural Information Processing Systems , volume 16, 2003. ) . A layer’s output has $d_{\mathrm{out}}$ entries and its weights have $d_{\mathrm{out}} \times d_{\mathrm{in}}$, so activation noise lives in a much smaller space. Naively, the dimensionality argument doesn’t hold for transformers with large activations across many tokens. The noise on one sequence is a $T \times d_{\mathrm{out}}$ tensor, which has at least as many entries as the weight matrix once $T \ge d_{\mathrm{in}}$. However, with per-token independent perturbations and rewards, what perturbing activations gives is a new, efficient population along the token axis that is orthogonal to the batch axis EGGROLL already relies on.

2.2 Credit Assignment

For a linear layer $y_t = W x_t$ we jitter its output at all tokens, $y_t \to y_t + \sigma a_t$ with $a_t \sim \mathcal{N}(0, I)$ and $\sigma$ the noise scale, and run the forward pass. At each token $s$ we calculate the centered loss reduction $c_s = \tilde{\ell}_s - \ell_s$, where $\ell_s$ is the perturbed loss and $\tilde{\ell}_s$ is the mean perturbed loss at that token across draws evaluated together in the same batched forward. The reward of the jitter at token $t$ is the loss reduction at $t$ and, with a decay $\gamma$, the loss reductions at the tokens after it, which the jitter also reaches through attention,

$$r_t = \sum_{s \ge t} \gamma^{\,s-t} c_s .$$

(1)

With $\gamma = 0$ a jitter is rewarded by its own token alone. We leave it to the tuning of Section 2.3 to decide which layers see the future tokens. One independent jitter of all tokens is a draw , and a population is $K$ draws. Averaged over the draws, the reward-weighted noise

$$\hat g_t = -\frac{1}{K\sigma}\sum_{i=1}^{K} r_t^{(i)} a_t^{(i)}$$

(2)

is the estimated error at the layer’s output, and its outer product with the layer’s input, which the forward pass already computed, summed over tokens, is the weight gradient,

$$\widehat{G}_W = \sum_t \hat g_t\, x_t^\top .$$

(3)

Backprop forms the same outer product with the same input. The only difference is that it gets the output error from the chain rule and we get it from the population. For an embedding layer $x_t$ is one-hot, so the outer product is a scatter-add of $\hat g_t$ into the token’s row.

With unlimited population the estimator above is the whole method, and every perturbation could go into one forward pass. Each draw’s reward-weighted noise is the gradient plus an error with no preferred direction. Over draws the gradient adds up linearly while the errors add up as the square root, so their ratio falls as the population grows and the interference vanishes in the limit.

2.3 Interference and Tuning

At a population we can afford, the main cost is interference, i.e. we jitter many layers at many tokens in the same forward pass, so the loss change that rewards one token’s noise also picks up the effect of every other perturbation in that pass. We reduce such interference in three ways. First, different layer types are jittered in separate forward passes, each with its own noise scale, and each block gets its own passes. These passes are cheaper than full forwards because the clean forward is cached and a draw for block $l$ only reruns blocks $l$ onward. Second, the attention internals (query, key, value, gate, value embedding) are jittered separately. Token losses barely register their jitters, so they are rewarded through the attention output instead, as described below. Third, the language modeling head is jittered directly on the cached logits, re-evaluating only the cross-entropy and only a slab of the vocabulary per draw, which costs a small fraction of a forward pass and lets the head run a much larger population.

For the attention internals we recompute only their block’s attention output with the jitter, from the cached clean activations. We score the jitters using the alignment with the estimated attention output gradients,

$$c_s = -\langle \hat g_s, \Delta o_s \rangle ,$$

(4)

where $\Delta o_s$ is the change the jitter makes in the attention output at token $s$ and $\hat g_s$ is that output’s estimated error from Equation 2 . The reward is Equation 1 with these scores, computed per head, in place of the loss reductions.

The hyperparameters, the noise scale of each layer type, the credit decay of the attention internals and the share of the population each layer gets, can be tuned in two ways. One is a grid search that trains with each setting on a small token budget and keeps the setting that lowers the loss most. It is reliable but expensive. The other is a grid search that maximizes the cosine between our estimate and the backprop gradient on a single batch, which needs no training at all. A larger cosine on one batch doesn’t always lower the loss after training, so the cosine picks candidates and training decides. Either way the tuning is mostly a one-time cost, since the settings it finds largely generalize across token budgets and population sizes, with one exception. At the largest population at 10M and 20M tokens, a slower credit decay and a shift of draws toward the attention side still pay (Appendix F ). Hence, this search recovers general principles of the method rather than settings for one run. As expected, all layers require no credits from future tokens except the keys, values, gates and value embeddings, which are read by the later tokens that attend to them and get a $\gamma$ close to one.

3 Pretraining Without a Backward Pass

3.1 Setup

We train GPT-style transformers on FineWeb with a 4096-token BPE tokenizer, a batch of 16k tokens (8 sequences of 2048 tokens), one epoch, and SGD with momentum at a constant learning rate. The base model has 8 layers and width 512. Every method gets the same protocol and three seeds per cell, and is tuned separately at every token budget and population. Dust and backprop share one momentum and learning-rate grid; we implement EGGROLL with the same transformer architecture (named EGGROLL-Transformer) and it is tuned over its own grid of step size, momentum, noise scale and fitness shaping. Validation and test are held-out sets of 544 sequences each. We report the test loss at best validation checkpoint.

We count population in draws for Dust and in forward passes of the batch for EGGROLL ( Sarkar et al., 2025 Bidipta Sarkar, Mattie Fellows, Juan Agustin Duque, Alistair Letcher, Antonio León Villares, Anya Sims, Clarisse Wibault, Dmitry Samsonov, Dylan Cope, Jarek Liesen, Kang Li, Lukas Seier, Theo Wolf, Uljad Berdica, Valentin Mohl, Alexander David Goldie, Aaron Courville, Karin Sevegnani, Shimon Whiteson, and Jakob Nicolaus Foerster. Evolution strategies at the hyperscale. arXiv preprint arXiv:2511.16652 , 2025. ) , our weight-space baseline. A draw is one jitter of activations across all tokens of selected layers, rewarded with the token losses, and the population $K$ is the number of draws per update. A draw is slightly cheaper than a forward pass, because the clean forward is cached and a draw that jitters block $l$ reruns only the blocks from $l$ on. $K$ leaves out the draws of the head and the attention internals, which are a small fraction of the update’s FLOPs (Appendix E ). Taken together, from a population of 256 up Dust uses less compute than EGGROLL at the same population, so the comparison is lenient to EGGROLL.

3.2 Main Results

6.0 6.5 7.0 7.5 8.0 Validation loss 1M tokens 0 0.25 0.5 0.75 1 Training fraction 5.0 6.0 7.0 8.0 Validation loss 10M tokens 0 0.25 0.5 0.75 1 Training fraction Dust 64 1k 16k EGGROLL- Transformer 64 1k 16k Backprop

Figure 1: Validation loss over training at 1M and 10M tokens for Dust ( violet ), EGGROLL at the same populations ( gray ) and backprop (black, dashed). The x-axis is the fraction of the token budget consumed. Shade is population, 64, 1k and 16k, darker for larger. Curves are means over three seeds.

7.2 7.4 7.6 7.8 8.0 Test loss 100k tokens 64 256 1k 4k 16k Population 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 1M tokens 64 256 1k 4k 16k Population 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 Test loss 10M tokens 64 256 1k 4k 16k Population 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 20M tokens 64 256 1k 4k 16k Population ∞ 4.43 fit 1.53 P −0.145 + 4.43 limit as P → ∞ 95% interval of the limit EGGROLL-Transformer Dust Backprop

Figure 2: Test loss at the validation-selected checkpoint against population, one panel per token budget, for Dust ( violet ) and EGGROLL-Transformer ( gray ). Points are the cells of Table 1 , the dotted black line is backprop tuned at the same budget, and the dashed curve is a power law fit through Dust’s five cells. The 100k panel has its own y range; the other three share one. At 20M the dash-dotted violet line is the fit’s limit, 4.43, an extrapolation from a ladder that is still falling, and the bar at $\infty$ is its 95% bootstrap interval, with the seeds resampled at each population.

We sweep the token budget from 100k to 20M against populations from 64 to 16k, with backprop tuned on the same grid at every budget (Table 1 , Figure 2 ). At 100k and 1M tokens Dust ends below backprop, from a few hundred draws at 100k and from a thousand at 1M. At 10M and 20M tokens, the gap with backprop shrinks with population. At 10M the ladder has flattened and its fitted limit lands just above backprop. At 20M the ladder is still falling at 16k draws. Its power law fit puts the limit at 4.431 (95% interval 3.89 to 4.58), below backprop’s 4.633, but with the ladder still falling the fit is loosely constrained, so we read it as evidence that the gap keeps closing with population rather than as a measured limit.

Weight-space ES is far less efficient . With 256 times the population, EGGROLL at 16k still does not reach Dust at 64 draws. It comes within 0.02 at 100k tokens and stays 0.4 to 0.6 above at 1M, 10M and 20M (Figure 2 ). EGGROLL’s ladder is still falling steeply at 16k, so it would keep improving with more population, but continuing the ladder puts what it needs to match Dust’s smallest population at several thousand to about $10^4$ times that population (Appendix D ). Figure 1 also shows the gap during training; Dust at 1k and 16k draws stays close to backprop’s curve throughout, while every EGGROLL curve falls behind early and flattens well above Dust at 64 draws.

3.3 Dust Under Adam

5.5 6.0 6.5 7.0 7.5 Test loss 1M tokens with Adam 64 256 1k 4k 16k 64k ∞ Population EGGROLL-Transformer Dust 5.248 Backprop 5.361

Figure 3: Test loss against population at 1M tokens with Adam, for Dust ( violet ), EGGROLL ( gray ) and backprop (black, dotted ). The violet curve is a power law fit in population over the populations run, and the dash-dotted violet line is its limit, 5.248, below backprop. The bar at $\infty$ is the limit’s 95% bootstrap interval, 5.17 to 5.30, with the seeds resampled at each population. Means over five seeds for Dust and backprop and three for EGGROLL; the Adam runs of Dust give the token embedding more draws, about $1.4\times$ the listed population in total.

While we focus primarily on SGD for the rest of the paper, we repeat the 1M token ladder with Adam on all three methods (Figure 3 ), re-tuning backprop’s learning rates, Dust’s own hyperparameters and EGGROLL’s step size, momentum and fitness shaping at every population. Interestingly, EGGROLL gains almost nothing from Adam and its tuned Adam ladder lands within 0.01 of its SGD ladder in Table 1 at every population. However, Adam improves both Dust and backprop and leaves the ladder’s shape similar to before. Dust closes on backprop with a large population and the interval on its limit sits below backprop (Figure 3 ). So Dust’s estimate already works with modern optimizers , even though modern optimizers were optimized for backprop gradients. We suspect that coevolution of optimizers with Dust could lead to further gains and leave this to future work.

4 Search in High-Dimensional Space

4.1 Overparameterization

Test loss within each row lowest in its row highest in its row

Figure 4: Four model sizes at a fixed 10M tokens, 2M to 243M parameters (L2/d128, L4/d256, L8/d512, L16/d1024), test loss against population. The table has the same cells and backprop per size, with the best size at each population in bold.

The conventional view is that zeroth-order methods cannot train large networks ( Lillicrap et al., 2020 Timothy P. Lillicrap, Adam Santoro, Luke Marris, Colin J. Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience , 21 (6): 335–346, 2020. doi: 10.1038/s41583-020-0277-3 . ) . A forward pass returns a single scalar, so the variance of the gradient estimate grows with the number of perturbed dimensions, and with it the population needed for a useful update ( Nesterov and Spokoiny, 2017 Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics , 17 (2): 527–566, 2017. doi: 10.1007/s10208-015-9296-2 . ; Werfel et al., 2003 Justin Werfel, Xiaohui Xie, and H. Sebastian Seung. Learning curves for stochastic gradient descent in linear feedforward networks. In Advances in Neural Information Processing Systems , volume 16, 2003. ) . We test this directly by training four sizes, 2M, 7M, 38M and 243M parameters (a $120\times$ range in parameter count), at a fixed 10M tokens across the population sweep (Figure 4 ; see Appendix F for tuning details). We make two striking observations that challenge conventional wisdom.

  • Bigger models are often more population-efficient, not less. At every population from 256 up the loss falls from 2M to 7M to 38M parameters, and the 243M model is only slightly worse. Even at the smallest population we test, a model $120\times$ larger does about the same, and is better at every other population. Up to a point, a larger model gains more from its size than it loses to variance. What we observe, however, is that the gap to backprop at the same population and model size increases with model size, but barely noticeable, esp at large population sizes.
  • Bigger models keep improving with population where small ones flatten. Small models saturate sooner, while big models keep improving with large population sizes. Past 1k draws the 38M and 243M models gain about 30% more than the 2M and 7M ones, and no amount of population makes the tiny model competitive.

The right way to think about model size is therefore as the size and the geometry of the search space . A larger model has a larger space to search over, which is what lets it put a large population to use, and potentially a better-conditioned loss landscape geometry, which could be why the search is more effective even at small populations.

4.2 Emergence of Backprop-Like Gradients

0.00 0.05 0.10 Cosine to backprop gradient EGGROLL, 10M tokens 64 1k 16k 128k 0.00 0.25 0.50 0.75 1.00 Dust, 10M tokens 64 1k 16k 128k Dust, 100M tokens 64 1k 16k 128k Dust, 1B tokens 64 1k 16k 128k Population MLP avg Q K V head

Figure 5: Cosine between the estimated and the backprop gradient on the same batch, per layer type (mean over layers; MLP averages the input and output projections), against population, for Dust on backprop-trained checkpoints at 10M, 100M and 1B tokens and for EGGROLL at 10M tokens at the same compute budget (left, note the axis). For Dust the lines are the fitted law $\cos(K) = c_{\max}/\sqrt{1 + c/K}$ through measurements at populations of 64 to 128k; for EGGROLL they connect the measurements. The dotted vertical line is the 16k population the ladders train at. Initialization and EGGROLL at 100M tokens are in Appendix B .

We measure the cosine between Dust’s estimate and the backprop gradient on the same batch, per layer type and per layer, on backprop-trained checkpoints spanning two orders of magnitude in tokens, i.e. from 10M to 1B tokens, with the population growing from 64 to 128k forward passes per step (Figure 5 ). The cosine rises with population for every layer type at every stage of training, and a two-parameter law

$$\cos(K) = \frac{c_{\max}}{\sqrt{1 + c/K}}$$

(5)

fits each layer type with RMSE below 0.06, with $c_{\max}$ the ceiling and $c$ the population at which the layer reaches $c_{\max}/\sqrt{2}$. Useful gradients emerge from the large population alone, with nothing about the chain rule built in, just light tuning of the hyperparameters to maximize the cosine similarity as described in Section 2.3 . For the 100M token checkpoint, the fits to layer type mean cosine similarities have RMSEs of $0.0032$–$0.0367$ (see Appendix G for details). Dust’s gradients approach backprop’s in cosine, but how close they get varies by layer type and by layer. EGGROLL’s estimate, measured the same way at the same number of forward passes, is growing with population but is still below 0.05 in cosine at 128k for every type but the head, which might explain why training on it went nowhere in Section 3.2 .

Importantly, the cosines hold up across most layers as the token count grows, which is encouraging for scaling. Section 3 shows the population required to match and exceed backprop grows with tokens. However, the cosine stays flat across two orders of magnitude in tokens at large populations, which means that at a large enough population it’s possible that this requirement no longer grows with more tokens . Furthermore, the fact that Dust’s gradients approach backprop’s but don’t converge to them exactly is in fact a good property. The estimate points in a similar direction without being backprop’s gradient, which leads to a different optimization trajectory, and in our experiments that trajectory can even be better than backprop’s (Section 3.2 ).

5 Conclusion

Since Rumelhart et al. (1986) David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature , 323: 533–536, 1986. , backprop has been the algorithm that trains neural nets, and the architectures, optimizers and hardware we have were all built around it. As compute becomes more abundant, we think much better alternatives are possible. We introduced Dust, an algorithm that drastically improves upon existing ES algorithms and approximates backprop closely at pretraining transformers, even exceeding it with large amounts of computation.

There are many interesting open questions. The first is whether, and how, Dust can find better directions than backprop’s first-order gradient by implicitly exploring the loss landscape , picking up higher-order curvature that pulls the search toward flat regions. We have hints that it can, since at large populations it sometimes ends up below backprop, but the mechanism is not clear. The second is that Dust opens up the search space over architectures, since it does not need the network to be end-to-end differentiable, and it may do better where backprop is known to struggle, like recurrent or looped computation trained by backpropagation through time. The third is compute efficiency, which was not the focus of this paper. We must have orders of magnitude more compute efficiency before Dust becomes a practical alternative to backprop at current levels of compute.

Early work on zeroth-order pretraining of language models ( Allaire et al., 2025 Nathan Allaire, Mahsa Ghazvini Nejad, Sébastien Le Digabel, and Vahid Partovi Nia. Zeroth order optimization for pretraining language models. In Proceedings of ICPRAM , pages 113–121, 2025. doi: 10.5220/0013261100003905 . URL https://doi.org/10.5220/0013261100003905 . ) examined the difficulty of training transformers from scratch with weight perturbations. Their later method, KronZO ( Allaire et al., 2026 Nathan Allaire, Sébastien Le Digabel, Dominique Orban, and Vahid Partovi Nia. Zeroth-order Kronecker optimization for pretraining language models. SN Computer Science , 7, 2026. URL https://www.gerad.ca/en/papers/G-2025-44 . Article 162. ) , uses compact perturbations with Kronecker structure and selective directional updates to improve pretraining while reducing memory use. EGGROLL ( Sarkar et al., 2025 Bidipta Sarkar, Mattie Fellows, Juan Agustin Duque, Alistair Letcher, Antonio León Villares, Anya Sims, Clarisse Wibault, Dmitry Samsonov, Dylan Cope, Jarek Liesen, Kang Li, Lukas Seier, Theo Wolf, Uljad Berdica, Valentin Mohl, Alexander David Goldie, Aaron Courville, Karin Sevegnani, Shimon Whiteson, and Jakob Nicolaus Foerster. Evolution strategies at the hyperscale. arXiv preprint arXiv:2511.16652 , 2025. ) makes large populations of weight perturbations efficient on GPUs through low-rank structure. Dust instead searches over activations and uses rewards for each token to extract more credit from each forward pass.

For fine tuning, MeZO ( Malladi et al., 2023 Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. In Advances in Neural Information Processing Systems , volume 36, 2023. ) showed that language models can be adapted with forward passes alone and memory use close to inference. Evolution Strategies at Scale ( Qiu et al., 2026 Xin Qiu, Yulu Gan, Conor F. Hayes, Qiyao Liang, Yinggan Xu, Roberto Dailey, Elliot Meyerson, Babak Hodjat, and Risto Miikkulainen. Evolution strategies at scale: LLM fine-tuning beyond reinforcement learning. In Proceedings of the International Conference on Machine Learning , 2026. URL https://arxiv.org/abs/2509.24372 . arXiv:2509.24372. ) demonstrates fine tuning of all parameters with ES in language models with billions of parameters. Neural Thickets ( Gan and Isola, 2026 Yulu Gan and Phillip Isola. Neural thickets: Diverse task experts are dense around pretrained weights. arXiv preprint arXiv:2603.12228 , 2026. URL https://arxiv.org/abs/2603.12228 . ) finds useful task experts by randomly perturbing pretrained weights, selecting the best candidates and ensembling their predictions. These results show how much search can achieve around a pretrained model. Our experiments address learning the representations themselves through pretraining from scratch.

Our activation perturbations build on node perturbation ( Werfel et al., 2003 Justin Werfel, Xiaohui Xie, and H. Sebastian Seung. Learning curves for stochastic gradient descent in linear feedforward networks. In Advances in Neural Information Processing Systems , volume 16, 2003. ) . GEMINI ( Le Cun et al., 1988 Yann Le Cun, Conrad C. Galland, and Geoffrey E. Hinton. GEMINI: Gradient estimation through matrix inversion after noise injection. In Advances in Neural Information Processing Systems , volume 1, pages 141–148, 1988. URL https://papers.neurips.cc/paper_files/paper/1988/file/a0a080f42e6f13b3a2df133f073095dd-Paper.pdf . ) injected noise into the first hidden layer and recovered layerwise gradient estimates through iterative matrix inversion. Zoop ( Hu et al., 2025 Xixi Hu, Bo Liu, Qiang Liu, Xiaocong Du, Bhargav Bhushanam, Louis Feng, Chengyue Gong, and Kaizhao Liang. Zoop it! Efficient zero-order optimization with output perturbation. In ICML Workshop on Tiny Titans: The next wave of On-Device Learning for Foundation Models , 2025. URL https://openreview.net/forum?id=Tc8vFyRhPO . ) uses output perturbations for language model fine tuning, converting estimated output gradients into parameter updates with local derivatives. Scaling Forward Gradient with Local Losses ( Ren et al., 2023 Mengye Ren, Simon Kornblith, Renjie Liao, and Geoffrey Hinton. Scaling forward gradient with local losses. In International Conference on Learning Representations , 2023. ) combines activation perturbations and local losses with forward mode automatic differentiation to reduce estimator variance. Forward gradients with multiple tangents ( Flügel et al., 2025 Katharina Flügel, Daniel Coquelin, Marie Weiel, Charlotte Debus, Achim Streit, and Markus Götz. Beyond backpropagation: Optimization with multi-tangent forward gradients. In International Joint Conference on Neural Networks , 2025. URL https://arxiv.org/pdf/2410.17764v2 . ) also use forward mode differentiation, combining multiple directional derivatives through orthogonal projection to improve gradient estimates.

A separate line of work replaces the global backward pass with local learning dynamics: Sakana AI’s PC-ALM ( Seely and Gould, 2026 Jeffrey Seely and Julian Gould. Augmented Lagrangian predictive coding. arXiv preprint arXiv:2605.31022 , 2026. URL https://arxiv.org/abs/2605.31022 . ) propagates credit through local predictive coding dynamics and Lagrange multipliers. It uses local derivatives and is evaluated on image classification tasks. Dust estimates credit from forward perturbations during transformer pretraining, combining rewards for each token with local targets for attention outputs.

References

Nathan Allaire, Mahsa Ghazvini Nejad, Sébastien Le Digabel, and Vahid Partovi Nia. Zeroth order optimization for pretraining language models. In Proceedings of ICPRAM , pages 113–121, 2025. doi: 10.5220/0013261100003905 . URL https://doi.org/10.5220/0013261100003905 .

Nathan Allaire, Sébastien Le Digabel, Dominique Orban, and Vahid Partovi Nia. Zeroth-order Kronecker optimization for pretraining language models. SN Computer Science , 7, 2026. URL https://www.gerad.ca/en/papers/G-2025-44 . Article 162.

Katharina Flügel, Daniel Coquelin, Marie Weiel, Charlotte Debus, Achim Streit, and Markus Götz. Beyond backpropagation: Optimization with multi-tangent forward gradients. In International Joint Conference on Neural Networks , 2025. URL https://arxiv.org/pdf/2410.17764v2 .

Yulu Gan and Phillip Isola. Neural thickets: Diverse task experts are dense around pretrained weights. arXiv preprint arXiv:2603.12228 , 2026. URL https://arxiv.org/abs/2603.12228 .

Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, Runjin Chen, Anna Soligo, Paul Bogdan, Euan Ong, Rowan Wang, Ben Thompson, David Abrahams, Subhash Kantamneni, Emmanuel Ameisen, Joshua Batson, and Jack Lindsey. Verbalizable representations form a global workspace in language models. arXiv preprint arXiv:2607.15495 , 2026.

Xixi Hu, Bo Liu, Qiang Liu, Xiaocong Du, Bhargav Bhushanam, Louis Feng, Chengyue Gong, and Kaizhao Liang. Zoop it! Efficient zero-order optimization with output perturbation. In ICML Workshop on Tiny Titans: The next wave of On-Device Learning for Foundation Models , 2025. URL https://openreview.net/forum?id=Tc8vFyRhPO .

Yann Le Cun, Conrad C. Galland, and Geoffrey E. Hinton. GEMINI: Gradient estimation through matrix inversion after noise injection. In Advances in Neural Information Processing Systems , volume 1, pages 141–148, 1988. URL https://papers.neurips.cc/paper_files/paper/1988/file/a0a080f42e6f13b3a2df133f073095dd-Paper.pdf .

Timothy P. Lillicrap, Adam Santoro, Luke Marris, Colin J. Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience , 21 (6): 335–346, 2020. doi: 10.1038/s41583-020-0277-3 .

Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, et al. On the biology of a large language model. Transformer Circuits Thread , 2025.

Shengchao Liu, Dimitris Papailiopoulos, and Dimitris Achlioptas. Bad global minima exist and SGD can reach them. In Advances in Neural Information Processing Systems , volume 33, 2020.

Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. In Advances in Neural Information Processing Systems , volume 36, 2023.

Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics , 17 (2): 527–566, 2017. doi: 10.1007/s10208-015-9296-2 .

Xin Qiu, Yulu Gan, Conor F. Hayes, Qiyao Liang, Yinggan Xu, Roberto Dailey, Elliot Meyerson, Babak Hodjat, and Risto Miikkulainen. Evolution strategies at scale: LLM fine-tuning beyond reinforcement learning. In Proceedings of the International Conference on Machine Learning , 2026. URL https://arxiv.org/abs/2509.24372 . arXiv:2509.24372.

Mengye Ren, Simon Kornblith, Renjie Liao, and Geoffrey Hinton. Scaling forward gradient with local losses. In International Conference on Learning Representations , 2023.

David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature , 323: 533–536, 1986.

Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864 , 2017.

Bidipta Sarkar, Mattie Fellows, Juan Agustin Duque, Alistair Letcher, Antonio León Villares, Anya Sims, Clarisse Wibault, Dmitry Samsonov, Dylan Cope, Jarek Liesen, Kang Li, Lukas Seier, Theo Wolf, Uljad Berdica, Valentin Mohl, Alexander David Goldie, Aaron Courville, Karin Sevegnani, Shimon Whiteson, and Jakob Nicolaus Foerster. Evolution strategies at the hyperscale. arXiv preprint arXiv:2511.16652 , 2025.

Jeffrey Seely and Julian Gould. Augmented Lagrangian predictive coding. arXiv preprint arXiv:2605.31022 , 2026. URL https://arxiv.org/abs/2605.31022 .

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature , 550 (7676): 354–359, 2017. doi: 10.1038/nature24270 .

Richard S. Sutton. The bitter lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html , 2019. Blog post.

Akshay Vegesna and Samip Dahal. Decoupling search and learning in neural net training. arXiv preprint arXiv:2509.10973 , 2025.

Justin Werfel, Xiaohui Xie, and H. Sebastian Seung. Learning curves for stochastic gradient descent in linear feedforward networks. In Advances in Neural Information Processing Systems , volume 16, 2003.

Bernard Widrow and Michael A. Lehr. 30 years of adaptive neural networks: Perceptron, Madaline, and backpropagation. Proceedings of the IEEE , 78 (9): 1415–1442, 1990. doi: 10.1109/5.58323 .

← Back to Q

Opus 5.5 agents discover 2 room-temperature magnetic semiconductor candidates

Hacker News
www.vals.ai
2026-10-05 17:00:21
Comments...
Original Article

We’re all used to two types of magnet. The common one, the fridge magnet, is ferromagnetic — its atomic magnets all point the same way (up or down), adding their magnetic effects. The less well known one, the antiferromagnet (AF), has neighbouring atomic magnets that point opposite ways and exactly cancel out magnetically.

For a long time, there’s been an intense drive in computer memory research to create materials in between these two extremes. For this purpose, it helps to have a clear picture of what these extremes mean.

Today, I’ll share what we found. A team of AI agents and I designed one candidate magnet and found another, first made in 1999, that our calculations predict has the properties we were after.

But before diving into the details, let me first lay out a magnet primer that takes all of 90 seconds, assuming you are not an undergraduate in physics or chemistry.

A 90 second primer in magnets

Spin

Each electron has a quantum mechanical property called ‘spin’, which is responsible for its magnetic moment. We can model the spin direction for each electron as either pointing up or down.

Spintronics

We use spin for storage. A magnetised material stores information based on the spin up/spin down orientation of its electrons, in the same way classical magnets store information based on pointing up/down. The most prominent example of spintronics is the hard drive read head, the device that reads the magnetic bits on the disk. MRAM is another type of spintronics that uses the same principle to store binary information in a non-volatile way.

In the world of spintronics, we want to sort electrons by their spin orientation so we can read/store their information. Ferromagnets do this naturally: the electrons that carry current are mostly of one spin, up or down. Ordinary antiferromagnets, however, cannot distinguish up/down electrons.

This leads us into the next section.

Three kinds of magnets

Ferro

Ferromagnetic materials are characterised by having a macroscopic magnetic field, or a field that leaks out from the surface. This is why a fridge magnet sticks to your refrigerator door.

The problem, however, is that the magnetic field interferes with nearby materials and is difficult to control for storage purposes. In addition, switching magnets back and forth is relatively slow and consumes a lot of power.

Another feature of ferromagnetic materials is that the spins are sorted (by up/down orientation) according to energy level. We can see this when looking at the energy spectrum of the electrons: near the edges of the spectrum, the electrons all have the same spin.

Antiferro

If we take the above and flip the logic, meaning, the spins are unsorted, we end up with an ordinary antiferromagnetic material.

Here the electrons of the same energy level will instead have mixed spins. This leads us to two properties of interest:

  • Because spins are not sorted according to energy level, it is hard to read/store information using spintronics techniques.
  • However, the lack of macroscopic magnetic field allows us to pack these materials closer together, enabling higher performance for storage devices. In terms of speed, AF materials are also about a thousand times faster to switch.

The only problem is, if the spins are mixed at each energy level, how would we ever sort them by energy? How do we get a way to read/store information using spintronics with this lack of sorting?

Luttinger compensated

This brings us to a third type, Luttinger compensated (LC).

LC materials are antiferromagnets where the spin-up atoms and spin-down atoms have the same magnitude of magnetism, making the net spin moment zero (i.e. they cancel out). However, unlike in ordinary antiferromagnets, the up and down atoms sit in inequivalent environments: they can be different elements (e.g. one element points up and another points down), or the same element in two different kinds of site. The name comes from Luttinger’s theorem: in an insulator, the net spin moment of each repeating unit of the crystal must be a whole number, so once it is zero it stays locked at zero. Strictly, that holds for a perfect crystal near absolute zero; smaller effects such as spin–orbit coupling, and heat, can leave a slight imbalance.

Since the up and down atoms are not equivalent, up/down spins can now be separated (sorted) by energy, just like in ferro materials.

Let’s take the previous sections and look at how spin is distributed across the energy landscape for ferro, antiferro and LC magnets.

Three kinds of magnet: a ferromagnet, an antiferromagnet and a Luttinger-compensated magnet, with their net magnetism and how their electron levels are sorted by spin

What matters for storage is the “spin window”: the slice of energy at the edge of the band gap where every available electron state has the same spin. The larger this window compared with the thermal jiggling at room temperature (about 26 meV), the better the electrons stay sorted.

This means we’d love to have a semiconductor with a band gap, without losing the ability to separate spins by their energy levels, and with zero net magnetism.

What a spin window is

This is where our AI agents (and me) enter the scene. Let’s dive into how these agents found two promising materials for next generation computer memory.

The agents ran quantum-mechanical simulations of each crystal with the standard method for this, density functional theory, at two levels of approximation: a faster one (PBE+U) and a slower, usually more accurate one (HSE06). The band gaps and spin windows below come from the more accurate one.

Candidate 1: Designed a Luttinger Compensated Magnet, YBaMnFeO₅

First, let’s see what our AI agents designed:

  • A new compound made of only 5 elements (yttrium, barium, Mn, Fe, O). As far as we could find, it has never been made, nor proposed as this kind of magnet
  • The compound was predicted to be a semiconductor
  • This Luttinger-compensated magnet is predicted to have a 2.35 eV band gap, where spin sorting occurs at both sides of the band gap: a window of 1.0 eV for holes and 1.4 eV for electrons. Note that thermal agitation at room temperature only causes around 26 meV of fluctuation
  • Finally, our agent predicted the material to retain magnetism up to an elevated temperature: about 420 K in the raw simulation, or about 490 K after calibrating the simulation against a known magnet

While this seems like a pretty promising candidate, the design needs the Mn and Fe atoms to sit in a perfect checkerboard. When our agents simulated how the atoms arrange themselves at different temperatures, that checkerboard fell apart into a random mix at around 950 K. Making this kind of oxide takes about 900–1300 °C, and at lower temperatures the atoms barely move, so standard synthesis would likely give a scrambled crystal.

Why the ordered form of YBaMnFeO5 may be hard to reach

A scrambled crystal loses the spin sorting, so this design may be hard to make in its useful form.

Candidate 2: Identified a Luttinger Compensated Magnet in KV[Cr(CN)₆] from 1999

Shortly after, my agent found this interesting material, previously reported only in 1999, that matches many of the requirements above. Its zero net magnetism is not new: the chemists who made it designed the two metals’ magnetism to cancel. Even its spin sorting was already on paper. A 2008 study using hybrid functionals like ours plotted its electron states spin by spin, and both band edges in that plot carry the same spin. But that study was about magnetic coupling under pressure and never remarked on it. As far as we found, nobody had pointed out that this makes KV[Cr(CN)₆] a Luttinger-compensated semiconductor, put numbers on its spin-sorted windows, or tested how robust they are. It had been hiding in plain sight.

KV[Cr(CN)₆] belongs to the same family as Prussian blue, the 300-year-old pigment.

Our agent identified it as a Luttinger-compensated material:

  • It predicts the material will have a band gap of about 2.1 eV, with both band edges sorted into the same spin: windows of 2.6 eV for holes and 1.6 eV for electrons.
  • The 1999 sample stayed magnetically ordered up to 376 K (103 °C), above room temperature, as measured by the chemists who made it (365 K after the sample had been heated).
  • And most importantly, its structure locks each metal into its own site: chromium bonds to the carbon end of each cyanide and vanadium to the nitrogen end, which is exactly what YBaMnFeO₅ lacked.

These predictions are for a perfect, dry crystal. The only sample so far, from 1999, is a powder with water in its pores, and it showed a small leftover magnetic moment (0.125 Bohr magnetons per formula unit, where the perfect crystal would have zero). Our two simulation methods disagree on how much the water weakens the effect: the more accurate one (HSE06) says the spin sorting survives, while the faster one (PBE+U) says the hole window shrinks by more than half. Neither the band gap nor the spin sorting has been measured yet.

The bottom line

We identified a material that combines properties of the two familiar kinds of magnet and could be useful in spintronic devices. We believe the two materials discussed in this article are promising examples of a class of materials that have both antiferromagnetic and ferromagnetic properties.

The key features for both materials: in our calculations for perfect crystals they have zero net spin moment, yet they are semiconductors whose electrons are sorted by spin over windows far larger than the thermal jiggling at room temperature. With little stray magnetic field, they shouldn’t disturb their neighbours much.

Here is the score of our two materials:

  • YBaMnFeO₅: designed candidate; may be hard to make.
  • KV[Cr(CN)₆]: first made in 1999. Near-zero net magnetic moment in the 1999 sample (zero predicted for a perfect crystal), magnetic up to 376 K.

This suggests that materials for the next step in spintronics may already exist, waiting to be recognized. A 2025 study that predicted two other Luttinger-compensated semiconductors found that both lose their magnetic order below room temperature, and named a room-temperature one as the open goal. KV[Cr(CN)₆] has already been made, and its 1999 sample stayed ordered up to 376 K. Identifying room-temperature Luttinger-compensated semiconductors is a step toward practical spin-based technologies, particularly if their magnetic properties can be controlled through chemistry. The next step is to make KV[Cr(CN)₆] again and measure its spin sorting directly.

In the spirit of transparency, I invite the reader to go through all the computations that produced the above predictions, including the calculations behind the candidate designs. The input files, raw outputs and analysis code behind the numbers in this post, along with a one-command checker, independent re-runs and a list of known caveats, have been shared on GitHub in a public repository I created to document my research journey: github.com/spicylemonade/compensated-magnet-ledger .

Desperate to Cling to Power, Zombie Brooklyn Dems Resort to Sham, Late-Night Zoom Election

hellgate
hellgatenyc.com
2026-10-05 16:47:46
“It’s a pretty flagrant move to just say, ‘Fuck this court, fuck these orders.’”...
Original Article

Great! You’ve successfully signed up.

Welcome back! You've successfully signed in.

You've successfully subscribed to Hell Gate.

Your link has expired.

Success! Check your email for magic link to sign-in.

Success! Your billing info has been updated.

Your billing was not updated.

Unroll a loop, make shader 10x faster

Lobsters
aras-p.info
2026-10-05 16:37:27
Comments...
Original Article

Loop unrolling is complexicated. Usual knowledge is that the compiler has way more knowledge than you might have about source code and the target machine, and so you should just write a loop, and let the compiler decide what to actually do with it. This is a good rule of thumb, most of the time.

The other day I made one GPU compute shader in Blender 10 times ( ten times! ) faster by unrolling a loop though. (ten times on one GPU… on another GPU it is two times faster, but still!)

Kuwahara image filter

Blender has a Kuwahara compositor node, that produces a “painterly” look from an input image. Sometimes it is used as a building block for more complex painterly effects.

Fun fact! Original Kuwahara algorithm was developed for biomedical image processing needs, to smooth out noise while preserving boundaries. Kuwahara, Hachimura, Eiho, Kinoshita “Processing of RI-Angiocardiographic Images”, 1976. Much later, Kyprianidis, Kang, Döllner “ Image and Video Abstraction by Anisotropic Kuwahara Filtering ” (2009) extended it to look better and the goal was non-photorealistic image & video processing.

Anisotropic Kuwahara filter is quite expensive to calculate, and most of the cost comes from the fact that each and every output pixel must look at an oriented ellipse around itself (ellipse orientation & shape changes from pixel to pixel), sample input pixels inside of it, assign the inputs to eight sectors, calculate sector statistics, and then blend their average colors, preferring sectors with lower variance. The sectors are also actually overlapping with each other, ugh.

But, it produces quite nice “painterly” results, here’s a result that we can consider to be ground truth because the dog’s name is True :

The problem

Anyway, I was looking at it for some small optimizations. On my PC (Ryzen 5950X / RTX 3080Ti), doing Kuwahara with size 8 on a 2048x1536 input image took 408ms on the CPU, and 19ms on the GPU. So the GPU is about 20x faster at this, which sounds fine.

But then! On an Apple M4 Max device, the CPU took 385ms (faster than this Ryzen, eh), yet the GPU took a whopping 220ms. Like, what? One could go “well obviously this Apple GPU is slower than that nvidia GPU! And GPU path on the Apple device is still faster than CPU, so good” and move on. But the ratio felt way off; why the first machine would have 20x performance advantage by using GPU, whereas the second one would not even be twice as fast? And also, the M4 Max GPU might be slower than RTX 3080Ti, but it should not be ten times slower.

Let’s investigate a Metal shader

Apple Xcode can do a frame capture of an application that uses Metal graphics API, and then you can use it to do debugging, performance insights and so on. I’m using Xcode 26.3 on macOS 15.7.3. Capture a frame with performance analysis enabled, and click Show Performance button:

The Top Shaders view shows the most expensive shaders, where we find our main Kuwahara compute shader. It uses 192 registers ( a lot! ), and even spills 64 bytes since it’s kinda out of registers ( that is not great at all ):

Clicking on Counters tab makes the whole computer almost unresponsive for several minutes, while Xcode is really busy doing something . Then it shows that our compute shader only has 28% occupancy (not great), but more curiously, the instruction mix is like:

  • ALU float instructions: 12% (huh? that low? source code was full of math!)
  • ALU half instructions: 0% (yes, Blender today does not use FP16 math in shaders… perhaps it should, but that’s for another day)
  • Conditional, Integer and Boolean instructions: 87%. What?!

With our compute shader still selected in Overview tab, right click on it and pick Reveal in Cost Graph , and it shows a flamegraph of the shader code & function calls (pretty much everything in just main entry point), but more importantly, it shows the shader source code with approximate per-line costs below. Scroll to the expensive part:

This is the inner shader loop, where for each image sample we check which sector(s) it is in, weigh them, calculate their statistics and so on. It would make sense that this is the heaviest part of the shader, since it is the innermost loop after all. But hovering on each of the line-attribution-bubbles shows something like this:

Wait what? Why would a line like weighted_mean_of_color_of_sectors[upper_index] += upper_color * weight be mostly “select” instructions, and not just math?!

At least this version of Xcode just stops there. It says there are a ton of “select” instructions, but does not actually show them. What is going on there? Why? No idea, but it would be really nice if Apple had tools that could enable us to drill deeper. Since Xcode can not help us further, we’ll have to go heavy metal 🤘

Dougall Johnson has a project with Apple M1 (“G13”) GPU reverse engineered, with extensive documentation and disassembler and other tools at github.com/dougallj/applegpu . It is not exactly the same GPU that I am on (which is M4), but this will have to do.

After some goofing around, something like this seems to achieve a “GPU disassembly for M1 GPU” from my Metal source code:

  1. Compile Metal source code into a metal library: xcrun -sdk macosx metal metal_source.metal -o test.metallib
  2. Create a test.mtlp-json file (this describes the pipeline state) with contents like: 
    {"pipelines":{"compute_pipelines":[{
"compute_function":"_compositor_kuwahara_anisotropic_constant_size_comp",
"threadgroup_size_is_multiple_of_thread_execution_width":true,
"max_total_threads_per_threadgroup":1024}]}}
    and then compile to M1 GPU ( applegpu_g13g ) code with xcrun metal-tt -arch applegpu_g13g -o test_g13g.gpubin test.mtlp-json test.metallib
  3. This produces a .gpubin file which is a Mach-O file, and we are interested in the __TEXT,__compute section. That section itself is another Mach-O file! And from that file, we need the __TEXT,__text section. With that extracted into a separate file like g13g.bin , we could finally disassemble it.
  4. But the disassemble.py tool stops on the first stop instruction, which is placed in some sort of shader preamble. Disable this stopping to get full shader disassembled: 
    cd applegpu
python3 -c "
import disassemble as d
d.STOP_ON_STOP = False
d.disassemble(open('../g13g.bin','rb').read())" > ../g13g.asm

And yes, majority of instructions in the shader disassembly are icmpsel . There’s also a bunch of stack_store and stack_load , which seem like that’s the “spilled bytes” that Xcode was pointing out. Here’s a snippet of actual instruction sequence from the GPU shader disassembly: 

mov              r9, r51.cache
icmpsel          seq, r51, r1l.cache, 2, r7.cache, r34.discard
icmpsel          seq, r18, r1l, 3, r7, r18
stack_store      i16, 1, 0, xy, 4, r18l_r18h, 88, 0
icmpsel          seq, r16, r1l, 4, r7, r44.discard
stack_store      i16, 1, 0, xy, 4, r16l_r16h, 84, 0
icmpsel          seq, r7.cache, r0h.cache, 1, r14.cache, r12.cache
icmpsel          seq, r14.cache, r0h.cache, 1, r39.discard, r6
icmpsel          seq, r7.cache, r0h.cache, 2, r51, r7.cache
icmpsel          seq, r14.cache, r0h.cache, 2, r24.discard, r14.cache
icmpsel          seq, r7.cache, r0h.cache, 3, r18, r7.cache
icmpsel          seq, r14.cache, r0h.cache, 3, r13, r14.cache
icmpsel          seq, r7.cache, r0h.cache, 4, r16, r7.cache
icmpsel          seq, r14.cache, r0h.cache, 4, r5, r14.cache
icmpsel          seq, r7.cache, r0h.cache, 5, r38, r7.cache
icmpsel          seq, r14.cache, r0h.cache, 5, r3.cache, r14.cache
icmpsel          seq, r7.cache, r0h.cache, 6, r20, r7.cache
icmpsel          seq, r14.cache, r0h.cache, 6, r11, r14.cache
icmpsel          seq, r7.cache, r0h.cache, 7, r19, r7.cache
icmpsel          seq, r14.cache, r0h.cache, 7, r8, r14.cache
fadd32           r7.cache, r7.cache, r31
fmadd32          r50, r32, r29, r14

Out of 22 instructions, only two do some sort of math ! The rest are integer comparison-selects and some stack data movement. What seems to be going on, is that the final GPU shader compiler did not unroll this loop (every loop iteration has the array index [i] that is just a variable), and shader variables like float sector_weights[8] , float4 weighted_mean_of_color_of_sectors[8] and float4 weighted_mean_of_squared_color_of_sectors[8] become actual arrays inside the GPU registers. At least this GPU cannot index registers! So in order to do any array index the compiler emits a sequence like if i==0 use this, else if i==1 use that, else if i==2 use this, ... . All of that expands to about 200 integer select instructions per loop iteration .

Madness!

Let’s unroll the loop

Blender’s shaders are written in a curious shading language tentatively called “BSL” (see BCON26 talk about it, “ How Not to Build a Shading Language ”). For explicit loop unrolling it has a C++-style attribute [[unroll]] , which literally copy-pastes the loop body before passing it to a platform compiler. 

for (int k = 0; k < 8 /* number_of_sectors */; k++) [[unroll]] {
  float weight = sector_weights[k] * radial_gaussian_weight;
  sum_of_weights_of_sectors[k % (number_of_sectors / 2)] += weight;

  int upper_index = k;
  weighted_mean_of_color_of_sectors[upper_index] += upper_color * weight;
  weighted_mean_of_squared_color_of_sectors[upper_index] += upper_color_squared * weight;

  int lower_index = (k + number_of_sectors / 2) % number_of_sectors;
  weighted_mean_of_color_of_sectors[lower_index] += lower_color * weight;
  weighted_mean_of_squared_color_of_sectors[lower_index] += lower_color_squared * weight;
}

This makes the shader run 10 TIMES faster .

It was taking 220ms before, and now it runs in 20.4ms. The shader instruction mix went from (12% float math, 87% conditional and integer) to (79% float math, 17% conditional and integer), which feels way more sensible. It still uses 192 registers and spills (64 bytes -> 48 bytes), but that’s a topic for another day. And the (unrolled) inner loops are now all just math instructions:

Okay, but what about other GPUs?

Fair question! I only have two GPUs around here, the other one being RTX 3080Ti. Which, funnily enough, is another GPU where the GPU vendor does not easily allow you to see “what is the actual GPU assembly” bit :)

Nsight Graphics allows me to debug and profile graphics applications, but for example when using Vulkan, it can show me which of the SPIR-V assembly instructions were costly, but how they map into the underlying GPU instructions, is anyone’s guess.

In this particular case, speedrun of Kuwahara through Nsight (2026.3 version): Start Activity, pick “GPU Trace Profiler” option, turn on “Real-Time Shader Profiler”, launch the application, do a capture. Find the expensive compute shader dispatch in the timeline (it takes 17.8ms here):

In lower right corner, under Shader Pipelines section, it displays some data about this compute pipeline:

It can run 8 warps, uses 204 registers (198 live registers), average warp latency is 134K, instruction mix: 44% FMA FP32, 36% data movement.

With unrolled loop, now our compute dispatch takes only 9.5ms:

And it can run 16 warps, uses 113 registers (103 live registers), average warp latency is 45K, instruction mix: 60% FMA FP32, 13% data movement.

So yes even on RTX 3080Ti this particular shader gets twice as fast by just unrolling the inner loop. Half the registers results in twice the occupancy here, but TBH I did not dig much deeper into why exactly the register usage has decreased.

What’s the lesson?

Yes the loop unrolling should be left to the compiler… except when it should not :)

That by itself is not very useful, so perhaps a better lesson is - it helps to have at least a vague understanding of what sort of performance should be expected. This way you can go “wait a minute, this feels way too slow” and investigate. In this case, I wanted to fix seemingly very bad compute shader performance on an Apple GPU (and sped it up ten times ), but a side effect was that on an NVIDIA GPU the shader also got twice as fast.

It helps if the performance analysis tools can point out some sort of data that makes you go “huh wait, this should not be here”, like in this case, majority of shader instructions being integer compare/selects. It would be even better if these tools could actually show more data; for Apple GPUs in Xcode frame capture and NVIDIA GPUs in Nsight, I’d like to see GPU assembly. One can dream!

And now I’ll go and ship these Kuwahara node performance improvements in Blender ( PR 164617 , just merged for upcoming Blender 5.3). There’s probably more work to do in that area, but that’s for later.

Rejetto HFS servers now actively scanned for critical RCE flaw

Bleeping Computer
www.bleepingcomputer.com
2026-10-05 16:20:05
Hackers are actively scanning for a Rejetto HFS weak signing key vulnerability, tracked as CVE-2026-61500, that allows session forgery, account takeover, and remote code execution (RCE). [...]...
Original Article

Hacker

Hackers are actively scanning for a Rejetto HFS weak signing key vulnerability, tracked as CVE-2026-61500, that allows session forgery, account takeover, and remote code execution (RCE).

VulnCheck VP of Security Research Caitlin Condon posted on LinkedIn over the weekend that the company's Canary Intelligence honeypots had observed probes targeting CVE-2026-61500.

Condon said the observed activity appears to be small-scale reconnaissance from a single China Telecom IP address probing deployments in Japan and the United States.

Rejetto HFS (HTTP File Server) is a free and open-source file-sharing server tool used for self-hosted file sharing on Windows, Linux, and macOS.

CVE-2026-61500, first published on July 13, 2026 , is a session-cookie signing weakness and leakage issue fixed in Rejetto HFS  version 3.2.1.

"Rejetto HFS 3.0.0 through 3.2.0 derives its session-cookie signing key from the non-cryptographic Math.random() generator and discloses outputs of the same generator to unauthenticated clients during login," reads the flaw description on the NIST NVD.

"A remote attacker can collect a small number of login responses, reconstruct the generator's state, recover the signing key, and forge a valid administrator session cookie, leading to full administrative access and remote code execution via the server_code configuration feature."

Horizon3 researchers discovered the flaw using Anthropic's Mythos model, which identified both the weak signing-key generation and the leak that enabled key recovery.

Horizon3 published more details about the flaw and a proof-of-concept (PoC) exploit in a write-up on September 30, 2026.

"Mythos didn't just flag the insecure PRNG in isolation – it simultaneously identified that the application leaked raw Math.random() outputs through a separate code path, recognized those two facts as a chain, and determined the leak produced exactly the observations needed to make state recovery feasible," explained Horizon3 .

Recovering the session key
Recovering the session key
Source: Horizon3

The researchers' exploit demonstrates the chain to abuse HFS's built-in ability to execute custom server-side JavaScript to achieve remote code execution.

The release of these technical details may have prompted the probing activity targeting CVE-2026-61500.

Possible attack scenarios include accessing, stealing, or deleting HFS files, installing malware on the server, or using the compromised host to access internal systems.

However, VulnCheck has not shared details on successful exploitation or any post-exploitation activity.

Users of Rejetto HFS are recommended to upgrade to version 3.2.1 or, ideally, the latest stable release, 3.3.4, as soon as possible.

article image

Build your security blueprint for AI-powered attacks

Join Mikko Hyppönen and security leaders from the NFL, CHANEL, and Atlassian for a two-hour digital summit on what AI-speed attacks change, what defenders should stop doing, and how to validate, decide, fix, and re-validate at machine speed.

Save your seat

Dostoevsky: Better Space-Time Trade-Offs for LSM-Tree Based Key-Value Stores via Adaptive Removal of Superfluous Merging

Lobsters
nivdayan.github.io
2026-10-05 16:18:23
Comments...
Original Article
No preview for link for known binary extension (.pdf), Link: https://nivdayan.github.io/dostoevsky.pdf.

A third way of using Linux

Hacker News
hisvirusness.com
2026-10-05 16:17:07
Comments...
Original Article

When you want to use Linux, you do so in two ways. You either use the command-line interface and type out every single command, or you use the graphical user interface, which is much more user-friendly (on average). However, there is a third way, and it's not talked about as much because it never had enough time to pick up steam.

"What 'third way'?"
Think about it: What gives you the benefit of a visual interface, without the bloat that comes with Wayland or X11 ? What mode of computing seemed to have come and gone so fast, that modern applications in its form seem noteworthy and bizarre because of the form factor alone? The Third Way is the text user interface , a.k.a., the TUI .

The True Middle Ground

The TUI experience is text-only, but is more approachable than the command line, despite technically being run inside of a CLI session. And because it doesn't require services like X11 or Wayland on the lower level, it frees up even more RAM for the system to use, while still giving an accessible UX.

You typically see it used in isolation: htop/btop , Midnight Commander , and distro installers like archinstall and slackinstall . What we've never seen is a unifying system that treated the TUI itself as the primary user experience. I think, in 2026, it's time to consider making the TUI a first-class UX for *nix OSs.

Modern-Day Considerations

I don't know if you've checked RAM prices lately, but they're at a premium. Users are now urging devs to optimize their applications and are largely working with what they have, because you never know when your memory could start to fail. Add in other factors, like the shift away from bloated frameworks whenever possible, and people are starting to realize something that a few of us already knew: Obsolete doesn't automatically mean bad .

Linux repos still contain GUI-based desktop environments and window managers from almost 30 years ago, yet there have been no notable TUI-based equivalents to them from any point in time. Why? Personally, I think it's because the concept was considered obsolete before it could gain traction at a time when "obsolete" meant "not worth navigating". GUI toolkits had evolved and been standardized early on, so all of the attention had been placed on them. Approaching the TUI in new ways was never considered, so because of that, it has a generational gap that stops us from using our devices in a different way entirely.

"Doesn't a tiling WM like i3 or sway accomplish this concept for the most part?"
Yes. However, they are still GUIs , not true text mode . They still have the same heavy graphical dependencies as any GUI, with a UX and aesthetic that is explicitly tied to the GUI, regardless of how hard you rice your setup. No matter how much you cut down a tiling WM, it's still not going to be as light as a pure TUI.

Obvious Flaws

Of course, there are drawbacks. A/V and graphics editing are out of scope, but modern-day web browsing could still be done... so long as you're a fan of the IndieWeb, since Electron apps are also out.

Gaming is mostly stuck to emulation. Both RetroArch and EmulationStation could be run without the need for either X11 or Wayland , though doing so does mostly break the TUI UX concept.

But the limitations are what make this "The Third Way" : Of course the actual experience is going to have its differences, and is obviously more geared toward uses that don't involve graphics .

How To Accomplish?

Something like TUIOS . It's a great showcase for the concept, but still has visual dependencies. Keeping it in TTY, without the assumption of a mouse, and with "windows" taking up the full screen instead of a small "windowed" section would be how I would envision it. You'd have the main window, with ASCII panels at the top and bottom giving up-to-date system info. And to switch between running apps, just Alt+Tab like you would any other WM.

Linux is not short of TUI tools. They are pretty common and mature. What's missing is a unifying environment to bring them all together in a cohesive way, and hopefully inspire more TUI-native development. Then over time, the actual UX could evolve on its own, ideally without just becoming another GUI.

Could it be done? Absolutely. Could I do it? ...Check back with me in a few years.

So, what do you think? Is The Third Way something you'd be interested in? Am I looking at the concept all wrong? Was this whole idea dump just a gigantic waste of time and energy?
Leave a comment , thanks for reading and stay tuned for more bullshit .

[ ~/third-is-the-way ]

GitHub Actions Has Problems

Hacker News
www.githubstatus.com
2026-10-05 16:09:12
Comments...
Original Article

Update

Actions is experiencing degraded availability. We are continuing to investigate.

Posted Oct 05 , 2026 - 20:47 UTC

Update

We’re continuing to investigate job failures and delays affecting GitHub-hosted runner assignment and workflow start times. Our teams are actively working to mitigate the impact and will provide another update as we learn more.

Posted Oct 05 , 2026 - 20:39 UTC

Update

We’re still investigating delays in assigning GitHub-hosted runners, affecting workflow start times across multiple runner configurations. Our teams are working to mitigate the impact; we’ll share another update as we learn more.

Posted Oct 05 , 2026 - 19:50 UTC

Update

We’re investigating an issue causing delays when assigning GitHub-hosted runners to Actions jobs. Some workflows may take longer to start across runner configurations. Our teams are working to mitigate the issue, and we’ll provide an update as we learn more.

Posted Oct 05 , 2026 - 19:15 UTC

Investigating

We are investigating reports of degraded performance for Actions

Posted Oct 05 , 2026 - 19:11 UTC

This incident affects: Actions.

The mental health of young men is declining. Experts warn it could get worse

Hacker News
www.cbc.ca
2026-10-05 16:07:25
Comments...
Original Article

Young men are struggling. Can Canada fix the problem?

Canadian men and boys are facing an escalating societal crisis as their rates of depression, anxiety and addiction rise. For The National, CBC's Omar Dabaghi-Pacheco investigates the roots of the struggle and whether a new federal mental health strategy goes far enough.

On the surface, Nick Cholmsky looked like a thriving teenager.

Growing up in Ottawa, he says he was athletic and heavily involved in basketball, had friends and a close relationship with his family.

But underneath all that he says he was deeply depressed. The symptoms began in middle school and became much worse by Grade 11.

At 16, he had begun entertaining suicidal thoughts.

A young man in a red shirt.
Nick Cholmsky, who grew up in Ottawa, experienced depression as a teen. He says online messaging encourages boys and men to stay silent about their psychological struggles. (CBC)

Even as his depression and anxiety grew stronger, Cholmsky believed he couldn’t say anything, because "being a man" meant pushing negative thoughts down, and that any sign of weakness would cause him shame.

"Looking back, so much of that [congenial attitude] was for sure just a performance, me acting the way I thought I was supposed to act as a man," said Cholmsky, now 21.

There’s a common problem everywhere and it’s how men and boys are feeling. And they are not feeling great. - Federal Health Minister Marjorie Michel

Earlier this year, the federal government reported on a wide range of data showing that the mental health of young men has deteriorated since 2012. Between 2012 and 2022, the number of men aged 15 to 24 who self-reported mood disorders increased from 11 per cent to 16 per cent, while self-reported generalized anxiety disorders increased from four per cent to 10 per cent.

Those who reported their mental health as "very good" or "excellent" plummeted from 70 per cent to 52 per cent in that same time period.

A separate report shows that men are three times as likely as women to kill themselves and that it's the second-leading cause of death in boys aged 15 to 34 (after unintentional injuries).

"There’s a common problem everywhere and it’s how men and boys are feeling. And they are not feeling great," federal Health Minister Marjorie Michel told CBC News.

"There is a lot of anxiety [about] their role as men and boys."

Social media use, gambling major issues

Research implicates a variety of factors in men’s depression, most prominently social isolation, social media use and online sports betting.

Data from the Canadian Medical Association Journal shows a huge rise in young men calling helplines with gambling addictions. (Health Canada reports men are three times more likely than women to have a gambling problem.)

Canada legalized single-event sports betting in 2021. In 2022, Ontario became the first province to allow private companies to operate alongside government-regulated industry.

A separate report released this month by the Institute for Clinical Evaluation Sciences (ICES) showed that since 2022, emergency room visits for gambling disorders had doubled — largely with young men showing up in a psychological crisis, such as suicidal ideation or psychosis, as a result of gambling losses.

"The gambling stats didn’t surprise me at all when I heard them," said Cholmsky. "I saw kids in middle school gambling."

WATCH | Investor and podcast Scott Galloway on male potential:

Scott Galloway on male potential

U.S. investor and podcaster Scott Galloway talks to CBC News about the impacts of the manosphere and the loss of 'off-ramps to the middle class,' and how young men are being impacted.

Whether the stress is the result of gambling or something else, Cholmsky says online messaging encourages boys and men to stay silent.

"When I was going through some of my worst mental health times, I was online a lot and it was really the height of Andrew Tate," Cholmsky said.

The U.S.-born Tate is one of the best-known influencers in the manosphere, famous for espousing strict rules for being an alpha male, such as maximizing fitness, maintaining control over relationships and emotions and making as much money as possible.

He retains a large following despite being in U.S. detention on multiple criminal charges, including sex trafficking.

"[Tate's] message, and a lot of that manosphere messaging for boys, is 'be a man, be tough, make lots of money,'" Cholmsky said.

There are also trends like "looksmaxxing." Influencer Braden Peters, also known as Clavicular, has amassed a huge following telling young boys sometimes to smash their own faces with hammers to be more conventionally beautiful. His content has more than 40 million likes. Peters was charged with rape last month.

Experts warn dangerous role models like this can have real-world impacts.

"The manosphere is radicalizing young boys who don’t know where they fit  — and that should concern us all," said John Oliffe, Canada research chair in men’s health promotion at the University of British Columbia.

Oliffe says this was "amplified" by the earliest parts of the COVID-19 pandemic when global lockdowns increased isolation and the loss of social connection. Vulnerable teenagers often found themselves lost online.

"There continues to be a whole world of pain in some of that unregulated content," Oliffe said.

A man with grey hair in a button-down shirt.
John Oliffe, Canada research chair in men’s health promotion at the University of British Columbia, says the radicalizing effects of the manosphere should worry us all. (Nav Rahi/CBC)

Adam Kunder, a 33-year-old former firefighter who lives in Toronto, says he also felt he needed to keep his problems quiet. He suffered mental health issues and hit "rock bottom" before beginning to find help in the summer of 2024.

"As a man, you think you are supposed to be strong, push any bad feelings down, suppress depression or sadness or fear," he said.

Kunder says he has lost multiple friends to suicide and says you never would have known they were suffering.

Gender roles and 'toxic masculinity'

Kunder said the world is changing in ways that are becoming confusing for many men.

The emphasis on women's parity in the labour force and society, as well as a greater acceptance of non-traditional gender identities, has upended expectations.

"At one point we knew we were supposed to be the provider and protector, and the idea of ‘toxic masculinity’ wasn’t a thing," he said. "I think men are really trying to figure out what it means to be a man in today’s world."

WATCH | Why manosphere influencers appeal to young men:

Why ‘manosphere’ influencers are appealing to young men

Critic Amil Niazi joins Commotion with Elamin Abdelmahmoud to discuss Louis Theroux: Inside the Manosphere, a documentary about the online world of toxic masculinity, and examine the rising popularity of "manosphere" influencers among young men.

Oliffe pointed to other troubling patterns for young men.

"There are increasing lower rates of boys heading to universities, and very serious challenges in male-dominated industries such as construction, that as a society, we should not ignore."

Oliffe said that while the trades offer good careers, they often see high rates of substance use and mental health issues, as the necessary physical labour and harsh working environments can "heighten these risks."

Oliffe said that in the last decade, there has been "low government investment in men's mental health."

"I think the theme of silence is a very regrettable period in history, where on one side we talked about equity and inclusion, and then in effect ostracized 50 per cent of the population," Oliffe said.

Health Minister Michel said that when she brought this issue to the federal cabinet, she was met with some resistance. Some colleagues told her it took "courage" to broach the mental health of boys and men, given it could even be politically risky to talk publicly about the issue.

"I said 'courage'? This is not courage. We are ministers of health, it is our job to take care of the entire population," she said.

"Men don't know how to ask for help. And when they are struggling, they don't know how to express it."

Government strategy to come by early 2027

The federal government will soon l aunch its first-ever mental health strategy for boys and men.

The agency told CBC that some of what they hope to invest in will be " mentorship, peer support, sport and recreation," while "promoting healthy relationships, positive social norms and safer online environments."

A young man in a green shirt.
Toronto resident Adam Kunder says he feels it's his duty 'to show up for men who are suffering and to hear them out and just let them know it’s all OK.' (CBC)

For Robert Whitley, a men’s health expert and professor of psychiatry at McGill University in Montreal, part of the problem is the way society now views masculinity.

"Traditional masculinity has been denigrated in the popular mind whether it's through the media, through influencers, through commentators," he said. "The problem sometimes is a lack of masculinity."

He says there's a need for positive male role models.

"I’ve said many times before, it’s not a crisis of masculinity, there’s a crisis of mentorship," said Whitley.

For example, data from Statistics Canada shows only 25 per cent of teachers across elementary and high school systems in Canada are men. And at the organizations that step in to fill the mentorship gap — like the Big Brothers and Big Sisters of Canada — 70 per cent of volunteers are women.

"We're getting young men who are getting to the age of 18, 19, 20, who have never had a male role model in their lives," Whitley said.

Kunder and Cholmsky both say their mental health began to recover after they started asking for help, and they're now working to be role models for other young men.

"I feel like it’s my duty in a way to show up for men who are suffering and to hear them out and just let them know it’s all OK," said Kunder, who coaches and works with other men.

Last month in Ottawa, Cholmsky was awarded top 21 under 21 for his work in mental health advocacy and challenging "harmful stereotypes around masculinity and mental health."

"We all need to be doing our part to talk about this stuff," he said.

Wait, Are Any of These AI Restaurant Reservation Apps Even Legal?

hellgate
hellgatenyc.com
2026-10-05 16:00:00
New York regulated services that use bots to book coveted tables in 2024, we thought....
Original Article

Great! You’ve successfully signed up.

Welcome back! You've successfully signed in.

You've successfully subscribed to Hell Gate.

Your link has expired.

Success! Check your email for magic link to sign-in.

Success! Your billing info has been updated.

Your billing was not updated.

Golang tool to check SPF, DKIM, TLSA, and TLS settings for mailservers

Lobsters
git.sig-io.nl
2026-10-05 15:50:52
Yes, I admit, this is 100% vibecoded, but it scratches an itch i've had for a while. Seems to work for my usecases at this time, but might add some more features in the future. Comments...
Original Article

mailcheck is a small command-line SMTP/DNS/TLS diagnostic tool written in Go. It checks whether a domain appears correctly configured to receive SMTP mail and can emit Nagios/Icinga-compatible output.

Checks

  • MX records, sorted by preference
  • implicit-MX fallback to A/AAAA when no MX exists
  • A and AAAA resolution
  • IPv4/IPv6 selection
  • TCP SMTP connectivity
  • SMTP greeting and EHLO
  • STARTTLS support and negotiation
  • optional implicit TLS ( --implicit-tls , useful for port 465)
  • certificate parsing, hostname/PKIX verification, issuer, dates and expiry
  • TLS 1.2/1.3 and modern AEAD cipher classification
  • TLSA lookup and certificate/public-key matching for TLSA usages 0-3
  • DNSSEC indication using the AD bit returned by the selected recursive resolver
  • SPF record discovery and basic syntax/lookup-limit checks
  • DMARC record parsing and policy checks
  • DKIM selector checks, including RSA and Ed25519 public-key validation
  • human, JSON and Nagios/Icinga output

Build

Requires Go 1.23 or newer. 

go build -o mailcheck .

For a Linux install: 

install -m 0755 mailcheck /usr/local/bin/mailcheck

Examples 

./mailcheck example.com
./mailcheck example.com --show-expiry
./mailcheck example.com --ipv4
./mailcheck example.com --ipv6
./mailcheck example.com --host mail.example.net
./mailcheck example.com --host mail.example.net --address 192.0.2.25
./mailcheck example.com --dkim selector1,selector2
./mailcheck example.com --json
./mailcheck example.com --nagios
./mailcheck example.com --port 465 --implicit-tls
./mailcheck example.com --dns 1.1.1.1

When --host is supplied, that hostname is used for TLS SNI/name validation and TLSA lookup. --address controls the TCP destination without changing the TLS hostname. If only --address is supplied, the domain itself is used as the TLS hostname; for accurate SMTP certificate/DANE testing, supplying the actual MX hostname is preferable.

Nagios exit status 

0 OK
1 WARNING
2 CRITICAL
3 UNKNOWN

Example: 

MAILCHECK WARNING - mx.example.net/192.0.2.25 SMTP=OK TLS=OK TLSA=WARNING | duration_ms=142

DNSSEC / DANE note

The program requests EDNS DNSSEC data and reports DNSSEC as secure when the recursive resolver returns the DNS AD (Authenticated Data) bit. It does not implement a complete DNSSEC validator itself. For production DANE monitoring, point --dns at a validating recursive resolver you trust.

TLSA matching implements the TLSA selector and matching-type operations and checks usages 0-3 against the certificate chain. The result should be read together with the DNSSEC status: a matching TLSA record obtained without DNSSEC validation is not equivalent to authenticated DANE.

SPF / DKIM / DMARC scope

These checks are deliberately DNS-policy diagnostics, not a complete mail-authentication implementation.

SPF checks the record syntax, multiple-record condition, terminal policy, and the ten-DNS-lookup limit at the top level. It does not simulate every possible sender IP through a complete recursive SPF evaluation.

DKIM requires selectors because DNS does not provide a universal way to enumerate all selectors. Example: 

./mailcheck example.com --dkim selector1,google

The DKIM check validates the public-key encoding and RSA key size / Ed25519 key length, but it does not verify a signed message.

DMARC validates the record and key policy tags but does not evaluate an actual message's SPF/DKIM alignment.

Security / operational considerations

The SMTP test performs a real TCP connection to the target's port 25 (or the explicitly selected port). It does not send mail or authenticate.

TLS certificate verification is performed separately from the TLS handshake so that a certificate that is not publicly trusted can still be inspected and compared with a DANE TLSA record. This is intentional for SMTP/DANE diagnostics.

The default TLS client minimum is TLS 1.2. A server that only supports TLS 1.0/1.1 will therefore fail the modern TLS check rather than being silently accepted.

Suggested future extensions

  • full recursive SPF evaluator with RFC lookup accounting
  • complete DNSSEC chain validation instead of relying on resolver AD
  • more exact RFC 7672 DANE policy evaluation
  • TLS version capability matrix (separate probes for TLS 1.2 and 1.3 and optional legacy probes)
  • SMTP MTA-STS policy retrieval/validation
  • TLS-RPT record checks
  • CAA checks
  • PTR / forward-confirmed reverse DNS diagnostics
  • SMTP banner hostname diagnostics
  • optional concurrency and retry controls

DKIM input

--dkim accepts either selectors or complete DKIM owner names. For example: 

mailcheck --dkim 2025a,2025b maniac.nl
mailcheck --dkim 2025a._domainkey.maniac.nl.,2025b._domainkey.maniac.nl. maniac.nl

A complete owner name is used as-is; it is not prefixed with the domain a second time.

DNS implementation

DNS queries are implemented using the upstream github.com/miekg/dns library (v1.1.70), not a hand-written DNS packet encoder/decoder. It provides native RR parsing, UDP/TCP transport, EDNS0 and DNSSEC-related support.

The tool requests DNSSEC data with the EDNS DO bit and reports the resolver's Authenticated Data (AD) bit separately. An AD=true result means the configured recursive resolver authenticated the response; it does not mean mailcheck itself is acting as a validating recursive resolver.

DKIM selectors

--dkim accepts either selectors: 

mailcheck maniac.nl --dkim 2025a,2025b

or complete owner names: 

mailcheck maniac.nl --dkim 2025a._domainkey.maniac.nl.,2025b._domainkey.maniac.nl.

The latter are queried exactly as supplied (apart from normalizing the trailing DNS root dot).

Command-line option order

Both forms are accepted: 

mailcheck maniac.nl --json --dkim 2025a
mailcheck --json --dkim 2025a maniac.nl

SMTP/TLS diagnostics

The SMTP probe reports the connection as a sequence of stages: TCP, SMTP greeting, EHLO, STARTTLS, and TLS. A failure at an earlier stage no longer incorrectly marks later stages as failures; later stages are reported as UNKNOWN because they could not be tested.

Use --debug to print the SMTP/TLS exchange diagnostics to stderr without putting protocol chatter into JSON or Nagios stdout: 

mailcheck --debug --host mail.example.net example.com

For example, a TCP timeout is reported as SMTP=CRITICAL while TLS and TLSA remain UNKNOWN , with the underlying socket error included in the summary.

TLSA is reported separately as:

  • WARNING when TLS succeeds but no TLSA record exists or the TLSA RRset is not DNSSEC-authenticated.
  • CRITICAL when a DNSSEC-backed TLSA RRset exists but does not match the negotiated certificate.
  • UNKNOWN when TLS was not negotiated, because the certificate cannot be matched to TLSA.

This distinction prevents a failed TCP connection from being reported as a TLS certificate failure.