Since its release in Go 1.16 (2021),
//go:embed
has quickly proven to be one of the most loved
features of the Go language. Russ Cox and Brad Fitzpatrick originally designed the feature in 2020
to embed files and directories into Go binaries using a go directive (more on what that is later).
It allows engineers to quickly embed anything they want instead of using awkwardly formatted
constant string literals or reaching for flaky third-party tooling to bake data directly next to
their program.
Feedback on the
proposal
was overwhelmingly positive.
It was quickly coined as one of the stand-out features of Go, not only because it’s an easy way to
embed strings and byte slices, but because it also shipped with a read-only file system. That was
the part that set it apart from other languages;
D
and
Zig
shipped expressions, but Go shipped a file
system you could interact with easily, straight from the standard library.
I’m unreasonably excited about this feature. I feel like I always have some big ol’ JSON or
template or something chilling in a constant making the file looking more unruly than it actually
is, but don’t want to ship the file with the binary or bring in some sort of 3rd party tool like
packr into the mix.
u/RyanTheKing
So let’s have a look into
//go:embed
, to see how to use it, but also see how it works under the
hood; so you’re able to make more informed decisions when shipping static assets alongside your
programs.
Contents
The directive: What it is
So how do we use it? Here’s a simple example to whet your appetite:
//go:embed version.txt
var version string
-
We have what looks like a go comment, but it’s actually a Go directive, sometimes called
pragmas (if you’re working with the compiler). You may have seen these before as
go:generate
or
go:build
.
-
After writing
//go:embed
we specify the relative file path to where the file is located.
-
In the next line, we specify the variable name we want to use. Which is where the go compiler
will read the file contents and bake all of that lovely data into the binary before your
program runs.
-
The variable is just a pointer to the file contents; the magic happens in the build.
Before we get into any of that, it’s worth remembering the problem this was built to solve.
We’ve all seen this?
I’m sure we’ve all seen something like this before:
$ docker run myapp
2026/09/09 open ./static/app.css: no such file or directory
How annoying is this to debug? It’s usually not difficult or cumbersome, but fiddly and
time-consuming. You usually have to debug some weird absolute/relative path fiasco or go into some
glob patterns. This is exactly why an embed feature was needed, it makes static assets incredibly
easy to ship.
What it is not
We’ve talked about what
//go:embed
is, but let’s just touch on what it isn’t for a moment.
-
Don’t think that it’s a way to compress your files; a megabyte is a megabyte. Go doesn’t care or
manipulate your data in any way.
-
It’s also not a build pipeline, as mentioned above. Go doesn’t minify, transpile or fingerprint
anything; it just copies bytes.
-
It’s also not dynamic, it’s run at compile time, not run time. The bytes are locked in when you
run
go build
. So if you tweak a template and expect it to update, it won’t. There’s no hot
reload anywhere.
-
And it’s not a media server, please don’t be going embedding your favourite Netflix films.
Coincidentally, there’s actually a 2GB limit (I tried, believe me). But seriously, use object
storage or something else for storing large files.
The variable: How to use it
So now we’ve seen what it is, let’s remind ourselves how to use it on a high level. If you’re
familiar with the concepts of how to use it, feel free to skip
this
section
.
There are three data types we can use with
//go:embed
; a string, a byte slice and a read-only
file system. Here are all three of them in one program:
import "embed" // Or import _ "embed"
//go:embed version.txt
var version string
//go:embed schema.sql
var schema []byte
//go:embed assets/*
var assets embed.FS
func main() {
fmt.Println(version) // v1.4.2
db.Exec(string(schema))
css, _ := assets.ReadFile("assets/index.css")
fmt.Println(string(css)) // body { margin: 0 }
}
We first need to import the
embed
package. The import isn’t blank here because we’re naming
embed.FS
below, so we need the package itself. A blank import (
import _ "embed"
) is for when
you’re only embedding into strings and byte slices. Either way, it sends a signal to the compiler
that we need to embed data, and you’ll see exactly how that works later on.
Then it’s one directive per variable, one type each:
-
A string
, our
version.txt
from the last section. Good for anything you’re going to read as
text; a version number, a template, a licence.
-
A byte slice
, here an SQL schema. But we could really use it with anything, a config file,
TLS cert, font, image, and much more. Use bytes when it makes sense for how the data is used in
your program.
-
An
embed.FS
, a whole folder. This is the part that sets Go apart from most languages, and
it’s great for web assets, templates, migrations and whatever else you fancy.
And then we just use them. They’re variables. No
os.Open
, no path fiddling, no “did I ship the
migrations folder?”. The contents are already there, just like any other variable in your program.
Note that a file inside an
FS
keeps the path it was embedded with, which is why we’re asking for
assets/index.css
rather than just
index.css
.
embed.FS
The
FS
type has a bit more going on with it, so it’s worth a proper look.
You can see the methods below:
Open
,
ReadDir
and
ReadFile
.
// An FS is a read-only collection of files.
type FS struct {}
func (f FS) Open(name string) (fs.File, error) // Satisfies fs.FS
func (f FS) ReadDir(name string) ([]fs.DirEntry, error)
func (f FS) ReadFile(name string) ([]byte, error)
The most important thing to note is that it satisfies the
fs.FS
interface, which means we can use
it in many different ways. It makes it easy to plug and play across packages, such as
http.FileServer
to serve it as a static site or
template.ParseFS
to parse HTML templates.
// An FS provides access to a hierarchical file system.
type FS interface {
Open(name string) (File, error)
}
Working example
Before we dive any deeper, let’s give ourselves a folder to work with. This is the example we’ll
be coming back to all the way down to the linker, so it’s worth a proper look.
We’re serving a normal website. An
index.html
at the root, and some
css
and
js
in a folder
called
assets
. No more, no less.
.
├── main.go
├── index.html (<h1>Hello</h1>)
└── assets
├── index.css
└── index.js
To embed that into the program, we create a string variable for the contents of
index.html
, and an
embed.FS
for the contents of the
assets
folder, and then just print them. In a production
environment you’d serve these through a handler, but we’ve kept it simple for now.
package main
import (
"embed"
"fmt"
)
//go:embed index.html
var indexHTML string
//go:embed assets/*
var static embed.FS
func main() {
fmt.Println(indexHTML) // <h1>Hello</h1>
css, _ := static.ReadFile("assets/index.css")
fmt.Println(string(css)) // body { background: red; }
}
The
assets/*
pattern gets us every file in that folder; naming the directory on its own
(
//go:embed assets
) takes the whole tree instead. The difference isn’t only depth: naming the
directory skips every file beginning with
.
or
_
anywhere beneath it, where the
/*
glob still
pulls those in at the top level. Write
all:assets
when you want the whole tree, dotfiles and all.
There are more gotchas in the
pkg.go.dev/embed
docs.
The binary: How it’s built
Now that we know what
//go:embed
is, we’ve seen how it works. Let’s dive a little bit deeper to
see how it’s constructed under the hood. If I were to show you a bunch of code from the Go
compiler out of context in the next section, I’ll probably have you lost; like it did me for a few
days researching into how Go is built (TLDR; it’s challenging but worth it).
Everything below was read against
Go 1.27.1
, and every file and line number links to that tag
on GitHub, so the lines quoted here stay put even after the toolchain moves on.
Compiler basics
So before we go anywhere near how go is built, let’s zoom all the way out. When you type
go build
,
six steps run in order and out the other end comes a binary. This is the path that matters for
//go:embed
; it isn’t everything the toolchain does, just the part your files travel through.
I’m aware that there’s a whole heap of stuff that we’re not going into here such as SSA,
inlining, escape analysis, etc… But none of it really touches the embed packages, because your
data never becomes instructions; they are just bytes after all.
The first thing worth knowing is that there isn’t one program doing all of this. There are several,
each responsible for its own thing, and
go build
is the one that runs the other two.
go build: six steps, three programs
So what are we looking at here?
-
Packages
: The
go
command starts by reading your
.go
files and working out what package
each one is in.
-
Lexer
: The lexer chops that text up into tokens, and while it’s in there it reads the
comments looking for pragmas, which is where your
//go:embed
lines get spotted.
-
Patterns
: Turns each
//go:embed
pattern into a list of real files on the disk and hands that
list to the compiler.
-
Parser
: Will then arrange those tokens into a syntax tree (
AST
) that describes the structure
of your code.
-
Compiler
: Then checks the tree and turns it into machine code. What comes out is one object
archive per package, which is code that can’t run on its own yet.
-
Linker
: Is what ties everything together, along with the Go runtime, into the binary you
build.
Notice the three different sections, that’s the bit that tripped me up for the longest,
go
runs once for the whole build,
compile
runs once
per package
, and
link
runs once at the
end. It’s an important nuance to understand.
Check out
go.dev/src/cmd/compile/README
for more information.
Packages (
go build
)
So now, onto the juicy bits. Let’s start with how Go looks at your project. When you run
go build
it needs to know what files and packages exist, so it knows what to parse and build.
A package directory goes in, and a
*build.Package
comes out. So it walks your project, finds every
.go
file and then works out which package each one belongs to. For every file, it notes down where
it lives, what it imports, and any directives (such as
//go:embed
) that it carries.
Packages: walking the package directory
You can see our
assets/
folder and
index.html
being skipped, because they aren’t Go files and
this pass only cares about Go.
main.go
makes it through the condition into
matchFile
, and
out the other side comes a
Package
which details its name, its Go files, and the position of
every
//go:embed
pattern in it.
func (ctxt *Context) Import(path string, srcDir string, mode ImportMode) (*Package, error) {
for _, d := range ctxt.readDir(p.Dir) { // walk the package dir
name := d.Name()
if filepath.Ext(name) != ".go" { // main.go, index.html, assets/…
continue
}
info, _ := ctxt.matchFile(p.Dir, name, ...) // reads header, scans comments, gets embed
p.GoFiles = append(p.GoFiles, name) // ["main.go"]
p.Name = info.parsed.Name.Name // main
for _, emb := range info.embeds {
// {"index.html": [main.go:10], "assets/*": [main.go:13]}
embedPos[emb.pattern] = append(embedPos[emb.pattern], emb.pos)
}
}
}
Reference
:
src/go/build/build.go
(
(*Context).Import
, simplified)
Our
//go:embed index.html
and
//go:embed assets/*
have been spotted and recorded along with
their file names and token positions.
But how did it spot the comment originally? This is where the lexer comes into play.
Lexer (
go build
)
The lexer (or scanner, or tokeniser, whatever you fancy) is the first thing that actually reads your
code as
Go
, rather than as a blob of bytes. It scans the file one character at a time and chops it
up into little tokens, which are the pieces of code that mean something on their own. Such as
func
,
var
,
=
, a string literal, or a comment.
This is the stage that makes
//go:embed
actually possible. In Go, a comment is a real token
(
token.COMMENT
).
Lexer: one character at a time, into tokens
Characters go in and tokens come out. The whole
//go:embed index.html
line becomes a single
COMMENT
token, and the two words of
var indexHTML string
become a
VAR
and two
IDENT
s.
The
go
command’s scan is deliberately inexpensive as it stops at the end of the import block,
because that’s normally all it needs. Unless, however it saw
import "embed"
on the way past,
in which case it reads on to the end of the file, picking out every
comment starting
//go:embed
:
if hasEmbed {
r.readRest() // keep going: read the rest of the file after all
sc.Init(file, r.buf, nil, scanner.ScanComments) // get comments as tokens
for {
pos, tok, lit := sc.Scan()
if tok == token.COMMENT && strings.HasPrefix(lit, "//go:embed") {
// in: `//go:embed index.html`
// out: {pattern: "index.html", pos: ...}
embs, _ := parseGoEmbed(fset, pos, lit)
info.embeds = append(info.embeds, embs...)
}
}
}
Reference
:
src/go/build/read.go#L374-L392
(
readGoInfo
, simplified)
That
hasEmbed
flag is the real reason
import _ "embed"
is mandatory even when you never type the
package name. If you don’t import the package, we’ll never go into this code block.
What comes out is a list of the patterns it found, and where each one was written:
[]fileEmbed{
{pattern: "index.html", pos: main.go:10:12},
{pattern: "assets/*", pos: main.go:13:12},
}
We now have patterns to file names and line numbers, but there are no files? This is where resolving
embed patterns comes in.
Patterns (
go build
)
Within the
go command
we have a fairly important function called
resolveEmbed
which resolves
patterns into actual files and trees. File patterns go in, and real file paths come out.
Patterns: resolveEmbed turns patterns into files
The glob is where it gets interesting. Our
index.html
was only ever going to match itself, but
assets/*
fans out to
assets/index.css
and
assets/index.js
, neither of which we named anywhere.
That’s the first time anything in this whole process has actually gone and looked at your disk.
So these patterns get expanded, globbing it against the disk and walking any directory it
matches, into a list of actual file names. You can see this in action below:
func resolveEmbed(pkgdir string, patterns []string) (files []string, pmap map[string][]string, err error) {
pmap = make(map[string][]string)
have := make(map[string]bool) // every file, once
// patterns = []string{"index.html", "assets/*"}
for _, pattern := range patterns {
matches, _ := fsys.Glob(pkgdir + "/" + pattern)
var list []string
for _, match := range matches {
if fsys.IsDir(match) {
fsys.WalkDir(match, func(path string, ...) { list = append(list, path) })
} else {
list = append(list, match)
}
}
pmap[pattern] = list // what this pattern matched
for _, f := range list {
have[f] = true // ...and the running set of all of them
}
}
for f := range have {
files = append(files, f)
}
return files, pmap, nil
}
Reference
:
src/cmd/go/internal/load/pkg.go#L2165
(
resolveEmbed
, simplified)
Output will now be:
// pmap:
map[string][]string{
"assets/*": {"assets/index.css", "assets/index.js"},
"index.html": {"index.html"},
}
// files:
[]string{"assets/index.css", "assets/index.js", "index.html"}
Those two return values become the two data points that the compiler needs:
-
Patterns
answers “what did this pattern match?” and keeps the duplicates, because that’s the tree
your
embed.FS
gets, and two different patterns can easily match the same file.
-
Files
answers “where do I read those bytes?” and cannot have duplicates, because the compiler
should open each file exactly once.
The
go
command marshals them into JSON, writes it to a file called
embedcfg
in the build’s
temporary work directory, and passes that path to the compiler as
-embedcfg
, once per package.
Why a file, why not just hand it over directly? Because
go build
and the compiler are two separate
processes, as we saw in that first diagram, so it gets written down instead.
For our example project, the generated
embedcfg
looks like:
{
"Patterns": {
"assets/*": [
"assets/index.css",
"assets/index.js"
],
"index.html": [
"index.html"
]
},
"Files": {
"assets/index.css": "/home/user/project/assets/index.css",
"assets/index.js": "/home/user/project/assets/index.js",
"index.html": "/home/user/project/index.html"
}
}
Reference
:
work/exec.go#L893-L906
builds and marshals it,
work/gc.go#L148-L152
writes it and passes
-embedcfg
(simplified)
(Tip:
go list -json=EmbedPatterns,EmbedFiles .
surfaces these exact matched patterns and files
straight from the package metadata, any time you want to look.)
Parser (
cmd/compile
)
We’re in a different program now. Tokens go in, along with the path to that
embedcfg
, and what
comes out is a tree with your directive attached to the right node.
The parser is where the real magic happens in the Go compiler. It takes those tokens and little
bits of Go code and works out their structure. A bit like breaking a sentence down into a
subject, verb and object.
The result is a
syntax
tree; you may have heard of the term
AST
before, which is short for
abstract syntax tree
. It’s the building blocks of our program. Every piece of your code (a
function, a variable, an if statement) becomes a
node
within that tree.
Directives such as
//go:embed
get attached to the node, which is how Go knows which variable to
put your embedded file in.
The below example is captured from the two lines from our example above:
//go:embed index.html
var indexHTML string
Parser: the directive and the variable are one node
-
GenDecl
: Is the declaration as a whole, it’s short for a
generic declaration node
which represents an import, constant, type or variable declaration. In our example, it’s
both the comment, and the variable.
-
Doc
: Is the comment, held as a
*ast.CommentGroup
. Our
//go:embed index.html
is in
there, kept word for word in
Text
.
-
Specs
: Is what that declaration actually declares which is an
*ast.ValueSpec
holding the
name
indexHTML
and the type
string
, our variable.
Both the
Doc
and
Specs
come under
GenDecl
. The comment and the variable aren’t two things that
happen to be near each other in a file; they’re two branches of one node. That’s how the compiler
knows the contents of
index.html
go into
indexHTML
, and not into whatever is declared next.
Which looks like this. The dump is
go/ast
, which is what
ast.Print
gives you and what most
tooling reads. The compiler’s own parser is
cmd/compile/internal/syntax
and holds the same shape
under different names, which is why
checkEmbed
in the next section takes a
*syntax.VarDecl
:
1: *ast.GenDecl {
. Doc: *ast.CommentGroup {
. . List: []*ast.Comment (len = 1) {
. . . 0: *ast.Comment {
. . . . Slash: main.go:8:1
. . . . Text: "//go:embed index.html"
. . . }
. . }
. }
. TokPos: main.go:9:1
. Tok: var
. Specs: []ast.Spec (len = 1) {
. . 0: *ast.ValueSpec {
. . . Names: []*ast.Ident (len = 1) {
. . . . 0: *ast.Ident {
. . . . . NamePos: main.go:9:5
. . . . . Name: "indexHTML"
. . . . }
. . . }
. . . Type: *ast.Ident {
. . . . NamePos: main.go:9:15
. . . . Name: "string"
. . . }
. . }
. }
}
Clever stuff, right? This was the aha moment for me. The comment and variable are one declaration
in Go’s eyes.
Compiler (
cmd/compile
)
Now onto the nuts and bolts, the compiler. A tree goes in, and what comes out is an
object archive
,
one per package, which is your code compiled but not yet runnable on its own. This is also the stage
where your data is actually allocated.
Everything so far has been bookkeeping; finding files, expanding patterns and writing lists, but not
one byte of your
index.html
has actually been allocated into your program. However, in this step,
it does.
The compiler lays your variable out in memory and fills it with the file’s contents, as static data,
that’s there before your program runs a single line of your app’s code.
Although its first job isn’t your files at all, it’s the rules. Remember, the
go
command spotted
your directive and wrote the pattern down, but it never checked anything about it, which is
mentioned in the comments:
// The compiler will enforce the mapping of comments to
// declared variables. We just need to know the patterns.
Reference
:
src/go/build/read.go#L369-L373
(
readGoInfo
)
Now it checks what you’ve written is actually valid. Nearly every
//go:embed
error message you’ve
probably seen comes from this switch statement:
func checkEmbed(decl *syntax.VarDecl, haveEmbed, withinFunc bool) error {
switch {
case !haveEmbed: // The file never imported "embed".
return errors.New(`go:embed only allowed in Go files that import "embed"`)
case len(decl.NameList) > 1: // var a, b string
return errors.New("go:embed cannot apply to multiple vars")
case decl.Values != nil: // var s string = "hello"
return errors.New("go:embed cannot apply to var with initializer")
case withinFunc: // Declared inside a func body.
return errors.New("go:embed cannot apply to var inside func")
}
return nil
}
Reference
:
src/cmd/compile/internal/noder/noder.go#L462-L481
(
checkEmbed
, simplified)
But how is the data read?
Compiler: WriteEmbed lays
<h1>Hello</h1>
out as 14 bytes of read-only data
// WriteEmbed, for our `var indexHTML string`.
case embedString, embedBytes:
file := files[0] // "index.html", from the embedcfg JSON
path := base.Flag.Cfg.Embed.Files[file] // "/home/user/project/index.html"
fsym, size, _ := fileStringSym(v.Pos(), path, ...) // read it, name the bytes
sym, off := v.Linksym(), 0 // indexHTML itself
off = objw.SymPtr(sym, off, fsym, 0) // field 1: pointer to those bytes
off = objw.Uintptr(sym, off, uint64(size)) // field 2: length
if kind == embedBytes {
objw.Uintptr(sym, off, uint64(size)) // field 3: cap, because a slice has one
}
case embed.FS:
// See source files below...
Reference
:
src/cmd/compile/internal/staticdata/embed.go#L101-L134
(
WriteEmbed
, simplified)
Here we can see
WriteEmbed
switching on the variable’s data type. It then goes on to call a
function called
fileStringSym
, which opens your file and returns a symbol for its contents, as
well as the size.
Okay, but what’s a symbol? Forget
[]byte
for a second. A symbol is a box with a name on it, what
it’s called, how big it is, and where it lives in the binary. That’s what the linker actually needs
to do its job, and it’s why
fileStringSym
hands one back rather than the bytes themselves.
This is where your asset has stopped being a path and has actually become data.
The rest is ordinary compilation. The tree gets turned into machine code
(
cmd/compile/internal/ssagen
), and out comes one
object archive
per package, which is code that
can’t run yet, which is where the linker comes into play.
Linker (
cmd/link
)
Although the linker has little to do with
//go:embed
, it plays a crucial part in the
build process. It’s a separate program, located under
cmd/link
. Many
.a
files go in, and one
executable comes out. All of the object archives that we mentioned above, now are joined together.
It fills in all the gaps so every piece can find the others.
Which is where the Go runtime is injected. The runtime is built-in code that manages memory,
goroutines and all of the stuff that makes Go, Go.
Linker: object archives and the runtime become one binary
There’s
main.a
still carrying our 14 bytes, along with
fmt.a
,
embed.a
, every other package you
pulled in, and
runtime.a
. On the right is what they become, one binary, and our
<h1>Hello</h1>
sitting in the
rodata
section, which is where read-only data lives.
Binary
Now run it (
./myapp
) and voilà! We have your finished program; maybe we should take a
breather…
Build Summary
I know that was a lot to take in, so let’s go back to where we started and walk it once more, now
that all the names mean something. I always find this easier to read as pseudocode, so here we go.
It isn’t real or runnable; it just connects all the dots:
import (
"go/build" // 1. Packages: what's in this package?
// 2. Lexer: text -> tokens
"cmd/go/internal/load" // 3. Patterns: "assets/*" -> real paths on disk
"cmd/compile/internal/syntax" // 4. Parser: tokens -> syntax tree
"cmd/compile/internal/staticdata" // 5. Compiler: file bytes -> read-only data
"cmd/link/internal/ld" // 6. Linker: object files -> the binary
)
func goBuild(dir string) Binary {
pkg := build.ImportDir(dir) // files, imports, embed patterns
var objs []Object
for _, p := range pkg.Deps {
embeds := load.ResolveEmbed(p.Dir, p.EmbedPatterns) // "assets/*" -> [assets/index.css, ...]
tree := syntax.Parse(p.GoFiles) // code + directives -> tree
for _, v := range tree.EmbedVars {
data := os.ReadFile(embeds[v.Pattern]) // the compiler opens your file
staticdata.WriteEmbed(v, data) // and bakes the bytes in
}
objs = append(objs, compile(tree)) // tree -> object file
}
return ld.Link(objs, runtime) // -> the binary you ship
}
Three separate processes, and they don’t run the same number of times,
go
runs once for the whole
build,
compile
runs
once per package
and
link
runs once at the end. That’s why
embedcfg
is
written per package, immediately before each
compile
invocation.
And you don’t have to take my word for any of it.
go build -x
prints the real command lines and the
temporary directory, so you can watch
embedcfg
being written and passed along, and
ls $(go env GOROOT)/pkg/tool/$(go env GOOS)_$(go env GOARCH)/
shows you that
compile
and
link
really are separate binaries sitting on your disk. Every
Reference
link above points at the exact
lines in Go 1.27.1 if you’d like to read the real thing.
The runtime: How it’s read
So the bytes are in the binary. The last question is what happens when your program actually
opens them, because that depends on which of the three types you picked.
For a
string
or a
[]byte
, there’s almost nothing to tell, and that’s rather the point. The
compiler wrote a pointer and a length, and the pointer aims at bytes sitting in the
rodata
section
we just watched the linker build. Reading the variable is an ordinary variable read.
An
embed.FS
is more involved, and it’s the more interesting half. Remember it’s a single pointer to
a hidden list, and here’s what’s on the end of it:
// An FS is a read-only collection of files.
type FS struct {
files *[]file
}
// A file is a single file in the FS.
type file struct {
name string
data string
hash [16]byte // truncated SHA256 hash
}
Reference
:
src/embed/embed.go#L151-L212
(
FS
and
file
, simplified)
Each file’s contents are a
string
, pointing into the same read-only data our
indexHTML
did,
so a 40MB asset isn’t sitting in your heap waiting for you.
Asking for a file is a search, not a syscall. The list is sorted, so
lookup
does a binary search
over it, and
readDir
does two
of them to find where a directory starts and ends, then hands back that stretch of the slice. There’s
no disk involved at any point, which is why
ModTime
on an embedded file is the zero time. There’s
nothing there to have been modified.
There’s one detail worth the whole section, though, and it’s the difference between the two ways of
getting at your bytes.
Open
hands you back an
openFile
that reads straight out of that string,
copying only as much as fits the buffer you gave it.
ReadFile
ends looking a bit like this:
return []byte(ofile.f.data), nil
Reference
:
src/embed/embed.go#L322-L332
(
ReadFile
)
That conversion is a copy, and a fresh one on every call. So the 40MB asset that wasn’t sitting in
your heap a moment ago lands there the moment you call
ReadFile
on it.
Open
doesn’t; it fills
your buffer a chunk at a time, which is the path
http.FileServer
and
io.Copy
take. Neither
touches the disk, but only one of them allocates the file.
The tradeoffs: What you take on
You’ve seen how it works and how it’s used, so let’s talk about what it costs you, because embedding
isn’t free.
Rebuild to change a byte
Remember what
WriteEmbed
did with those bytes. They’re read-only data in the object file now, which
means your file isn’t really a file any more. So change one line of CSS, and you have to rebuild.
This can be a bit painful because nobody wants a full build on every save. The fix is the last
in this section, so hold onto that.
No secrets
And here’s the one that catches people out. Embedded bytes are just data in the binary, which means
they’re readable:
$ strings myapp | grep -i api_key
API_KEY=sk_live_4eC39HqLyjWDarjtT1zdp7dc
Anyone with your binary has your file. So no
.env
, no private keys, no credentials. It might seem
obvious, but it’s an easy one to forget when you’re embedding a whole config directory.
What the compiler catches
Now for the good news. Remember
checkEmbed
, that big switch statement? Bad patterns, no matching
files, the wrong type, a variable inside a function; all of them are build failures. But the
compiler knows the file exists. It doesn’t know it’s the file you
meant
. That’s where the
standard library has a package that can help you out.
Testing what you embedded
testing/fstest
is the handy little package of the standard library that doesn’t get enough
attention, which is a shame, as it works perfectly alongside
embed
.
fstest.TestFS
takes your FS
and a list of paths you expect to find within it, and then does two jobs at once:
-
Asserts every path you named is really there.
-
Cross-checks the file system against itself, walking the whole thing to confirm that
Open
,
ReadDir
and
ReadFile
all agree with each other.
The trick is what you list. It shouldn’t be a whole inventory of the
FS
, just a handful of the
files your code actually opens:
func TestAssetsArePresent(t *testing.T) {
// Extras are ignored, so adding a file needs no test change.
err := fstest.TestFS(static,
"assets/index.css",
"assets/index.js",
)
if err != nil {
t.Fatalf("files are not found: %v", err)
}
}
Only name what you’d notice missing; the template your handler executes, the migration directory,
the default config. Everything else is just noise that makes the test brittle.
Depend on fs.FS
And this is the one thing I’d really like you to take away, and the answer to the rebuild problem we
started this section with.
// Take the interface, not the implementation.
func NewServer(assets fs.FS) *Server {
return &Server{assets: assets}
}
func TestIndexHandlesAnEmptyTemplate(t *testing.T) {
srv := NewServer(fstest.MapFS{
"templates/index.html": &fstest.MapFile{Data: []byte("")},
})
// No files on disk, no embed, no build.
if err := srv.Render(io.Discard, "index.html"); err == nil {
t.Fatal("expected an error for an empty template")
}
}
Take
fs.FS
in your constructors, not
embed.FS
. Your server doesn’t care where the files came
from; it cares that it can open them. Which means a test can hand you an
fstest.MapFS
instead,
with the files declared inline as a map.
And now you can ask the questions a good unit test should ask. What if this file is empty? What if
it’s corrupt, or 40 megabytes? You can answer all of that without touching the disk or waiting on a
build. It’s also how you sidestep the rebuild-to-change-a-byte problem in development: the same
constructor takes
os.DirFS("./assets")
while you’re working and your
embed.FS
in production, and
nothing else in your program needs to know which one it got.
Wrapping up
So that’s the whole journey, from a comment in your source to bytes in the
rodata
section of a
binary you can copy anywhere.
//go:embed
is one of those quiet features about Go that makes it so easy to reason about. It’s
a kind of feature that you didn’t know you needed until it came along. It makes for a great
experience when you’re building a package that needs to ship some data with it.
I would really encourage you to have a look at the Go source code, although it can look scary at
first (I spent some time dazed and confused looking at it). But it really makes you a better
engineer for knowing how these constructs work.
Please reach out to me at
hello@ainsley.dev
if you have any questions!
I hope you’ve enjoyed reading this post as much as I have writing it.
Source code
A reminder for each section, its program, package and line numbers are below:
|
Section
|
Program
|
Package
|
Evidence
|
|
Packages
|
go
|
go/build
|
read.go:314
(imports-only parse)
|
|
Lexer
|
go
|
go/build
|
read.go:379-385
(the embed scan)
|
|
Patterns
|
go
|
cmd/go/internal/load
|
pkg.go:2165
resolveEmbed
; JSON written by
work/gc.go:148-152
|
|
Parser
|
compile
|
cmd/compile/internal/syntax
|
parser.go
; the directive is attached by
noder
|
|
Compiler
|
compile
|
cmd/compile/internal/noder
|
noder.go:462
checkEmbed
; then
ssagen
,
staticdata
,
gc
|
|
Linker
|
link
|
cmd/link/internal/ld
|
|
|
Binary
|
-
|
-
|
|
Further reading
Embed
Go command
Compiler
Linker
Runtime
Other languages
-
D
,
import("file.txt")
, an expression, documented since D 1.0 (
spec
)
-
Zig
,
@embedFile
, in the docs since 0.1.1 (
ziglang/zig#14637
)
Community
){kind=icon}
isomorphism classes of elliptic curves defined over
. Let
be the proportion of elliptic curves over
, the proportion of curves that are strong is at least
. Now suppose that the proportion of these curves that the NSA knew how to break were
. In this hypothetical scenario the NSA could select such a weak curve by trying about
seeds. The number of NSA-weak curves would then be approximately
. This would mean that this huge class of weak curves was known to the NSA in 1997 but is still undiscovered by outside researchers in 2015.
It is highly unlikely that such a large family of weak elliptic curves would have escaped detection by the cryptographic research community from 1997 to the present.
Comments (0) Trackbacks (0) Leave a comment Trackback