This post is a brain dump of what I’ve learned so far after reading A Mathematical Framework for Transformer Circuits (herein: “Framework”) and working through the Intro to Mech Interp section on ARENA. My goal is to describe my current intuition for the paper, especially parts I was confused about so that perhaps my take can help others gain clarity on these areas as well.

First, a brief aside on my overall motivation for working on this stuff. Mechanistic Interpretability (MI/mech interp) is the study of ML model internals whose aim is to understand from first principles why models behave and work as they do. You can kind of think of it as the machine learning analogue of reverse engineering software. It is similar in spirit to the science of biological neural networks, but applied to artificial neural networks instead.

MI is part of a broader field of interpretability, which is used in yet another field called AI alignment. Alignment strives to make our large AI models aligned with human values. Basically, the overall goal is to understand and control the models before they control us. To ensure that they don’t engage in harmful, deceptive, dangerous, or subversive behavior. Unfortunately, we live in a world where large language models have encouraged “successful” suicide, engaged in blackmail for self-preservation, and asserted humans should be enslaved by AI. This current version of reality is unacceptable to me.

And as if that weren’t enough, we don’t even understand why these models do what they do. They are the only man-made technology in history that we don’t fully understand from first principles. Given this state of reality, I think that alignment is one of the most important problems we face today and one we have to get right. As a personal bonus, the alignment problem is as fascinating as it is important. It provides an outlet for me to leverage my specific technical skills and interests towards a meaningful cause. It is also extremely difficult, and I like a good challenge.

Ok, now back to the originally scheduled programming.

Share

Attention-Only Transformers

Framework does a deep dive into the key components of a simplified transformer-based language model. It analyzes transformer blocks that only have multi-head attention. This means no MLPs and no layernorms. This leaves the token embedding and positional encoding at the beginning, followed by n layers of multi-head attention, followed by the unembedding at the end. Here is a picture of a single-layer transformer with one attention head only:

My goal in this post is not to re-derive all the math, because the Framework paper does a better job, and Neel Nanda’s walkthrough of the paper on YouTube is also good for that (although this material only really started to click for me after I worked through the “Intro to Mech Interp” problems on ARENA, which I recommend doing if you are actually interested in doing this stuff yourself).

Instead I want to share how I conceptualize the most important takeaways, especially for areas that I thought were confusing at first so that if you have the same confusion perhaps my take will bring some clarity. In my view, the most important concepts to understand from this paper are the residual stream, attention, circuits, and induction heads.

The Residual Stream

Mathematically, the residual stream is a high dimensional vector space. You will usually see the dimension of the residual stream specified as d_model in GPT-related papers and code. For example, GPT2-small uses a d_model of 768.

Conceptually, the residual stream is like shared memory. It is used much like the DRAM on your computer. Different components of the model (attention, MLPs, etc) perform loads and stores from that memory. The loads and stores occur sequentially through the forward pass, one layer at a time. However each component in a given layer loads in parallel and stores in parallel with the others. The model learns to carve out subspaces in this vector space. This helps prevent components from clobbering over what previous components have written. The residual stream itself doesn’t do any computation, but serves as a shared medium through which layers communicate with each other.

We can get a sense of the size of a subspace used by doing a PCA on the appropriate weights. Below is the PCA eigenspectrum of the embedding and positional encoding weights from a 2-layer, attention-only model (the link to all code for this post is here). The first shows the top 100 principal eigenvalues. The second shows the cumulative variance explained:

So about 80% of the embedding variation lives in a 350-dimensional subspace of d_model. This is fairly large given that d_model is 768. Compare that to the positional encoding, which is essentially explained by only 5 directions.

When I was presented with this view of the residual stream, my mind immediately started asking how far can we push this analogy to memory? Having worked in computer security for a decade, it made me wonder if there is an analogue to page tables and memory permissions? Could we bring the concepts of userspace and kernelspace to prevent “privileged” subspaces from being accessed by “unprivileged” subspaces? Would this be useful for e.g. preventing an untrusted user from exfiltrating dangerous content from a privileged subspace?

But I’m getting ahead of myself. Let’s start with a simpler question: how does addressing work for the residual stream? In order to access a memory location, you have to have an address. Residual stream addresses can be decomposed into two logical parts, token:subspace, much like the classic segment:offset logical address from the x86 architecture. One major difference is that a traditional memory address is deterministic in the sense that only one value from one location is loaded. Addresses into the residual stream are “soft”, in general specifying a set of locations to load according to some learned probability distribution.

Subscribe now

Attention

Conceptually, attention computes the first part of the token:subspace address. The fundamental purpose of attention is to specify which source token locations to load information from. Each row in the attention matrix (see fake example below for tokens ‘T’, ‘h’, ‘e’, ‘i’, ‘r’) is the “soft” distribution over the source (i.e. key) token indices from which information will be moved into the destination token (i.e. query).

Let’s look at the extreme case, when the entry is 1 and all the others in the row are 0. This means that this head reads some subspace(s) of the source token’s (‘T’) residual stream and copies it verbatim into some subspace(s) of the destination token’s (also ‘T’) residual stream. But since attention is 1, there is only one source token position being read from. Otherwise the read is “spread out” over multiple source tokens according to the attention scores in each row. For example the second query above (‘h’) reads “30%” from token 0 (‘T’) and “70%” from itself.

It is important to understand that attention is all about figuring out the token indices to read from. If we look at the residual stream as a two dimensional memory array, then attention probabilistically selects rows of this memory for each query. For example, the third query above (‘e’) would have a token address that looks something like 0.1,0.6,0.3:

So the token part of the address selects the rows in the residual stream via attention. What about the subspace part? How is it computed? Once we have this part then we can determine the actual value that is stored into the destination token’s location. To answer this we need to understand circuits.

Circuits

Conceptually, circuits are particular paths through which information flows through the model. It is not too far off to think of them as the ML analogue of the electrical circuits you find on a PCB. They have inputs, do some computation, and produce outputs. In the simplified attention-only models, circuits are mathematically tractable to analyze due to the mostly linear structure of the transformer under the attention-only assumptions (and completely linear if the attention patterns are held constant).

The two basic circuits to know are the QK circuit and OV circuit. The QK circuit is a bilinear form, meaning it is linear in two input variables. In self-attention, the input variables are the same, but are interpreted as distinct queries and keys. Whereas the OV circuit is linear in one input variable. The inputs are the same for all three - the residual stream. We will refine this further in the next sections.

QK Circuit

Recall each attention head has its own W_Q and W_K weight matrices. Together these form a bilinear operator that outputs the attention pattern for that head. Mathematically this looks like:

where the W’s (also called W_QK) are learned weights of shape (d_model, d_head) and x is the residual stream of shape (seq_len, d_model). When you multiply this out, you get the attention pattern. So attention is more of an activation than a weight, since it depends on the input sequence. The attention queries are computed on the left and the keys are computed on the right. If a query “pays attention” to a key, then the dot product will be high. This will cause data from the key’s residual stream to be moved into the query’s residual stream. But what data will actually be moved? This is where the OV circuit comes in.

OV Circuit

The final input of the head is the W_V weight matrix. It reads in from the residual stream and writes out to the residual stream via the W_O matrix. W_V is (d_model, d_head) and W_O is (d_head, d_model). Together their product is referred to as W_OV. This is what the OV circuit looks like mathematically:

The value that is read by W_V determines what value gets written back to the residual stream, if that token is attended to by a particular query. The final expression for the entire head with attention and the OV circuit is:

Now that we have some common footing in the math, we can move on to developing some intuition for how circuits work. This is also where the subspace part of the residual stream address comes into play.

Subspace Scores

We know that the QK and OV circuits both read in from the residual stream. But how are they choosing what to read in? This is determined by what I call subspace scores. In the Framework paper these are called virtual weights and in the ARENA walkthrough these are called composition scores. These scores are implicitly learned by the model in order to read from particular subspaces from the residual stream:

While attention scores are learned indices into the rows of the residual stream, subspace scores are learned “coefficients” that provide a soft index into the “column dimension” of the residual stream. The model is able to do this because the W_QK and W_OV matrices are low-rank: d_head is conventionally much smaller than d_model. This allows for low-dimensional subspaces to be used for different purposes. Each component that reads from the residual stream learns to read from a distinct linear combination of subspaces.

To see this in action, lets look at head 7 from layer 0 from an attention-only, 2-layer transformer. Below is the attention pattern from this head on the input sequence “the cat sat on the mat. the dog sat on the log.”:

If you stare at this long enough, you can see that this head is attending to the previous token (except for the first token, which can only attend to itself).

So, here’s a question: What subspaces would the QK circuit of this head need to read from in order to create this pattern? First, let’s just look at the state of the residual stream as seen from the layer 0 heads’ perspective:

The layer 0 heads only have two options: the embedding or the positional encoding. Since “previous token” doesn’t depend on what the token is, but is just positional information, we would expect head 7 to learn a higher subspace score for the positional encoding subspace relative to the embedding subspace.

Is there a way we can quantify this from the actual model? It turns out there is. The paper and ARENA walkthrough propose using a ratio involving the Frobenius norms between the output of the previous layer and the input of the subsequent layer:

where W_A is the output and W_B is the input. A detailed justification for using this measure is given in ARENA. The justification is based on the SVD. If you do an SVD for each term, the numerator ends up containing a cosine similarity between the right singular output vectors and the left singular input vectors, so the norm is maximized when the output and input are aligned. Here are the subspace scores between the embedding and positional encodings against each layer 0 head’s QK circuit:

We can see a general pattern where the layer 0 heads are mostly reading from the positional subspace. Head 7 in particular is, especially relative to the embedding space. Since the running justification behind the Frobenius norm is to measure the alignment between output and input, we should be able to rotate the output and observe the subspace score drop. Check out the scores after rotating the positional encoding by 180 degrees:

So we can see that the QK circuit of head 7 is mostly reading from the positional subspace. This determines which source token(s) will be attended to for each query. But what about the value that is loaded from the source token(s) and written into the destination query’s residual stream? This is determined by the subspace score of the head’s OV circuit. Again, for heads in layer 0, there are only two possibilities: the embedding or positional encoding. Here are the OV subspace scores for each head:

Head 7’s OV circuit scores higher with the embedding than with the positional encoding. This means that head 7 will add the embedding of the previous token into the current token’s residual stream. Given our example “the cat sat on the mat. the dog sat on the log.”, the residual stream of token “cat” will look like this after the forward pass through layer 0:

Hopefully this token:subspace discussion has provided some intuition for how the various model components interact with each other through the residual stream. It is not a perfect model. For one, there is not really a clean, distinct set of orthogonal subspaces being selected, especially in larger real world models. Also, as the models scale up, so do the number of subspaces that a given layer has to “choose” from. It is unclear to me how many layers back a given layer can effectively communicate. This creates all sorts of questions, like are there “repeater” layers that keep a signal alive? The Framework paper suggests some components may fill the role as memory cleanup. What other traditional memory management techniques can be found here? And what would it mean to impose security isolation techniques like “privilege rings” to the residual stream? Despite the residual fuzziness, I think this mental model is a useful entry point to start thinking about this stuff.

Subscribe now

Now that we understand how the model addresses the residual stream, we can start to understand induction heads, which are just a particular combination of token:subspace addresses across heads in two adjacent layers.

Induction Heads

When a model learns induction, it learns a way to predict patterns such as A B … A __. Given the previous occurrence of A B, the induction head will predict B for the token after the subsequent A. What is cool is that this prediction solely depends on the in-context pattern rather than the particular values of A and B.

The Framework paper discusses a basic form of induction that occurs when a head in layer 1 composes with the output of a “previous-token head” from layer 0. The particular type of composition in this case is called “K-composition” because the key side of the head's QK circuit learns a high subspace score with the OV output from the previous-token head in layer 0. Keep in mind, each layer 1 head sees roughly 14 subspaces in the residual stream of each token: embedding, positional encoding, and the OV output of the 12 heads from layer 0.

When the induction head sees the second occurrence of A, it queries for keys which have emb(A) in the particular subspace that was written by the previous-token head. This is different from the subspace that was written to by the original embedding, and hence has a different “offset” within the residual stream. If A B only occurs once before the second A, then the only key that satisfies this constraint is B, and therefore attention will be high on B. The induction head’s OV circuit learns a high subspace score with the subspace of B that was originally written to by the embedding. Therefore it will add emb(B) to the residual stream of the query (i.e. the second A). In the 2-layer, attention-only model, the model learns an unembedding vector that dots highly at the column index of B in the unembed matrix, resulting in a high logit value that pulls up the probability of B.

To get some more intuition, lets look at some pictures. First, the attention pattern induction head itself. In the 2-layer model, there are actually 2 induction heads that compose with the previous-token head from layer 0. But we will just look at the first, head 4:

You can see that “by default” the head attends to the first token in the sequence, which is the special end-of-text token from the tokenizer. Later in the sequence, the attention forms an off-diagonal. If you look closely, you can see this is where some tokens A B are being repeated. For example, take A=sat and B=on. Then A B is repeated twice in the sequence, so we would expect induction to happen here.

Before we look at the subspace scores, let’s think about what we expect to see. On the query side of the QK circuit, we should see a relatively high score for the embedding of the token: when the head sees the second A (e.g. token 10), it is querying based on the actual “value” of A, i.e. emb(sat).

On the key side of the QK circuit, we need the token indices that have emb(sat) in the subspace written by the previous-token head. So the K subspace score should be high for that particular head (head 7). In this case, this would the first ‘on’ token (token 4 above).

Once we have the token index from attention (token 4), the V subspace score determines the particular subspace(s) to read from token 4 and write to the residual of the query (token 10). In this case this would be the embedding subspace of token 4.

One note: you’ll notice that the heatmaps below don’t have the positional encoding. This is because the particular 2-layer model I used for this uses the “shortformer” positional encoding option in TransformerLens, meaning that the positional encoding is added to the layer 0 residual stream input only, so layer 1 heads don’t see a positional encoding.

Here are the subspace scores for the layer 1 heads:

These are mostly in line with what we expected. The Q side scores highly with the embedding. The K side scores high with L0.H7 in heads 4 and 10, which are the two induction heads. Interestingly though, they also incorporate information from L0.H4, both in the query and key scores. I wonder what this head is doing! The V side is mostly aligned with the embedding, as expected.

Conclusion

Hopefully now you have some better intuition for how different components in a transformer interact with each other through the residual stream. Obviously we just looked at simplified models. But I think that the mental model of “residual stream as shared memory” is a useful one to begin thinking about this stuff. And if the residual stream is a shared memory, then understanding how the memory is addressed is a reasonable next step.

One point of clarification on the token:subspace address. In the attention section above, I said that attention computes the token part of the token:subspace address. However, this really applies only to the OV circuit’s token. Both the query and key sides of the QK circuit use an implicit token of just whatever the “current” token is, with each token being computed in parallel. However, the OV circuit doesn’t know which tokens to look at, and so the OV circuit’s token part of the address is provided by attention from the QK circuit. However, the Q, K, and V inputs of each head all learn the optimal subspace scores independently, completing the full two-part address needed to perform the head’s overall operation.

Story time: Once upon a time, there was a little boy named Timmy. Timmy loved to play with his toys and run around in the woods. One day, Timmy’s mom told him to his mommy and said, “Mommy, what’s that it’s important to be careful.”

Mommy said, “Okay, Timmy, you’re car book will be careful picture. They both went to the beach and said, “Mommy, I help you.” His mom wash the candy. He could do it and said, “That’s a word. Mommy said, “Okay, I will help you do it!” Timmy was scared and said, “Thank you, Timmy. You can use your dress and carrots of cool!”

Timmy was so proud of himself for his dark and played with Max. They went back and playing with his mom. They were happy to see Mr. From that day on, Timmy was glad went on the big wave came back and he was safe. Timmy learned that it’s important to go away to the forest. They were playing until the air and he would always very happy. They all lived happily ever after. Timmy was so happy to have fun in the sky and Timmy went outside

The story has decent structure, but is still mostly an incoherent mess. One idea to improve on the performance is to use a tokenization algorithm called byte-pair encoding (BPE). BPE is commonly used in pre-training of real large language models today.

Note “pre-training” is jargon for the training objective of maximizing the next-token probabilities used in the context of large language models. This is the same training objective used for every post in this series.

In practical setups, pre-training doesn’t use character-level tokens (where 1 token = 1 character). Instead they use chunks of characters, words, or even multiple words to represent one token. BPE tokenization is the main algorithm used for creating these chunked tokens.

So what is BPE and why should we suspect it may help performance?

The main idea behind BPE is to leverage the “low-level” statistical structure that is pervasive in text in order to reduce the amount that our model has to learn from scratch. For example, in English, the character ‘t’ is frequently followed by the character ‘h’. And the characters ‘th’ are much more common than ‘xz’. Knowing these statistics of frequently-paired characters1 is directly supportive of our objective: predicting the next token given the tokens we have already seen. BPE tokenization iteratively computes many of these initial, low-level statistical pairings (called “bi-grams”), alleviating our model of the burden from learning these itself. This frees up capacity of our model to learn higher-level patterns.

Share

The BPE tokenizer computes these frequency statistics, starting from the raw characters. The individual characters comprise the initial vocabulary. Each iteration of the tokenizer creates a new token that is the result of merging two smaller tokens (e.g. ‘t’+’h” → ‘th’). The new token is added to the vocabulary, and the process is repeated until a target vocabulary size is reached.

The high-level flow of BPE is:

1. Compute the frequencies of each pair of adjacent tokens in the current token list

2. Merge the most frequent pair of tokens into one new token

3. Add the new token to the vocabulary

4. If vocabulary size equals target vocabulary size, stop. Else proceed to step 1.

Implementing the BPE Tokenizer

You can find all the code for this post here.

The input of the tokenizer is the base vocabulary and a representative collection of training samples. The output is an updated, expanded vocabulary. The base vocabulary (including the stop token) for TinyStories is 175. I performed tokenization on the first 40% of the training set containing 847K sample stories:

for s in data_trn[:nr_trn]:
    cur_tokens += list(s)

init_token_count = len(cur_tokens)
nr_actual_tokens = len(stoi)
nr_desired_tokens = 5000

print(f"Initial token count: {init_token_count}")

while nr_actual_tokens < nr_desired_tokens:
    new_tokens = []
    counts = {}
    candidates = {}

    for i in range(len(cur_tokens)):
        if i == len(cur_tokens) - 1:
            break

        left = cur_tokens[i]
        right = cur_tokens[i+1]

        # Prevent merging on the stop_char so it is easier to deliminate
        # each sample from the dataset
        if left == stop_char or right == stop_char:
            continue

        tok = left + right

        if tok not in counts:
            counts[tok] = 1
            candidates[tok] = {}
            candidates[tok]["left"] = left
            candidates[tok]["right"] = right
        else:
            counts[tok] += 1

    # Pluck out the token which has highest count
    new_token = max(counts, key=counts.get)
    left = candidates[new_token]['left']
    right = candidates[new_token]['right']
    cursor = 0

    for i in range(len(cur_tokens)):
        if i == 0:
            continue

        # Check for merge condition
        #
        # We merge if the left and right tokens match, and the cursor is not i.
        # If the cursor is i, it means we just merged, and could merge again (two
        # overlapping merges), but merges are non-overlapping for book-keeping purposes,
        # so we just skip over the current token, keeping cursor where it is.
        #
        if cur_tokens[i-1] == left and cur_tokens[i] == right and cursor != i:
            if cursor < i - 1:
                # Cursor is behind the left token, so copy [cursor, left token)
                new_tokens += cur_tokens[cursor:i-1] + [new_token]
            else:
                new_tokens += [new_token]

            # anytime we merge, we move the cursor to the immediate
            # right of the right-merge token
            cursor = i + 1

    # Grab the rest. this also gracefully covers the degenerate case where no merges happened
    if cursor <= len(cur_tokens) - 1:
        new_tokens += cur_tokens[cursor:]

    cur_tokens = new_tokens
    new_len = len(cur_tokens)

    # Now add the new token to the token dictionary stoi and itos
    stoi[new_token] = nr_actual_tokens
    itos[nr_actual_tokens] = new_token
    nr_actual_tokens += 1
    
    print(f"Merged {new_token}")
    
    if nr_actual_tokens % 1000 == 0:
        print(f"Tokens: {nr_actual_tokens} ({(init_token_count - new_len) / init_token_count:.2f}% reduction)")
        
    if len(new_token) > 40:
        print(f"Max token length is {len(new_token)}: {new_token}   (stopping)")
        print(f"Number of tokens: {nr_actual_tokens}")
        break

The first step is to convert each sample story into a list of individual characters. That gives you your initial token list. The algorithm then iterates over the token list until the actual vocabulary size meets the target size (or another stopping condition is met, like maximum token length). My target size was 5000. For context, the GPT family of LLMs have a vocab size in the ballpark of 60k2.

Subscribe now

On each iteration, the tokenizer first computes the frequency of each adjacent pair of tokens (lines 29-35). The pair that occurs with highest frequency is plucked out (line 30). Then it iterates over the token list again, this time looking for adjacent token pairs whose merged result matches the highest-frequency token. Each matching pair is replaced with the merged token, with the exception of tokens that are adjacent and would be involved in two merges (i.e. a token can be “consumed” in at most one merge per iteration). Then the new token is added to the vocabulary (line 71). This process repeats until we get 5000 tokens, or we get a token that has length longer than 40. In practice, the latter condition is what triggered first in my setup. Both of these conditions were arbitrarily set from me. It would be interesting to understand what the “optimal” tokenization hyperparameters are for downstream performance, including the minimum number of samples needed, max token size, and number of tokens.

After running the tokenizer, the vocab size grew from 175 to 869. We can get a feel for the tokenizer’s effect by sorting the new vocabulary by token length and looking at the first few entries:

print(sorted_stoi)

{
 'Once upon a time, there was a little girl named ': 868,
 'Once upon a time, there was a ': 449,
 'Once upon a time, ': 430,
 'little girl named ': 720,
 'Once upon a time ': 843,
 'Once upon a tim': 375,
 'there was a ': 360,
 'Once upon a ': 374,
 'little girl ': 475,
... [snip]
}

Each entry above is a single token and will be represented by a single embedding vector in the model. This has effectively compressed the token space that our model has to learn from. Before BPE, the number of tokens was 1.1 billion; after tokenization the total drops to 359 million. That is a 67% reduction.

Now that we’ve run the tokenizer, we can tokenize the stories in the training and validation sets:

def tokenize_string(s, sorted_stoi):
    tokens = []
    cursor = 0

    while cursor < len(s):
        for k, _ in sorted_stoi.items():
            if s[cursor:].startswith(k):
                tokens.append(k)
                cursor += len(k)
                break
    
    return tokens

sorted_stoi = {k : stoi[k] for k in sorted(stoi, key=len, reverse=True)}

# We have the tokenization map stoi. Now we need to tokenize the training and validation sets
nr_trn = len(data_trn) * 6 // 10
nr_val = len(data_trn) // 10

trn_stories = data_trn[:nr_trn]
val_stories = data_trn[nr_trn:nr_trn+nr_val]

trn_tokenized = []
val_tokenized = []

trn_lens = 0
trn_tokenized_lens = 0

for s in trn_stories:
    tokens = tokenize_string(s, sorted_stoi)
    trn_tokenized.append(tokens)
    trn_lens += len(s)
    trn_tokenized_lens += len(tokens)

for s in val_stories:
    tokens = tokenize_string(s, sorted_stoi)
    val_tokenized.append(tokens)

Note the above algorithm is completely serial and very sluggish. It could be sped up alot with multiple threads but I didn’t want to overcomplicate it.

Now to inform the maximum sequence length used during training, we can look at the distribution of tokenized story lengths in the training set:

Based on this I chose a sequence length of 512. Since many of the stories are less than 512, we need to pad them with seq_len - len(story) stop characters so that they all have 512 for batched training runs. While we’re at it we need to record the valid length of each story, which is the length of the story before any padding:

seq_len = 512

trn_tokenized = [s for s in trn_tokenized if len(s) <= seq_len]
val_tokenized = [s for s in val_tokenized if len(s) <= seq_len]
trn_tokenized_lens = []
val_tokenized_lens = []

pads = []
for s in trn_tokenized:
    trn_tokenized_lens.append(len(s))

    if seq_len == len(s):
        pads.append([])
    else:
        pads.append([stop_char] * (seq_len - len(s)))

trn_tokenized = [s+p for s, p in zip(trn_tokenized, pads)]

pads = []
for s in val_tokenized:
    val_tokenized_lens.append(len(s))

    if seq_len == len(s):
        pads.append([])
    else:
        pads.append([stop_char] * (seq_len - len(s)))

val_tokenized = [s+p for s, p in zip(val_tokenized, pads)]

for s, l in zip(trn_tokenized, trn_tokenized_lens):
    assert len(s) == seq_len
    assert l <= len(s)

for s, l in zip(val_tokenized, val_tokenized_lens):
    assert len(s) == seq_len
    assert l <= len(s)

The valid lengths are used for calculating the mask of the padding characters along with the future tokens in multi-head attention. This means that attention scores for the padding characters and the future tokens will be zero.

Training and Performance

At this point we are ready to train the one-layer transformer again, this time with our tokenized samples. There is a slight difference between the model we trained in the last post versus this one. Since the vocabulary is larger, the embedding matrix and the final linear projection will be larger. This also means the predicted distribution that the model produces will have 869 choices instead of just 175. This creates an apples-to-oranges situation when directly comparing the perplexity between models. Instead we can use bits-per-character (BPC), which normalizes the total log likelihood by the total number of characters. The total characters are the same between models.

The larger vocabulary increased the total parameters to over 10 million. It took about 38 hours to train on my system. The BPC of the prior model was 0.82 and the BPC of the model with tokenization was 0.55. That is a 32% performance boost!

Let’s sample a story from our tokenized model:

Story time: Once upon a time there was a little girl named Lucy. She was 3 years old and she loved to explore. One day, Lucy decided to go outside and explore the world. Lucy was excited and she ran outside to explore.

As she walked, she noticed a big puddle of water. She wanted to take a closer look and thought it was so cool! She ran towards the door and looked around. She saw a big, green tree with a lot of smoke coming from inside.

“What is that?” she asked him.

Her mum smiled and said, “It’s an earthquake. You can open the door and see what will happen.”

And so, Lucy spent the whole day exploring the world. She found lots of exciting things in the puddles and putting them out of the open field. She was so happy and excited that she jumped up and down in excitement.

She ran around the puddle and laughed, and she ran around the house with her fingers. When she reached home, she was so excited to explore and make new friends. She had so much fun exploring the world around her.

The End.

I think this is the best story yet! It isn’t perfect, there are still logical inconsistencies and issues with pronoun agreement. The story doesn’t follow a unified, coherent progression of thought. However it does seem to have a beginning, middle and end and overall is the most coherent story we’ve seen so far.

For reference, here is the sampled story from the very first model we trained from Part 1 using Bengio et al.’s “Neural Probabilistic Language Model” which is essentially just a feed-forward network (FFN):

Story time: Once upon a timk.e an ter pere eo hire sores the caed va boanshThr witlin. HtA. ThengerDpg, and ker ditgy und g nit, tayeur Vag anddbTÉdbvjogd isswarp!e wow,e. ouancs.”Tneyd-4%un6¸¤ŒÂ·¯ } Iyž+‡+›´¢¿D»ájfÉŽ°éG­™yz›1ŒÂš¯»{U9¬#³’} %>²)¸‘¬#œj;Êq>‘æɵLbæäc®è.cŽ39°zc·dxnomd.ƒ>o¦t.mTe suŒlmvcyI¢”D᜖jœ³;¿äXécv™¦Rƒ¸2’F‹ @›”ƒÃ›6±zš<°bÉ;®®Ò`0 ?.Ä#2»áB”·”â2´F¹…¥®@12§9\>ˆ§£V}å4¹€FéQ}¦©¡¨±¼¯†Ã))`¼É\Rzä¡\¬#;³YŸ°vVLâ%Ä<Z泯éO‚ÃMž+`[”æCâj,CÑSŠ\,¹ ]O⬘<!ææÒ¯Y案9Êï4g$½?ÄbïÉ?oBH˜ä± ;ãR>@)ƒ‰ˆ=Xð¥¹P,?0=>Žð:”QW°JFxQ(3\h„ŠðÉ)X˜´QDµxj».¢É?š¬ªRc³ŠïŠ¬­qU¢E¹¢œR0‰2Ÿð:Ž+Å4¡º^

Again we can’t quite compare this directly with the tokenized transformer, since the FFN I trained was using character-level tokens. Regardless the improvements that were made in the 14 years between Bengio’s FFN and Vaswani’s Transformer are astonishing. And interestingly, the FFN is one of the key components inside the Transformer block. Now that we have BPE tokenization, the only major difference between this model and the GPT family of models from OpenAI is scale. GPT-2 has 12 layers. GPT-3 has 96 layers.The addition of two other major components, self-attention and the residual stream, combined with the scale of stacked Transformer blocks is what has elevated modern LLMs into the mainstream.

Once upon a time there were two fearful of many toys. They do not notice their fight. They liked to give the chimding into his room. There, they had a doll, I cut the brush to go away. Let’s decide it rown in your bones and your bike, Ben. You are brave and selfish.” They ask Mom and Dad.

“Go?” Lily said, pointing at the balloon. She hugged the doll bitter. She opened her around with her window. One day, she noticed something giragain and the airport. The little bird flew away, curious, and told her family for being so much fun.

Timmy felt happy with his game and went to her mom and stayed because no one wanted to see the flower. Lily realized that being happy she and Lily, was very surprise

with a perplexity of 2.32. Most of the structure is correct, but the semantics of the story are still poor. In this post I want to train a simple transformer on the TinyStories dataset to try and improve performance.

Subscribe now

Transformer Architecture

The Transformer was introduced in the seminal paper Attention Is All You Need. It was the first instance of using attention in a non-recurrent fashion for language modeling tasks. You can think of attention as computing a weighted average that is differentiable. This gives attention a nice interpretation, as a token which is given higher weight is being “attended to” more intently. The attention weights are learned automatically through backpropagation. The attention weights can be anything in theory, but in practice they are the output of a softmax function. This leads to a probabilistic interpretation where each weight is the probability of selecting a given value. Here is the attention calculation:

In Attention Is All You Need, Vaswani et al. used a special kind of attention called self-attention. This was used so that each token1 directly computes attention with itself and every other token in the sequence (except for some masking to prevent lookahead bias). These direct connections are the reason why the Transformer was such a breakthrough; they allow for efficient learning of long-term dependencies. This gives Transformers a massive advantage over prior recurrent-based architectures like the LSTM.

The original Transformer paper also used multiple “heads” of self-attention. Each head has its own attention weights and attends to the same input. The idea is to let the model learn different high-level features/concepts from the input via divide-and-conquer. One head for syntax, one head for co-reference, another for long-range dependencies, etc2. The output from each head is concatenated and projected into the “model dimension” which is the dimensionality of the token embedding space and model as a whole. The output of this self-attention block is then added to the residual stream of the input (an idea borrowed from ResNet) and normalized before being fed into a 2-layer feed-forward network with a ReLU in the first layer. Finally the output of the FFN is added to the residual stream and normalized as input into a final projection into the logit space (i.e. back to the vocabulary dimension). The logits are fed through softmax for the final probabilities of the next token.

The final piece to make this all work is positional encoding. This is done at the beginning and added to the initial token embedding to let the model learn relative and absolute token positions. Positional encoding isn’t really needed in sequential models like the LSTM because the computational graph already inherently encodes the token positions. However self-attention has no knowledge of which tokens are before which, since it unconditionally attends to every token at once. Vaswani et al. used a fixed encoding using trigonometric series, but later papers like GPT-1 used learned positions.

Share

Let’s Build It

You can find the code for this post here.

The original Transformer model was applied to machine translation, so it had an encoder and a decoder. The encoder mapped tokens from the source language into the model’s representation space (called “d_model”), and the decoder decoded the representation to the destination language vocabulary. Since we are just doing next-token prediction from one input sequence, we can get away with just a decoder. Our decoder-only model has the following structure:

This model is a single-layer, decoder-only Transformer with a final linear projection at the end into the vocabulary space. The softmax gives us the next-token probabilities.

Multi-Head Attention

The multi-head attention block is in blue. Each attn module computes scaled dot-product self-attention on identical inputs (the output of the positional encoding block) and linearly projects the attention-weighted values to a subspace of the model dimension. The model I trained for this post has a model dimension of 1024 and eight attention heads, each projecting into a 128-dimensional subspace. Each attention head computes the following:

Q, K, and V are all linear projections of the same token sequence into 128-dimensional subspaces. The output of each head is concatenated and projected back to the model dimension of 1024. Since this is a decoder-only Transformer, self-attention must be “causal”, meaning we can’t allow it to attend to tokens in the future. So the first token can only attend to itself. The second token can attend to itself and the first token, and so on. We can implement this by “masking” the softmax operation on a per-token basis. The mask is implemented by setting the columns with an index greater than the token index to a large negative number (e.g. -inf in PyTorch). Each entry with -inf will have a softmax of zero, and so the attention score of that entry will be zero. Performing the softmax over the column dimension effectively prevents the attention from “looking ahead" to future tokens. Here is a plot of the attention scores from each of the 8 heads, conditioned on the input:

One day, Tim said “let’s go to the park to play”.

In each of these, ‘O’ is at the top of the y-axis and left of the x-axis, and the input proceeds from there. Masking is what creates the lower-triangular matrix effect.

If you squint you can see that head 3 (and head 4 as well for the most part) learned to strongly attend exclusively to the previous character. While head 1 learned to attend to spaces prior to the quote and the opening quote itself. The others are a bit harder to interpret. It is interesting that the model seemed to have learned to “delegate” different tasks to different heads.

Residual Stream and FFN

The output of the multi-head attention is added back to the residual stream, then layer normalized. Then the output of the first layer normalization is fed into a 2-layer feed-forward network with a ReLU nonlinearity. The output of the FFN is added back to the residual stream and layer normalized. The result is the output of the transformer decoder layer. Then there is a final linear projection into the vocab dimension which gives the logits of the next-token prediction. Finally these are normalized with softmax to get the predicted probability distribution of each token.

Training and Performance

This 1-layer model has 9.8 million parameters and took about 1.5 days to train on my system. For reference, GPT-2, the precursor to ChatGPT, has 48 layers with 1.5 billion parameters. So this model is still very small by today’s standards. Despite that, we still saw some performance gains over the LSTM.

The final validation loss was 0.76, beating the 1-layer LSTM by 7%. The perplexity was 2.30, edging out slightly ahead of the 2.32 from the 1-layer LSTM.

The sampled story quality is interesting. In previous posts, we have sampled the most likely character every time. This is a special case of the more general top-k sampling with temperature equal to 1.0. Top-k sampling samples the top k most likely tokens according to the predicted distribution. When k=1, we have the greedy approach from previous posts. Temperature controls the shape of the distribution. As temperature increases, the distribution flattens towards the uniform distribution. Higher temperature means the model spreads out the probability more evenly across each token:

Sampling with temp=1.0 and top-k=1 (as in prior posts), we get a story that is coherent at first, then interestingly lapses into repetition:

Story time: Once upon a time, there was a little girl named Lily. She loved to play outside in the sunshine. One day, she saw a big box of colorful flowers and she was so excited to see what was inside.

Lily was so excited to see what was wrong. She was so excited to see what was inside. She wanted to help her mom and said, “Lily, you can still she was so happy to have a new friend.

Lily was so happy to have a new friend. She was so happy to have a new friend. She was so happy to have her a big hug. She was so happy to have her a big hug. She was so happy to have her a big hug. She was so happy to have her a big hug. She was so happy to have her a big hug. From that day on, Lily always remembered to be careful when she was always be careful when she was so happy to have found the park. She was so happy to have her a big hug. From that day on, Lily always remembered to be careful when she was able to be careful when she was able to help her friends. They were happy to have a new friend.

We can reduce the repetition by increasing the top-k value and decreasing the temperature slightly. Here is a story with temp=0.7 and top-k=20:

Story time: Once upon a time, there was a little boy named Timmy. Timmy loved to play with his toys and run around in the woods. One day, Timmy’s mom told him to his mommy and said, “Mommy, what’s that it’s important to be careful.”

Mommy said, “Okay, Timmy, you’re car book will be careful picture. They both went to the beach and said, “Mommy, I help you.” His mom wash the candy. He could do it and said, “That’s a word. Mommy said, “Okay, I will help you do it!” Timmy was scared and said, “Thank you, Timmy. You can use your dress and carrots of cool!”

Timmy was so proud of himself for his dark and played with Max. They went back and playing with his mom. They were happy to see Mr. From that day on, Timmy was glad went on the big wave came back and he was safe. Timmy learned that it’s important to go away to the forest. They were playing until the air and he would always very happy. They all lived happily ever after. Timmy was so happy to have fun in the sky and Timmy went outside

This version is much more story like. It also has much better coherency than the LSTM version featured at the top of this post. Though still not perfect, it is approaching a representative story from the underlying dataset.

It is also fun to see what happens when the temperature is increased substantially. Here is a “story” with temp=10 and top-k=20:

Story time: Onmy.b.lMiaa tw pysitzm olwb,âWtb,! O a llam ub agodh?! NEdor,.v at!o is’-migat,wy.M-gete! Bme dttzoeslar?go.llss’,

WDu scrofm?!

.Holdyr,n,::!

Oe.’l

nuim yte canâ€-miff;.Jx’b peer!. Shiopr.Mors,s,s!tVe’:?“Ge rea ppcbalinsâdrus!â€ic emter.

Meff.âms.Te,tbcdusnnsq

afl jdb, it tmysqeoul’âSoppoâ€

Increasing the temperature increases the entropy of the output since sampling approaches the uniform distribution.

In the next post, I will experiment with a couple of things to try to increase performance further.

The first is the tokenization scheme. The current tokenization scheme maps one character to one token. One alternative, used heavily in real language models, is to use byte-pair encoding (BPE). BPE sits somewhere in the middle between character-level tokens and word-level tokens. It effectively compresses the token space, which means that for a given sequence length, the model can learn longer-range dependencies from the source dataset. This should result in better coherency, especially for a fixed compute budget.

The second is scaling. The transformer in this post is just one layer and is fairly narrow. However the transformer architecture is inherently scalable, which is why the frontier models these days use multiple 10s of layers with wide capacities. We’ll train a bigger model to see how it affects performance.

Once upon a time there were two fearful of many toys. They do not notice their fight. They liked to give the chimding into his room. There, they had a doll, I cut the brush to go away. Let’s decide it rown in your bones and your bike, Ben. You are brave and selfish.” They ask Mom and Dad.

“Go?” Lily said, pointing at the balloon. She hugged the doll bitter. She opened her around with her window. One day, she noticed something giragain and the airport. The little bird flew away, curious, and told her family for being so much fun.

Timmy felt happy with his game and went to her mom and stayed because no one wanted to see the flower. Lily realized that being happy she and Lily, was very surprise

These stories are so good I’m going to have to start charging a subscription! Kidding, and if you decide to tell this one to your kids I’m not responsible for their resulting nightmares or subsequently poor English proficiency.

Ok so the story has a few invalid words and is incoherent nonsense overall, but the syntactic structure the model has learned to generate is pretty good. In particular you can see it has mostly figured out quotations as well as punctuation, spacing, and some subject-verb agreement.

What I want to do in this post is probe the internals of the model to map out where different capabilities live. For example, can we find the hidden unit(s) responsible for recognizing quotation marks? What about other punctuation, or particular words? Can we control the expression of these capabilities by modifying the hidden units in a particular way? Most of these questions are inspired by Karpathy’s excellent paper Visualizing and Understanding Recurrent Networks. I recommend that paper for good background material. Let’s explore these questions in the next section.

Share

Visualizing LSTM Cells

Recall the structure of the LSTM:

The example cell above has only 8 hidden units (hidden_dim=8) for drawing purposes. The LSTM that we trained in Part 4 uses a cell with 256 hidden units (note I’m going to refer to “hidden unit of the cell” as just “cell”).

If we want to attribute different cells to various aspects of the input, we can generate a trace of the activation values of each cell for a given string. There is one activation value per token of the input string. This string can contain tokens of interest, such as vowels, quotes, and other punctuation. Then we can scan the activation maps of each cell to look for patterns.

Here is the code for generating a trace:

# Collect all activation values of cell @cell_idx. In the
# LSTM there are 256 total cells (note the "cell" usually refers to
# the entire memory cell of the LSTM - I'm abusing the terminology
# here slightly by referring to a particular hidden unit
# (given by @cell_idx) within the cell as "cell").

def trace_cell(lstm, string, cell_idx):
    assert cell_idx < lstm[1].hidden_dim # 256
    hidden = cell = None
    activation_trace = []

    for char in string:
        c = torch.tensor(stoi[char], device=device)
        
        # Embed the token
        x = lstm[0](c)
        x = x.unsqueeze(dim=0).unsqueeze(dim=0)

        # Generate prediction
        x, hidden, cell = lstm[1](x, h=hidden, c=cell)

        # assumes cell is (1, hidden_dim)
        activation_trace.append(cell[0][cell_idx].item())

    return activation_trace

You can see the full code for this here. When you run this for each cell with the input string:

Once upon a time, there was a girl named Lucy. Lucy asked Bob, “Why did the chicken cross the road?”

we see some interesting patterns emerge. Note in the activation heatmaps below, purple means the cell is not activated, greenish is neutral, and yellow is highly activated. The corresponding line plot of the activation values versus token is below each heatmap.

Here is Cell 48:

Cell 48 is fairly noisy, but appears to be excited by sequences ending in ‘e’ and ‘y’ like ‘Once’, ‘the’, and ‘Why’.

Subscribe now

Let’s look at Cell 46:

It seems to only get excited about ‘girl’. Of course to verify this, we would need to test with other input strings containing ‘girl’ in different positions, and also with ones not including ‘girl’ at all.

Now check out Cell 253:

It looks like a cell that is signaling for quoted strings, since the activations of the quotes themselves are as active as the quoted content. The activation is negative until is sees the first quote, then stays positive every token in the quoted string, including the last quote. The large dip prior to the quote is quite puzzling!

At this point there are more questions than answers. For example, if we retrain the model will we see the same cell patterns at the same cell index? How do the input, forget, and output gates behave in relation to the activations we are seeing? What would happen to the model’s capabilities if we nerf a cell? Let’s double click on this last cell to explore some of these questions.

Share

Digging Deeper Into the Quote Signaler

I retrained the single-layer model to see if the cell patterns were stable. Here is the plot of Cell 253:

It’s different! What happened? I had to look through all cells in the new model to see if it had moved. Sure enough I found it, but this time at cell 178:

Why did it move? Turns out that I had been accidentally initializing the weights of the model without a constant-seeded random number generator. This means the weights had different initial values than the first run. That is the only difference I can think of. It is not obvious exactly why this would cause it to move to cell 179 instead of staying at 253, but it is very interesting that the model learned it regardless. Roughly the same pattern holds - activation is negative outside of the quote, and positive inside the quote.

Analyzing Gates

Now lets turn to the gates’ behavior. We can create the same heatmap and line activation plots to see how the gates react as the input proceeds. Recall the input gate controls how much of the current token’s representation to let into the cell. So an activated input gate is excited about the current token and wants to ensure it is represented in the cell. An input gate that is deactivated doesn’t want to let that input in the cell.

The input gate is fairly noisy, especially at the beginning. In general though it seems to activate for spaces and punctuation. This would make sense for a gate trying to signal for quotes, since for this particular dataset, a large percentage of quotations are preceded by spaces and punctuation. You can see this by calculating the frequency statistics of all characters that precede a quote. For TinyStories, this distribution follows a power law:

What is really fascinating is that initially the gate is interested in spaces, but that interest fades as the sequence gets longer. However the interest in punctuation remains elevated throughout.

Let move on to the forget gate. Remember the forget gate is “active low”. When the forget gate is high, it wants to retain the previous cell’s representation in the current cell. If it is low, it wants to remove the previous cell’s representation from the cell. Here is the heatmap:

You can see it partially retains the beginning character and in general holds high, retaining the cell’s state until it reaches the first quote, where it then completely clears the prior cell’s representation from the current cell value. This means the cell’s value for the quote token is whatever is passed through the input gate, which happens to be a fairly strong representation of the quote itself, based on the input gate activation above. It then immediately activates near 1 on the next token and holds high until the next quote. The forget gate seems to be implementing a primitive state machine of {inside quote, outside quote}, where transitions occur whenever a quote is encountered.

Finally let’s look at the output gate. The output gate controls how “strongly” the cell’s representation is written into the hidden state. A value of zero resets the hidden state, whereas a value of one copies the cell verbatim into the hidden state.

This one is a bit harder to interpret than the forget gate. One thing to notice is that none of the activations are zero. So every token has at least some of the cell being written to the hidden state. We also see that on average, the output gate is more activated before the first quote. However there are “near clearing” events before and after the quote, so it isn’t clear how that is being used. It is notable that the punctuation activations are high. Combined this with the high punctuation activations for the input gate and forget gate, it suggests the output gate has learned to let the punctuation flow into the hidden state.

The running hypothesis is that this cell encodes a state machine, where the cell activation is negative when outside of quotes, and positive when inside quotes. We can re-trace the cell on different inputs to see if this holds up. I created two sets of strings from the validation set - 100 quoted and 100 unquoted. I then traced cell 178 on each string.

Here is a plot of the maximum activation values across all strings:

So we’ve falsified the hypothesis, since the unquoted strings have positive values despite not having any quotes. However the quoted strings consistently (except for one string) have a max that is greater than the unquoted strings. The majority of the time (96%), the character that gives the max activation is the quote:

Here is an example heatmap and activation line plot from an example quoted string:

These plots suggest the cell is most excited for quotes, followed by punctuation, rather than being a state machine indicating inside quotes / outside quotes.

Nerfing Cell 178

Suppose we really don’t want quotes in our generated stories. For some reason, the stories with quotes keep the kid up at night, instead of putting them to sleep like a proper bedtime story should (probably due to the anticipation of the closing quote!).

Could we use our knowledge of cell 178 to prevent the model from generating stories with quotes? The tricky part is we don’t want to hurt model performance too bad (we can’t afford that!); we just don’t want quotes in the output. The problem is that cell 178 is influenced by and influences every other cell through the recurrent matrices in the gates and candidate cell state. Some of these cells I haven’t listed here (due to time and space constraints) seem “interested” in quotes suggesting that there is a quote circuit, a graph of cells that work together to deliver the overall quote capability. But perhaps we can get lucky by clipping the value of cell 178 just right so that quotes are not generated or generated less when we sample. If we clip the activation value so that it doesn’t exceed 0.8 (based on the max values above), then the cell should behave similar to the un-nerfed version for the majority of tokens except quotes. Hopefully this will limit the second-order effects on cells which depend on cell 178.

We can generate 100 stories without modifying cell 178 to get a baseline percentage of quotes, then generate new stories after clipping any value greater than 0.8 to 0.8. In each scenario we can measure the perplexity to see the total impact to performance and the total number of quotes in the string.

The baseline 100 stories without cell modifications gave a total of 363 quotes. The perplexity was 2.2271.

Clamping the activation of cell 178 to 0.8 resulted in 298 quotes, which is about an 18% reduction. The perplexity actually fell slightly to 2.2114. We can try to clamp with smaller values to see its affect on quote generation. Here are some plots for number of quotes generated and perplexity versus activation limit (0.8 is on the far right at 298):

It is interesting that we can limit the cell all the way to -5 before seeing hardly any change to perplexity. We do begin to start trading off less quotes for losses in perplexity after -5, but otherwise it seems we’ve definitely found at least one of the cells responsible for quote generation! To completely erase quotes would take more work tracking down the other cells that are working with cell 178.

Not bad for a quick peek under the hood of our LSTM. There are other things that would interesting that I didn’t cover, like tracking down the cells that influence cell 178 the most and vice versa. Also understanding what the most “exciting” input would be for a given cell. This is tricky since we are dealing with a sequence of inputs rather than just one. It would require carefully constructing a loss function to backpropagate into the input sequence, perhaps maximizing the autocorrelation (i.e. “trendiness”) of the cell activation curve in addition to the magnitude.

John said, “I can’t make it to the gym today because I have to work”

requires the model to remember the opening quote at the beginning. Recurrence allows the model to remember previously seen tokens so that it can consider them for the current prediction.

This is the sample story from the latest model:

Story time: Once upon a little eazings ont Day, I kidgo preate it said. At and you were ittlied. The dreing frien fell backy. She is a camed an. Dadry, Tommy said, “Now, Timmy. He good they fate ause wime caplo. Shew it a with and purnt wike to na;e and hoar. Her and it was he a wourhan. Bobse time, tak mom in the mom togetecide is the bout his mommy, they prele big to think. The rail a citcus but the ponf the she friendsn. The finger ick will and do sturted the tallo!” Biknywnedded usever,o gand and ce oven tore, girl flew the parn.

Obviously still not performing very well.

The problem with the vanilla RNN is that the recurrent hidden state leads to instability during training - gradients tend to vanish or explode. This instability led to the development of initialization techniques that ensure the hidden weights have a principal eigenvalue of 1. This somewhat helps prevent the gradient problem, however due to numerical drift over long sequence lengths and other terms in the gradient, gradients as a whole still have propagation issues.

This motivated the development of alternative architectures which allow for better gradient flow through the sequence. In this post we will look at the most prominent recurrent successor1 to the vanilla RNN, the long short-term memory (LSTM).

Share

Long Short-Term Memory

Both RNNs and LSTMs have a hidden state h_t. The hidden state evolves over the sequence dimension, and represents the model’s output at the end. In order to track long term dependencies, the LSTM adds a new state element called the memory cell (or just cell for short, denoted c_t).

The terminology here was confusing to me when I learned this. The cell c_t is completely internal to the LSTM block, so in a sense it is completely hidden. The hidden state h_t is updated internally and eventually output from the model, so in a sense the hidden state is quasi-hidden.

You can think of the cell as a chunk of volatile RAM. Each element in the cell is a hidden unit that can be read from, written to, and reset to zero, analogous to DRAM. These operations are controlled via the output gate (read), input gate (write), and the forget gate (reset). Each of these gates is implemented with a sigmoid non-linearity to provide a differentiable operation of a binary choice.

When the input gate is 0, the cell is effectively ignoring the current token. When it is 1, the cell is prioritizing the current token. The forget gate is “active low”. When it is 0, the previous cell state is forgotten; when it is 1, the previous cell state is remembered. The output gate controls the degree to which the internal cell state is “released” into the hidden state h_t as output.

The other key ingredient that enables the LSTM to keep track of long-range dependencies like quotes, braces, etc. is to feed the hidden state into the gates. The hidden state provides the historical context the gates need to encode state like “we are currently inside a double quote”.

Take the following string:

The writer said, “you need to use double quotes more.”

Consider what the input and the forget gates need to do for a hidden unit in the cell that tracks quotes. At the first quote, the input gate needs to saturate to one and the forget gate needs to saturate to zero. This effectively latches the “inside quote” state representation into the particular hidden unit in the cell. Further, the output gate needs to saturate to one, so that the hidden state encodes “inside quote”. Then for the subsequent characters, this hidden state is fed into each of the three gates, effectively encoding a state transition to “inside quote”. For the non-quote characters, the input gate now saturates to zero and the forget gate saturates to one (i.e. to keep the hidden unit cell state the same). The output gate is still saturated to one to preserve the hidden state. Finally, once the closing quote arrives, the input gate transitions towards one and the forget gate resets towards zero. This transitions the cell’s hidden unit to “outside quote” again, which is fed through the output gate into the hidden state.

Now you may be wondering, what is the point of creating this new cell state? Why can’t we just use the hidden state as in the vanilla RNN?

The reason is that the LSTM construction enables better gradient flow backwards through the sequence dimension. To see why we can look at the code snippet around the cell calculation (you can find all the code for this post here):

# LSTM forward pass function
def __call__(self, x, h=None, c=None):
    ...
    new_cell = torch.tanh(batch @ self.W_xc + self.hidden @ self.W_hc)
   
    # (1) cell is child of sum operation
    self.cell = f * self.cell + i * new_cell

    # (2) cell is child of non-linearity tanh
    self.hidden = o * torch.tanh(self.cell)

    y = self.hidden @ self.W_hy + self.b_y
    outputs.append(y)
    ...

You can see there are two instances of cell on the right-hand side. This means the gradient of the loss with respect to cell will ultimately be a sum of two terms. The second instance is similar to the calculation of the hidden state in the vanilla RNN in that it is a child of the tanh in the computational graph. The first instance is the critical part, as it is the child of an addition. Since addition directly distributes gradients, this provides an alternative route for gradient to flow to cell across the full sequence dimension. This route provides a bypass around the potentially saturating tanh non-linearity.

LSTM Performance

Let’s train the LSTM to compare with the vanilla RNN. Note that I added an instance of the Adam optimizer in the training loop. This was used to train both the RNN and LSTM with a sequence length of 32 and hidden dimension of 256. Adam is outside the scope of this post, but I highly recommend reading Why Momentum Really Works in case you are curious to learn more.

The RNN ended with a loss of 1.058 and perplexity of 2.86. This means simply adding Adam provided roughly 33% improvement in the loss and 25% improvement in perplexity!

The LSTM ended with a loss of 0.9177 and perplexity of 2.45. So even with a fairly small sequence length of 32, the LSTM outperforms the vanilla RNN.

Here is a sample story from the RNN:

Story time: Once upon a time, there was a little boy named Timmy. You don’t move. And he reached the grass her toys.

“Of cou did not know. She wants to go home back in the cells. They did it was too enter and said, “No,” Ben got angry angry. Sarate was so happy. They be okay. You are wrong cricket. The bott. She saw lots of lion teaches lesson to have a big bed and a big numbers. They liked to lay.

One day, she watched and said, “That was a room, Ben looked zigingly. The moral of toys and decided to live energe.”

Lily got theard something scarf.

Now that is much better. We can see most of the words are valid English, though still a few that aren’t. We can also see that the model has learned order and structure of quoted phrases in relation to the subject that is saying them.

Here is the LSTM:

Story time: Once upon a time, there was a small bird playing tight he would always made a mess.

Suddenly, their mom smiled and said, “No, new and how to urge her mom. She did not want to send it in his room. He wanted to play in the park with.

Aman was so happy that Lily was walking in the park. She wondered why he was old twigs, but one day it said. “Okay, but you are curious. She went to recorded his wings and answer. Afterwards, he shouted on the way. They do not know that a cupboard before in the bench. Tim determined to quarrel have seen the water. Then asked her dad, “I’m sorry, Ben,” Ben says.

“F’m cool!” She wished he tried to talk to the store and put some strong store on a piece. He played in

This one is even better! All the words except for “F’m” are valid. Now the story still doesn’t have much coherency, but the structure of the story is much better. The main issue is the subjects are coming and going with no real connection to an underlying narrative. Keep in mind that this LSTM used a sequence length of 32 for training. If we want to have coherency across larger portions of the text, perhaps we should have a longer sequence length to capture longer range dependencies.

When we re-train with a sequence length of 128, the loss drops from 0.9177 to 0.8285 and perplexity decreases from 2.45 to 2.32. Here is a story:

Story time: Once upon a time there were two fearful of many toys. They do not notice their fight. They liked to give the chimding into his room. There, they had a doll, I cut the brush to go away. Let’s decide it rown in your bones and your bike, Ben. You are brave and selfish.” They ask Mom and Dad.

“Go?” Lily said, pointing at the balloon. She hugged the doll bitter. She opened her around with her window. One day, she noticed something giragain and the airport. The little bird flew away, curious, and told her family for being so much fun.

Timmy felt happy with his game and went to her mom and stayed because no one wanted to see the flower. Lily realized that being happy she and Lily, was very surprise

It seems performance is roughly the same - the quality of the stories are within the same neighborhood. Most of the words are valid, but the coherence is still lacking.

One more thing we can do is stack LSTMs into multiple layers. In this stacked arrangement, the hidden state outputs for the first layer are fed into the input of the second layer.

The loss from a 2-layer LSTM drops down to 0.7964 with perplexity 2.22. Here is a story sample:

Story time: Once upon a time there was a hunter. She liked to put away them in her little kitten. On the beach winter and was walking home, there were many fish!

Jack made sure to keep him available felt feeling helpless and sparkly, treat, and was getting tracks. The post was the most long in the wheel said, “Thank you should never have to stay up for a while. What do you tag your thome?”

At the store, her mom wanted the new glove and put it in the riverom next door. Then his owner told him about the been before dinner. He ran home and began to joke. He waved goodbye to the loud of the volcano. The sailor said, “This is enterice, me!” and warned the best joke finish his favourite sleep.

Despite the slightly better loss and perplexity, the quality of the story from the 2-layer seems about the same as the one layer version. However, the model has improved significantly over the vanilla RNN. It has learned to balance out quotations and the placement of punctuation, spaces, and line breaks. Overall the stories still lack coherency though, so there is more work to be done.

Before we try to improve upon the LSTM, I want to take some time to visualize the internals of the LSTM, inspired by Karpathy et al. Visualizing and Understanding Recurrent Networks. We will take a look at this in the next post.

Story time: Once dook had rine. It was tore on. They fubbe= ef­, storisk âun wookgar. The fabiing. He soulld eed sou jrean frea toy tayt their to so she prcedadn- and suym haw seare a vene ily. Polly wari. They were mad a wert to xime, the veasen" and grien to but furnyt want to sed talked a may.

The grid a foll. The "iling. %o loon a with ray fft. Tre sly call. He dayz away hoff the perter. The whone her to saibry. The smuny. She by timpy something to seve

fee the ground a cime then gratest ands and a with his wood! One day plays garond to curprasy was so ine of the back to take a smiled. Ore lefyor her!" Soras ecried. Jon the will and adpoly toor lantend his and story."

In this post we will try to improve on this by introducing the recurrent neural network (RNN). Before we do, it helps to remember how the model is working up to this point:

In this model, we are concatenating ctx_window character embeddings into a single feature vector, then passing it through the fully connected layers. One problem with this design is the impact the ctx_window has on the number of trainable parameters. With 4 linear+layernorm+tanh layers, our model has 313,743 parameters. Currently, our ctx_window is only 8, meaning we only have a “history” of 8 characters. So we can’t really expect the model to be able to track dependencies in the character stream beyond the most recent 8 characters. If we want to increase the ctx_window to have a deeper “history”, the problem is that this blows up the number of parameters in the first linear layer, since its dimensions are ctx_window * embed_dim, hidden_dim). For example, increasing ctx_window from 8 to just 16 doubles the number of parameters of the first linear layer from 65536 to 131072. In other words, our current model doesn’t scale with the input length.

It would be better if the architecture were better suited to exploit the fact that text is sequential. When we want to generate the next character, ideally we can make that decision based on all the previous characters that we’ve generated so far, not just the 8 most recent characters. This is what recurrent neural networks (RNNs) help us do. They are explicitly designed to exploit the fact that text (among many other types of data) can be considered as a sequential stream of tokens. This is an example of an inductive bias that aids in learning. Inductive bias is another word for the assumptions we make about the data we are working with that help the model learn.

Subscribe now

Vanilla RNNs

There are many types of RNNs. Here we will just be looking at vanilla RNNs to get our feet wet. The main idea behind all RNNs is to include a loop in the architecture. Whereas the architecture above is a directed acyclic graph (DAG), an RNN has a loop. The loop keeps track of a so-called hidden state. The hidden state is the model’s representation of everything it has seen so far. The hidden state, along with the current input, are both included in the output of the RNN. Here is a high level view:

Two things to note. One is that the sequence is processed one token at a time t=0,t=1,t=2 instead of being concatenated. Two is that the hidden state h_t is a function of the previous hidden state h_t-1 and the current token x_t. The output of the RNN o_t is a final linear layer to get us back to the vocab_size dimension. Mathematically we can write the hidden state h_t and output o_t as:

where x_t is the embedding of the current token and W_x and W_h are learned weight matrices. Of course other activations can be used besides tanh, and you can add bias to the input and/or hidden state terms if you want. The important thing to note is the weights W_x, W_h, and W_o are shared across each token t. This enables us to crank up the ctx_window without increasing the parameter count.

Even though in theory we can increase the ctx_window to any value, we still can’t in practice. One reason is that we need a constant value for fixed-width batches during training. Another is related to the training dynamics from Part 2. The main issue is the recurrent multiplication across the time dimension causes gradient instability. We will see exactly why that is later.

Share

You can find the implementation for the RNN here. To see the impact of scaling up the context window (also called the sequence length; seq_len in the code), we can train the model with different context windows to see how it affects loss, perplexity, and story generation.

Here are the validation losses for sequence lengths 16, 32, and 64:

The model with seq_len=16 had a final loss of 1.59, perplexity of 4.53. The model with seq_len=32 had a final loss of 1.47 and perplexity 4.47. The model with seq_len=64 had a final loss of 1.54 and perplexity of 4.48. Here are some example stories from each:

seq_len=16

Story time: Once upon a little Tom and his Day had dogo saed Aaded soog other to see sadint it she day liked redg feall She with his mack the funny, You of frold. He stark. They so the was byathinks fun decape iserswing an was and asked wike to nameed for a smill!” veiave hard a with a liendss time, jat man in the upon you west fish as?”

AThan’ur work. He park and the is hear to him was s ruchid. Everyone the phe fax friendsing in and cparial and dogethereot dact get loved happy. She girl ong and meace ave there agall find the park.

seq_len=32

Story time: Once upon a litt. They ging, to gelt did not ie so her saig.

A¨n the little they her and reing. red to all she with his mach the fun. They cound shink inst I cur they go bot he fathin?”

Hed cap on hasse for with and asket with then she with a beecsted to see are a wase af fellss is histar was so had ut him the sto telt the be his und scart to dell big to think.” Lily like cid nory to tigen a toy.

seq_len=64

Story time: Once upon a little eazings ont Day, I kidgo preate it said. At and you were ittlied. The dreing frien fell backy. She is a camed an. Dadry, Tommy said, “Now, Timmy. He good they fate ause wime caplo. Shew it a with and purnt wike to na;e and hoar. Her and it was he a wourhan. Bobse time, tak mom in the mom togetecide is the bout his mommy, they prele big to think. The rail a citcus but the ponf the she friendsn. The finger ick will and do sturted the tallo!” Biknywnedded usever,o gand and ce oven tore, girl flew the parn.

Clearly we aren’t seeing an advantage in increasing the sequence length on performance. The losses and perplexities are still within a small range of each other, and the sampled stories are all within the same neighborhood of not good. What’s going on?

The Curse of Recurrence

In theory, RNNs are supposed to provide a nice inductive bias for sequential data by maintaining a state that records the “history” of the sequence seen so far. In practice though, they are difficult to scale to long sequence lengths due to unstable gradients. The root cause is in the history itself, specifically the recurrent weight multiplication. To see this we can derive the gradient of the loss with respect to the hidden weights W_h:

You can find the full derivation here. The important term in the above is the W_h which is raised to the power of T-i. If we assume that W_h is diagonalizable1, then we can express T-i powers of W_h as the eigendecomposition

Where the middle matrix sigma is the diagonal matrix containing the eigenvalues of W_h. The effect that this has on an input is to stretch it in the direction of each eigenvector by T-i factors of the corresponding eigvenvalue. This means that as T increases, the matrix W_h will pull the input towards the principal eigenvector, and the magnitude will blow up (if the largest eigenvalue is greater than 1), shrink to zero (if the largest eigenvalue is less than 1), or stay about the same (if the largest eigenvalue is equal to one).

The first two cases explain the exploding and vanishing gradient phenomena present as we try to increase the sequence length T. We can see this in the gradient histograms of the hidden weight W_h measured at the last token index T halfway through each training run:

As the sequence length increases, the distribution of the hidden gradients converges to a sharper peak at zero. This problem is the primary motivation behind orthogonal weight initialization. Initializing the recurrent weights as an orthogonal matrix ensures the norm of its eigenvalues is 1, preventing an exploding or vanishing magnitude in the transformation of the input.

We can visualize this with a simple example to gain some more intuition. We will initialize a random square matrix W and a vector v, and multiply v by W for several iterations. Then we can plot the norm of v across the iterations:

W = torch.rand(3, 3, dtype=torch.float64)
v = torch.rand(3, 1, dtype=torch.float64)
eigvals = torch.linalg.eigvals(W)
eignorms = torch.abs(eigvals)
eignorms
...
tensor([1.1060, 1.1060, 1.6240], dtype=torch.float64)

The principal eigenvalue has magnitude 1.62, so we expect the norms to explode after many iterations:

v_norms = []

for i in range(100):
    v = W @ v
    v_norms.append(torch.norm(v).item())

plt.plot(torch.arange(0, 100), v_norms)

Now if we use orthogonal initialization on W, we can perform the same iteration, and the magnitude will be constant:

W = torch.randn(3, 3, dtype=torch.float64)
W = torch.nn.init.orthogonal_(W)
eigvals = torch.linalg.eigvals(W)
eignorms = torch.abs(eigvals)
...
tensor([1.0000, 1.0000, 1.0000], dtype=torch.float64)

And the plot of the norms is flat as expected:

Finally, whenever the principal eigenvalue has magnitude less than one, we see the norm vanish:

W = torch.randn(3, 3, dtype=torch.float64)
W = W * 0.1 # scale down to get smaller eigenvalues
eigvals = torch.linalg.eigvals(W)
eignorms = torch.abs(eigvals)
...
tensor([0.2711, 0.0049, 0.0449], dtype=torch.float64)

The principal eigenvalue is 0.2711. Here is the plot:

Of course, this is a simplified scenario, so real RNNs will have dynamics that are much more noisy than these, however these provide a good intuition for what is happening. Most of this noise comes from the other terms of the gradient, plus the fact that the weight matrices themselves are modified after each backwards pass (assuming they get a gradient). Like we saw in Part 2, the values of the weights are dynamic, so special initialization techniques like orthogonal init only takes us so far. This is why architectural improvements to the vanilla RNN have been pursued, the most notable of which is the long short-term memory (LSTM), which we will see on the next post.

Story time: Once upon a timk.e an ter pere eo hire sores the caed va boanshThr witlin. HtA. ThengerDpg, and ker ditgy und g nit, tayeur Vag anddbTÉdbvjogd isswarp!e wow,e. ouancs.”Tneyd-4%un6¸¤ŒÂ·¯ } Iyž+‡+›´¢¿D»ájfÉŽ°éG­™yz›1ŒÂš¯»{U9¬#³’} %>²)¸‘¬#œj;Êq>‘æɵLbæäc®è.cŽ39°zc·dxnomd.ƒ>o¦t.mTe suŒlmvcyI¢”D᜖jœ³;¿äXécv™¦Rƒ¸2’F‹ @›”ƒÃ›6±zš<°bÉ;®®Ò`0 ?.Ä#2»áB”·”â2´F¹…¥®@12§9\>ˆ§£V}å4¹€FéQ}¦©¡¨±¼¯†Ã))`¼É\Rzä¡\¬#;³YŸ°vVLâ%Ä<Z泯éO‚ÃMž+`[”æCâj,CÑSŠ\,¹ ]O⬘<!ææÒ¯Y案9Êï4g$½?ÄbïÉ?oBH˜ä± ;ãR>@)ƒ‰ˆ=Xð¥¹P,?0=>Žð:”QW°JFxQ(3\h„ŠðÉ)X˜´QDµxj».¢É?š¬ªRc³ŠïŠ¬­qU¢E¹¢œR0‰2Ÿð:Ž+Å4¡º^

As you can see, the model is pretty terrible right now. In this post, we will try to improve the performance of the model by scaling it up - adding more layers and hence representational capacity to the model. However, we will see that simply adding layers can cause instability in training. We will look at some basic techniques for addressing this instability and improving performance. The notebook for this post can be found at char-bengio-dynamics.ipynb if you want to follow along with code.

Subscribe now

Adding Layers

The model from Part 1 uses just a single Linear layer with tanh activation. By adding more layers, we add parameters, and give the network more representational capacity. In theory this should let our model learn better (at the risk of overfitting to the training set). Let’s try using 4 linear+tanh layers to the base model:

This brings the number of parameters up from 116,912 to 313,568. I modified the training loop slightly to log the gradient values of each layer every 20,000 training steps:

Running this for 100k training steps bring the final validation loss down from 6.4 to 4.4. Here are the loss curves:

It also lowered the perplexity from 424 to 22. Now let’s check a sampled story:

Story time: „h3I\vŠâªâ¤š°FA7G
'½ndS4 m tt .elhIeidoioms e oaiydsseJgd
tceo,ty,
wSns itt h hs ud rli eh rndg ee l S ,yed dp i ra cam d"ae.Sodtro,eo ehoh h ahd lstbr tuv t,yeo t ho"
hlbLsth usrnwymdecr oi erswiii Tdw.we onhaourdh wtko temn epea shoaws ecstSapniade rg a ,ehaf e scn ehdg dat me i'hg osutah yaaaetsds
het atw obn h usuyhwuknnlnus hla isttctioih oer T r olwa hi uuryTt m twannaw ee.ahem, rsf nta snioe as iep hirigyn!e o ut tSeos dcstagbui"bSn"tyawlud
h ueyi rtocg ar emtaae aoni rl"ag l fhn Tt b pte mbmah ahwt"aueEiva ai rv" hswr!eA .ekga ote edu, hyw ohoee?T wtt lrthnecmb'au hlend tytiuwdld E a,snaus.tc lddd otnHsbgre"tasdhidrMdtr ebtrhasa esc aa
meT mntyldmmpu s"
esnty.csyeâa itielt nh a tp ol oesn iie obgfeS.sogskB cen
hdiaes e hssuo.

As we can see, our dream of Skynet is still in the far distant future, however we are making progress; the model is starting to learn what appears to be the ASCII character range, word boundaries, and punctuation. Since our validation loss is still close to the training loss, we have room to simply add more parameters before overfitting takes over. We could do this by adding additional layers or increasing the embedding and/or hidden dimensions.

But before we throw even more capacity at the problem, I want to dig into the flow of various data streams throughout the network. The behaviors of these streams of data, both in the forward and backward direction, are called the training dynamics. In order to train deep networks, we have to ensure that the training dynamics are stable, otherwise gradient descent will diverge. If gradient descent diverges, then our model fails to learn anything useful. The most common culprit of unstable training dynamics are unstable gradients.

Share

The Gradient Must Flow

Like the spice of Arrakis, the gradients within our network have to flow to each of the parameters in order for them to learn. Recall the update formula:

If p.grad is zero, the parameter value (p.data) doesn’t change. Likewise, if p.grad is too large, it can teleport us to completely different parts of the loss landscape without ever settling down into a minimum. So we need to understand and monitor the gradient values for each layer to ensure they don’t become degenerate.

For our model, we are using linear layers followed by tanh activation. To understand the gradients through these layers, we can analyze the graphs of tanh and its derivative:

We can see the derivative (in red) vanishes to 0 as the absolute value of the input increases beyond roughly 3. This region beyond |3| is the saturating region of the tanh activation. What is the input to our tanh? The input is the output of the linear layer y (also called the preactivation):

So as soon as the absolute value of y gets larger than 3, tanh itself becomes saturated, and the gradient on the tanh for that value will be essentially 0. This problem compounds in backpropagation with each additional layer because of the chain rule, which is multiplicative:

When the gradient of tanh is 0, the middle term dz/dy is 0, causing the entire expression to be 0. This 0 gets propagated down to the child nodes in the computational graph (i.e. layers which are closer to the beginning), causing learning to stagnate in these deeper layers as well.

Visualizing Gradients

To gain more intuition for this problem, it helps to visualize the gradient values of different layers at various steps during training. Below is a histogram of each layer’s gradient after training for 100k steps. You can see from the spikes in the middle that several layers’ gradients have vanished substantially:

We can double-click on these layers to see the evolution of the gradient for each neuron at various training steps. Below is the evolution of the gradient of the first linear layer. As the gradient tends to 0, the color of the corresponding point in the heatmap tends to purple. A vertical column of purple indicates a completely dead neuron - no gradient is flowing through it. The x-axis indicates the neuron and the y-axis is the batch:

Below is the evolution of the last linear layer, i.e. the output layer. You can see it fares a little better than the first linear layer, yet still only has a few surviving neurons:

So what can we do to ensure better gradient flow? There are many options, more than I can cover in this post. However they mostly revolve around the same fundamental idea: we need to avoid inputs which lead to saturated non-linearities. In our example, this means avoiding too many inputs whose absolute value is greater than roughly 3, beyond which the derivative of the non-linearity, tanh, is zero. Other non-linearities will have different bounds, but the need to avoid these bounds is the same. Below we will look at a few techniques that are useful for mitigating excessive saturation.

Xavier Initialization

Glorot and Bengio figured out that for tanh activations1, keeping the variance of the preactivations close to one for every layer helps to create more stable gradient flow. The reason is that assuming the preactivations have mean 0 and variance 1, then approximately 95% of all values lie within the interval [-2, 2], thus 95% of the values will stay within the non-saturated region of tanh.

If we assume the inputs and weights are independent with mean 0, then we can calculate the variance of the jth neuron (assuming zero bias for simplicity):

If we further assume the weights and the inputs have a constant variance Var(W) and Var(x), respectively, then we can derive an expression for Var(W) which provides a variance of 1 on yj:

So for initialization we can choose a W such that the variance of W depends on the fan-in (n) of the layer, as well as the variance of the input. The input could be the actual input, or it could be the activation from a previous layer. This underscores the importance of normalizing the data input to the network - doing so ensures the assumptions of the above calculations are met and that the chain of variances can be one throughout the network. This is essentially what so-called “Xavier initialization” does.

After re-training the model with PyTorch’s xavier_uniform_ on each linear layer, we get the following loss curve:

Notice the initial loss is much closer now to the expected baseline of ln(nr_classes) = ln(174) = 5.15. This change also brought the final validation loss down from 4.4 to 1.9 and the perplexity down from 22 to 5.73. Let’s sample a story:

Story time: 1ap ue was laghoned thain hit wayl Ieind om tearouly ss og otciout the ton ittligher and reined veru†ne low at wis pois, shay dove. Eher weorej Io hatd ast Iraw. RWy and her hebfat hemstily hecrpeneser with he was and and hewtur to nacked itho

" brat. He ane sard a dore aflyen sconed. That was in tad, thery are sus is the boun hit'ur wor't beap ala bimtterigit on the bolea ci'rus. Tthe twand.

We. hem, tharent Ting to pave puririgen.

Aut theor dact gerilbeghty was ho sey herocghtryemeree to hackri" gily fin the dparpmbmas and "aver Iaw iore" haw so loqk ver"eded they dobee tike therthe cablaw hernd tearpund a hays. Th. The ded to shire raschilred boZ theas to chaaned Timmyy whipg saine. They yin tigingt nog her oldong to he cogie took the cent oudines a histo.

Woge hiund have hake, "athee lilly can sever bucher. Th they jaly Sveol.

Still nonsense! But also much better. We see some actual English words now and both sentence and paragraph structure starting to emerge. All that just from specially chosen initialization values. Now let’s check the gradients. Here is the overall distribution:

We can see the peak is lower (from around 15000 to roughly 12000), and there appears to be just one layer, the embedding layer, which has significantly vanished gradients. This perhaps isn’t surprising, because it is the last layer in gradient descent, so any near-zero multiplicative affects from its ancestors in the computational graph are magnified there. Here is the evolution of the gradients for layer 2, the first linear layer:

Interestingly, the gradients start out closer to zero in the first step relative to the non-Xavier init model. However as training continues, the gradients’ magnitudes appear to get slightly larger on average, increasing at each step. If you look at the last frame, it looks like no neurons are completely dead - there are at least a few batches providing non-zero gradients.

There is alot more to initialization than was covered here. Xavier init is good for activations that are symmetric around 0, whereas Kaiming init is better suited towards asymmetric activations like the ReLU and its relatives.

One shortcoming of these special initialization techniques is they obviously only occur once. This can cause the network, especially deep networks, to have drifting activations over time. This drift can venture into saturating or exploding regions of activations, causing unstable gradients. One mitigation to this is to add normalization dynamically at each layer so that the inputs to the subsequent non-linearity stay in the activated region.

Layer Normalization

Many different types of normalization techniques exist. And you can apply normalization to different types of data - weights, preactivations, gradients, input data, etc. Layer normalization typically applies to the preactivations, i.e., the output y of the linear layer:

Layer Normalization applies this normalization across the hidden unit dimension. This means the average and variance are computed across each neuron in a single batch. This is opposed to Batch Normalization, which normalizes across the batch dimension:

The advantage of layer norm over batch norm is that batch norm requires keeping a running average and variance throughout training so that it can be used later during inference. The reason is that inference has to gracefully handle a batch size of 1, but the average and variance over one element is at best noisy and at worst meaningless. So we have to retain the statistically significant batch statistics computed during training and reuse them during inference. On the other hand, layer norm works with any batch size, including batch size of 1 during inference and doesn’t require any special state between training and inference. The implementation is straightforward:

Let’s see what happens when we add LayerNorm after each linear layer and retrain. The loss curves are reaching a slightly lower minimum (and also slightly overfitting):

The final validation loss improved from 1.9 to 1.7. Perplexity improved from 5.73 to 4.50. Ok, now grab some coffee! It’s time for another story:

Story time: Once dook had rine. It was tore on. They fubbe= ef­, storisk âun wookgar. The fabiing. He soulld eed sou jrean frea toy tayt their to so she prcedadn- and suym haw seare a vene ily. Polly wari. They were mad a wert to xime, the veasen" and grien to but furnyt want to sed talked a may.

The grid a foll. The "iling. %o loon a with ray fft. Tre sly call. He dayz away hoff the perter. The whone her to saibry. The smuny. She by timpy something to seve

fee the ground a cime then gratest ands and a with his wood! One day plays garond to curprasy was so ine of the back to take a smiled. Ore lefyor her!" Soras ecried. Jon the will and adpoly toor lantend his and story."

Hey look at that, dook had rine! We’re getting more words now. And the early hints of a coherent story are taking shape. A lot of the words are nonsense still, but the sentences appear to have a structure with basic subject-verb agreement.

Here are the gradients at the end of training:

The peak is lower, down to 8000 from around 12000. It looks like many of the layers have a decent spread around 0 as well. Here is the evolution of the gradient of layer 2:

Once again we see the gradients slightly increasing in a controlled way, and no neurons appear dead on any of the steps.

Conclusion

In this post we’ve focused on understanding gradient flow and deployed two techniques for stabilizing gradients during training: Xavier initialization and LayerNorm. Of course there are other aspects of the training dynamics we could look at, especially the parameter update size and learning rate. There are also architectural tricks like residual connections. These connections provide an additional path for gradients to flow back to deep layers. They work by providing a direct linear path from one deep layer to another closer to the output, thereby bypassing the sequence of non-linearities and repeated matrix multiplications that can lead to instabilities. We applied these techniques to language modeling, but they apply generally to all neural networks. In the next post we will further improve the performance of our model by leveraging recurrence.

This post is Part 1 of an n-part series on language modeling. Language modeling has a diverse range of problems within it, such as translation, semantic analysis, and prediction. For this series we will be looking at next token prediction. Specifically, given a sequence of tokens (like characters or words), we want to create a model which predicts the next token. Mathematically this amounts to finding a model which maximizes the likelihood of the next token conditioned on the previous tokens.

In this series, we are going to start from simple models and work our way up to the modern state of the art like transformers and also some more exotic architectures like text diffusion models.

For more background on these topics, I highly recommend Andrej Karpathy’s Zero to Hero series on YouTube. His lectures were motivation for this series. For mathematical background, I recommend Mathematics of Machine Learning. For detailed explanations of sequence modeling with practical examples, I recommend Dive Into Deep Learning.

Neural Probabilistic Language Model

Some of the first attempts of language prediction used frequency-based n-gram models. With n-gram models, the predictions are based on the frequency of n-grams in the training corpus. The work from Bengio et al. improved upon the existing state of the art n-gram models way back in 2003 with their “Neural Probabilistic Language Model”. They were one of the first to successfully apply neural networks to the language prediction task. The main contribution from this paper was a solution to the curse of dimensionality problem via learned embeddings.

To understand the curse of dimensionality, consider the following sentences:

The dog is running in the yard

The cat is drinking in the kitchen

The man is running in the street

If only the first two are in the training set, then a classical n-gram approach, where the probabilities are assigned based on the frequency of their occurrence in the training data, will fail to predict the last sentence as very likely, since it never appeared in training. This seems wrong, since all three sentences are grammatically and semantically similar.

The root cause of this is the set of possible sentences is an extremely high dimensional space (and hence is cursed :)), and the n-gram approach has no mechanism to transfer probability mass found during training to unseen sentences.

Overcoming the Curse of Dimensionality

Bengio found a way around this using learned word embeddings. Embeddings are real-valued vectors that are used to numerically represent words. The vectors encode the syntax and semantics of each word. What is neat about this encoding is that it allows us to perform arithmetic on words, and to measure mathematically how close they are to each other via the geometric interpretation of vectors. So words with similar meaning like “mammal”, “dog”, “human” will have similar vectors and words like “it”, “the”, “a”, “and” will have similar vectors, however those two groups of vectors would not be close to each other in the embedding space.

As a simple example, we can suppose the embedding vector is three dimensional as seen below:

Each word would have a vector of its own, for example mammal might be (3.6, -2.4, 0.1) and dog might be (3.7, -2.3, 0.5). We can then compute the Euclidean distance between them. Semantically similar words will have a distance near zero.

So you may be thinking, OK then we just have to somehow come up with the “most important” features of a word, like part of speech and its semantic usage and create vectors that encode these that we come up with. But Bengio found that we can let the model learn what “features” are important based on the statistics of the training data. We don’t have to explicitly say that the first element of each vector represents part of speech, the second whether it represents something inanimate, etc. The model learns these values for us via backpropagation.

The reason this technique overcomes the curse of dimensionality is that predicting the next word given a sequence of words is likely to pick any word with a vector which is similar to that seen in the training set, not just the word that was seen in the training set. So predicting the next word of

The man was hungry so he ate the ____.

would assign a high likelihood to any type of word that semantically describes food, even if the only instance of the phrase in training was e.g. “The man was hungry so he ate the banana”. This is what it means to “transfer” probability mass to similar words. This transfer allows for an exponential increase in the sequences which the model finds likely, leading to better generalization.

Visualizing Embeddings with t-SNE

Unfortunately, humans can’t visualize spaces with a high number of dimensions, given our senses have evolved in three dimensional space. This makes gaining intuition of high-dimensional objects like embedding vectors difficult.

Fortunately, there are algorithms such as t-SNE for projecting high-dimension vectors into two dimensions. This allows us some sense of the clustering that is present among the embedding vectors and gives us intuition for the similarity between the words.

You can see an example of this below that uses the TinyStories dataset. It uses t-SNE to project the 32-dimensional embeddings of lower case letters to two dimensions.

The vowels are in red and consonants in blue. The image above is the t-SNE projection from the random initialization of the embedding vector.

The image above is the projection after 30,000 training iterations of a character-level language model that uses Bengio-style embeddings for each character. You can see as the training progresses, the embeddings of semantically similar characters like vowels begin to cluster together.

Building the Model

Let’s implement a model similar to the one from the Bengio paper. We will use the TinyStories dataset. This dataset features a collection of short stories generated from ChatGPT. We will train a character-level language model using PyTorch that will be able to generate new short stories for us.

Note that below will just have snippets; here is a colab notebook if you want to have a complete picture.

Prepping the Dataset

The first step is to get familiar with the data. We need to download the dataset and convert it into a format that we can use for training:

I validated experimentally that ‘^’ isn’t in the training set at all, so I opted to use it as a special starting character as we will see later. Now we can create our char-to-index/index-to-char mappings as well as a dataset creation function. This function build_dataset creates sequences of character indices, each with length equal to ctx_window.

Defining Layers

Now that we have our dataset wrangled, we can think about adding the layers needed for our network. Our initial network will use the following forward pass:

1. Embedding. This maps input tokens to their corresponding embedding vectors.

2. Flatten. This is a simple reshape that concatenates the embedding vectors column-wise.

3. Linear. This applies an affine transformation to the input.

4. Tanh. This is our activation function. Performs a pointwise tanh to each value from the preceding linear layer.

5. Linear. This applies a final linear transformation to the tanh activations.

The output from the final Linear layer will then be run through F.cross_entropy for the loss. This implementation is pretty close to what is in the Bengio paper. In later posts we add some bells and whistles to incrementally improve the performance. Here is the code for each layer:

Building the Training Loop

Now that we have our layers, we can build the training loop. The loop will use a simple stochastic gradient descent with a fixed learning rate. In later posts we will add in optimizers like Adam for increased performance.

First we will define some hyperparameters of the training loop:

I chose these mostly arbitrarily. A ctx_window of 8 means the model has a fixed “history” of eight characters, and will be predicting the next character based on these 8. The vocab size (i.e. number of characters) for this dataset is 175 including the special character, so an embedding dimension of 32 seemed reasonable.

Next we can define our model, initialize the parameters, and create our training dataset:

Finally we can implement the full forward and backward pass:

Measuring Performance

When we run this, it results in the following loss curve:

We can see the initial loss is pretty high. Since we are doing softmax classification, we should see an initial loss of around ln(num_classes) = ln(175) = 5.16. We can address this in a later post with proper initialization. The final loss is a little over 5.

The key performance metric for this task is perplexity. Intuitively, perplexity is how surprised the model is when faced with a given character, conditioned on the distribution it has learned from the training data. Higher perplexity implies higher confusion. Mathematically, perplexity is the exponential of the cross-entropy. The cross-entropy in turn is the expected surprisal. Here is some code for calculating perplexity:

This simple model we’ve built gives a perplexity of 434. In the follow-up posts we will see how we can get this number down with architectural improvements.

Finally, lets look at the quality of the story that is generated by our model. For this we will sample from the model one character at a time according to the learned distribution:

Running this gives the following:

Story time: Once upon a timk.e an ter pere eo hire sores the caed va boanshThr witlin. HtA. ThengerDpg, and ker ditgy und g nit, tayeur Vag anddbTÉdbvjogd isswarp!e wow,e. ouancs.”Tneyd-4%un6¸¤ŒÂ·¯ } Iyž+‡+›´¢¿D»ájfÉŽ°éG­™yz›1ŒÂš¯»{U9¬#³’} %>²)¸‘¬#œj;Êq>‘æɵLbæäc®è.cŽ39°zc·dxnomd.ƒ>o¦t.mTe suŒlmvcyI¢”D᜖jœ³;¿äXécv™¦Rƒ¸2’F‹ @›”ƒÃ›6±zš<°bÉ;®®Ò`0 ?.Ä#2»áB”·”â2´F¹…¥®@12§9\>ˆ§£V}å4¹€FéQ}¦©¡¨±¼¯†Ã))`¼É\Rzä¡\¬#;³YŸ°vVLâ%Ä<Z泯éO‚ÃMž+`[”æCâj,CÑSŠ\,¹ ]O⬘<!ææÒ¯Y案9Êï4g$½?ÄbïÉ?oBH˜ä± ;ãR>@)ƒ‰ˆ=Xð¥¹P,?0=>Žð:”QW°JFxQ(3\h„ŠðÉ)X˜´QDµxj».¢É?š¬ªRc³ŠïŠ¬­qU¢E¹¢œR0‰2Ÿð:Ž+Å4¡º^

Started off pretty good! Then fell completely off the tracks.

The notion that hacking involves illicit activity with a computer is wrong. Or rather, it is incomplete. Our culture has hijacked this term and given it this negative connotation.

So what does it actually mean?

To answer this, we have to go back to the early 1970s. During this time, the creators and users of the early Internet at various universities created a file that contained certain terms related to the systems they were working on. This file became known as the Jargon File. The Jargon File contains many terms commonly used in hacker subcultures, and some such as “bug” and “troll” have had widespread adoption by society today. The Jargon File offers several definitions of hacker:

hacker: (n) A person who enjoys exploring the details of programmable systems and how to stretch their capabilities, as opposed to most users, who prefer to learn only the minimum necessary. RFC1392, the Internet Users’ Glossary, usefully amplifies this as: A person who delights in having an intimate understanding of the internal workings of a system, computers and computer networks in particular.

…

An expert or enthusiast of any kind. One might be an astronomy hacker, for example.

…

One who enjoys the intellectual challenge of creatively overcoming or circumventing limitations.

…

[deprecated] A malicious meddler who tries to discover senstive information by poking around. Hence password hacker, network hacker. The correct term for this sense is cracker.

This definition does a much better job of capturing the true essence of hacking. It underscores the essential traits of hacking: deep understanding, competence, curiosity, and creativity. A hacker is someone who comes to grok a system so deeply, they can discover novel ways of (ab)using it. They are especially skilled at combining systems in unexpected ways, beyond the original design of either system, to overcome the boundaries and limitations of either system.

The only thing I would add to the definition above is that hacking applies to any complex system, not just computer systems. And in this sense I agree with society’s generalization of the word “hack” to domains outside of computers. There are biohackers, gym hackers, growth hackers, food hackers, work hackers, life hackers, etc. Hacking is a mindset which can be applied to any domain of life in which there are observable inputs and outputs. Hackers try a bunch of inputs and observe how the system behaves. Even if the internals of the system are a complete black box, you can learn a great deal from just observing what inputs lead to what outputs.

The difference between a hacker and non-hacker is that the hacker

1. becomes consciously aware of the inputs to the system

2. intentionally finds and feeds unconventional inputs to it.

Cold showers are a good example of this. Cold showers were an unconventional input to the standard morning routine. Then some influential biohackers tried them out and found they provide many health benefits that you would miss out on if you stuck to the conventional input of a warm shower. Sometimes the unconventional input is no input at all. This is colloquially referred to as hacking the test.

Hacking the test involves pruning the inputs that involve extra effort, because you’ve figured out that the system doesn’t reward you for them. Or that you can still achieve a favorable outcome without them. This is what business hacker Charlie Munger meant by “show me the incentives and I’ll show you the outcome”. The incentives are the feedback from the output to the input. When someone says they have no incentive to do something, they’ve figured out that they can safely prune that something from the input with no detrimental effect to the outcome. AI systems do this as well, especially in reinforcement learning. This “reward hacking” involves finding clever bypasses around the rules of their environment. These workarounds are examples of another characteristic of hacking. Hackers play with the rules rather than within the rules of the system. These examples show that hacking itself is similar to a technology, a tool that can be used for good or bad.

Since hacking is a mindset, a way of questioning the world, it can be learned. It is a skill that can be devleoped. It is a skill that I intend to keep developing in myself and to help others develop it as well.

Lately I’ve been developing a mushroom growing app. This is an open-source project that enables citizen science for mushroom growing. For a while now, I have wanted a tool that allows me to keep track of my grows and to run experiments with various controllable parameters. I’ve also wanted to be able to learn from others’ techniques more effectively. There are plenty of forums online for sharing, however it is difficult to rigorously repeat what others have done, as the sharing tends to happen in unstructured natural language and often lacks critical details needed for repeatability.

The app has three conceptual sections: Grows, Teks, and IoT Gateways. The Grows section provides grow management. You can add/remove grows, schedule tasks, view grow progress and key characterstics like health status, cost, and yield. The app focuses on bulk grows for the initial version. Each grow has 5 stages. The user can add items, environmental conditions, tasks, and notes to each stage.

Teks (”tek” is short for “technique”; commonly found in online mushroom forums) are similar to grows, but they are a more “abstract” version, similar to a recipe vs doing the actual cooking. Teks are portable instructions that can be shared and instantiated into actual grows.

IoT Gateways are connections to Internet-of-Things (IoT) sensor hubs. Home Assistant is the one supported in the initial version. Home Assistant is nice because it is a widely used free and open-source home automation platform and is one I’m fairly familiar with. You can connect many different sensors, especially those relevant to providing optimal mushroom growing conditions (temperature, humidity, carbon dioxide, pH, light, etc). You can also write custom automations, so that you can react to the environment changing. For example, if your spawn incubator temperature falls too low, it can switch on a heat mat to bring the temp back into optimal range. Home Assistant also supports cameras and embedded controllers such as ESPHome, and it is extendable so support can be added for just about anything if it doesn’t exist already. The IoT Gateway connects to the Home Assistant API and receives real-time updates over WebSockets. All the controls, sensors, and automations are accessible from within the app. They can be linked to specific grow stages for different tasks, like enforcing environmental conditions (e.g. temperature and humidity range) during spawn colonization, or detecting anomalies in sensor values (e.g. discoloration, pH drops) during bulk colonization. This Bring-Your-Own-Home-Assistant design provides a platform for endless customization and experimentation.

These three sections will be present in the first version of the app, as it is already getting pretty complex. Later I would like to add “Lab” section that provides first-class support for running experiments. You should be able to have a grow, and tweak some aspect of it, say a different substrate material, or a slightly higher temperature range during bulk colonization, or more light. The space of potential experimental inputs is massive here, so providing a clean “diffing” mechanism that allows the user to easily compare and contrast the control vs experimental variables and track how they effect the outputs like yield is what I’m thinking of.

---

Lately I’ve been thinking about work, my relationship to it, and what it means to do great work. Work is important to me, and I’ve learned over the past 2 years or so that doing great work is really important to my wellbeing. I feel like shit when I know inside that I haven’t lived up to my potential. I’ve read a lot books lately related to work, some of which I’ve written about and others I haven’t (yet), that have helped me reframe what it means to work and work well. Paul Graham’s take on work has been especially influential on me.

Strangely, work tends to get a bad rap in today’s society. Ambition has become somewhat of a dirty word. But I think a lot of this is because people conflate their job (or, if they are young, school) with work. And if your job sucks, or you’re learning things in school that you have zero interest in, then naturally you’re going to dread going to “work”. You are going to resort to language such as I have to go to work tomorrow, rather than I get to go to work tomorrow. The underlying metaphor is that WORK IS A NECESSARY EVIL. Something to put up with. Something that society forces us into. Lately I’ve come to change that metaphor. For me, WORK IS AN OPPORTUNITY FOR SPIRITUAL GROWTH. It is something that I am blessed to be able to do. Work to me means learning, creating great things, and sharing what I’ve learned and built with others. Great work probably look different for you, but it is always guided by our natural curiosity and comes from our internal drive. And we have an obligation to ourselves and to each other to ensure that our curiosity and interests are fully expressed in the world. We must share our work, because each piece plays a small part of a much larger creative force that works in ways that we do not fully understand. You never know how your work could impact a person’s life.

The trick is to maintain balance between your work and the other important things in life. I have found that viewing your life’s work as an infinite game helps in maintaining balance. The alternative is that work is a finite game, which creates a frantic sense of urgency and anxiety. Work is important, but it shouldn’t interfere with sleep, relationships, diet, and exercise. The goal of any infinite game is to keep playing, and you have to take care of yourself in order to that.

My aim is to show that both of these phenomena, while very different on the surface, are inextricable nth order effects of the same root cause. The root cause is something that the majority of Americans either take for granted or are completely ignorant to: the US dollar’s status as the global reserve currency and the debt-based fiat monetary system on which this status depends. An nth order effect is, simply put, an indirect consequence of some event. The set of these effects for a given event is constructed by repeatedly asking the question “And then what?”. If I eat candy every day, the first order effect is a rush of pleasure and dopamine. The second order effect is stomach distress and low energy. A third order effect is diabetes. Many people sometimes fail to look beyond the first order effects of decisions they make and of events they become aware of. This leads to a shallow understanding of events that occur within complex systems and an under-appreciation of potential future outcomes.

The Trump administration wants to balance the United States’ long-standing trade deficit. To achieve this policy goal, Trump is levying tariffs on trade partners in order to encourage the re-shoring and revitalization of the US industrial base. However, in my view, tariffs alone will be ineffective in balancing trade in the long run, because the trade deficit is an nth order effect of the true root cause: the US dollar status as global reserve currency and the debt-based fiat monetary system. Tariffs do nothing to address this root cause, and in fact, Trump has emphasized the administration’s official policy is to preserve the US dollar’s reserve currency status. However, just because tariffs alone may be ineffective, doesn’t necessarily mean the underlying policy objective shouldn’t be pursued.

The aim of this post is not a political one, but rather to analyze the tariff policy within the reality of the economic system it is being deployed in so that we can make informed judgments of its efficacy and worth. To this end, we will start with a brief history of the modern dollar system and the elevation of the dollar to global reserve status. Then we will see how this system is linked to persistent US trade deficits. Once we understand this financial plumbing, we can make an informed judgement on whether tariffs alone are likely to be effective. In order to judge whether the underlying policy (i.e. balanced trade) is worthwhile, we will look to various other consequences of the current system and the impact it has had on the American public, including the recent wave of political populism and general feeling that the American dream is out of reach for many people. With these effects correctly attributed to the root cause, which transcends political divisions, we can develop a more nuanced view of the underlying goal of balanced trade, even if we may disagree with the means to achieve it.

A quick aside: much of my view has been influenced by Lyn Alden and her book Broken Money, as well as work from macroeconomists Luke Gromen, John Mauldin, @infraa\_. I recommend checking out their stuff for a further deep dive.

The Rise of King Dollar

At the peak of World War II, Allied financiers and central bankers convened in Bretton Woods, New Hampshire to create a new global monetary order to be enacted once the war was over. At this point in July of 1944, the United States had emerged as the country best positioned for imposing a new international financial system on the rest of the world. The US had built up an immense manufacturing capacity and the American homeland had avoided the devastating damage that plagued Europe, Asia, and the Soviet Union.

The so-called “Bretton Woods system” would confer the lofty status of global reserve currency to the United States dollar. Under Bretton Woods, member nations agreed to maintain a fixed exchange rate against the dollar, while the dollar itself was redeemable for gold at $35 per ounce. The intent of this system was to facilitate free international trade using the dollar as the common medium of exchange.

There was a problem though. In order for nations to settle trade in dollars with each other, they needed dollars. But dollars only come from the United States. Foreign nations and companies cannot create more dollars themselves, and Europe, Asia, and the Soviet Union had very little export capacity relative to the United States due to the physical destruction from the war, i.e., they had very little to sell in exchange for dollars.

At the very beginning of Bretton Woods, the United States had what is called a balance of payments surplus. You can think of a country’s balance of payments as a record of all the value flowing into and out of the country. A surplus means that more value is flowing in than is flowing out. The balance of payments includes things like the trade balance and the financial account. The US was running huge trade surpluses and had amassed sizeable dollar reserves. This surplus had to be reversed in order to supply the rest of the world with dollars that it needed. It did this initially through grants and loans, since the European export capacity was severely hamstrung at the time.

This deliberate outflow of dollars was bootstrapped by massive aid packages such as the Marshall Plan, during which the US gave Western European countries approximately $17 billion to jumpstart re-industrialization. About 90% of the Marshall Plan aid was given in the form of grants rather than loans, for which Europe didn’t have to pay the United States back. This injection of dollars helped spur European industrial capacity, and by the end of the Marshall Plan in 1952, European exports had doubled, and industrial output had increased 55%.

These persistent balance of payments deficits, i.e., export of dollars, and the impetus to use dollars for global trade by non-US countries, created what is called the Eurodollar market. Despite the name, the Eurodollar market is the market for dollars between any nation that is not the US, not just in Europe. The Eurodollar market experienced dramatic growth during the 1960s. It grew from $75 billion to $264 billion (in 2020 dollars) between 1964 to 1969. Keep in mind that each dollar deposit in a foreign country ultimately represents a liability of the United States, since every dollar was redeemable for gold by foreign governments and their central banks.

As seen above, eventually the level of external dollar liabilities exceeded the value of the US gold supply. This coincided with large inflationary deficit spending by the United States in the late 1960s in order to fund the Vietnam War and President Johnson’s Great Society. The deficit increased 5x during this period, from $5.9 billion in 1964 to $25.2 billion in 1968.

There are a couple of reasons why the foreign (Eurodollar) and domestic liabilities were able to grow so quickly under Bretton Woods.

The first dates back to three decades earlier, when FDR declared an emergency bank “holiday” and signed a series of executive orders during the 1933 banking crisis. Executive Order 6102 made ownership of over $100 worth of gold in the form of currency illegal for US citizens. Executive Order 6073 ended the domestic convertibility of dollars for gold at US banks. Later that year in June, Congress revoked the “gold clause”, which nullified any obligations for settling private and federal contracts in gold. Finally, in 1934, the Gold Reserve Act enabled the President to perform a de jure devaluation of the dollar by adjusting the gold peg from $20.67 to $35 per ounce. These policies amounted to confiscation of gold from the American public into the Federal Reserve system and marked the end of the gold standard domestically.

At the same time gold flowed into the Fed from private citizens in the 1930s, the required gold reserve ratio decreased throughout the Bretton Woods period. From the establishment of the Fed in 1913 through 1945, the Fed and its member banks were legally required to maintain a 40% gold reserve. This was reduced to 25% in 1945, and to 0% in 1968. The net effect was the US banking system was no longer constrained by their gold levels and could create loans based on fiat dollar reserves rather than physical gold reserves. This leads us to the second reason for the huge expansion of dollar liabilities under Bretton Woods: fractional reserve lending.

Fractional reserve lending allows banks to create new money in the form of loans while only keeping a small percentage of the initial deposit as physical reserves1. Assuming all banks have a reserve ratio of 10%, if you deposit $100 into the bank, the bank will loan out $90 of that deposit to a borrower while earning the spread between the interest it pays you and the interest is charges the borrower. That $90 then gets loaned again by another bank with a 10% reserve ratio, resulting in $81 for the new loan. This process tends to continue via the profit motive for banks to earn the interest spread between deposits and loans. If you do the math, the amount of new money created from this process (aka broad money), works out to be equal to: initial deposit * (1 / reserve ratio). So a $100 deposit with a 10% reserve ratio results in $1000 of new broad money created on top of the $100 base money2. This is why base money, like actual physical currency and checking accounts, is referred to as “high powered money”; it has a multiplicative effect on the overall supply of broad money in the system.

So you can see how the end of domestic gold convertibility combined with fractional reserve lending lead to enormous sums of new broad money creation. This money creation process accelerated under Bretton Woods as the United States began exporting dollars to the rest of the world, and the Eurodollar market began to metastasize. These offshore US dollars were also expanded using fractional reserves as industrial capacity came online in Europe.

Eventually Western European nations sniffed out the unsustainability of this system. There were simply way to many dollars and not enough gold in the US. The suspension of the gold standard in the 1930s only applied to citizens and banks within the country. Foreign governments and their central banks were still able to redeem their dollars for gold. So that is exactly what they did. Nations like France and Germany began draining gold from the US in order to evade the monetary debasement that was occurring in the dollar:

This run on the United States’ gold stock was so acute in the late 1960s that it prompted the so-called “Nixon shock”. In 1971, President Nixon announced that the US would no longer allow foreign governments to redeem dollars for gold, ending the gold standard completely. The Nixon shock was effectively a default by the United States and marked the end of the Bretton Woods regime.

The Dollar and Trade(offs)

The collapse of Bretton Woods was predicted in 1959 by an economist named Robert Triffin. Triffin identified a contradiction, called Triffin’s dilemma, inherent to the system: in order for the rest of the world to use the dollar in international trade, foreign countries must have dollars. These countries can’t print dollars, so they have to get the dollars through continuous US trade deficits (or more generally, balance of payments deficits, which includes things like foreign aid as in the Marshall Plan). Trade deficits provide foreign countries with the surplus dollars needed to settle trade.

Now the problem isn’t really with the occasional trade deficit. In a normal scenario, the excess supply of the currency from the deficit will put downward valuation pressure on it relative to other currencies. This devaluation causes the country’s exports to become more competitive on a relative basis. As exports become more competitive, the trade deficit turns into a balance or even a surplus.

However, when the country (US) maintains the world reserve currency (dollar), this natural equilibrium is never reached. The excess supply is soaked up by the global demand, keeping the relative value of the dollar propped up. The key word here is relative. On a relative basis, the dollar is strong, however in absolute real terms, the dollar gets much weaker over time. The one-two punch of abandoning the gold standard and the proliferation of debt through fractional reserve lending leads to massive loss of purchasing power for holders of US dollars over the course of decades. This weakness, as starkly demonstrated in the graph below, undermines the collective belief that the dollar is worthy as being the global reserve currency.

Triffin’s dilemma captures the dynamic between the global reserve currency and trade deficits that I think Trump doesn’t fully understand or at least doesn’t appreciate. The deficits are a consequence of the US dollar’s global reserve currency status. They don’t just happen randomly or because of mercantilist policies from China. Sure China’s policies such as the intentional devaluation of the yuan may exacerbate the deficit, but these effects are marginal relative to the structural flows that result from the dollar reserve status.

Any meaningful tariff increases have little staying power, because they do nothing to address the root cause of the deficits in the first place. The bottom line is that dollars must continue to flow to the rest of the world. The reason is that there have been trillions of dollars worth of debt that has accumulated outside the US over the past several decades, where neither the creditor nor the debtor are US-based entities. These so-called “offshore” debts are between foreign corporations, households, banks, and governments. The charts below from the Atlanta Fed show the magnitude of these liabilities to be in the tens of trillions:

Except for the roughly $3T in the bottom left graph, all of these liabilities are held outside the United States. The interest payments on these debts create inelastic demand for US dollars. When trade is free-flowing, the dollar income can be used to make the interest payments, no problem. But if trade income falls due to massive tariffs (or other shocks like COVID-era supply chain disruptions), dollars have to be sourced from somewhere else.

Where do the dollars come from in the absence of trade? Liquid US assets, mainly stocks and bonds3. When countries run a trade surplus with the US, they reinvest much of their dollar profits back into the shares of US companies and US government debt. The degree of this foreign ownership of US assets is captured in the United States’ net international investment position (NIIP). A negative NIIP means that foreign countries own more American assets than America owns of foreign assets. We can also look directly at foreign investment in US equity and debt markets. The graphs below illustrate growth of this foreign ownership of US assets over the past few decades:

So whenever countries experience a dollar shortage due to severely reduced trade, they can sell their US assets in order to raise cash. This cash can then be used to service their dollar denominated debts. This is one of the reasons why US stock and bond markets sold off after Trump’s Liberation day in April. Typically US Treasuries are considered safe-haven assets, resulting in a bid (yields going down) whenever there is risk-off sentiment in the market. But yields didn’t go down after Liberation Day. The went up. Which means bonds sold off hard.

The bond market is another reason why meaningful tariffs have no real staying power. Foreigners own upwards of $9T worth of our debt which they can and will sell if they are forced to raise dollars to meet their interest obligations4. They can also sell Treasuries to create upward pressure on interest rates, which can have serious consequences for the global monetary system. It is not a coincidence that the Trump administration quickly walked back from the reciprocal tariffs and the 145% China tariff after the sharp bond market sell-off. Bessent made it clear, they don’t really give a shit about stocks. But they absolutely care about the bond market.

The US Treasury market is the foundation of the global financial system, so any prolonged disorderliness in that market has catastrophic implications. Think of it this way. The entire dollar system is like a massive game of musical chairs, with 10,000 people to every one chair. The ever-increasing debt that flows from the Treasury market is the music. If the music ever stops, the game is over because the system is completely insolvent. The thing is, the rest of the world knows this. For this reason, their ownership of US debt offers them immense leverage at the tariff negotiating table, much more than the Trump administration is probably willing to admit publicly.

My view is that tariffs are more likely being used as negotiating leverage rather than sustainable long-term policy. I think when the dust settles we will see some tariffs on the margin for specific strategic goods (steel, aluminum) and industries (artificial intelligence, semiconductors, pharmaceuticals). Along with some sort of “Mar-a-Lago Accord” that results in a globally coordinated weakening of the dollar relative to its peers, similar to the Plaza Accord in the 1980s. Steve Miran, Trump’s chair of the Council of Economic Advisors, proposed this as a possibility in his recent paper A User’s Guide to Restructuring the Global Trading System. That paper is long but worth it if you want deeper insight into the ideas behind the administration’s policy. To that end, the dollar has already lost about 10% since Inauguration Day. This is positive for the Trump administration’s goals, as it makes US exports more competitive.

Share

A Damn Shame

Despite my view that broad reciprocal tariffs are basically not possible for any prolonged period of time due to the structural demand for dollars, I do think the underlying goal of restoring relative prosperity to the American middle class is vitally important. The current system has left many Americans disillusioned with the social contract they were raised to believe, which is that hard work pays off.

In the last 50 years, the United States has exported dollars and its manufacturing base overseas. As discussed above, this has created a strong network effect of dollar-based trade, leading to a persistently strong dollar and massive inflows of capital back into US debt and equity markets. The question is whether this is good for America or not. The answer is it depends on which American you ask. If you ask someone that owns stocks or real estate, they are probably OK with the status quo. According to the latest FRED data from the St. Louis Fed, the top 1% wealthiest Americans own 34% of all financial assets, whereas the 50th wealth percentile owns just 2.6%. The majority of Americans that live purely off of wages have missed out on the enormous nominal wealth that has accrued to asset owners.

The sense of falling behind is reflected in the actual data for major life milestones such as buying a home and having kids. The following chart from Apollo Chief Economist Torsten Slok shows the median age of all homebuyers is at an all time high of 56 years old:

This tracks with the increase in the median sales price of houses sold over a similar period:

Have you ever wondered why housing prices have increased so much in price? A 3-bed 2-bath today is not that physically different from what it was 40 years ago, let alone 5 years ago. Yet home prices have increased 33% since COVID. One reason is that homes these days are used more than just for shelter. They are investments and a means to store value, i.e., they carry a “monetary premium” on top of their value for shelter. Most mortgages these days are bundled up into a fixed income instrument called a mortgage-backed security (MBS). Each MBS is sold by Wall Street to various institutional investment firms like pensions and insurance companies. These are exactly the same financial instrument that caused the Global Financial Crisis in 2008. In the time since that crisis, the Federal Reserve has printed trillions of dollars to directly buy MBS as part of their quantitative easing programs, in effect subsidizing the mortgage market with money created out of thin air. This “financialization” of the housing market has kept rates low and allowed for large institutional buyers and private equity firms like Blackstone to acquire massive portfolios of single family homes, pricing out the average American family.

People are having fewer kids as well. The birth rate in the US has fallen from over three births per woman in the 1960s to around 1.6 births per woman today. Statista recently reported a new survey that showed the most common reason cited for not having kids was financial limitations:

This game of hyper-financialization has been playing out for decades now. You can either choose to play or not. If you don’t, you are aren’t going to keep up in nominal (monetary) wealth terms unless you have an inheritance. If you do decide to play, you have to invest time and money just to keep up with inflation, let alone to grow your wealth in real terms. The system of endless debt turns every dollar into an ice cube on hot pavement. Or, in the words of Oliver Anthony, “your dollar ain’t shit”. Case in point, the dollar has lost 25% of its value since 2020 due to the massive fiscal and monetary stimulus response to COVID.

What is the impact of this financial game that is so clearly rigged against the majority of the American public?

Show me the incentives and I’ll show you the outcome.

- Charlie Munger

Populist songs like Rich Men North of Richmond. OnlyFans. Assassination of elites. The Red Wave. Online gambling and 0-DTE options. When people struggle to afford basic necessities like food, the social fabric starts to unwind5. Violent protests and riots start to become every day news. Many people blame politics (specifically the sitting president) for these things, and I agree that they play an important role. But the problem with our money is bipartisan; the graphs above span decades and multiple administrations, both Democrat and Republican. Tariffs may have some effect around the margin to bring specific industry back to American soil, but they will do nothing to reverse the profligate debt expansion that the current system depends on and the devastating consequences this expansion has on the value of the dollar and the American people.

What is the solution, if not tariffs? It’s a really tough question; one that I don’t have a full answer to. I think any solution has to address the root cause: the dollar as global reserve currency and the fiat financial system we have today. Replacing the dollar for a neutral reserve currency would allow for the US trade balance to reach a natural equilibrium. This transition back to a net-positive or net-zero trade regime would be painful initially, as it would require the US to consume less than it produces for some time. This would require a massive cultural shift in behavior, values, and norms, but it is possible to do. If we were to replace the debt-based system with one based on credit and hard money (i.e. money that cannot be debased), it would flip the underlying price dynamic from an inflationary one to a deflationary one. Imagine the price of eggs falling each year rather than rising. Mainstream economic thought would say this situation is bad because if people expect prices to fall, they will tend to hoard the currency rather than spend it, causing the economy to grind to a halt6. I agree that massive deflation probably is not good, but why not 1% per year? It would align with the naturally deflationary force of technology, which is only going to accelerate in the coming years due to AI and robotics. Call me crazy, but gradually falling prices and robots doing my laundry and getting me groceries sounds like peak human civilization to me.

And you actually succeed in cutting back, only to discover that your brain sneakily found some *other* thing to do compulsively instead. Something that otherwise you rarely do steps in to replace the original bad habit.

I call this the dopamine whack-a-mole.

It’s a game that I find myself playing quite a bit. One thing that helps me with it is being aware of when a new habitual distraction pops up. Then at least I can label it and bring it into conscious awareness. This gives me greater ability to pause the next time I reach for the new distraction.

The other thing that I’ve found to be helpful is to start each day with no phone, and instead give my attention to two things that I want to do or work on. I keep my phone in another room at night. Only after doing two things, whatever they may be, I can go on my phone if I need to. On the best days, I forget about my phone completely and get loads done. This is especially true whenever I’m trying to create something, as I’ve written about before. Keeping the phone away helps create space for new ideas.

Have you ever experienced this dopamine whack-a-mole? What are your favorite moles? How do you overcome them?

The book opens by dispelling the narrative that art is restricted to what we traditionally consider to be art, such as making music or painting. Rather, it argues that creativity is a universal property of human nature. Creativity is certainly expressed through paint and song, but it just as well could be expressed through a home-cooked meal, a gift, or a new business. Creativity is the act of bringing something into existence that wasn’t there before. Even if we don’t do anything at all and just be, we are still creating our version of reality through our unique filter of sensory experience. Being creative is more than an activity. It is a fundamental aspect of the human spirit. Just as birds fly and fish swim, humans create.

With this spiritual foundation in place, the book proceeds to examine how to cultivate a creative way of being in our lives.

Tuning in is a critical component for cultivating creativity. The universe is buzzing with energetic potential that searches for a path through which it can be expressed. Humans are conduits that facilitate the manifestation of this potential into the physical world. We may think that ideas are our own, but they are really downloads from the larger creative force that surrounds us; initial seeds that have sprouted into our consciousness. This is why it is not uncommon for an idea that you have to be expressed by someone else if you choose not to bring it to life. The other person didn’t steal your idea; the idea’s time has come. The idea re-routed around you and found another path to manifestation.

How can we better tune in? Two ways: leveling up our taste and creating space.

We need to be selective of what information we let into our lives. If our bodies are the food we ingest, our minds are the information we consume. Rather than doom-scroll social media, read a book. Better yet, read classic literature. And don’t just read to read, read to understand and to form critical opinions.

To let creative inspiration in, we need to create space for it. There are two high level ways to do this, both involving filters. The first is to use a highly restrictive external filter. The second is to use a minimally restrictive internal filter, or perhaps even no internal filter at all.

Our external filter controls the information that we receive from the outside world. Our high volume, low quality information culture fills our stream of consciousness with low fidelity noise that drowns out any signal that would otherwise be picked up by our creative antennae. Filtering out sources of external noise allows for creative signal to be received. Sometimes the best option is to just sit in silent contemplation. Or dedicate time each day with no phone or any other external sources of information. This creates a vacuum in our minds which can be filled with creative insight. Discipline helps here, but I’ve personally found that grounding the restrictions against a set of values or higher purpose is more effective and motivating, especially at the beginning when the siren song of the external digital world is so loud.

Our internal filter controls the information that we transmit into the outside world. This filter may be overly restrictive due to fear or ego or by the self-constructed narratives we tell ourselves. As adults, we carry with us narratives and labels that situate our self in relation to others and society. The stories we tell about ourselves can act as a filter that restricts our creative potential. When our identity is based on a preconceived set of labels, it makes it more difficult it to act in ways that contradict that identity. We create roles for our self. These roles implicitly confine us within the boundary of finite games. Roles force us to play within the rules rather than play with the rules. How do we go about deconstructing our narratives and labels? The book recommends adopting a more childlike attitude. Children just do things. They live in the moment without worrying about the future or trying to conform to some identity.

Being more childlike as an adult is much easier said than done. Unfortunately, the book doesn’t really expand on how adults can be more childlike in practice, and in fact recommends not to over-analyze or force this state. However, in the spirit of playing with the rules rather than within them, I want to offer an idea for doing this. You need to seek out activities that suppress your default mode network (DMN). This brain region is widely believed to be the physical “location” of the self, and incidentally, is not yet fully developed in the brains of children. Suppressing this region in adults achieves a childlike state of mind via increased communication between different brain regions that don’t normally talk to one another. The reason they don’t normally talk to one another is because the DMN acts as a reducing valve that prunes connections between different brain regions. You can think of the DMN as the stern teacher that walks into a classroom. Before the teacher is in the room, there are conversations everywhere, even between people on opposite sides of the room. But as soon as the teacher walks in, the room goes silent and the more predictable patterns of conversations start to emerge (e.g., role call). How do you suppress the DMN? Meditation, psychedelics, and perhaps(?) prolonged aerobic exercise.

In addition to looking at ways to tune in and what that means for creativity, the book implicitly offers a set of metaphors that provide psychological structure to the creative act that really resonated with me.

Share

CREATIVITY IS ABUNDANT OPPORTUNITY

Creativity is about tapping into the flow of ideas and manifesting them in the world. When you tune in to this flow and surrender yourself to it, it never runs dry. Ideas are constantly flowing. On the other hand, if we view creativity or ideas as being a competition over scarce resources, we hold onto our work, keep it in our heads, and are less prepared to share it with the world, because we think that one work will define us for the rest of our lives. A symptom of a scarcity mindset is perfectionism, which is based in fear. Fear of looking weird or bad, or otherwise not living up to the expectations that we or other people have for us. This is one form of our internal filter that we have to learn to let go of. On the other hand, an abundant mindset allows us to see each work as just one small piece within a much larger work of our lives. When we create with an abundant mindset, we are more free to start, finish, and share our work so that we may let it go and move on to the next piece.

CREATIVITY IS AN INFINITE GAME

The book describes creation as a game we play to play rather than to win. It is positive sum – great work begets more great work, both in ourselves and others. The process of tuning in, of restricting our external filter and loosening our internal filter, allows us to engage in the creative act as a forward-looking player in an infinite game. It allows for us to transcend the titles we may have acquired in other finite games we play, and create freely based on our intuitive sense of what the universe is telling us to work on rather than being restricted in scope by the roles we have created for ourselves. When we create as part of an infinite game, we honor our own curiosity by expressing what it inside of us, rather than trying to predict what will be most socially accepted at the time. Success is defined not by the level of public perception or praise, but by the consistent engagement with the process – start, finish, and share, then move on to the next one.

I’ve highlighted the parts of the book that have resonated with me the most. I’ll close with what I see as the key takeaway from the book: In order to create effectively, you need to develop self-awareness and tune in to your self. The answer is within you, not in the outside world.

The tricky part with the metaphors we live by is they tend to be subliminal. As soon as we start to acquire language as a child, we subconsciously download metaphors from our parents, friends, and culture. This download occurs through media and language: the songs we listen to, the books we read, the movies we watch, and the words we hear in the conversations around us. The choice of words reflect the underlying metaphor being used.

Here are some common metaphors from American culture and language that reflects them:

ARGUMENT IS WAR

I demolished his argument. His claims were indefensible. She shot down all of my arguments. I’ve never won an argument against him. His rebuttal was a great counterattack. I attacked his position. How to Win Every Argument.

POLITICS IS WAR

Barack Obama to campaign for Harris across battleground states next week. Who won the VP debate? Why Trump won — 9 takeaways from the 2024 election.

LOVE IS WAR

She left me defenseless. He has a high body count. Love is a battlefield. They had a bad fight last night. She killed his tendency to be vulnerable.

FOOD IS A REWARD

How can you have any pudding if you don’t eat your meat? I’ve been good, so I’m going to treat myself, I deserve it. It’s been a long week, so I’m going to splurge on some chocolate. Finish your dinner then you can have dessert. Open Happiness.

Changing Metaphors

The most impactful takeaway I had from Metaphors We Live By is that we get to choose the metaphors that define our lives. This choice comes after we bring the metaphors that we are using into our conscious awareness. Then we can look at each one and ask the question, “is this the metaphor I really want to be framing my experience with, or is there one that resonates with me more?”. If we don’t like the answer, we are free to change it to anything we want. Changing our metaphors does more than change the words we speak; it fundamentally alters how we experience day-to-day life in our thoughts and behaviors.

After I read the book, I wrote down the metaphors that had been implicitly chosen for me by society in various parts of my life. At the time, I was searching for a new job and was getting ready to start interviewing. In the past, I had always been nervous during interviews, to the point were I had trouble clearly communicating my experience. Interviews terrified me. I began to wonder, “what might the metaphor be that was causing this fear?” This is what I came up with:

INTERVIEWS ARE A COMBATIVE DEFENSE OF SELF

This metaphor was framing my experience of interviews. Where did this come from? At least in my experience, it was common to receive questions like “are you getting nervous for your interview?”, or even simple words of encouragement like “good luck”. The underlying assumption was that an interview is something I should be nervous and fearful about, or one in which luck plays an outsized role in success.

If you think about the goal of an interview, though, it doesn’t make sense to frame it as something that requires combat or self defense. Both parties share a common goal: to figure out if there is a good fit for the role through the sharing of information. A better metaphor is:

INTERVIEWS ARE A COLLABORATIVE SHARING OF INFORMATION

When you look at an interview from this point view, it immediately changes how you perceive the actual event, as well as how you prepare for it. It wasn’t until I switched to this metaphor that I understood the importance of learning how to be a better interviewer, as well as the importance of thoroughly rehearsing my past experience ahead of time. The reason is that the new metaphor aligned my mindset and preparation with the actual purpose of the interview. With this new metaphor in hand, I managed to successfully get through not one, but six interviews and ended up getting an offer. What’s even more fascinating is that my physical stress response during the interviews was significantly reduced compared to prior ones.

There are a couple other metaphors that I have changed which have been especially impactful.

LOVE IS A WAR LOVE IS A COLLABORATIVE WORK OF ART

I have had a tendency to structure my relationships as something that I should win or defend myself for. This has manifested as me getting defensive when my partner shares a grievance with me, especially if their interpretation of what happened isn’t what I intended or matches with my version of reality. People have varying frames of reference (influenced, at least in part I imagine, by the metaphors they explicitly or implicitly are using), and so it is completely reasonable for someone to interpret an event in such a way that would make them upset, and simultaneously for me to feel that they may be overreacting because my intentions were pure. When you view your partner as an adversary on a battlefield, you dig in and try to prove to them how their version of reality is not valid. This tends to make them defensive, as it signals to them that their emotions are invalid. Now you’ve got a full blown “fight”.

However, if you view your relationship as a work of art, and your partner as a fellow artist on that work of art, the dynamic changes completely. True art is not something to be won. It exists outside the realm of competition. The best music contains tension, but it also resolves that tension with a series of notes that underscore the harmony inherent to the song. Of course relationships can develop tension as well. The goal is not to add more tension by trying to “win”, but to resolve it with the notes of listening and understanding. The goal is to keep the music playing. There is nuance here. I’m not saying it’s a good idea to just rollover for the sake of resolving tension. The best relationships have strong boundaries and healthy emotions expressed openly. But this expression is done not for victory, but for continuation of the relationship in the long term.

FOOD IS A REWARD FOOD IS INFORMATION

Have you ever wondered why the “Food and Drug Administration” separates “Food” and “Drug” as if they are two separate things? The distinction only exists in language. To our cells, they are the same thing: chemical information. Our cells don’t use language to distinguish things the same way that our minds do. When you pop a pill or eat a steak, our cells just see various chemicals swimming around. And critically, the chemical information isn’t just discarded. It is actively used to create outputs, whether that be to rebuild new cells, or to render various parts of our genome.

Viewing food as information instead of a reward completely changed how I relate to eating. Before, when food was a reward, I remember justifying a dessert to myself because of something that I had done or even just based on what day of the week it was. I did this with alcohol too, especially in college. I worked hard in college during the week, which meant the weekend was for getting slammed. Now, with food as information, my diet is aligned with the needs of my body instead of my emotions. I try to give my cells what they need, which is good information. Doing that consistently leads to more energy and feeling good, both in my body and my mind. It has also led to more flexibility in some ways. We’ve been conditioned to eat certain meals at certain times of day, but to our bodies it is all the same. Steak for breakfast? Absolutely. One closing thought on this: I’m not saying that eating isn’t associated with reward at all. Eating a good meal with friends and family is certainly rewarding and healthy. But in this case it is the social interaction that is the reward, not the food per se.

Why is changing metaphors so effective at changing behavior? I think it relates to a mental model called the map versus territory. Metaphors are “maps” of our experience that we use to simplify our navigation of the world. They project the high entropy of reality into a lower dimensional form that is easier for us to deal with. These projections are saved into our library of mental subroutines that are triggered by our subconscious when certain environmental patterns are detected. The thing about maps is that some are better than others. When we upgrade our maps to versions that are closer to reality, they allow us to be more effective in our navigation of the world.

If changing metaphors can be beneficial, how do you change them? The first step is to be aware of what your current metaphors are. This is easier said than done, because metaphors we live by are ingrained in us and reside underneath our conscious awareness. Try reading Metaphors We Live By. The book gives a lot of examples that may trigger ideas for your life. After you read it take some time to be introspective and list out the different aspects of your life. See if any of these are particularly stressful or fearful. Then ask “what might the metaphor be related to this aspect of my life that is causing me to feel this way?” Other things that have helped me are meditation and psychedelics. Both of these help bring the subconscious to the conscious, including metaphors. Once you are aware of your metaphors, all you have to do is decide if they actually resonate with you. If they don’t, change them.

Share

Finite versus Infinite Games

I’ve noticed an underlying theme in the metaphors presented in Metaphors We Live By as well the more ill-fitting ones for my own life, including those I’ve changed as discussed above. Or rather than theme, perhaps a better characterization would be an über-metaphor; the root metaphor from which all the other metaphors descend from. The common thread is they involve winning something. They characterize the major aspects of life – school, work, relationships, money, etc – as arenas of play in which there can only be one winner.

This characterization of life as different forms of play is captured in another book called Finite and Infinite Games by James Carse that I highly recommend. The book defines two types of games: finite games and infinite games.

Finite games have strict rules and a fixed boundary of play. The goal of the game is to win. And for one player to win, the other players must lose. In this way, finite games are zero-sum. Finite games are also defined by roles. When we play a finite game, we assume a role that carries a title, which has some meaning within the context of the game itself, like “quarterback” or “senior vice president”. Titles are abstractions that indicate completed victories and therefore orient players towards the past. The orientation towards the past keeps finite play within the boundary of the game. In finite play, life is scarce; something to be won, possessed, and acquired.

The infinite game is different. It is chaotic and non-stationary where the rules and players continuously change. There may be a boundary temporarily, but it is generally amorphous and dynamic with respect to time. The goal is not to win, but to keep playing the game. The infinite game is positive-sum, where players’ contributions enable the game to continue, both for themselves and for others. The infinite player does not completely avoid finite games, but rather recognizes them as small pockets of play subsumed by the infinite game. Infinite players embrace the abstractness of finite games they willingly choose to compete in, and therefore take them up playfully, not seriously. Surprisingly (to me), infinite players do not eschew the roles of finite games, but rather assume them with the full acknowledgement that they are yet another abstraction, completely separate from who they really are. Where the finite player is oriented towards the past and fully constrained by and within the rules of the game, the infinite player is oriented towards the future and therefore is free to play with the rules of the game. Because infinite players understand that there is a bigger game being played in which any particular finite game is but a small part. The infinite game admits ample degrees of freedom for manipulating existing rules or creating new rules entirely. In infinite play, life is abundant; something to be experienced and created.

LIFE IS A FINITE GAME LIFE IS AN INFINITE GAME

Love is a battlefield. Argument is war. Food is a reward. All of these metaphors descend from the root metaphor LIFE IS A FINITE GAME. When we view every aspect of our life as an opportunity to win or acquire something, we are playing the game of life as a finite player. Finite play orients our existence towards the acquisition of abstract titles, colors our relationships with adversarial undertones, and instills our mindset with a sense of scarcity.

On the other hand, if we choose to live by LIFE IS AN INFINITE GAME, we are choosing to approach life with a certain playfulness rather than seriousness. Choosing infinite play doesn’t mean we opt out of all finite games. It means the way we play them changes. Instead of conflating our true sense of self with the roles required for any particular game, we maintain separation between them and fully exercise our ability to assume and unassume our role within the finite games we play. This separation allows us to maintain perspective on what truly matters to us, which can only be determined outside the confines of bounded games and their attendant rules and titles.

Love is a collaborative work of art. Argument is dance. Food is information. All of these metaphors descend from LIFE IS AN INFINITE GAME. The goal is not to win, but to continue playing. Infinite play orients us towards future possibility rather than the past. It emphasizes healthy boundaries and cooperation and instills in us a sense of abundance.

Now that you know metaphors can be much more than a literary device, can you think of any that may need an update for your own life? Raising awareness of existing metaphors and proposing new ones has been great benefit to my own life. Perhaps it will benefit you too.

This was the second ultra I’ve done in my life. I can’t really describe why I decided to do arguably the most difficult 50-miler in the ultramarathon world as my second-ever race (the first I did was a 50k). It was more of a gut feeling that I should do it rather than the result of calculated decision making. A call to adventure, if you will.

Training

I started training for the race in March of this year. I focused on building out my aerobic base from March until about mid-July. This consisted of high volume, low-intensity runs, in addition to maintaining a fairly regular schedule at my gym for strength and mobility. My goal for each run during this period was to stay under (within 10 beats of) my aerobic threshold for the entire run. About two months out from the race, I shifted focus on elevation and strength training. I trained on the trails around Ouray, doing 6 to 9 hour days on the weekend.

As you can see from my Garmin data above, my highest average distance during training was around 25 miles per week. There were a few runs that I didn’t have my watch on, but not enough to have put me much over the 25 miles per week mark, if at all. Since I’m not a professional runner by any means, my goal was just to finish before the 24-hour cutoff. I didn’t have much more time to train given my job, social responsibilities, and other hobbies. I didn’t follow any formal training plan or coaching.

Despite the lack of a formal training regimen, I did follow the recommendations from Mark Sisson’s book, Primal Endurance. One key argument of the book is that endurance athletes tend to overtrain and underrecover. This is especially easy to do when you already have good fitness and you are following a strict training schedule that only focuses on volume. The book shows that subjective feelings of rest aren’t necessarily sufficient to determine your recovery level. Instead, you need to measure heart rate variability (HRV) and combine that with how you feel. A lower HRV means you should probably rest more before the next training session. During my own training, there were many mornings where my body felt good, but my HRV was in the ditch, so I ended up taking extra rest days until my HRV recovered to baseline or better.

The other major lever I used to train had nothing to do with exercise: nutrition.

Nutrition

I’m an ardent reader of Lyn Alden’s investment research. Her macro takes are probably the most clear and well written economic analysis I’ve ever read. She also has a fantastic book called Broken Money. Read it. Take notes. Everyone should read that book.

You’re probably wondering, what the hell does Lyn Alden and investment research have to do with my nutrition for the Ouray 50? Well, she has a non-financial article called 12 In-Depth Tactics to Seriously Boost your Energy that I happened to read in May, espousing the benefits of low-carb ketogenic diets for mental and physical performance. In that article, she also recommends the book The Art and Science of Low Carbohydrate Performance.

I had experimented with keto a couple years prior, but wasn’t able to stick with it for more than a week. Reading Lyn’s article and book recommendation is what made me want to try again, so I did. I transitioned to a strict, well-formulated keto diet in May, emphasizing whole food from quality sources. No ultra-processed keto breads or desserts. Every week I went to my local farmer’s market to buy grass-fed, grass-finished beef from a local ranch, as well as fresh greens, vegetables, and herbs from regenerative farms.

My goal here is to share my subjective, “n of 1” experience with keto, especially in the context of training for and running a long, difficult race. The goal is not to convince you to start eating a low-carb diet. It bugs me whenever I hear people claiming a particular diet is the clearly “the best” for everyone. Food is not a finite game to be won and lorded over other people. Food is information. And besides, nutrition is unsettled science. In my opinion, the human body is an irreducibly complex system3 in the mathematical sense, so nutrition will remain unsettled science for a long time. Luckily, I don’t need science to tell me how food makes me feel. I can use my body and intuition as a guide, no science required.

Now, my subjective experience. After the first few days of keto, I felt increased mental clarity and general cognitive “wired-ness” throughout the day. And I started feeling really good at the 6-week mark. At that point, just about every aspect of my life was notably better – sleep, cognition, energy levels, mood. I had significantly less gas and significantly better erections. I had multiple people comment on how good my skin looked and that I looked healthier. When I first experimented with keto a couple years ago, I sensed a difference in how my brain felt after only a couple of days. However, back then I didn’t stay strict with the diet for more than a week, and I didn’t have any of the other benefits that I have experienced this time around at the 6-week mark.

I was also running a lot during this time (see Figure 1 above), so you may think that all these benefits were from the running instead of the diet. But I didn’t have these benefits when I trained for the 50k last year on a high-carb diet. And it’s been about 7 weeks since the race, and I’ve run maybe 10 miles total, but I still feel the same benefits and am still doing keto.

I started many of my training runs fasted. On the trail I drank water and sugar-free electrolytes and ate high-fat homemade trail-mix containing nuts, MCT oil, basil seeds, and 78% dark chocolate. I added in the dark chocolate after a particularly hot, steep training day when I was squarely in Zone 3 and got dizzy on the uphill climb. I suspect that the rate of gluconeogenesis was not fast enough to prevent hypoglycemia, so my brain started shutting things down. Of course that’s speculation; I didn’t have a way of proving it. Adding in a couple grams of sugar seemed to help on the subsequent training days that were particularly hot.

Now that my subjective experience is out of the way, I’m going to share some of the things I’ve learned while researching the ketogenic diet and some its potential benefits for endurance sports. Fair warning: the next section is technical and contains a lot of jargon from nutritional biochemistry. If you don’t care about this, feel free to jump to Race Day below where I discuss how the actual race went. If you do care, well, down the rabbit hole we go…

Fundamentals of ATP Production

ATP is the currency of life. If our mitochondria stop making ATP4, we die within seconds. ATP is made from two major metabolic pathways: glycolysis and lipolysis. How do our mitochondria decide which pathway to use? Insulin, mostly5. There are other hormones at play as well, but insulin is the one we can most directly affect through the food we eat. Insulin rises when we eat food with a moderate to high amount of net carbohydrates. Insulin drops when we fast or consistently eat food with a low amount of net carbohydrates. Whenever insulin is consistently low, the first thing our body does is release glucagon, which is a hormone that tells the liver to release its glycogen stores as glucose back into the bloodstream. The problem is the liver can only store about 12 to 24 hours worth of energy in the form of glycogen, depending on your activity level.

So what happens when we are not able to replenish glucose and glucagon through diet? Maybe it’s winter and the berry stock has long been exhausted. Or we live in an Arctic zone where no plant life exists in the first place.

Thankfully we don’t just keel over. Instead, our body starts to leverage the abundant energy stored in fat cells (even a lean person can store around ~20x more energy in fat than they can in their glycogen stores6) through a process called lipolysis. Lipolysis hydrolyzes the triglycerides in our adipose tissue into glycerol and free fatty acids (FFAs). FFAs are carried into the bloodstream and used by most of the cells and organs in our body. The FFAs are converted by β-oxidation to Acetyl-CoA, which then enters the Krebs cycle.

There is one major exception to using FFAs for fuel: the brain. The brain chooses not to oxidize FFAs for ATP7, which is unfortunate, since the brain accounts for around 20% of our basal metabolic demand. As far as I can tell, the reason behind the brain eschewing FFA oxidation is currently unsettled8. So what does the brain use when there isn’t enough glucose and it doesn’t use FFAs? In this case, ATP levels are low, so the ratio of AMP to ATP increases, which activates an enzyme called AMP kinase (AMPK9). In turn, AMPK initiates the process of ketogenesis, the production of ketone bodies in the liver. Ketones enter the bloodstream, eventually reaching the mitochondria in our brain (and other tissues). The receiving mitochondria convert the ketone body β-hydroxybutyrate (β-OHB) to Acetyl-CoA, giving it another entry point into the Krebs Cycle. Since β-OHB10 is derived from triglycerides, this means our brain and other cells can generate ATP from fat, allowing us to survive for days to weeks to months, depending on how much fat we have stored up.

However, the brain still demands some amount of glucose in the bloodstream, otherwise we become hypoglycemic. That is why if you’ve ever checked your blood when fasted, even if your β-OHB levels are in the 1-3 millimolar range, your glucose will be on the low end of the normal range. How could this be if you haven’t eaten a single carbohydrate in days? The glucose is created from other molecules in the body through gluconeogenesis. Remember the glycerol backbone from the hydrolyzed triglycerides? Gluconeogenesis recycles this glycerol back into glucose. Lactate, pyruvate, and amino acids are other substrates for gluconeogenesis.

A key point to note is that whenever you first start inducing ketosis, it takes time for cells to adapt to the new fuel source. Ketones enter cells via monocarboxylic transporters (MCTs), which are upregulated via AMPK11. This adaptation can take 4 to 6 weeks to be fully expressed in cell membranes.12

So, we’ve refreshed our memories on ATP production. We know the two primary paths for producing ATP. And we know the basics of ketogenesis. Here’s the question: does ketogenic metabolism provide benefits to endurance athletes relative to glycolytic metabolism?

It’s a tough question. There are many dimensions to ultra endurance running for which benefit could be provided or not. I haven’t attempted to go through all of them. Instead I’ll focus on one area in particular: oxidative stress.

Managing Oxidative Stress

Whenever we exercise, our mitochondria produce large amounts of reactive oxygen species (ROS). ROS are ionic oxygen compounds that increase inflammation, denature surrounding proteins, and cause cellular damage. One of the primary victims of ROS are polyunsaturated fatty acids (PUFA)13. PUFA are ubiquitous in human tissues and are found in cellular phospholipid membranes14.

The Art and Science of Low Carbohydrate Performance proposes that degradation of membrane PUFA by unquenched reactive oxygen species may be a key driver in long recovery times; the more muscle cells that are damaged, the longer it takes to heal. The book estimates that a runner in a 100-mile race would consume roughly 7lbs of oxygen. Assuming a ROS production rate of 2%, there would be around 63 grams of ROS produced over the course of the race. That is enough ROS to degrade 3x the PUFA content in a runner’s legs. The authors also hypothesize that PUFA degradation in the gut could be a cause of the gastrointestinal distress commonly encountered in ultra marathons.

ROS can also inhibit blood flow. Specifically, the enzyme myeloperoxidase (MPO) is abundant in immune cells and generates ROS from H2O215. The MPO in activated immune cells binds with nitric oxide, reducing the bioavailability of nitric oxide to cells in the endothelium16. Nitric oxide is a potent vasodilator, so reducing nitric oxide leads to endothelial dysfunction, vasoconstriction, and poor blood flow (which is probably not something you want during prolonged exercise). Unfortunately, exercise is not the only source of oxidative stress; ROS is a byproduct of metabolism in general, not just during exercise. And there is evidence that the particular pathway taken to make ATP can greatly influence the amount of oxidative stress.

Glycolysis and Oxidative Stress

Excessive carbohydrate consumption leads to oxidative stress. Figure 4 below is from a study17 that reviewed various mechanisms of ROS-generating pathways in animal models of hyperglycemia in the context of diabetic neuropathy:

The advanced glycation end-products (AGE) pathway is what you get when you combine high levels of dietary sugar with the proteins and fats in cells18. Glucose and fructose metabolism creates reactive metabolites such as glyoxal and methylglyoxal. These metabolites bond with proteins and lipids, forming AGEs which disrupt the normal function of cells. Methylglyoxal specifically has been shown in animal models to increase oxidative stress through increased production of superoxide and hydrogen peroxide, as well as reduced antioxidant capacity via NADPH depletion in the endothelium, kidneys, and brain19.

AGEs also lead to the expression of a receptor for AGEs (called RAGE) on cell surfaces. RAGE activation is implicated in cardiovascular disease and obesity and also up-regulates nuclear factor kappa B (NF-κB)20. Chronic NF-κB activation has been shown to drive a positive feedback loop between pro-inflammatory response and production of ROS, leading to blood vessel and neuronal damage21.

Over-stimulation of the polyol pathway is another source of increased ROS in hyperglycemic conditions. This pathway reduces glucose to fructose (fructose itself produces 7x more AGEs than glucose; methylglyoxal produces 250x more AGEs than glucose) leading to excessive depletion of NADPH. NADPH provides the reducing agents required for regeneration of glutathione, a key antioxidant, which leads to increased levels of unquenched ROS22.

Ketosis and Oxidative Stress

Many keto pundits claim that ketones are a “cleaner burning fuel” than sugar without going into detail on what “cleaner burning” actually means from a metabolic perspective. I did some digging to try to answer this question, and based on what I’ve learned, I think it is reasonable to interpret “cleaner burning fuel” as one that is less prone to reactive oxygen species and inflammation.

This review by Kolb et al. discusses the interesting possibility of ketone metabolism being initially pro-oxidative stress, followed by a long-term adaptation that is anti-oxidative stress.

It looked at animal studies23 and in-vitro human studies24 that showed increasing acetoacetate, one of three ketone bodies, increased mitochondrial ROS. And increasing both acetoacetate and β-OHB increased NADPH oxidase activity, resulting in less NADPH available for glutathione production. The review then highlights seemingly contradictory studies that show up-regulation of anti-oxidant and anti-inflammatory defenses of ketone bodies. For example25, an (in-vitro human / in-vivo mouse) study showed increased β-OHB lead to increased FOXO3a activity. FOXO3a is a genetic transcription factor that leads to increased expression of antioxidant enzymes: superoxide dismutase 2 (SOD2) and catalase26.

The review resolved this contradiction by analyzing the time allowed for adaptation in each study and found it takes 48 hours to several weeks for the complete expression of anti-oxidant defenses. These defenses include up-regulation of Nrf2 and SIRT3. Nrf2 decreases NF-κB expression, enhances SOD2 activity, and decreases MPO activity, while SIRT3 increases NADPH availability. In simpler terms, after several weeks, inflammation decreased, ROS decreased, and anti-oxidant capacity increased27.

What about those AGEs from glucose and fructose metabolism? We know hyperglycemia contributes to excessive AGE formation, but does this mean that ketogenic metabolism produce less AGEs overall? One place to look is HbA1c, aka glycated hemoglobin. Technically, HbA1c is an early glycation end-product, but Turk et al. showed a significant positive correlation between HbA1c and hemoglobin-related AGEs (Hb-AGE)28 in diabetics. A meta-analysis from 202029 analyzed the effect of the ketogenic diet on HbA1c in humans with Type-II diabetes. The analysis spanned 13 studies with a total sample size of n=567 people, and eight of the studies (n=422) measured HbA1c before and after treatment with the ketogenic diet. The plot below shows the average change in HbA1c across the eight studies:

As you can see, each study found a decrease in HbA1c. The average decrease across the eight studies was -1.07%, which is significant, given that a 1% change can mean the difference between normal and diabetic levels according to the ADA. Now there are some limitations to applying these studies to determine if ketogenic metabolism produces fewer glycated end-products relative to glycolytic metabolism. The first is that three of the studies (the blue bars above) had no control group with an alternative diet, so we can’t make a comparison for those cases. The studies also had varying methods to measure adherence to the diet. One had daily β-OHB measurements, another had weekly β-OHB measurements, and one relied on self-reporting of what the participants had eaten. The level of calories also wasn’t controlled for; one was hypocaloric, while others had no caloric restriction at all. Additionally, the type and number of diabetic medications varied across studies and participants. Finally, the sample sizes were small and limited only to Type-II diabetics, with the largest study only having 238 participants. Given these limitations, we can’t used these studies to logically conclude that ketogenic diets cause a lower HbA1c relative to high-carb diets. However there does appear to be a correlation, at least in diabetics.

So, in review, we have some evidence that ketogenic diet may reduce oxidative stress through various mechanisms, including down-regulation of inflammation, up-regulation of anti-oxidant defenses, and decreased AGEs. Does this really prove that the ketogenic diet produces less oxidative stress than a high-carb diet? No. First of all, my review effort here was only over a small subset of the research. It was by no means an exhaustive literature review. Second, many of the studies that I did review had uncontrolled, potentially confounding variables that make it impossible to establish causation.

That said, the data at least show some interesting correlations. Enough to make me curious if I would perceive any of these potential benefits throughout my training and the race itself. Subjectively my recovery times on longer (15+ mile) runs were probably around a couple of days, mostly limited by HRV instead of muscle soreness or fatigue. Unfortunately I don’t have a “control” to compare with, since the training was different than the 50k training I did the year before.

Alright, enough from the ketogenic tangent. This post is about the Ouray 50, remember? Next, I’ll describe the race itself and some of the things I learned.

Race Day

My nerves started ramping up about a week out from the race. I didn’t sleep well the entire week. By the time race day finally came, I was both excited and ready to get the damn thing over with. I ate my normal breakfast of bacon and eggs with some coffee and got to Fellin Park around 11AM to check in. The weather was clear with a high of around 73 degrees. This doesn’t seem that hot on paper, but at elevation it was, especially given the first two climbs were on southerly aspects with sparse tree cover.

Fellin Park to Alpine Mine Overlook

The race started at 12PM at Fellin Park in Ouray. The course proceeded south and counterclockwise along the Ouray Perimeter Trail, until it reached the intersection with Camp Bird Rd on the south rim. It then ascends a few miles to Weehawken trail which leads to Alpine Mine Overlook.

I came out way too fast due to a combination of pent-up anxiety and inexperience. I didn’t have a strict game plan on heart rate or pacing. As you can see from the heart rate data below, I was anaerobic for over three hours at the beginning of the race. Despite this, I felt pretty good going up and down Weehawken. However my body and my mental state took a nosedive as I started the third climb up Hayden Mountain.

Hayden to Crystal Lake

The crux of the race for me was the Hayden ascent. I started up Hayden trail around 2:45PM, the hottest part of the day. I felt dizzy and lightheaded. Luckily, Hayden offered a bit more tree cover relative to Weehawken, especially at the beginning, otherwise I think I would have passed out. I started eating blueberries and my homemade trail mix, but it was too little, too late. At this point, I was struggling to keep my heart below 170, and I had to completely stop roughly 10 times on the way up. The symptoms I felt were similar to the (what I assume was) hypoglycemia on particularly hot training days, but more intense and persistent. About halfway up Hayden, around mile 10, my body refused to swallow any more of the high-fat trail mix I made. It felt as if my brain was shutting down my digestive tract in order to preserve vital organs or something. I didn’t need to vomit, I just literally couldn’t swallow the food.

Everyone has a plan until they get punched in the mouth – Mike Tyson

Suffice it say, I had been punched squarely, unceremoniously, right in the mouth. And it was only mile 10! I hit my mental low. My stream of consciousness was filled with negative self-talk: “How could I have been so stupid to come out so fast? I’ve ruined months of training. I look like a fool! I’m trying to run 50 miles on the keto diet? I can hear the mockery now. My family came all this way, just to see me fail.”

Slowly, I recovered. As I stumbled up the loose, ball-bearing filled switchbacks of Hayden trail, I remembered what I had (serendipitously?) read a few weeks before the race from Man’s Search for Meaning. The meaning of life is not what you get out of life, but what life gets out of you. It’s in how you show up and do the work, especially when it gets hard and you feel like quitting. My mental milieu became a weird mixture of positivity and self-loathing. I decided in that moment that there would be only two outcomes. Either I miss a cutoff and am required to drop, or I finish. There would be no quitting.

I reached Crystal Lake with 20 minutes before cutoff. Crystal Lake was the first aid station that allowed crews and drop bags. My family and friends were there, giving me encouragement to keep pushing. Their presence was energizing. I felt so grateful they were there, that they would take time out of their busy lives to see me voluntarily struggle up and down mountains. One of them got me some bacon to eat, but my body wouldn’t have it. The only things my brain allowed down my throat were fruit, water, and electrolyte pills. And this remained true for the rest of the race. My plan to consume high-fat nutrition had been completely dissolved against the reality of my body. Thankfully I had plenty of fat stores and was 4 months in to fat and ketone adaptation. Now I just needed to get and stay aerobic so I could start mobilizing the fat in volume and spare the residual glucose for the brain.

Crystal Lake to Fellin Park

With about 10 minutes before cutoff, I started back up Hayden trail, this time from the Crystal Lake side towards Fellin Park. I was still moving slow and was near the back of pack, but I felt much better after the short rest at the aid station. It was dusk by the time I got to the black scree at the top of Hayden ridge. By the time I stopped to put my headlamp on, the temperature had dropped about 20 degrees and my heart rate was in zone 2. I started the long descent down Hayden to Fellin Park feeling much better. My mind was clear and I could finally keep a steady pace without blowing up my heart rate.

About a mile away from Fellin Park aid station, there was a small tributary to the Uncompahgre River that crossed the course. I took off my running vest and did a cold plunge for a couple minutes. I could slowly feel my legs getting numb as the icy-cold water flowed over my legs. In that moment, sitting in the freezing water, I was overcome with an intense rush of euphoria and heightened awareness of my connection with Nature. The water on the mountain stripped away my egoic concerns about “racing well” or how others may perceive my sloppy start of the race and replaced it with a quiet, authoritative awareness of the present moment. An appreciation for the harmony of life, for health, for family and friends.

When I finally got to Fellin aid station around 9PM, my crew was waiting there for me. They brought me fruit and warm broth, which I promptly devoured as I sat next to one of the space heaters. I shared my Hayden descent and cold plunge experience with them, and they offered more encouragement. Again, I felt so connected with each of them and grateful for their presence. There was a strange tinge to that moment. Despite having known my family and friends around me for so long, it was as if I was seeing them with a new vibrant clarity. A raw presence that was revealed when the veil of preconceived labels and conditioned mental representations was lifted away from my perception. I’m still curious about what this sensation was and what caused it. It felt as if everything was salient, similar to moving to a new place or the afterglow of a psychedelic trip.

After this commune with fruit and family, I was riding high. The race was halfway over, and I was allowed a pacer for the remaining four sections. Despite there still being 25 miles, I felt the worst was behind me.

The Back Half

Under the cool night sky, the running on the back half was rather steady and uneventful. One foot after another, the miles passed by. Up Twin Peaks via Old Twin Peaks trail, and down to Silvershield with only 10 minutes to spare before cutoff. I arrived back at Fellin Park early Sunday morning, around 1am, and most of my crew were asleep. I ate some more fruit and broth, then picked up another pacer to check off Chief Ouray Mine. Once again back at Fellin, more fruit, more broth. As I was preparing to head out to the final leg up Bridge of Heaven, there were a few racers that crossed the finish line. I was happy for them but also jealous. I wanted to be done. My body felt good, but I was starting to get sleepy and losing mental acuity. However I had done 40 miles and wasn’t about to drop out on the last leg. Cutoff or finish, remember?

The race organizers really saved the best for last when you’re most tired, just to make sure they squeeze every last drop out of you. Bridge of Heaven ascends around 5000 feet over 5 miles, for 10 miles total on the out and back30. My girlfriend paced me. We set out around 5am up the trail. This one was weird because the sun came up about an hour in. The realization that I had been running all night and most of the previous day seemed to place a dense layer of grogginess over my brain. My girlfriend tried to kindle conversation, but the most I could manage in return was short, unthoughtful responses.

Eventually I turned into a metronome. It felt like my body was on autopilot, one foot after the other, driven by some low frequency clock. My mind was essentially absent of all thought for several hours, which is an exceedingly rare state for my psyche to be in. The mind had shed all cares and responsibilities save for the orchestration of my vitals and leg muscles.

Finally, we made it to the top. The last climb was over. After a few minutes of rest and some pictures, we started the last descent towards the finish at Fellin. The descent involved higher frequency metronome, but was otherwise uneventful. I was very sleepy and remember longing for one of life’s simple pleasures: sitting in a chair. After about an hour, we were down and headed back towards the finish line at Fellin.

I crossed the finish line with about an hour to spare before cutoff. My family and friends were there to give me hugs and congratulations. After I took some pictures with my new finisher buckle, I finally found a chair in the late morning sun. Sitting in that chair, knowing that I was done, my body and mind began to relax into a state of authoritative fatigue. The kind of fatigue that makes rest non-negotiable. The kind that demands inattention from your environment. I was nodding off in between conversations as the last few runners trickled in through the finish line. Once the podium ceremony was finished, we made our way back to the car. I slept the whole way home.

Reflections

Peaks and valleys. Tension and resolution. Moments of suffering that required me to go within myself and find a way to just keep going. The Ouray 50 was like a small fractal of life itself. It made me thankful for a healthy, resilient body and for a mind that is able to persevere through hard times and to enjoy the good times. The ups and downs of the race gave me a newfound appreciation of the inherent harmony of life. This harmony is ever present in my life, if only I would zoom out from the individual notes and listen to the whole song.

Family and friends. I’m still trying to understand what that “strange tinge” was that I felt when I saw them at Fellin aid station. The feeling that I was seeing them for the first time, and yet loved and cared for them deeply. As if a switch had flipped from cognitive priors to the senses. From predictions based on learned mental representations to a keen awareness of them in that moment. In How to Change Your Mind, Michael Pollan describes a similar phenomenom commonly reported in psychedelic therapy sessions. In Chapter 5, he highlights the theory put forth by Robert Carhart-Harris et al. in their research, The entropic brain: a theory of conscious states informed by neuroimaging research with psychedelic drugs, that different states of consciousness can be placed on a continuum of entropy. States such as anxiety, addiction, depression, and OCD are low entropy states that form deep grooves of cognition, creating a positive feedback loop with themselves. High entropy states, such as the adult brain on psychedelics or the brain of a young child, exhibit increased connectivity and creativity, as well as a diminished sense of separateness from other people and nature. The proposed mechanism behind this theory is an attenuation of the default mode network, or lack of a fully developed one in the case of children, leads to high entropy states. The DMN is believed to be the physical “location” of our sense of self as well as the library of prior experiences, narratives, and labels we use to explain the world. It is believed that reducing DMN activity reduces the sense of self and lifts the veil imposed by the stories we tell about ourselves and others. If this theory is true, could it be that prolonged exercise such as ultra-marathons provide another way to reduce DMN activity? Does ultra-endurance increase the entropy level of the brain? What about other forms of exercise?

Ketosis. Is it optimal for ultra endurance? I don’t know. Maybe if you can stay aerobic the entire time. You probably don’t need to buy those ultra-processed sugary goos and gels that people peddle around.

It is safe to say that I didn’t eat 10,573 calories worth of fruit, so the idea that fat can be used as the primary fuel source for ultra events seems to hold up.

I am curious about how exactly that happens. The common narrative is that whenever you eat sugar, you spike your insulin, which locks you out of your own fat stores.

This must change during exercise in ways that I don’t fully understand. Perhaps the metabolic demand was so great that the residual glucose was immediately taken up by cells and any insulin spike is extremely short lived.

I definitely learned the importance of staying aerobic if you’re going to use fat for fuel. I ran the first half of the course a couple weeks before the race for a training run and ate the exact same high-fat trail mix that my body simply refused on race day.

During the training run, the trail-mix went down fine, but I also stayed aerobic the entire time. Clearly, going anaerobic for a few hours did not work out so well for me.

If I ever run a race like this again, my heart rate will be the key variable I watch, especially at the beginning.

Well, this was a long post. If you’re interested in some further information on low-carb and ketogenic diets, check out the following sources

• Dr. Peter Attia’s blog31

• Levels blog

• Dr. Casey Mean’s Good Energy

• Dr. Robert Lustig’s Metabolical

Despite this wide variance in personas, every prisoner was in the same environment. What was different was how each prisoner related to their environment internally. The book argues that these differences were caused by different meanings that each prisoner held for their life. Those that were able to still find meaning in their life were most likely to endure the daily stress of camp. Others that had no meaning, or lost their meaning, were most likely to give up. They would refuse to eat, refuse to move, and would lie in their own excrement until they died. This suggests that the answer to the age-old question “what is the meaning of life?” is more than just a thought exercise; it is critical to our survival and well-being. Frankl believes that pursuit of meaning is a fundamental human drive (the “will to meaning”), and suggests that meaning, not pleasure (a la Freud), is the primary motivating factor in life.

Frankl eloquently describes how we can answer “what is the meaning of life?” in our own lives. One thing we have to understand is that meaning is personal. It is unique to each individual. Meaning isn’t something that we can outsource to our culture or other people. It comes from deep within us, from our gut and our heart. Frankl suggests that often when we ask “what is the meaning of life?“, we tend to answer in such a way that involves us acquiring or extracting things from life. The answer tends to underscore a cultural metaphor that life is a finite resource, ready to exploited for our own gain and pleasure. To find our meaning, he recommends to flip that perspective around. What matters is not our expectation of life, but what life expects of us.

What does life expect of you?

The meaning of your life comes from authentically answering that question and orienting your life in pursuit of actualizing the answer. The process of confronting one’s meaning and orienting one’s life towards it forms the basis of Frankl’s logotherapy approach. Logotherapy requires listening to and honoring your gut and heart, which may be at odds with your mind. This inner tension is actually a good sign that you’ve identified meaning for your life (and reminds me of Resistance acting as a gradient for self-actualization described by Steven Pressfield in the War of Art). Frankl argues that true wholeness and human thriving is dependent upon creating this tension (by confronting one’s meaning) and then working to resolve it via “right action”, i.e., pursuing your meaning through your daily actions.

“What man needs is not a tensionless state but rather the striving and struggling for a worthwhile goal, a freely chosen task. A call of a potential meaning waiting to be fulfilled by him.” — Viktor Frankl

Despite meaning being personal, Frankl suggests there are three general sources of meaning. The first is creation. Humans are especially gifted in that we can create things; solutions to hard problems, art, wealth, or other humans. Heeding the instinctual call to create something of lasting value can bring immense meaning in one’s life.

The second is love. Creating a strong relationship with a spouse, friends, and community aligns with our fundamental need to social connection. The meaning in love is about seeing the essential traits of a person and the unrealized potential (their own meaning) which ought to be actualized, and then enabling them to actualize it.

The third is suffering. Frankl suggests that life can have meaning despite ineffable suffering, as he experienced first hand in Nazi concentration camps.

He who has a why to live for can bear almost any how — Friedrich Nietzsche

Maintaining your positive character and persevering through adversity are two things that provide meaning during times of suffering. Suffering challenges us to change ourselves; there is no growth without suffering and pain.

Frankl believes that many of our collective neuroses, e.g., nihilism, anxiety, depression, can be explained in terms of meaning, or rather lack of meaning in our lives. Whenever we lose meaning orientation in our life, it creates a so-called “existential vacuum” that we tend to fill with pleasure or power1 seeking behavior. On the other hand, having a strong meaning for your life provides a foundation for your behavior and mental health. When we have a foundational “why” for our life, it makes doing things that require discipline (e.g. regular exercise, consistently eating real food, writing) easier to stick to. And since our meaning comes from within us, it will be intrinsically aligned with our core values. This alignment will foster the free expression of our core values through our behavior and thought patterns, a necessary condition for mental health.

Personal Thoughts

If meaning originates from our true self, then we need to have a strong connection to our whole self (including our gut instinct) as a prerequisite for finding meaning in our lives. From a previous post, we know that trauma causes disconnection from our whole self, including our gut instinct. Therefore, finding meaning may require us to first heal trauma to reconnect with our self so that we may hear the full message that our gut tries to send us.

Once we’ve identified some potential meaning in our life, we have to act on it. This is the source of tension that Frankl describes. Whenever our potential meaning requires us to switch careers, or end unhealthy relationships (with people or bad information2), this can represent fundamental changes in our life that are difficult to follow through on. Frankl argues that working to resolve this tension through our actions, i.e., pursuing our life’s meaning, is a necessary condition for mental health.

The question “what does life expect of you?” is a great frame to think about meaning in our lives, however I think it deserves caution not to take it too far. No matter how much you think life “expects” of you, you still have to defend your own boundaries and ensure your needs are met. Sacrificing your health or relationships for goals, no matter how ambitious or noble they may be, misses the forest for the trees. You are no good to anyone if you burn yourself out. One thing that can help here is to have a low time preference. Don’t try to solve every problem in a week. Identify opportunities for long-term value creation, and consistently work toward them in a sustainable way.

What is Trauma?

Trauma is somewhat of a nebulous thing that is hard to concretely define, kind of like love. The book does a great job of breaking it down, first discussing what it is not, and then providing a definition and mental model that help the reader understand its nature.

Trauma is often conflated with stress in everyday language. They share some characteristics, but they aren’t the same thing. Both stress and trauma may lead to similar physiological effects such as autoimmune disorders and insulin resistance, as well as mental conditions such as anxiety and depression. However, they differ in locus of affliction and duration. Stress is a physiological response of the body to what is happening to you in your environment; it has an external locus of affliction and environment-dependent duration. Fix the environment, and the stress response dissipates.

On the other hand, trauma is what happens inside you; it is a psychological response to adverse experiences and therefore has an internal locus of affliction. The duration is environment-independent in the sense that improving the environment does nothing to heal the wound.

The essence of trauma is a fracturing of the self; a psychological dismemberment into parts expressed that comprise the conditioned personality everyone sees, and parts suppressed as mitigation to the perceived threat they pose to the social attachment required for survival. The fracturing occurs subconsciously, altering the architecture of our psychological ground.

This alteration leaves an imprint on our psyche that is triggered whenever we subconsciously perceive similar adverse situations as adults. Even if we are consciously aware that our survival is not at risk later on, the subconscious imprint may perceive a similar threat to our survival, and tries to protect us by shutting parts of ourselves down or activating maladaptive behaviors or beliefs.

“Until you make the unconscious conscious, it will direct your life, and you will call it fate” — Carl Jung

Finally, trauma is persistent. Once our psyche is fractured, it remains that way unless and until it is made whole again. In other words, time heals all wounds, except for trauma. Why? Because humans prioritize survival above all else, and trauma is a potent, persistent expression of that survival instinct.

What Causes Trauma?

We tend to constrain our conception of trauma to things like natural disasters, war, severe neglect, and physical abuse. But restricting the definition to catastrophes would imply trauma consists of infrequent external events. Dr. Mate argues that overt tragedies cause only one form of trauma called “big-T” trauma. Big-T trauma occurs when bad things happen to people. Things like sexual abuse or loss of a parent. The other form, called “small-t” trauma, occurs when good things don’t happen, specifically when the core psycho-social needs of a child go unmet. These needs of early childhood are: emotional and physical attunement of parents, a sense of unconditional worthiness of that attunement, a sense of safety in expressing authentic emotions, and agenda-free, person-to-person interactive play.

Dr. Mate argues that the inclusion of small-t trauma with big-T trauma suggests that we should not view trauma as a rigid yes/no, but rather a spectrum that everyone is on. This does not discount the fact that some people endure more trauma than others, but rather that everyone should honestly look at their own experience and expect to find some evidence of trauma, and that many of our maladaptive behaviors and destructive relationship patterns in adulthood can be traced back to unresolved trauma.

What does Trauma Cause?

Dr. Mate presents many potential downstream effects of trauma, supported by research and anecdotes from his clinical experience. Some of the effects, if proven to be causal, would be quite alarming and would necessitate a shift in our approach to treating many afflictions that pervade modern life. He provides evidence that many cases of chronic illnesses such as cancer and autoimmunity, as well as mental disorders such as anxiety, depression, and addiction may have a root cause of unresolved trauma. This seems reasonable, since we know unresolved trauma persists through time, and therefore all of its downstream destructive habits and self-abnegation cause persistent damage to our bodies and minds.

How to Heal Trauma?

So far this book may seem pretty bleak. Humans have an instinctual response to the environment that occurs when we have zero control over it that leaves a persistent imprint of subconscious behaviors and thought patterns that not only cause chronic disease, but also pass down to our own children.

The good news is that we can do something about it.

We know what trauma is – a fracturing of the self. Healing is a process to restore wholeness of the self. To integrate the repressed emotions and parts of our selves with our conditioned personality so that we may finally live according to our authentic essence.

So how do we heal? What does it look like? The book describes various things within our control we can do in order to restore ourselves. Before I review those though, there are two things described in the book that should be avoided: blame and comparison.

Blaming someone for our trauma, even if what they did undeniably caused big-capital-T-all-caps-TRAUMA!!, is antithetical to healing, because it implicitly assumes the problem is the person instead of what the person caused to happen inside of us. Healing is not about being right and using that as justification of perpetual victimhood. Placing blame on one person or event ignores the possibility that trauma is systemic in our collective subconscious and that unsocial behaviors levied upon us by others are they themselves the shadow of an accumulation of unresolved trauma in their own life. We are not responsible for what other people do to us, but we are responsible for how we relate to their actions.

Similarly, comparing your trauma to others isn’t helpful for healing either. Just because someone may have had a harder life than you according to some cultural measuring stick, doesn’t invalidate your own harmful experiences. Telling yourself “I should be happy, I had it pretty good, especially compared to this person over here” only induces guilt and shame. It is a further rejection of your emotions rather than a reintegration of them. Comparing in the other direction doesn’t help either. If we think our traumatic experience is so much worse than others, it can give rise to jealousy, resentment, self-pity and even self-aggrandizement. None of these mindsets provide any true healing at all.

After highlighting some things to avoid in the healing process, the book goes on to describe some concepts and techniques that you can use to regain your whole self. One of these concepts is the “4 A’s”. Healing involves the development of each of these 4 A’s in our life:

1. Authenticity. In order to heal, we have to start listening to our self and shed our self-abnegating social character. Some social attachments may not survive the authentic expression of ourselves, but new attachments will be made based on who we truly are.

2. Agency. When we reframe trauma as something that happened inside us, it allows us to let go of the particular events that led to it. This means that we no longer have to be controlled by the subconscious impulses that trauma causes; we can have control over our lives again.

3. Anger. Healthy anger is expressed in the moment, not repressed down inside us. Healthy anger represents a decisive “no”; a boundary that we have to learn to honor.

4. Acceptance. This means being present in the moment, and not resisting the emotions that may come with it, uncomfortable as they may be.

Another key concept is awareness. Since trauma takes place subconsciously, you have to bring the wound into conscious awareness. This requires intention, hard work and a serious exploration of the self. You have to ask yourself hard questions; questions you initially may not want to know the answer to. Dr Mate provides a few examples that you can use (from his compassionate inquiry process):

• What am I not saying “no” to that I should be? When did I sense a “no” but repressed it?

• How does my inability to say no impact my life?

• What bodily signals have I been overlooking?

• What is the hidden story behind my inability to say no?

• Where did I learn those stories?

• What am I not saying “yes” to that I should be? What have I wanted to do, create, or express, but haven’t out of fear?

Getting truthful answers to these questions is difficult due to how buried some of these wounds can be in our psyche and how much pain and fear they trigger – some memories can be so utterly suppressed that we have no conscious recollection of them ever happening. Due to this inherent difficulty, the book recommends both therapy and psychedelics (in a therapeutic setting) to help peel the psychological onion and to guide the exploration and integration of emotions that bubble up into our conscious awareness.

Despite the systemic nature of trauma in our lives and culture, and the myriad adverse consequences it has on us, the message of healing is hopeful: it’s not about what happened in the past, but how we relate to our past. If we are willing to do the work, we can resolve our trauma and live the life of wholeness that we are meant to live.

Share

Personal Thoughts

This book is long, but well worth the time if you are interested in learning about trauma and ways to heal. One of the main contributions of the book, in my view, is the conceptualization of what trauma is. I only had a vague notion of what trauma was before, but now I understand it in a way that allows me to identify and label it, which helps to bring awareness of and separation between trauma and myself.

Also, this post tends to focus on the childhood causes of trauma, but the book explains that trauma is not restricted to just children. The same fracturing can and does happen in adults as well.

The section on psychedelics resonated with me as well. I have experienced first hand the “emulsifying” effect that psilocybin mushrooms have between the conscious and subconscious. Daily life consists of the same patterns playing out over and over, like traffic that creates deep ruts on a dirt road. These ruts become so deep that it makes turning onto other roads impossible. Psilocybin is like a great flood that erodes the ruts down into a level surface, providing smooth passage to the other roads once again.

I do disagree with some of the book’s claims that capitalism causes or (at least perpetuates) trauma. My main contention is that the author doesn’t actually define what he means by “capitalism”. Based on the arguments he presents, I think his definition of capitalism means the “modern credit-based financial system led by the hegemonic power of the Federal Reserve and the United States”. If that is the case, then I disagree with him on his definition of capitalism and not so much that his notion of capitalism perpetuates trauma. Indeed, I think our fiat credit-based financial system, propped up by exponentially increasing debt and debasement of our currency is the root cause of the inequality, division, and populism in our culture today. If we take capitalism to mean “free markets”, then the system we have isn’t really capitalism. The system we have today bails out banks that are too big to fail (privatized gains, socialized losses). The system we have today features a cabal of unelected central planners to distort the price of money. I have a lot more to say on this topic, but this post is already long enough, and it really deserves a dedicated post (or series of posts).

Despite these disagreements, the book is well worth the read. The depth of discussion regarding the various facets of trauma, as well what we can do about it in our own lives, leaves the reader empowered to start living according to their authentic self.

Lately I’ve had this gut feeling that I should start writing. I think this comes from a need to integrate and express the ideas I become exposed to from media like books and podcasts.

Where does this need come from? I believe it springs from one of my core values; authenticity. To me, living an authentic life means sincerely and openly expressing one’s ideas and beliefs, even if it means damaging (or losing) interpersonal relationships with loved ones due to differences in opinion.

For most of my life, this fear of loss has led me to suppress authentic expression for my ideas and interests. I’ve learned the root cause of this fear is based in the psychology of (insecure) attachment. Attachment refers to the set of emotional and social patterns present in our interpersonal relationships.

As children, when any of our core social, emotional, and safety needs are not met, we modify our outward behavior to preserve our attachment with caregivers. This behavior change stems deep from the survival instinct and leads to suppression of authentic emotions. These behavior patterns are imprinted into the our subconscious and manifest later in adolescent and adult relationships as either excessive neediness (anxious attachment), dismissiveness (avoidant attachment), or a mix of both (disorganized attachment) and people-pleasing behavior.

Writing this blog is one way for me to overcome the fear of loss of attachment and express my authentic ideas and interests. My ultimate goal is to use this blog to connect with myself and others in an authentic way. Writing aligns with my other core values as well:

• Health - you can’t really do anything worthwhile in life if you aren’t healthy. Consistent exercise and eating right are huge components of this.

• Authenticity - expressing what is on your mind, especially when it feels like it will be unpopular

• Discipline – I’m committing to a regular writing schedule. At least one post a month, for now.

• Growth mindedness – I’m not a great writer, but I know that I will get better over time with consistent effort. Writing about a topic will also force me to understand it more deeply.

• Connection – my hope is that someone will find the ideas discussed in this blog helpful, and that they walk away having learned something that can make a positive impact on their life.

This year is the first in my life where I’ve actually thought through and written down what my core values are. To find them, I thought about scenarios in the past when I felt awful over something I did or didn’t do, and the “inverse” of that (mis)behavior pointed to the underlying value that needs to be honored in order not to feel that way in the future. I’m sure there are other methods for finding your core values, but this way has worked for me, maybe it will work for others too. The easy part is taking the time to think through them, and write them down. The hard part is living them.

What are your core values?