# blog_post

title: QK Norm and the Curious Case of Logit Drift
slug: qk-norm-and-the-curious-case-of-logit-drift
date: 2024-10-03
body:
  Multimodality has led to some tweaks to the standard Transformer recipe. In this post we will cover [QK Normalization](https://arxiv.org/abs/2305.18234 "Query-Key Normalization for Transformers") (QKNorm), where we apply normalization to the query and key vectors in the attention mechanism. This has become an important step for unified input models where we have tokenized representations of text, images and other modalities present in a single sequence. These types of models are very prone to training instability without appropriate normalization, as the norms of the Q and K vectors can experience exponential growth during training. QKNorm is a necessary (although perhaps not sufficient) step to correct this problem.

  ## What's the problem?

  Attention logits are constructed via a scaled dot product in the Transformer attention mechanism:

  \[ \text{softmax}(\mathbf{x}) = \text{softmax}(\frac{\mathbf{q} \mathbf{k}^\top}{\sqrt{d_h}}) \]

  The softmax is invariant to translation. If we add a constant \(c\) to each element of the input vector \(\mathbf{x}\) then we can see that:

  \[ \text{softmax}(\mathbf{x} + c) = \frac{e^{\mathbf{x} + c}}{\sum_j e^{x_j + c}} = \frac{e^c \cdot e^{\mathbf{x}}}{e^c \cdot \sum_j e^{x_j}} = \frac{e^{\mathbf{x}}}{\sum_j e^{x_j}} = \text{softmax}(\mathbf{x}) \]

  The invariance property is a potential source of training instability when combined with unnormalized query and key vectors.

  To see why, we can write an individual logit of the attention matrix as:

  \[ \text{x}_{ij} = \frac{\|\mathbf{q_i}\| \|\mathbf{k_j}\| \cos(\theta_{ij})}{\sqrt{d_h}} \]

  Since additions have no impact on the relative magnitudes of the logits, the only way for the model to increase the relative distance between logits is by increasing or decreasing the norms of the \(\mathbf{q}\) and \(\mathbf{k}\) vectors, or adjusting the angle \(\cos\left(\theta_{ij}\right)\). It may be easier for the model to learn to increase the norms, as this requires uniform scaling of weights, as opposed to changing the angles which requires coordinated changes in multiple weight parameters.

  However, uncontrolled growth in the norms of these vectors can lead to instabilities in training, as large logits will collapse attention weights to one-hot vectors - "attention entropy collapse" ([Dehghani et al 2023](https://arxiv.org/abs/2302.05442 "Scaling Vision Transformers to 22 Billion Parameters"), [Zhai et al 2023](https://arxiv.org/abs/2303.06296 "Stabilizing Transformer Training by Preventing Attention Entropy Collapse")). If the training setup is overly sensitive to some hyperparameters, such as a high learning rate, then norms will be pushed into the exponential region of the softmax. This is the case discussed by [Wortsman et al (2023)](https://arxiv.org/abs/2309.14322 "Small-scale proxies for large-scale Transformer training instabilities").

  Recent empirical evidence suggests that introducing multimodal inputs increases hyperparameter instability, and makes training more prone to norm explosion. We cover the relevant literature in the next section.

  ## What's the evidence?

  [Dehghani et al (2023)](https://arxiv.org/abs/2302.05442 "Scaling Vision Transformers to 22 Billion Parameters") observed divergent training with an 8B Vision Transformer. The loss decreased as normal to begin with but then steadily increased after 2,000 steps. Without normalization they found that attention logits grew to over 50,000 in magnitude, resulting in one-hot attention weights post-softmax and subsequently unstable losses and gradients:

  ![Effect of QKNorm on an 8B parameter model](https://rossjtaylor.com/images/vitnorms.png)

  [Wortsman et al (2023)](https://arxiv.org/abs/2309.14322 "Small-scale proxies for large-scale Transformer training instabilities") found they could obtain attention logit instability with smaller models using higher learning rates. As a solution, they found that QKNorm enabled stable training across three orders of magnitude of learning rate (LR) variation.

  ![Learning rate and instability]("https://rossjtaylor.com/images/traininginstabilities.png width=400")

  [Lu et al (2023)](https://arxiv.org/abs/2312.17172v1 "Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action") experienced unstable training as they integrated additional modalities into their UnifiedIO architecture. They observed extremely large values in the multi-head attention logits when including image and audio modalities, leading to one-hot attention weights. They used QKNorm to stabilise training.

  ![Effect of multimodality and QKNorm](https://rossjtaylor.com/images/unifiedio2.png)

  [Chameleon Team (2024)](https://arxiv.org/abs/2405.09818 "Chameleon: Mixed-Modal Early-Fusion Foundation Models") experienced instabilities in the mid-to-late stages of training which they attributed to uncontrolled norm growth. They attribute norm growth to "norm competition" between modalities of differing entropy, which is encouraged by softmax translation invariance. They note this becomes problematic once training reaches outside the effective representation range of bfloat16. They do not observe the same problem for the unimodal text-only setting.

  ![Chameleon logit drift](https://rossjtaylor.com/images/chameleonnorm.png)

  Interestingly, they found that QKNorm alone was not sufficient for stabilising training with the LLaMA architecture. They opt for a Swin Transformer normalization strategy, which normalizes on the outside of the feedforward block (helping bound its norm growth).

  Some other recent work includes OLMoE by [Muennighoff et al (2024)](https://arxiv.org/abs/2409.02060 "OLMoE: Open Mixture-of-Experts Language Models"), where they find QKNorm increases training stability (at the cost of 10% throughput). Additionally [Ramapuram et al (2024)](https://arxiv.org/abs/2409.04431 "Theory, Analysis, and Best Practices for Sigmoid Self-Attention") find QKNorm significantly stabilises performance by making Sigmoid Attention and Softmax Attention less sensitive to learning rate changes.

  ## Why does multimodality lead to norm growth?

  Let's look at the \(\mathbf{q}\) and \(\mathbf{k}\) vectors again. Both are \(\left(B,T,C\right)\) size vectors constructed via linear layers (omitting biases):

  \[ \mathbf{h_{q,k}} = \mathbf{x} \mathbf{W_{q,k}} \]

  where \(x\) is the input embedding. Let's restrict ourselves to looking at query vectors for now for a single batch:

  \[ \mathbf{q} = \mathbf{x} \mathbf{W_{q}} \]

  where \(x\) is of size \(\left(T,C\right)\) and \(\mathbf{W_q}\) is of size \(\left(C, C\right)\). The norm of an element \(\mathbf{q_{t}}\) (vector of length \(C\)) is given by:

  \[ \|\mathbf{q_t}\| = \|\mathbf{x_t} \mathbf{W_{q}}\| \]

  where \(\mathbf{x_t}\) is the input embedding of size \(C\) for the \(t\)-th token.

  In the unimodal case, the expected norm is constant:

  \[ E[\|\mathbf{q_t}\|^2] = \sigma_{x}^2 \cdot \|\mathbf{W_q}\|^2_{F} \]

  But in the multimodal case, we will have different variances for modalities of differing statistical entropy. For example, images tend to have less structure than text and are more variable (higher entropy), so we would expect the norms to be higher for query vectors from image tokens than word tokens; i.e. \( \sigma_{x_{i}} >\sigma_{x_{j}} \). This means they will dominate the attention mechanism when included alongside text. To compensate the model will increase the norms of the text tokens.

  But the softmax is a competitive, zero-sum objective. If we increase the norms of one modality, we increase their attention weight at the expense of other modalities. This dynamic can lead to explosive norm growth if the learning rate is too high. More importantly, the range of learning rates where this dynamic becomes explosive is wider for the unnormalized attention mechanism, as the work by [Wortsman et al (2023)](https://arxiv.org/abs/2309.14322 "Small-scale proxies for large-scale Transformer training instabilities") showed.

  For these reasons, a simple fix is to normalize the query and key vectors before the attention mechanism. A reference PyTorch implementation is provided below.

  ## Implementation

  ```
  import torch.nn as nn
  import torch.nn.functional as F

  class SelfAttention(nn.Module):
      def __init__(self, dim_size, n_heads, norm=RMSNorm, use_qk_norm=False):
          super(SelfAttention, self).__init__()

          self.dim_size = dim_size
          self.n_heads = n_heads
          self.head_dim = dim_size // n_heads

          self.qkv_attn = nn.Linear(dim_size, dim_size * 3, bias=False)
          self.project = nn.Linear(dim_size, dim_size, bias=False)
      
          self.use_qk_norm = use_qk_norm

          if self.use_qk_norm:
              self.q_norm = norm(self.head_dim)
              self.k_norm = norm(self.head_dim)

      def forward(self, x: torch.Tensor):
          B, T, C = x.size()

          qkv = self.qkv_attn(x)
          q, k, v = qkv.split(self.dim_size, dim=2)

          q = q.view(B, T, self.n_heads, self.head_dim)
          k = k.view(B, T, self.n_heads, self.head_dim)

          if self.use_qk_norm:
              q = self.q_norm(q)
              k = self.k_norm(k)

          v = v.view(B, T, self.n_heads, self.head_dim).transpose(1, 2)

          y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
          y = y.transpose(1, 2).contiguous().view(B, T, C)
          y = self.project(y)
          
          return y
  ```

  ## Thanks

  Thanks to Marcin Kardas and Michiel de Jong for proofreading this post.

  Thanks to [@assesseth](https://x.com/assesseth) for noticing a mistake with the mathematical derivation for the norms.

  If you find my writing interesting, you can follow me at [@rosstaylor90](https://x.com/rosstaylor90) on X.


title: Notes on the Membrane
slug: notes-on-the-membrane
date: 2024-12-09
body:
  The outer membrane of a neuron is a lipid bilayer approximately 7-10nm thick, separating the cell's interior from extracellular liquid. The concentration of ions differs within and outside the neuron, which creates a concentration gradient across the membrane. The typical ion concentrations for a mammalian neuron ([Izhikevich et al., 2004](https://pubmed.ncbi.nlm.nih.gov/15142958/ "Spike-timing dynamics of neuronal groups"), [Delpire et al, 2014](https://pmc.ncbi.nlm.nih.gov/articles/PMC4215762/ "Novel determinants of the neuronal Cl− concentration"), [Doyon et al, 2016](https://www.sciencedirect.com/science/article/pii/S0896627316001471 "Chloride Regulation: A Dynamic Equilibrium Crucial for Synaptic Inhibition")) are:

  - Sodium: \([Na^+]_{out} = 145 mM\), \([Na^+]_{in} = 5-15 mM\)
  - Potassium: \([K^+]_{out} = 5 mM\), \([K^+]_{in} = 140 mM\)
  - Chloride: \([Cl^-]_{out} = 110-130 mM\), \([Cl^-]_{in} = 3-15 mM\)

  The concentration gradient is maintained by the selective permeability of the membrane to different ions.

  The high intracellular concentration of potassium (opposite for sodium) is maintained via a **sodium-potassium pump**, where an ATPase enzyme exports three sodium ions and imports two potassium ions ([Delpire et al, 2014](https://pmc.ncbi.nlm.nih.gov/articles/PMC4215762/ "Novel determinants of the neuronal Cl− concentration")). This acts in the opposite direction of the concentration gradient.

  Chloride ions are maintained at a low intracellular level through a \(K^+-Cl^-\) cotransporter known as KCC2, which is a protein embedded in the membrane ([Ben-Ari et al, 2012](https://journals.sagepub.com/doi/10.1177/1073858412438697 "The GABA Excitatory/Inhibitory Shift in Brain Maturation and Neurological Disorders")). KCC2 moves \(Cl^{−}\) against its gradient by "piggybacking" on \(K^{+}\) as it moves down the \(K^{+}\) electrochemical gradient and out of the neuron.

  The unequal distribution of ions creates an electrical potential across the membrane. At thermal equilibrium, the potential for a specific ion is described by a Nernst potential:

  \[ E = \frac{RT}{zF} \ln{\left(\frac{[ion]_{out}}{[ion]_{in}}\right)} \]

  where \(R\) is the gas constant, \(T\) is the temperature, \(z\) is the ion's charge, \(F\) is the Faraday constant, and \([ion]_out\) and \([ion]_in\) are the concentrations of the ion outside and inside the cell.

  We can use this information to calculate the Nernst potential for sodium and potassium:

  ```executed
  import numpy as np

  # Concentrations (mM)
  Na_out = 145
  Na_in = 15 # we'll use upper end of concentration (commonly used)
  K_out = 5
  K_in = 140
  Cl_out = 120
  Cl_in = 9

  # Constants
  R = 8.314472  # Gas constant in J/(mol·K)
  T = 310.15  # Temperature in Kelvin (37°C) - body temperature
  F = 96485  # Faraday constant in C/mol
  Z = 1  # Valence; e.g. 1 for a monovalent ion such as K+

  def nernst_potential(ion_out, ion_in):
      return (R * T) / (Z * F) * np.log(ion_out / ion_in)

  E_Na = nernst_potential(Na_out, Na_in)
  E_Na_mV = E_Na * 1000  # Convert from volts to millivolts
  print(f"Nernst potential for Sodium (Na⁺): {E_Na_mV:.2f} mV")

  E_K = nernst_potential(K_out, K_in)
  E_K_mV = E_K * 1000  # Convert from volts to millivolts
  print(f"Nernst potential for Potassium (K⁺): {E_K_mV:.2f} mV")

  E_Cl = nernst_potential(Cl_out, Cl_in)
  E_Cl_mV = E_Cl * 1000  # Convert from volts to millivolts
  print(f"Nernst potential for Chloride (Cl⁻): {E_Cl_mV:.2f} mV")
  ---
  Nernst potential for Sodium (Na⁺): 60.63 mV
  Nernst potential for Potassium (K⁺): -89.06 mV
  Nernst potential for Chloride (Cl⁻): 69.23 mV
  ```

  So the difference in concentration creates a Nernst potential of \(E_{Na} \approx +60mV\), \(E_{K} \approx -90mV\) and \(E_{Cl} \approx +70mV\).

  To derive the resting membrane potential for all ions, we need to consider the permeability of each ion. Permeability is the capability of ions to flow across the membrane (whether they are moving or not). In contrast, conductance refers to the actual movement of charge across the membrane. The membrane is semipermeable as ion channels can be open or closed.

  The resting membrane potential \(E_m\) is given by the **Goldman equation**, which takes into account all of the ions which are permeant through the membrane:

  \[ E_m = \frac{RT}{F} \ln{\left(\frac{P_{K}[K^+]_{out} + P_{Na}[Na^+]_{out} + P_{Cl}[Cl^-]_{out}}{P_{K}[K^+]_{in} + P_{Na}[Na^+]_{in} + P_{Cl}[Cl^-]_{in}}\right)} \]

  At rest, the neuron is a state of dynamic equilibrum with ionic currents flowing across the membrane but balancing out ([Koch, 1999](https://academic.oup.com/book/40820/chapter-abstract/348792599 "Biophysics of Computation: Information Processing in Single Neurons")). The membrane in this state is more permeable to potassium ions as more channels are open. In a normal cell, the \(Na^+\) permeability is about 5% of \(K^+\) permeability or less ([Chrysafides et al, 2023](https://www.ncbi.nlm.nih.gov/books/NBK538338/ "Physiology, Resting Potential")).

  We can information on ionic permeability to calculate the resting membrane potential:

  ```executed
  import numpy as np

  # Concentrations (mM)
  Na_out = 145.0
  Na_in = 15.0 
  K_out = 5.0  
  K_in = 140.0 
  Cl_out = 120.0
  Cl_in = 9.0  

  # Relative permeabilities (Pion)
  P_K = 1.0   
  P_Na = 0.04 
  P_Cl = 0.4  # Example value; would need confirmation

  # Constants
  R = 8.314472    # Gas constant in J/(mol·K)
  T = 310.15      # Temperature in Kelvin (37°C)
  F = 96485.0     # Faraday's constant in C/mol

  RT_over_F = (R * T) / F # volts

  # Goldman equation
  # For Cl- the concentrations are inverted due to its negative charge
  numerator = (P_K * K_out) + (P_Na * Na_out) + (P_Cl * Cl_in)
  denominator = (P_K * K_in) + (P_Na * Na_in) + (P_Cl * Cl_out)

  Em = RT_over_F * np.log(numerator / denominator)
  Em_mV = Em * 1000

  print(f"Resting membrane potential (Em): {Em_mV:.2f} mV")
  ---
  Resting membrane potential (Em): -68.75 mV
  ```

  So the resting membrane potential is around \(−69mV\), meaning the neuron is negatively charged relative to the outside.

  The lipid bilayer is a nearly perfect electrical insulator; it can sustain voltages in excess of \(\pm{150mV}\) ([UPenn Bio251, 2003](https://www.sas.upenn.edu/LabManuals/BBB251/NIA/NEUROLAB/MEMB/lipidbil.htm "The bilayer membrane is a capacitor")). Since it separates intracellular and extracellular ionic (conducting) fluids, it is functionally equivalent to a capacitor. A capacitor stores electrical energy by accumulating charge on two conductors separated by an non-conductive region, which can be a vacuum or an electrical insulator such as the lipid bilayer.

  :::image_text_left
  ![alt](https://rossjtaylor.com/images/capacitor.png width=200)
  A charge on one conductor will induce a charge of equal magnitude but opposite sign on the other conductor, which means the dielectric develops an electric field. Its capacity to store charge is known as capacitance, or how much charge needs to be distributed across the membrane for a certain potential to build up.

  An ideal capacitor has a constant capacitance, measured in farads (\(F\)), and is defined as the ratio between the positive/negative charge \(Q\) on the conductors and the voltage \(V\) across the dielectric:

  \[ C=QV \]
  :::

  The relevant quantity for the membrane is the **membrane capacitance** \(C_m\), which is the capacitance per unit area of membrane. Capacitance affects how quickly the membrane potential changes (and the speed at which electrical signals propagate along the axon of the neuron). When the voltage across the capacitance changes, a current will flow.

  The membrane capacitance of a cell is traditionally seen as a "biological constant" ([Cole, 1969](https://www.degruyter.com/document/doi/10.1525/9780520326514/html?lang=en "Membranes, Ions and Impulses")) at around \(\sim{1} \mu{F}/\text{cm}^{2}\), but in fact it has been estimated at various values in the literature ([Gentet et al, 2000](https://www.sciencedirect.com/science/article/pii/S000634950076293X "Direct Measurement of Specific Membrane Capacitance in Neurons")). A histogram of various values is plotted below:

  ![Histogram of membrane capacitances](https://rossjtaylor.com/images/cmhistogram.png width=500 desc=Values of \(C_m\) from <a href='https://docs.google.com/spreadsheets/d/e/2PACX-1vSlxz0glbl_DMKIb3NJB6nSHV7yu2nH2M6LJVkS0l8D3aR0G05TirkST58ZxpFJmCpvjNsMBMtFTFjy/pubhtml' class='text-blue-700'>various studies</a>)

  The mean value is \(0.98 \mu{F}/cm^2\). Combining this with the resting potential of \(−69mV\), this means a hypothetical spherical cell of radius \(5-\mu{m}\) would store \(−0.22×10^{−12}\) coloumbs of charge:

  ```executed
  import math

  def calculate_stored_charge(radius_micrometers, resting_potential_mV):
      radius_m = radius_micrometers * 1e-6  # meters
      V = resting_potential_mV * 1e-3  # volts
      C_m = 0.98e-2  # F/m²

      # Calculate the surface area (A) of the spherical cell
      A = 4 * math.pi * radius_m**2  # m²

      C = C_m * A
      Q = C * V

      return Q

  radius_micrometers = 5  # µm
  resting_potential_mV = -69  # mV

  charge = calculate_stored_charge(radius_micrometers, resting_potential_mV)
  print(charge)
  ---
  -2.155132560362598e-13
  ```

  The importance of capacitance for neural dynamics is its effect on the neuron's time constant, which is the time it takes for the membrane potential to respond to inputs (specifically, to reach 63% of its steady state value). This can be calculated as:

  \[ \tau = R_mC_m \]

  where \(R_{m}\) is the membrane resistance and \(C_{m}\) is the membrane capacitance. The larger the capacitance, the slower the change in the membrane potential.

  The membrane resistance is the inverse of the membrane conductance, which is the sum of the conductances of all the ion channels in the membrane.

  The high resistivity of the lipids prevents charge from passing across the membrane, so from an electrical perspective, the properties of membranes can be solely described by capacitance ([Koch, 1999](https://academic.oup.com/book/40820/chapter-abstract/348792599 "Biophysics of Computation: Information Processing in Single Neurons")).


title: The Invention of Mathematics
slug: the-invention-of-mathematics
date: 2024-12-21
body:
  ## Counting, and numerical paucity

  Modern mathematics began with the need for counting. Counting was useful for several activities including counting livestock, bartering or marking the passage of days. However the notion of number may have been very rudimentary. Some cultures, such as Aboriginal cultures, only counted to two with bigger amounts classified under "many". This is also reflected in South American tribes along the Amazon, who could count to six, but had no independent number ways of groups of three or four ([Burton 2009](https://jontalle.web.engr.illinois.edu/uploads/298/HistoryMath-Burton.85.pdf "The History of Mathematics: An Introduction")). In additionally the Bushmen of South Africa could only count to ten in groups of two (10 = 2 + 2 + 2 + 2 + 2) (beyond 10 the descriptions became too long).

  ## Tallying

  Tally is the idea of matching the count of a target with some objects, such as shells or stones. Sheep could be counted by driving them through a narrow passage and dropping a stone for each one that passes through. Pebbles moved between one pile to another until all had been accounted for.

  A significant step in abstraction was moving away from tallying as a collection of objects, to notches on a permanent material - as a written memory device with some permancy. It helps abstract the notion of number as well as aid with communication. Scratches on stones, wooden sticks, or pieces of bone, or tying knots in strings. When tallies became too numerous, simplifying abstractions like grouping in groups of five (number of fingers on a hand). 

  Wolf Bone in Czechoslovakia, 55 deeply cut notches grouped in fives. Hunting tallies or lunar calendars.

  ![Ishango Bone](https://www.math.buffalo.edu/mad/Ancient-Africa/ishango_bone.jpg)

  Ishango Bone in 17,500 BC also thought to be a tally system.

  Near East tallying - and token system.

  Knots - ancient persia - darius I by herodotus:

  Commenting in his History, he informs us that the Persian
  king Darius handed the Ionians a knotted cord to serve as a calendar:

  The King took a leather thong and tying sixty knots in it called together the Ionian tyrants
  and spoke thus to them: “Untie every day one of the knots; if I do not return before the last day to which the knots will hold out, then leave your station and return to your several
  homes.”

  In the New World, the number string is best illustrated by the knotted cords, called
  quipus, of the Incas of Peru.When the Spanish conquerors arrived in the sixteenth century, they observed that each city in Peru had an “official of the knots,” who maintained complex accounts by
  means of knots and loops in strands of various colors. Performing duties not unlike
  those of the city treasurer of today, the quipu keepers recorded all of cial transactions
  concerning the land and subjects of the city and submitted the strings to the central
  government in Cuzco. T

  The quipu was made of
  a thick main cord or crossbar to which were attached ner cords of different lengths and
  colors; ordinarily the cords hung down like the strands of a mop. Each of the pendent
  strings represented a certain item to be tallied; one might be used to show the number
  of sheep, for instance, another for goats, and a third for lambs. The knots themselves
  indicated numbers, the values of which varied according to the type of knot used and its
  speci c position on the strand. A decimal system was used, with the knot representing
  units placed nearest the bottom, the tens appearing immediately above, then the hundreds,
  and so on; absence of a knot denoted zero. Bunches of cords were tied off by a single
  main thread, a summation cord, whose knots gave the total count for each bunch. The
  range of possibilities for numerical representation in the quipus allowed the Incas to keep
  incredibly detailed administrative records, despite their ignorance of the written word.

  To appreciate the quipu fully, we should notice the numerical values represented by
  the tied knots. Just three types of knots were used: a gure-eight knot standing for 1, a
  long knot denoting one of the values 2 through 9, depending on the number of twists
  in the knot, and a single knot also indicating 1. The gure-eight knot and long knot
  appear only in the lowest (units) position on a cord, while clusters of single knots can
  appear in the other spaced positions. Because pendant cords have the same length, an
  empty position (a value of zero) would be apparent on comparison with adjacent cords.

  Also, the reappearance of either a gure-eight or long knot would point out that another
  number is being recorded on the same cord.
  Recalling that ascending positions carry place value for successive powers of ten, let
  us suppose that a particular cord contains the following, in order: a long knot with four
  twists, two single knots, an empty space, seven clustered single knots, and one single
  knot. For the Inca, this array would represent the number
  17024 D 4 C (2 Ð 10) C (0 Ð 102
  ) C (7 Ð 103
  ) C (1 Ð 104
  ):

  Over the long sweep of history, it seems clear that progress in devising ef cient
  ways of retaining and conveying numerical information did not take place until primitive
  people abandoned the nomadic life. Incised markings on bone or stone may have been
  adequate for keeping records when human beings were hunters and gatherers, but the food
  producer required entirely new forms of numerical representation. Besides, as a means
  for storing information, groups of markings on a bone would have been intelligible only
  to the person making them, or perhaps to close friends or relatives; thus, the record was
  probably not intended to be used by people separated by great distances.

  Deliberate cultivation of crops, particularly cereal grains, and the domestication of
  animals began, so far as can be judged from present evidence, in the Near East some 10,000 years ago. Later experiments in agriculture occurred in China and in the New
  World. A widely held theory is that a climatic change at the end of the last ice age
  provided the essential stimulus for the introduction of food production and a settled
  village existence. As the polar ice cap began to retreat, the rain belt moved northward,
  causing the desiccation of much of the Near East. The increasing scarcity of wild food
  plants and the game on which people had lived forced them, as a condition of survival,
  to change to an agricultural life. It became necessary to count one’s harvest and herd, to
  measure land, and to devise a calendar that would indicate the proper time to plant crops.
  Even at this stage, the need for a means of counting was modest; and tallying techniques,
  although slow and cumbersome, were still adequate for ordinary dealings. But with a
  more secure food supply came the possibility of a considerable increase in population,
  which meant that larger collections of objects had to be enumerated. Repetition of some
  fundamental mark to record a tally led to inconvenient numeral representations, tedious
  to compose and dif cult to interpret. The desire of village, temple, and palace of cials to
  maintain meticulous records (if only for the purposes of systematic taxation) gave further
  impetus to nding new and more re ned means of “ xing” a count in a permanent or
  semipermanent form. 

  Thus, it was in the more elaborate life of those societies that rose to power some
  6000 years ago in the broad river valleys of the Nile, the Tigris-Euphrates, the Indus,
  and the Yangtze that special symbols for numbers rst appeared. From these, some of
  our most elementary branches of mathematics arose, because a symbolism that would
  allow expressing large numbers in written numerals was an essential prerequisite for
  computation and measurement. Through a welter of practical experience with number
  symbols, people gradually recognized certain abstract principles; for instance, it was
  discovered that in the fundamental operation of addition, the sum did not depend on the
  order of the summands. Such discoveries were hardly the work of a single individual,
  or even a single culture, but more a slow process of awareness moving toward an
  increasingly abstract way of thinking.


title: Synthetic Data: Escaping the Ouroboros
slug: synthetic-data-escaping-the-ouroboros
date: 2023-04-21
body:
  :::image_text_left
  ![alt](https://upload.wikimedia.org/wikipedia/commons/7/71/Serpiente_alquimica.jpg)
  Synthetic data seems hard to grasp intuitively. How can a model trained on its own outputs improve its performance? The key idea is to introduce **side information** to transform the proposal model distribution to a more desirable distribution. So let's see how we can escape the Ouroboros and achieve meaningful improvement.
  :::

  Let's assume we have access to a language model policy \(\pi\) that can generate text. We can measure the performance of the model on a particular task, e.g. mathematical reasoning, as \(P_{\pi}\left(X\right)\), the probability of predicting the right answer \(X\).

  The synthetic data question can then be posed as follows:

  **Can we find an improved policy \(\pi^{new}\) by sampling from \(P_{\pi}\) and re-training on that data?**

  The answer is yes, but only if our synthetic data procedure leads to a desirable distributional shift in our model: i.e. \(P_{\pi^{new}}\left(X\right) > P_{\pi}\left(X\right)\).

  To achieve this distributional shift, we need to introduce side information \(I\) that can allow us to use the proposal distribution \(P_{\pi}\left(X\right)\) to sample from a desired distribution \(P_{\pi^{new}}\left(X\right)\).

  The value of the side information is the mutual information relative to the original policy. Write \(P_{\pi}(X\mid{I}) = P_{\pi^*}\left(X\right)\), then:

  \[ I(X; I) = \mathbb{E}_{I}\left[D_{KL}\left(P_{\pi}(X\mid{I})\mid\mid{P}_{\pi}(X)\right)\right] \]

  If we have no side information then the KL divergence is zero and the value is zero: this explains why some people find synthetic data unintuitive, because they assume there is no side information being introduced. In contrast, if we can use side information and the original policy to efficiently sample examples of a better policy, then useful synthetic data is possible.


title: The Invention Curriculum
slug: beginning-of-invention
date: 2024-12-22
body:
  *"Let us create a theoretical city from the beginning. Its real creator, as it appears, will be our needs." - Plato*

  The next frontier for AI is invention. Invention requires creativity, the ability to come up with new ideas, and execution, the ability to implement those ideas in the real world. In both senses, current AI systems fall short. Models have shown glimpses of creativity in [narrow domains like games](https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol "AlphaGo versus Lee Sedol - Wikipedia"), but the environment is highly constrained. More  general systems like LLMs have shown much more limited creativity, and yet are similarly constrained to environments such as chat interfaces and code editors.

  To crack invention we will need to get creative. But what makes humans creative? What makes humans good at invention where existing AI systems fail?

  The key is a rich environment that generates problems of incrementally increasing complexity. Seen on a grand scale, civilisation has presented humans with a curriculum of problems to solve, incrementally increasing in complexity, where each step has never been too hard to solve. In short: an **invention curriculum**.

  In this essay I will focus on the invention of writing as an example of this process. The emergence of writing was deeply connected to socio-economic change, from the agricultural revolution to the first large urban settlements. At each stage of development, problems arose incrementally as social complexity increased. Writing was developed in tandem to solve these problems as they appeared.

  A core contention of this work is that invention is incremental. Inventions like writing did not appear *ex nihilo*: they emerged gradually as a series of refinements that tracked the increasing complexity of the environment. This is an important insight because it relieves the creative demands on a human to solve the invention curriculum. If a leap is too large, let's say from a tally stick from a full writing system, then this would require an unreasonable amount of imagination to solve. And if invention is premature, it would not solve a pressing problem for society and would not persist as an idea.

  But this is a broad topic: the environment alone is insufficient for invention due to intelligence constraints. Humans have civilisation; dogs do not. So we begin by discussing the latent capabilities of humans. Then we focus on civilisational progress with a focus on writing: starting from counting in hunter-gather societies, token systems following the agricultural revolution, and then writing. We conclude with lessons for AI: what can we learn about the invention curriculum in developing the next generation of models and environments.

  ## The Dawn of Invention

  ![2001 Space Odyssey](https://i.ibb.co/fCKRrHZ/Screenshot-2024-12-23-at-07-02-35.png desc=The Dawn of Man from 2001: A Space Odyssey)

  In [2001: A Space Odyssey](https://en.wikipedia.org/wiki/2001:_A_Space_Odyssey), a tribe of hominids is driven from a watering hole by a rival group. With widespread drought and famine, their hopes for survival are slim. But the next dawn, something miraculous happens. A black, rectangular monolith appears from nowhere. The tribe approaches the object with fear. But slowly their fear turns to awe and wonder. One by one, they extend their hands to touch the cold, black surface. They gaze upwards and stare at the monolith's edge intersecting the rising sun.

  The encounter proves transformative. The next day a hominid plays aimlessly with a bone near an animal carcass. Suddenly, he makes a leap of imagination: the bone can be used as a weapon. Armed with this new invention, the tribe confronts their rivals and defeats them in combat. The tool is thrown to the air in celebration, and the scene cuts to a tool-shaped spaceship floating in space. Humanity has ascended.

  Putting aside its cinematic qualities, Kubrick's vision of invention illustrates several components that are worth analysing. When did invention begin in humans? Was it a sudden spark of inspiration? A Eureka moment? Was it triggered by conflict over resources? How important was the role of the environment? And what makes humans so special in this story?

  Humans are the main characters in this story because there was a selective pressure for invention. That is, environmental factors rewarded inventiveness. But we are not so special. Chimpanzees in the Ivory Coast use stones and sticks to pound open nuts ([Boesch 1996](https://www.thebritishacademy.ac.uk/documents/3956/88p251.pdf "The Emergence of Cultures among Wild Chimpanzees")). This shows they understand abstract properties of objects that don't look alike. So we should also consider the longer story of our evolution as primates.

  A full account of the evolution of primate brains would be too lengthy for this post, so I refer the reader to [Murray et al, 2017](https://global.oup.com/academic/product/the-evolution-of-memory-systems-9780198817130?cc=us&lang=en& "The Evolution of Memory Systems: Ancestors, Anatomy, and Adaptations") and [Passingham, 2021](https://www.amazon.co.uk/Understanding-Prefrontal-Cortex-connectivity-operations-ebook "Understanding the Prefrontal Cortex: Selective advantage, connectivity, and neural operations") for accessible and thoughtful accounts. In short, our ancestors originally foraged for food in the fine-branch niche. Eventually they came to the ground to forage, but patchy resources and heat stress meant foraging periods were short. This created selective pressure for efficient problem-solving and planning. These pressures were accelerated by environmental changes such as global cooling events and competition from other animals. Food scarcity and volatility, along with the need to forage over longer distances, created further selective pressure for human intelligence.

  These selective pressures led to upward grade shifts in the size of primate brains. When looking at grade shifts in brain volume we can see a clear upward shift from strepsirrhines (▲) to modern anthropoids (+):

  ![Gradeshifts](https://i.ibb.co/1npNCd7/Screenshot-2024-12-23-at-07-48-42.png width=400 desc=From <a href="https://anatomypubs.onlinelibrary.wiley.com/doi/10.1002/ar.a.20113">Bush et al (2004)</a>)



  at is the best measure for the cognitive capacity of an animal; some insist on absolute measures [Gibson et al, 2001](https://www.researchgate.net/publication/344507167_Bigger_is_better_primate_brain_size_in_relationship_to_cognition "Bigger is better: primate brain size in relationship to cognition
  "). Others argue for relative measures such as the [Encephalization quotient](https://en.wikipedia.org/wiki/Encephalization_quotient "Encephalization quotient -Wikipedia"), on the basis that larger animals usually have larger brains which reflect a need for more capacity to control the body (not more sophisticated cognition).


  When looking at relative measures, we can see grade-shifts in brain volume in the fossil record. ([Passingham, 2021](https://www.amazon.co.uk/Understanding-Prefrontal-Cortex-connectivity-operations-ebook "Understanding the Prefrontal Cortex: Selective advantage, connectivity, and neural operations")). 

  ![Neocortical Scaling](https://i.ibb.co/jMDvKGy/Screenshot-2024-12-22-at-11-54-24.png desc="Grade shift for anthropoids")

  We also see scaling over time of the neocortical ratio - the surface area of neocortex divided by the surface area of the brain - where the curve is much higher and steeper for primates than other mammals.



  How does this compare to AI? Is it right to think of the architecture as the weight connections, and societal influences as connection strengths? Well not quite again, as even humans have latent ideas. Beavers famously have an urge to built when they hear flowing water.

  I would argue the lesson argues in a combination of neocortical and specifically prefrontal prefrontal capabilities, in particular planning (simulation), generation and search. The size of the human brain helps with all of this. (Working memory: chimps vs humans?) -> The ability to simulate is imagination; different from trial-and-error reinfrocement learning. Goal memory.

  Chimps can use tool use rudimentary, but scale up the connections to get it into takeoff mode.

  ## Counting

  ## Simple Tokens

  ## Complex Tokens

  ## Later Mathematics

  ## Conclusion and Lessons for AI


# blog_postreference

inline_text: Murray et al, 2017
reference_title: The Evolution of Memory Systems: Ancestors, Anatomy, and Adaptations
url:
  https://global.oup.com/academic/product/the-evolution-of-memory-systems-9780198817130?cc=us&lang=en&


inline_text: Gibson et al, 2001
reference_title: Bigger is better: primate brain size in relationship to cognition
url:
  https://www.researchgate.net/publication/344507167_Bigger_is_better_primate_brain_size_in_relationship_to_cognition


# blog_postsection

section_type: text
body:
  Multimodality has led to some tweaks to the standard Transformer recipe. In this post we will cover [QK Normalization](https://arxiv.org/abs/2305.18234 "Query-Key Normalization for Transformers") (QKNorm), where we apply normalization to the query and key vectors in the attention mechanism. This has become an important step for unified input models where we have tokenized representations of text, images and other modalities present in a single sequence. These types of models are very prone to training instability without appropriate normalization, as the norms of the Q and K vectors can experience exponential growth during training. QKNorm is a necessary (although perhaps not sufficient) step to correct this problem.


section_type: text
body:
  The softmax is invariant to translation. If we add a constant \(c\) to each element of the input vector \(\mathbf{x}\) then we can see that:


section_type: text
body:
  \[ \text{softmax}(\mathbf{x} + c) = \frac{e^{\mathbf{x} + c}}{\sum_j e^{x_j + c}} = \frac{e^c \cdot e^{\mathbf{x}}}{e^c \cdot \sum_j e^{x_j}} = \frac{e^{\mathbf{x}}}{\sum_j e^{x_j}} = \text{softmax}(\mathbf{x}) \]


section_type: text
body:
  The invariance property is a potential source of training instability when combined with unnormalized query and key vectors.


section_type: text
body:
  Since additions have no impact on the relative magnitudes of the logits, the only way for the model to increase the relative distance between logits is by increasing or decreasing the norms of the \(\mathbf{q}\) and \(\mathbf{k}\) vectors, or adjusting the angle \(\cos\left(\theta_{ij}\right)\). It may be easier for the model to learn to increase the norms, as this requires uniform scaling of weights, as opposed to changing the angles which requires coordinated changes in multiple weight parameters.


section_type: text
body:
  However, uncontrolled growth in the norms of these vectors can lead to instabilities in training, as large logits will collapse attention weights to one-hot vectors - "attention entropy collapse" ([Dehghani et al 2023](https://arxiv.org/abs/2302.05442 "Scaling Vision Transformers to 22 Billion Parameters"), [Zhai et al 2023](https://arxiv.org/abs/2303.06296 "Stabilizing Transformer Training by Preventing Attention Entropy Collapse")). If the training setup is overly sensitive to some hyperparameters, such as a high learning rate, then norms will be pushed into the exponential region of the softmax. This is the case discussed by [Wortsman et al (2023)](https://arxiv.org/abs/2309.14322 "Small-scale proxies for large-scale Transformer training instabilities").


section_type: text
body:
  Recent empirical evidence suggests that introducing multimodal inputs increases hyperparameter instability, and makes training more prone to norm explosion. We cover the relevant literature in the next section.


section_type: text
body:
  [Dehghani et al (2023)](https://arxiv.org/abs/2302.05442 "Scaling Vision Transformers to 22 Billion Parameters") observed divergent training with an 8B Vision Transformer. The loss decreased as normal to begin with but then steadily increased after 2,000 steps. Without normalization they found that attention logits grew to over 50,000 in magnitude, resulting in one-hot attention weights post-softmax and subsequently unstable losses and gradients:


section_type: text
body:
  [Wortsman et al (2023)](https://arxiv.org/abs/2309.14322 "Small-scale proxies for large-scale Transformer training instabilities") found they could obtain attention logit instability with smaller models using higher learning rates. As a solution, they found that QKNorm enabled stable training across three orders of magnitude of learning rate (LR) variation.


section_type: text
body:
  [Lu et al (2023)](https://arxiv.org/abs/2312.17172v1 "Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action") experienced unstable training as they integrated additional modalities into their UnifiedIO architecture. They observed extremely large values in the multi-head attention logits when including image and audio modalities, leading to one-hot attention weights. They used QKNorm to stabilise training.


section_type: text
body:
  [Chameleon Team (2024)](https://arxiv.org/abs/2405.09818 "Chameleon: Mixed-Modal Early-Fusion Foundation Models") experienced instabilities in the mid-to-late stages of training which they attributed to uncontrolled norm growth. They attribute norm growth to "norm competition" between modalities of differing entropy, which is encouraged by softmax translation invariance. They note this becomes problematic once training reaches outside the effective representation range of bfloat16. They do not observe the same problem for the unimodal text-only setting.


section_type: text
body:
  Interestingly, they found that QKNorm alone was not sufficient for stabilising training with the LLaMA architecture. They opt for a Swin Transformer normalization strategy, which normalizes on the outside of the feedforward block (helping bound its norm growth).


section_type: text
body:
  Some other recent work includes OLMoE by [Muennighoff et al (2024)](https://arxiv.org/abs/2409.02060 "OLMoE: Open Mixture-of-Experts Language Models"), where they find QKNorm increases training stability (at the cost of 10% throughput). Additionally [Ramapuram et al (2024)](https://arxiv.org/abs/2409.04431 "Theory, Analysis, and Best Practices for Sigmoid Self-Attention") find QKNorm significantly stabilises performance by making Sigmoid Attention and Softmax Attention less sensitive to learning rate changes.


section_type: text
body:
  Let's look at the \(\mathbf{q}\) and \(\mathbf{k}\) vectors again. Both are \(\left(B,T,C\right)\) size vectors constructed via linear layers (omitting biases):


section_type: text
body:
  where \(x\) is the input embedding. Let's restrict ourselves to looking at query vectors for now for a single batch:


section_type: text
body:
  where \(x\) is of size \(\left(T,C\right)\) and \(\mathbf{W_q}\) is of size \(\left(C, C\right)\). The norm of an element \(\mathbf{q_{t}}\) (vector of length \(C\)) is given by:


section_type: text
body:
  But in the multimodal case, we will have different variances for modalities of differing statistical entropy. For example, images tend to have less structure than text and are more variable (higher entropy), so we would expect the norms to be higher for query vectors from image tokens than word tokens; i.e. \( \sigma_{x_{i}} >\sigma_{x_{j}} \). This means they will dominate the attention mechanism when included alongside text. To compensate the model will increase the norms of the text tokens.


section_type: text
body:
  But the softmax is a competitive, zero-sum objective. If we increase the norms of one modality, we increase their attention weight at the expense of other modalities. This dynamic can lead to explosive norm growth if the learning rate is too high. More importantly, the range of learning rates where this dynamic becomes explosive is wider for the unnormalized attention mechanism, as the work by [Wortsman et al (2023)](https://arxiv.org/abs/2309.14322 "Small-scale proxies for large-scale Transformer training instabilities") showed.


section_type: text
body:
  For these reasons, a simple fix is to normalize the query and key vectors before the attention mechanism. A reference PyTorch implementation is provided below.


section_type: code
body:
  <pre><code>import torch.nn as nn
  import torch.nn.functional as F

  class SelfAttention(nn.Module):
      def __init__(self, dim_size, n_heads, norm=RMSNorm, use_qk_norm=False):
          super(SelfAttention, self).__init__()

          self.dim_size = dim_size
          self.n_heads = n_heads
          self.head_dim = dim_size // n_heads

          self.qkv_attn = nn.Linear(dim_size, dim_size * 3, bias=False)
          self.project = nn.Linear(dim_size, dim_size, bias=False)
      
          self.use_qk_norm = use_qk_norm

          if self.use_qk_norm:
              self.q_norm = norm(self.head_dim)
              self.k_norm = norm(self.head_dim)

      def forward(self, x: torch.Tensor):
          B, T, C = x.size()

          qkv = self.qkv_attn(x)
          q, k, v = qkv.split(self.dim_size, dim=2)

          q = q.view(B, T, self.n_heads, self.head_dim)
          k = k.view(B, T, self.n_heads, self.head_dim)

          if self.use_qk_norm:
              q = self.q_norm(q)
              k = self.k_norm(k)

          v = v.view(B, T, self.n_heads, self.head_dim).transpose(1, 2)

          y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
          y = y.transpose(1, 2).contiguous().view(B, T, C)
          y = self.project(y)
          
          return y</code></pre>


section_type: text
body:
  Thanks to [@assesseth](https://x.com/assesseth) for noticing a mistake with the mathematical derivation for the norms.


section_type: text
body:
  If you find my writing interesting, you can follow me at [@rosstaylor90](https://x.com/rosstaylor90) on X.


section_type: text
body:
  The outer membrane of a neuron is a lipid bilayer approximately 7-10nm thick, separating the cell's interior from extracellular liquid. The concentration of ions differs within and outside the neuron, which creates a concentration gradient across the membrane. The typical ion concentrations for a mammalian neuron ([Izhikevich et al., 2004](https://pubmed.ncbi.nlm.nih.gov/15142958/ "Spike-timing dynamics of neuronal groups"), [Delpire et al, 2014](https://pmc.ncbi.nlm.nih.gov/articles/PMC4215762/ "Novel determinants of the neuronal Cl− concentration"), [Doyon et al, 2016](https://www.sciencedirect.com/science/article/pii/S0896627316001471 "Chloride Regulation: A Dynamic Equilibrium Crucial for Synaptic Inhibition")) are:


section_type: text
body:
  <ul class="list-disc text-gray-700 leading-relaxed list-inside text-xl p-2 pb-4 pt-4">
  <li>Sodium: \([Na^+]_{out} = 145 mM\), \([Na^+]_{in} = 5-15 mM\)</li>
  <li>Potassium: \([K^+]_{out} = 5 mM\), \([K^+]_{in} = 140 mM\)</li>
  <li>Chloride: \([Cl^-]_{out} = 110-130 mM\), \([Cl^-]_{in} = 3-15 mM\)</li>
  </ul>


section_type: text
body:
  The concentration gradient is maintained by the selective permeability of the membrane to different ions.


section_type: text
body:
  The high intracellular concentration of potassium (opposite for sodium) is maintained via a <strong>sodium-potassium pump</strong>, where an ATPase enzyme exports three sodium ions and imports two potassium ions ([Delpire et al, 2014](https://pmc.ncbi.nlm.nih.gov/articles/PMC4215762/ "Novel determinants of the neuronal Cl− concentration")). This acts in the opposite direction of the concentration gradient.


section_type: text
body:
  Chloride ions are maintained at a low intracellular level through a \(K^+-Cl^-\) cotransporter known as KCC2, which is a protein embedded in the membrane ([Ben-Ari et al, 2012](https://journals.sagepub.com/doi/10.1177/1073858412438697 "The GABA Excitatory/Inhibitory Shift in Brain Maturation and Neurological Disorders")). KCC2 moves \(Cl^{−}\) against its gradient by "piggybacking" on \(K^{+}\) as it moves down the \(K^{+}\) electrochemical gradient and out of the neuron.


section_type: text
body:
  The unequal distribution of ions creates an electrical potential across the membrane. At thermal equilibrium, the potential for a specific ion is described by a Nernst potential:


section_type: text
body:
  where \(R\) is the gas constant, \(T\) is the temperature, \(z\) is the ion's charge, \(F\) is the Faraday constant, and \([ion]_out\) and \([ion]_in\) are the concentrations of the ion outside and inside the cell.


section_type: executed_code
body:
  import numpy as np

  # Concentrations (mM)
  Na_out = 145
  Na_in = 15 # we'll use upper end of concentration (commonly used)
  K_out = 5
  K_in = 140
  Cl_out = 120
  Cl_in = 9

  # Constants
  R = 8.314472  # Gas constant in J/(mol·K)
  T = 310.15  # Temperature in Kelvin (37°C) - body temperature
  F = 96485  # Faraday constant in C/mol
  Z = 1  # Valence; e.g. 1 for a monovalent ion such as K+

  def nernst_potential(ion_out, ion_in):
      return (R * T) / (Z * F) * np.log(ion_out / ion_in)

  E_Na = nernst_potential(Na_out, Na_in)
  E_Na_mV = E_Na * 1000  # Convert from volts to millivolts
  print(f"Nernst potential for Sodium (Na⁺): {E_Na_mV:.2f} mV")

  E_K = nernst_potential(K_out, K_in)
  E_K_mV = E_K * 1000  # Convert from volts to millivolts
  print(f"Nernst potential for Potassium (K⁺): {E_K_mV:.2f} mV")

  E_Cl = nernst_potential(Cl_out, Cl_in)
  E_Cl_mV = E_Cl * 1000  # Convert from volts to millivolts
  print(f"Nernst potential for Chloride (Cl⁻): {E_Cl_mV:.2f} mV")

code_output:
  Nernst potential for Sodium (Na⁺): 60.63 mV
  Nernst potential for Potassium (K⁺): -89.06 mV
  Nernst potential for Chloride (Cl⁻): 69.23 mV


section_type: text
body:
  So the difference in concentration creates a Nernst potential of \(E_{Na} \approx +60mV\), \(E_{K} \approx -90mV\) and \(E_{Cl} \approx +70mV\).


section_type: text
body:
  To derive the resting membrane potential for all ions, we need to consider the permeability of each ion. Permeability is the capability of ions to flow across the membrane (whether they are moving or not). In contrast, conductance refers to the actual movement of charge across the membrane. The membrane is semipermeable as ion channels can be open or closed.


section_type: text
body:
  The resting membrane potential \(E_m\) is given by the <strong>Goldman equation</strong>, which takes into account all of the ions which are permeant through the membrane:


section_type: text
body:
  \[ E_m = \frac{RT}{F} \ln{\left(\frac{P_{K}[K^+]_{out} + P_{Na}[Na^+]_{out} + P_{Cl}[Cl^-]_{out}}{P_{K}[K^+]_{in} + P_{Na}[Na^+]_{in} + P_{Cl}[Cl^-]_{in}}\right)} \]


section_type: text
body:
  At rest, the neuron is a state of dynamic equilibrum with ionic currents flowing across the membrane but balancing out ([Koch, 1999](https://academic.oup.com/book/40820/chapter-abstract/348792599 "Biophysics of Computation: Information Processing in Single Neurons")). The membrane in this state is more permeable to potassium ions as more channels are open. In a normal cell, the \(Na^+\) permeability is about 5% of \(K^+\) permeability or less ([Chrysafides et al, 2023](https://www.ncbi.nlm.nih.gov/books/NBK538338/ "Physiology, Resting Potential")).


section_type: executed_code
code_output: Resting membrane potential (Em): -68.75 mV
body:
  import numpy as np

  # Concentrations (mM)
  Na_out = 145.0
  Na_in = 15.0 
  K_out = 5.0  
  K_in = 140.0 
  Cl_out = 120.0
  Cl_in = 9.0  

  # Relative permeabilities (Pion)
  P_K = 1.0   
  P_Na = 0.04 
  P_Cl = 0.4  # Example value; would need confirmation

  # Constants
  R = 8.314472    # Gas constant in J/(mol·K)
  T = 310.15      # Temperature in Kelvin (37°C)
  F = 96485.0     # Faraday's constant in C/mol

  RT_over_F = (R * T) / F # volts

  # Goldman equation
  # For Cl- the concentrations are inverted due to its negative charge
  numerator = (P_K * K_out) + (P_Na * Na_out) + (P_Cl * Cl_in)
  denominator = (P_K * K_in) + (P_Na * Na_in) + (P_Cl * Cl_out)

  Em = RT_over_F * np.log(numerator / denominator)
  Em_mV = Em * 1000

  print(f"Resting membrane potential (Em): {Em_mV:.2f} mV")


section_type: text
body:
  So the resting membrane potential is around \(−69mV\), meaning the neuron is negatively charged relative to the outside.


section_type: text
body:
  The lipid bilayer is a nearly perfect electrical insulator; it can sustain voltages in excess of \(\pm{150mV}\) ([UPenn Bio251, 2003](https://www.sas.upenn.edu/LabManuals/BBB251/NIA/NEUROLAB/MEMB/lipidbil.htm "The bilayer membrane is a capacitor")). Since it separates intracellular and extracellular ionic (conducting) fluids, it is functionally equivalent to a capacitor. A capacitor stores electrical energy by accumulating charge on two conductors separated by an non-conductive region, which can be a vacuum or an electrical insulator such as the lipid bilayer.


section_type: image_text
image: post_images/capacitor_GzA0HbR.png
image_position: left
body:
  A charge on one conductor will induce a charge of equal magnitude but opposite sign on the other conductor, which means the dielectric develops an electric field. Its capacity to store charge is known as capacitance, or how much charge needs to be distributed across the membrane for a certain potential to build up.
  An ideal capacitor has a constant capacitance, measured in farads (\(F\)), and is defined as the ratio between the positive/negative charge \(Q\) on the conductors and the voltage \(V\) across the dielectric:
  \[ C=QV \]


section_type: text
body:
  The relevant quantity for the membrane is the <strong>membrane capacitance</strong> \(C_m\), which is the capacitance per unit area of membrane. Capacitance affects how quickly the membrane potential changes (and the speed at which electrical signals propagate along the axon of the neuron). When the voltage across the capacitance changes, a current will flow.


section_type: text
body:
  The membrane capacitance of a cell is traditionally seen as a "biological constant" ([Cole, 1969](https://www.degruyter.com/document/doi/10.1525/9780520326514/html?lang=en "Membranes, Ions and Impulses")) at around \(\sim{1} \mu{F}/\text{cm}^{2}\), but in fact it has been estimated at various values in the literature ([Gentet et al, 2000](https://www.sciencedirect.com/science/article/pii/S000634950076293X "Direct Measurement of Specific Membrane Capacitance in Neurons")). A histogram of various values is plotted below:


section_type: image
image: post_images/cmhistogram_E9CVdY9.png
image_description:
  Values of \(C_m\) from <a href='https://docs.google.com/spreadsheets/d/e/2PACX-1vSlxz0glbl_DMKIb3NJB6nSHV7yu2nH2M6LJVkS0l8D3aR0G05TirkST58ZxpFJmCpvjNsMBMtFTFjy/pubhtml' class='text-blue-700'>various studies</a>


section_type: text
body:
  The mean value is \(0.98 \mu{F}/cm^2\). Combining this with the resting potential of \(−69mV\), this means a hypothetical spherical cell of radius \(5-\mu{m}\) would store \(−0.22×10^{−12}\) coloumbs of charge:


section_type: executed_code
code_output: -2.155132560362598e-13
body:
  import math

  def calculate_stored_charge(radius_micrometers, resting_potential_mV):
      radius_m = radius_micrometers * 1e-6  # meters
      V = resting_potential_mV * 1e-3  # volts
      C_m = 0.98e-2  # F/m²

      # Calculate the surface area (A) of the spherical cell
      A = 4 * math.pi * radius_m**2  # m²

      C = C_m * A
      Q = C * V

      return Q

  radius_micrometers = 5  # µm
  resting_potential_mV = -69  # mV

  charge = calculate_stored_charge(radius_micrometers, resting_potential_mV)
  print(charge)


section_type: text
body:
  The importance of capacitance for neural dynamics is its effect on the neuron's time constant, which is the time it takes for the membrane potential to respond to inputs (specifically, to reach 63% of its steady state value). This can be calculated as:


section_type: text
body:
  where \(R_{m}\) is the membrane resistance and \(C_{m}\) is the membrane capacitance. The larger the capacitance, the slower the change in the membrane potential.


section_type: text
body:
  The membrane resistance is the inverse of the membrane conductance, which is the sum of the conductances of all the ion channels in the membrane.


section_type: text
body:
  The high resistivity of the lipids prevents charge from passing across the membrane, so from an electrical perspective, the properties of membranes can be solely described by capacitance ([Koch, 1999](https://academic.oup.com/book/40820/chapter-abstract/348792599 "Biophysics of Computation: Information Processing in Single Neurons")).


section_type: text
body:
  Modern mathematics began with the need for counting. Counting was useful for several activities including counting livestock, bartering or marking the passage of days. However the notion of number may have been very rudimentary. Some cultures, such as Aboriginal cultures, only counted to two with bigger amounts classified under "many". This is also reflected in South American tribes along the Amazon, who could count to six, but had no independent number ways of groups of three or four ([Burton 2009](https://jontalle.web.engr.illinois.edu/uploads/298/HistoryMath-Burton.85.pdf "The History of Mathematics: An Introduction")). In additionally the Bushmen of South Africa could only count to ten in groups of two (10 = 2 + 2 + 2 + 2 + 2) (beyond 10 the descriptions became too long).


section_type: text
body:
  Tally is the idea of matching the count of a target with some objects, such as shells or stones. Sheep could be counted by driving them through a narrow passage and dropping a stone for each one that passes through. Pebbles moved between one pile to another until all had been accounted for.


section_type: text
body:
  A significant step in abstraction was moving away from tallying as a collection of objects, to notches on a permanent material - as a written memory device with some permancy. It helps abstract the notion of number as well as aid with communication. Scratches on stones, wooden sticks, or pieces of bone, or tying knots in strings. When tallies became too numerous, simplifying abstractions like grouping in groups of five (number of fingers on a hand).


section_type: text
body:
  Wolf Bone in Czechoslovakia, 55 deeply cut notches grouped in fives. Hunting tallies or lunar calendars.


section_type: text
body:
  Commenting in his History, he informs us that the Persian
  king Darius handed the Ionians a knotted cord to serve as a calendar:


section_type: text
body:
  The King took a leather thong and tying sixty knots in it called together the Ionian tyrants
  and spoke thus to them: “Untie every day one of the knots; if I do not return before the last day to which the knots will hold out, then leave your station and return to your several
  homes.”


section_type: text
body:
  In the New World, the number string is best illustrated by the knotted cords, called
  quipus, of the Incas of Peru.When the Spanish conquerors arrived in the sixteenth century, they observed that each city in Peru had an “official of the knots,” who maintained complex accounts by
  means of knots and loops in strands of various colors. Performing duties not unlike
  those of the city treasurer of today, the quipu keepers recorded all of cial transactions
  concerning the land and subjects of the city and submitted the strings to the central
  government in Cuzco. T


section_type: text
body:
  The quipu was made of
  a thick main cord or crossbar to which were attached ner cords of different lengths and
  colors; ordinarily the cords hung down like the strands of a mop. Each of the pendent
  strings represented a certain item to be tallied; one might be used to show the number
  of sheep, for instance, another for goats, and a third for lambs. The knots themselves
  indicated numbers, the values of which varied according to the type of knot used and its
  speci c position on the strand. A decimal system was used, with the knot representing
  units placed nearest the bottom, the tens appearing immediately above, then the hundreds,
  and so on; absence of a knot denoted zero. Bunches of cords were tied off by a single
  main thread, a summation cord, whose knots gave the total count for each bunch. The
  range of possibilities for numerical representation in the quipus allowed the Incas to keep
  incredibly detailed administrative records, despite their ignorance of the written word.


section_type: text
body:
  To appreciate the quipu fully, we should notice the numerical values represented by
  the tied knots. Just three types of knots were used: a gure-eight knot standing for 1, a
  long knot denoting one of the values 2 through 9, depending on the number of twists
  in the knot, and a single knot also indicating 1. The gure-eight knot and long knot
  appear only in the lowest (units) position on a cord, while clusters of single knots can
  appear in the other spaced positions. Because pendant cords have the same length, an
  empty position (a value of zero) would be apparent on comparison with adjacent cords.


section_type: text
body:
  Also, the reappearance of either a gure-eight or long knot would point out that another
  number is being recorded on the same cord.
  Recalling that ascending positions carry place value for successive powers of ten, let
  us suppose that a particular cord contains the following, in order: a long knot with four
  twists, two single knots, an empty space, seven clustered single knots, and one single
  knot. For the Inca, this array would represent the number
  17024 D 4 C (2 Ð 10) C (0 Ð 102
  ) C (7 Ð 103
  ) C (1 Ð 104
  ):


section_type: text
body:
  Over the long sweep of history, it seems clear that progress in devising ef cient
  ways of retaining and conveying numerical information did not take place until primitive
  people abandoned the nomadic life. Incised markings on bone or stone may have been
  adequate for keeping records when human beings were hunters and gatherers, but the food
  producer required entirely new forms of numerical representation. Besides, as a means
  for storing information, groups of markings on a bone would have been intelligible only
  to the person making them, or perhaps to close friends or relatives; thus, the record was
  probably not intended to be used by people separated by great distances.


section_type: text
body:
  Deliberate cultivation of crops, particularly cereal grains, and the domestication of
  animals began, so far as can be judged from present evidence, in the Near East some 10,000 years ago. Later experiments in agriculture occurred in China and in the New
  World. A widely held theory is that a climatic change at the end of the last ice age
  provided the essential stimulus for the introduction of food production and a settled
  village existence. As the polar ice cap began to retreat, the rain belt moved northward,
  causing the desiccation of much of the Near East. The increasing scarcity of wild food
  plants and the game on which people had lived forced them, as a condition of survival,
  to change to an agricultural life. It became necessary to count one’s harvest and herd, to
  measure land, and to devise a calendar that would indicate the proper time to plant crops.
  Even at this stage, the need for a means of counting was modest; and tallying techniques,
  although slow and cumbersome, were still adequate for ordinary dealings. But with a
  more secure food supply came the possibility of a considerable increase in population,
  which meant that larger collections of objects had to be enumerated. Repetition of some
  fundamental mark to record a tally led to inconvenient numeral representations, tedious
  to compose and dif cult to interpret. The desire of village, temple, and palace of cials to
  maintain meticulous records (if only for the purposes of systematic taxation) gave further
  impetus to nding new and more re ned means of “ xing” a count in a permanent or
  semipermanent form.


section_type: text
body:
  Thus, it was in the more elaborate life of those societies that rose to power some
  6000 years ago in the broad river valleys of the Nile, the Tigris-Euphrates, the Indus,
  and the Yangtze that special symbols for numbers rst appeared. From these, some of
  our most elementary branches of mathematics arose, because a symbolism that would
  allow expressing large numbers in written numerals was an essential prerequisite for
  computation and measurement. Through a welter of practical experience with number
  symbols, people gradually recognized certain abstract principles; for instance, it was
  discovered that in the fundamental operation of addition, the sum did not depend on the
  order of the summands. Such discoveries were hardly the work of a single individual,
  or even a single culture, but more a slow process of awareness moving toward an
  increasingly abstract way of thinking.


section_type: image_text
image: post_images/Serpiente_alquimica_bDhsSoy.jpg
image_position: left
body:
  Synthetic data seems hard to grasp intuitively. How can a model trained on its own outputs improve its performance? The key idea is to introduce <strong>side information</strong> to transform the proposal model distribution to a more desirable distribution. So let's see how we can escape the Ouroboros and achieve meaningful improvement.


section_type: text
body:
  Let's assume we have access to a language model policy \(\pi\) that can generate text. We can measure the performance of the model on a particular task, e.g. mathematical reasoning, as \(P_{\pi}\left(X\right)\), the probability of predicting the right answer \(X\).


section_type: text
body:
  <strong>Can we find an improved policy \(\pi^{new}\) by sampling from \(P_{\pi}\) and re-training on that data?</strong>


section_type: text
body:
  The answer is yes, but only if our synthetic data procedure leads to a desirable distributional shift in our model: i.e. \(P_{\pi^{new}}\left(X\right) > P_{\pi}\left(X\right)\).


section_type: text
body:
  To achieve this distributional shift, we need to introduce side information \(I\) that can allow us to use the proposal distribution \(P_{\pi}\left(X\right)\) to sample from a desired distribution \(P_{\pi^{new}}\left(X\right)\).


section_type: text
body:
  The value of the side information is the mutual information relative to the original policy. Write \(P_{\pi}(X\mid{I}) = P_{\pi^*}\left(X\right)\), then:


section_type: text
body:
  If we have no side information then the KL divergence is zero and the value is zero: this explains why some people find synthetic data unintuitive, because they assume there is no side information being introduced. In contrast, if we can use side information and the original policy to efficiently sample examples of a better policy, then useful synthetic data is possible.


section_type: text
body:
  <em>"Let us create a theoretical city from the beginning. Its real creator, as it appears, will be our needs." - Plato</em>


section_type: text
body:
  The next frontier for AI is invention. Invention requires creativity, the ability to come up with new ideas, and execution, the ability to implement those ideas in the real world. In both senses, current AI systems fall short. Models have shown glimpses of creativity in [narrow domains like games](https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol "AlphaGo versus Lee Sedol - Wikipedia"), but the environment is highly constrained. More  general systems like LLMs have shown much more limited creativity, and yet are similarly constrained to environments such as chat interfaces and code editors.


section_type: text
body:
  To crack invention we will need to get creative. But what makes humans creative? What makes humans good at invention where existing AI systems fail?


section_type: text
body:
  The key is a rich environment that generates problems of incrementally increasing complexity. Seen on a grand scale, civilisation has presented humans with a curriculum of problems to solve, incrementally increasing in complexity, where each step has never been too hard to solve. In short: an <strong>invention curriculum</strong>.


section_type: text
body:
  In this essay I will focus on the invention of writing as an example of this process. The emergence of writing was deeply connected to socio-economic change, from the agricultural revolution to the first large urban settlements. At each stage of development, problems arose incrementally as social complexity increased. Writing was developed in tandem to solve these problems as they appeared.


section_type: text
body:
  A core contention of this work is that invention is incremental. Inventions like writing did not appear <em>ex nihilo</em>: they emerged gradually as a series of refinements that tracked the increasing complexity of the environment. This is an important insight because it relieves the creative demands on a human to solve the invention curriculum. If a leap is too large, let's say from a tally stick from a full writing system, then this would require an unreasonable amount of imagination to solve. And if invention is premature, it would not solve a pressing problem for society and would not persist as an idea.


section_type: text
body:
  But this is a broad topic: the environment alone is insufficient for invention due to intelligence constraints. Humans have civilisation; dogs do not. So we begin by discussing the latent capabilities of humans. Then we focus on civilisational progress with a focus on writing: starting from counting in hunter-gather societies, token systems following the agricultural revolution, and then writing. We conclude with lessons for AI: what can we learn about the invention curriculum in developing the next generation of models and environments.


section_type: text
body:
  In [2001: A Space Odyssey](https://en.wikipedia.org/wiki/2001:_A_Space_Odyssey), a tribe of hominids is driven from a watering hole by a rival group. With widespread drought and famine, their hopes for survival are slim. But the next dawn, something miraculous happens. A black, rectangular monolith appears from nowhere. The tribe approaches the object with fear. But slowly their fear turns to awe and wonder. One by one, they extend their hands to touch the cold, black surface. They gaze upwards and stare at the monolith's edge intersecting the rising sun.


section_type: text
body:
  The encounter proves transformative. The next day a hominid plays aimlessly with a bone near an animal carcass. Suddenly, he makes a leap of imagination: the bone can be used as a weapon. Armed with this new invention, the tribe confronts their rivals and defeats them in combat. The tool is thrown to the air in celebration, and the scene cuts to a tool-shaped spaceship floating in space. Humanity has ascended.


section_type: text
body:
  Putting aside its cinematic qualities, Kubrick's vision of invention illustrates several components that are worth analysing. When did invention begin in humans? Was it a sudden spark of inspiration? A Eureka moment? Was it triggered by conflict over resources? How important was the role of the environment? And what makes humans so special in this story?


section_type: text
body:
  Humans are the main characters in this story because there was a selective pressure for invention. That is, environmental factors rewarded inventiveness. But we are not so special. Chimpanzees in the Ivory Coast use stones and sticks to pound open nuts ([Boesch 1996](https://www.thebritishacademy.ac.uk/documents/3956/88p251.pdf "The Emergence of Cultures among Wild Chimpanzees")). This shows they understand abstract properties of objects that don't look alike. So we should also consider the longer story of our evolution as primates.


section_type: text
body:
  A full account of the evolution of primate brains would be too lengthy for this post, so I refer the reader to [Murray et al, 2017](https://global.oup.com/academic/product/the-evolution-of-memory-systems-9780198817130?cc=us&lang=en& "The Evolution of Memory Systems: Ancestors, Anatomy, and Adaptations") and [Passingham, 2021](https://www.amazon.co.uk/Understanding-Prefrontal-Cortex-connectivity-operations-ebook "Understanding the Prefrontal Cortex: Selective advantage, connectivity, and neural operations") for accessible and thoughtful accounts. In short, our ancestors originally foraged for food in the fine-branch niche. Eventually they came to the ground to forage, but patchy resources and heat stress meant foraging periods were short. This created selective pressure for efficient problem-solving and planning. These pressures were accelerated by environmental changes such as global cooling events and competition from other animals. Food scarcity and volatility, along with the need to forage over longer distances, created further selective pressure for human intelligence.


section_type: text
body:
  These selective pressures led to upward grade shifts in the size of primate brains. When looking at grade shifts in brain volume we can see a clear upward shift from strepsirrhines (▲) to modern anthropoids (+):


section_type: image
image: post_images/Screenshot-2024-12-23-at-07-48-42_3OVLIWP.png
image_description:
  From <a href="https://anatomypubs.onlinelibrary.wiley.com/doi/10.1002/ar.a.20113">Bush et al (2004)</a>


section_type: text
body:
  at is the best measure for the cognitive capacity of an animal; some insist on absolute measures [Gibson et al, 2001](https://www.researchgate.net/publication/344507167_Bigger_is_better_primate_brain_size_in_relationship_to_cognition "Bigger is better: primate brain size in relationship to cognition
  "). Others argue for relative measures such as the [Encephalization quotient](https://en.wikipedia.org/wiki/Encephalization_quotient "Encephalization quotient -Wikipedia"), on the basis that larger animals usually have larger brains which reflect a need for more capacity to control the body (not more sophisticated cognition).


section_type: text
body:
  When looking at relative measures, we can see grade-shifts in brain volume in the fossil record. ([Passingham, 2021](https://www.amazon.co.uk/Understanding-Prefrontal-Cortex-connectivity-operations-ebook "Understanding the Prefrontal Cortex: Selective advantage, connectivity, and neural operations")).


section_type: text
body:
  We also see scaling over time of the neocortical ratio - the surface area of neocortex divided by the surface area of the brain - where the curve is much higher and steeper for primates than other mammals.


section_type: text
body:
  How does this compare to AI? Is it right to think of the architecture as the weight connections, and societal influences as connection strengths? Well not quite again, as even humans have latent ideas. Beavers famously have an urge to built when they hear flowing water.


section_type: text
body:
  I would argue the lesson argues in a combination of neocortical and specifically prefrontal prefrontal capabilities, in particular planning (simulation), generation and search. The size of the human brain helps with all of this. (Working memory: chimps vs humans?) -> The ability to simulate is imagination; different from trial-and-error reinfrocement learning. Goal memory.


# django_session

session_key: dons75o17b5qi45yhhpip33zu1ybj41p
expire_date: 2025-01-04 16:29:51.744346
session_data:
  .eJxVjMsOwiAQRf-FtSGMvF267zcQhgGpGkhKuzL-uzbpQrf3nHNfLMRtrWEbeQkzsQsDdvrdMKZHbjuge2y3zlNv6zIj3xV-0MGnTvl5Pdy_gxpH_dbKepfIRF-0sMolg9ZIlQGTRCANVhIaX6h4EQGcLVZ6p91ZQQaphWPvD9SkNv0:1tP2MR:8v2SKNKh2G85dCjcz-8q2uiKI53gPp-c9xy1DO8JgWQ


