{"notes":[{"id":"mamba/hippo","category":"mamba","legacyPath":"mamba/2008-hippo","legacySlug":"2008-hippo","legacyPaths":[],"title":"HiPPO","subtitle":"Hippo's Memory","author":"Z. Yuan","paperDate":"2020-08-17","publishedOn":"2024-12-07","paperTitle":"HiPPO: Recurrent Memory with Optimal Polynomial Projections","paperUrl":"https://arxiv.org/abs/2008.07669","bibliography":{"title":"HiPPO: Recurrent Memory with Optimal Polynomial Projections","authors":["Albert Gu","Tri Dao","Stefano Ermon","Atri Rudra","Christopher Re"],"url":"https://arxiv.org/abs/2008.07669","pdfUrl":"https://arxiv.org/pdf/2008.07669","codeUrl":null,"preprint":{"id":"2008.07669","submittedOn":"2020-08-17","url":"https://arxiv.org/abs/2008.07669"},"publication":{"venue":"NeurIPS","year":2020,"kind":"conference","url":"https://papers.nips.cc/paper_files/paper/2020/hash/102f0bb6efb3a6128a3c750dd16729be-Abstract.html"},"citation":"@misc{mambahippo2020,\n  title = {{HiPPO: Recurrent Memory with Optimal Polynomial Projections}},\n  author = {Albert Gu and Tri Dao and Stefano Ermon and Atri Rudra and Christopher Re},\n  year = {2020},\n  eprint = {2008.07669},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2008.07669}\n}\n","citationPath":"/content/papers/citations/mamba/hippo.b20e3393056b.bib"},"thumbnail":{"src":"/content/papers/assets/0edb34026ecfca3a51cdc9a978386d1f6a0aab6d88e101c0fd44d7a9db21462f.jpg","width":1440,"height":514},"locale":"en","path":"/content/papers/en/mamba/hippo.4ae9143e5667.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"problem-solution","text":"Problem Solution","depth":2},{"id":"compressed-representation-of-sequences","text":"Compressed Representation of Sequences","depth":3},{"id":"the-advantages-and-disadvantages-of-approximation","text":"The Advantages and Disadvantages of Approximation","depth":3},{"id":"hippo-framework","text":"HiPPO Framework","depth":2},{"id":"orthogonal-polynomials-and-coefficient-projection","text":"Orthogonal Polynomials and Coefficient Projection","depth":3},{"id":"transforming-into-a-differential-equation","text":"Transforming into a Differential Equation","depth":3},{"id":"hippo-abstraction","text":"HiPPO Abstraction","depth":3},{"id":"hippo-application-examples","text":"HiPPO Application Examples","depth":3},{"id":"hippo-discretization","text":"HiPPO Discretization","depth":3},{"id":"hippo-low-degree-projection","text":"HiPPO Low-Degree Projection","depth":3},{"id":"timescale-robustness-of-hippo","text":"Timescale Robustness of HiPPO","depth":2},{"id":"definition-of-the-legs-measure","text":"Definition of the LegS Measure","depth":3},{"id":"theoretical-properties-and-advantages","text":"Theoretical Properties and Advantages","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"long-term-memory-benchmark-task","text":"Long-Term Memory Benchmark Task","depth":3},{"id":"timescale-robustness","text":"Timescale Robustness","depth":3},{"id":"scalability-verification","text":"Scalability Verification","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/s4","relation":"next","topic":"From state-space memory to visual sequences"}]},{"id":"mamba/s4","category":"mamba","legacyPath":"mamba/2111-s4","legacySlug":"2111-s4","legacyPaths":[],"title":"S4","subtitle":"The Prelude of Mamba","author":"Z. Yuan","paperDate":"2021-10-31","publishedOn":"2024-11-17","paperTitle":"Efficiently Modeling Long Sequences with Structured State Spaces","paperUrl":"https://arxiv.org/abs/2111.00396","bibliography":{"title":"Efficiently Modeling Long Sequences with Structured State Spaces","authors":["Albert Gu","Karan Goel","Christopher Ré"],"url":"https://arxiv.org/abs/2111.00396","pdfUrl":"https://arxiv.org/pdf/2111.00396","codeUrl":null,"preprint":{"id":"2111.00396","submittedOn":"2021-10-31","url":"https://arxiv.org/abs/2111.00396"},"publication":{"venue":"ICLR","year":2022,"kind":"conference","url":"https://arxiv.org/abs/2111.00396"},"citation":"@misc{mambas42021,\n  title = {{Efficiently Modeling Long Sequences with Structured State Spaces}},\n  author = {Albert Gu and Karan Goel and Christopher Ré},\n  year = {2021},\n  eprint = {2111.00396},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2111.00396}\n}\n","citationPath":"/content/papers/citations/mamba/s4.bd8f80223f7e.bib"},"thumbnail":{"src":"/content/papers/assets/86eedb9a9143dff790f0e1e5af878b5b3eb93e2b6ea2d3c788567363967af802.jpg","width":1224,"height":424},"locale":"en","path":"/content/papers/en/mamba/s4.c1074bff2ec5.md","headings":[{"id":"state-space-models","text":"State Space Models","depth":2},{"id":"hippo-theory","text":"HiPPO Theory","depth":3},{"id":"discretization","text":"Discretization","depth":3},{"id":"convolutional-representation","text":"Convolutional Representation","depth":3},{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"s4-parameterization","text":"S4 Parameterization","depth":3},{"id":"s4-complexity","text":"S4 Complexity","depth":3},{"id":"architecture-details","text":"Architecture Details","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"efficiency-benchmarks","text":"Efficiency Benchmarks","depth":3},{"id":"stability-benchmarks","text":"Stability Benchmarks","depth":3},{"id":"visualization","text":"Visualization","depth":3},{"id":"a-general-model","text":"A General Model?","depth":3},{"id":"importance-of-hippo","text":"Importance of HiPPO","depth":3},{"id":"experiment-1-unconstrained-ssm-initialization","text":"Experiment 1: Unconstrained SSM Initialization","depth":4},{"id":"experiment-2-combining-nplr-parameterization-with-hippo","text":"Experiment 2: Combining NPLR Parameterization with HiPPO","depth":4},{"id":"experiment-3-regularization-and-full-s4-performance","text":"Experiment 3: Regularization and Full S4 Performance","depth":4},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"mamba/hippo","relation":"background","topic":"From state-space memory to visual sequences"},{"id":"mamba/mamba","relation":"next","topic":"From state-space memory to visual sequences"}]}],"text":"We have already read **S4** to better understand the research context, so let's revisit this paper.\n\nI have to say that the author's papers are very mathematical, and you need to muster some courage before opening them.\n\n\n> [!TIP]\n> \n> If you want to know more about S4, you can refer to another article of ours:\n> \n> - [**[21.11] S4: The Prelude of Mamba**](/blog/papers/mamba/s4)\n\n\n## Problem Definition\n\nMost people are familiar with the issue of vanishing and exploding gradients in RNNs.\n\nAlthough this is a characteristic inherent in the architecture, there are always some ways to mitigate it, such as adding gates like LSTM or GRU, or using special bases like Fourier Recurrent Unit or Legendre Memory Unit (LMU).\n\nHowever, this only treats the symptoms, not the root cause!\n\nThe authors do not agree with these approaches. They believe these methods share some common problems:\n\n1. **There is no unified theoretical framework to explain the memory mechanism.**\n2. **They often require prior knowledge about the input sequence's length or time scale.**\n\nCan we find a method that does not require prior knowledge of the time scale, has a theoretical foundation, and can address the long-term dependency issue?\n\n## Problem Solution\n\nFirst, let’s reconsider the scenario of an RNN.\n\n### Compressed Representation of Sequences\n\nGiven an input function $f(t)$ that changes over time, at each time $t \\geq 0$, we want to make decisions, predictions, or representations based on all past information (i.e., the entire sequence or signal segment from the start time 0 to time $t$, $f(x)$, $x \\le t$).\n\nHowever, directly storing the entire history $f_{\\le t} := \\{ f(x) \\mid x \\le t \\}$ is impractical, because as time progresses, the amount of data increases indefinitely, making storage computationally difficult.\n\nTherefore, we need a \"compressed representation with bounded dimensions\"!\n\nThat is, we hope to use a finite number of parameters (values) at each moment to approximate the entire historical information.\n\nNext, we must choose a function subspace $G$ with bounded dimensions in which to find the best approximation.\n\nThe authors choose a natural and commonly used option: **polynomial spaces**.\n\nSuppose we take $G$ as the space spanned by polynomials:\n\n$$\nG = \\{ \\text{polynomials } p(x) : \\deg(p) < N \\}.\n$$\n\nIn this subspace, we can write the approximate function $g$ as\n\n$$\ng^{(t)}(x) = \\sum_{n=0}^{N-1} c_n(t) p_n^{(t)}(x),\n$$\n\nwhere $p_n^{(t)}(x)$ are basis polynomials for the measure chosen at current time $t$, and $c_n(t)$ are the coefficients we need to maintain and update. Since $N$ is fixed, we only need $N$ coefficients to represent a compressed approximation of the history observed by time $t$.\n\n\n> [!TIP]\n> \n> **Why choose polynomials?**\n> \n> Because polynomials are very versatile bases:\n> \n> 1. Mathematically, polynomial density guarantees arbitrary approximation of many functions (as $N \\to \\infty$).\n> 2. Orthogonal polynomials have many good properties, allowing closed-form solutions for calculating optimal approximation coefficients.\n> 3. Other bases, like Fourier series, can be considered a special case of polynomials under appropriate transformations.\n> \n> Good stuff, why not use it?\n\n\n\n> [!TIP]\n> \n> **Let's consider another example of temperature monitoring:**\n> \n> Suppose $f(t)$ is the temperature recorded by a sensor at time $t$, and we want to predict future trends using the historical temperature data at any time $t$. Storing all past temperatures would require infinite memory; instead, we can use polynomial approximation to represent the overall trend of temperature changes over time. For example, if $N=3$, we might use\n> \n> $$\n> g(t) = x_0 + x_1 t + x_2 t^2\n> $$\n> \n> to represent the recent temperature change characteristics. As time progresses, we update $x_0, x_1, x_2$ in real-time, maintaining a \"compressed description\" of the temperature history.\n> \n> See, you can already sense what the HiPPO framework is trying to achieve, right?\n\n\n### The Advantages and Disadvantages of Approximation\n\nThis brings up the next core question:\n\n- **What method should we use to define \"approximation\" and quantify the \"quality of the approximation\"?**\n\nThe quality of the approximation, or \"the quality of the approximation,\" can be measured by the \"concept of distance.\"\n\nGenerally, we know that there is a length on the number line, a distance in the plane, and also a distance in higher-dimensional spaces. The function space (where each function is treated as a point) can also define a \"distance,\" but the way to define this distance requires the use of integrals.\n\nThe author's idea here is:\n\n> **If two functions are very close in value at every time point, we say they are \"close in distance\"; on the other hand, if they are inconsistent in many places, they are \"far apart\".**\n\nMathematically, we define the distance between functions by introducing a \"measure\" $\\mu$ and a corresponding \"weighted inner product\":\n\n1. **Measure $\\mu$**\n\n   Here, we can think of the measure as a kind of \"weighting system\" used to indicate which time points on the time axis are more important.\n\n   If we care more about the most recent data, we assign a larger weight to the \"latest time intervals\" and a smaller weight to the more distant past time points. Or, we could treat all time points equally.\n\n2. **Weighted Inner Product $\\langle f,g \\rangle_\\mu$**\n\n   We define the \"inner product\" between two functions $f$ and $g$ as follows:\n\n   $$\n   \\langle f,g \\rangle_\\mu = \\int_0^\\infty f(x)g(x) \\, d\\mu(x).\n   $$\n\n   The inner product itself is not a distance. To measure the difference between $f$ and $g$, calculate $\\|f-g\\|_{L^2(\\mu)}$; the measure gives more weight to errors at the times it emphasizes.\n\nSimilar to the length of a vector in two-dimensional space, which is $\\sqrt{x^2 + y^2}$, in function space, we use the inner product to define the \"length\" or \"norm\" of a function:\n\n$$\n\\| f \\|_{L^2(\\mu)} = \\sqrt{\\langle f,f \\rangle_\\mu} = \\sqrt{\\int_0^\\infty f(x)^2 \\, d\\mu(x)}.\n$$\n\nIf the value of function $f$ is large, the norm will be large; if $f$ is close to 0, the norm will be small.\n\nFinally, we try to approximate a simpler function $g$ to approach $f_{\\le t}$ (representing the historical data from 0 to $t$), and then, within this metric structure, we can calculate the difference between the two:\n\n$$\n\\| f_{\\le t} - g \\|_{L^2(\\mu)} = \\sqrt{\\int_0^\\infty [f(x) - g(x)]^2 \\, d\\mu(x)}.\n$$\n\nThis integral squares the difference where the measure $\\mu$ has weight and then integrates it. If $\\mu^{(t)}$ is supported only on $[0,t]$, later times are not included. The **$(f(x) - g(x))^2$** term tells us:\n\n1. If it is small for most of the time points, the integral result will be small, indicating that $f$ and $g$ are very close.\n2. If it is large for many time points, the integral result will be large, indicating that $f$ and $g$ differ greatly.\n\n## HiPPO Framework\n\nBy this point, we have defined the \"memory\" problem as an \"online function approximation problem,\" where, at any given time $t$, we approximate the accumulated history $f_{\\le t}$ of the input function over the interval $[0,t]$ using an $N$-dimensional subspace (usually spanned by polynomials).\n\nTo achieve this goal, the next task is to solve the following problem:\n\n> **As new data $f(t)$ keeps arriving, how do we update the approximation coefficients online, so that the polynomial stays close to $f_{\\le t}$?**\n\n### Orthogonal Polynomials and Coefficient Projection\n\n1. **Choosing an appropriate basis $\\{g_n\\}_{n<N}$:**\n\n   To ensure good mathematical properties, we choose normalized orthogonal polynomials for the measure $\\mu^{(t)}$ as the basis for $G$. Their orthonormality gives the optimal approximation coefficients directly as inner products and makes the updates more concise.\n\n2. **Solving for the \"best approximation projection\":**\n\n   If $g^{(t)} \\in G$ is the best polynomial approximation to $f_{\\le t}$, then it can be written as:\n\n   $$\n   g^{(t)}(x) = \\sum_{n=0}^{N-1} c_n(t) g_n^{(t)}(x)\n   $$\n\n   where $c_n(t)$ is the best approximation coefficient at time $t$. By definition, $c_n(t)$ can be obtained through the inner product:\n\n   $$\n   c_n(t) = \\langle f_{\\le t}, g_n^{(t)} \\rangle_{\\mu^{(t)}}.\n   $$\n\n   Due to orthogonality, obtaining $c_n(t)$ becomes a straightforward and feasible task.\n\n### Transforming into a Differential Equation\n\nThe key problem now is:\n\n> **How do we update $c(t) = [c_0(t), c_1(t), \\dots, c_{N-1}(t)]^\\top$ as time $t$ progresses?**\n\nSince $f_{\\le t}$ and the measure $\\mu^{(t)}$ change over time, we consider taking the derivative of $c_n(t)$ with respect to $t$.\n\nAfter detailed mathematical derivation, we find:\n\n\n> [!TIP]\n> \n> The author provides the full derivation in the appendix of the paper, and we apologize for skipping this part here.\n\n\n$$\n\\frac{d}{dt} c(t) = A(t) c(t) + B(t) f(t),\n$$\n\nwhere $A(t)$ is an $N \\times N$ matrix, and $B(t)$ is an $N \\times 1$ vector, both determined by the measure and the properties of the orthogonal polynomials.\n\nThis means that the \"evolution of the best approximation coefficients over time\" can be described by an ordinary differential equation (ODE).\n\nTheoretically, by solving this ODE, we can update $c(t)$ online.\n\nHowever, in practice, we cannot solve the ODE exactly in continuous time, but we can discretize it (using methods like Euler's method or other numerical techniques) to convert it into a linear recurrence form.\n\nThus, each time a new $f(t)$ value arrives, we can update\n\n$$\nc_{k+1} = \\bar A_k c_k + \\bar B_k f_k,\n$$\n\nsimilar to an RNN. The barred coefficients come from discretization; a generic matrix-vector update takes $O(N^2)$ time per step, with lower costs possible for specific structures.\n\n### HiPPO Abstraction\n\nIn the previous sections, we have defined the \"memory representation\" problem as a \"function approximation problem\":\n\n- At any given time $t$, we wish to represent the entire historical signal $f_{\\le t}$ in a finite-dimensional way, ensuring that this approximation is optimal under a selected metric.\n\nThe author introduces several key operators here:\n\n1. **Projection Operator $proj_t$:**\n\n   - **Input:** The historical function segment $f_{\\le t}$, which is the portion of the function $f$ over the time interval $[0, t]$.\n   - **Output:** A polynomial $g^{(t)} \\in G$, where $G$ is an $N$-dimensional polynomial subspace.\n\n   This $g^{(t)}$ is the polynomial that best approximates $f_{\\le t}$ under the $L^2(\\mu^{(t)})$ metric.\n\n   In simple terms, $proj_t(f)$ finds the \"best polynomial approximation\" to represent $f$ over the interval $[0, t]$.\n\n2. **Coefficient Extraction Operator $coef_t$:**\n\n   - **Input:** A polynomial $g^{(t)}$.\n   - **Output:** A vector $c(t) \\in \\mathbb{R}^N$, which is the coefficient vector of the polynomial in terms of an orthogonal polynomial basis.\n\n   Since $g^{(t)}$ is an element in an $N$-dimensional subspace, we can uniquely express $g^{(t)}$ as\n\n   $$\n   g^{(t)}(x) = \\sum_{n=0}^{N-1} c_n(t) \\, g_n(x),\n   $$\n\n   where the $g_n(x)$ are the orthogonal basis functions (determined by the measure $\\mu^{(t)}$).\n\nBy combining these two operators, we have $hippo = coef_t \\circ proj_t$, which maps the original function $f(t)$ directly to its best polynomial approximation coefficient vector $c(t)$:\n\n$$\n(hippo(f))(t) = coef_t(proj_t(f)).\n$$\n\nThus, we only need to maintain $c(t) \\in \\mathbb{R}^N$ to dynamically update and obtain a bounded-dimensional representation of historical memory.\n\nWhile the definition is simple, directly computing $proj_t(f)$ is not easy.\n\nThe author derives the update rule for $c(t)$ from theory, which can be described by an ODE:\n\n$$\n\\frac{d}{dt} c(t) = A(t)c(t) + B(t)f(t),\n$$\n\nwhere $A(t)$ and $B(t)$ are matrices and vectors determined by the measure $\\mu^{(t)}$ and the choice of the basis.\n\n\n> [!TIP]\n> \n> **Why is this important?**\n> \n> Through this ODE, we do not need to repeatedly solve a complex projection problem at every time step. Instead, we can simply integrate (or discretize) over time to keep $c(t)$ as the best approximation coefficient vector for the current history.\n> \n> Furthermore, when we discretize this ODE, we obtain a linear recurrence relation:\n> \n> $$\n> c_{k+1} = A_k c_k + B_k f_k,\n> $$\n> \n> where $A_k$ and $B_k$ come from discretizing $A(t)$ and $B(t)$ and may change with the step.\n> \n> With this discrete recurrence, we can efficiently update the memory representation online:\n> \n> - Every time a new input value $f_k$ arrives, we just compute $c_{k+1}$ based on the previous $c_k$.\n> \n> This approach makes the seemingly complex optimal projection problem practical.\n\n\nFinally, the author includes a cute hippopotamus and a chart to illustrate the process:\n\n\n\n![hippo](/content/papers/assets/0edb34026ecfca3a51cdc9a978386d1f6a0aab6d88e101c0fd44d7a9db21462f.jpg)\n\n\n\nThe process of simplifying the approximate function is broken down into four steps:\n\n1. **Arbitrary Function $f$**: The upper left shows a function curve $f(t)$, defined for $t \\geq 0$. This represents the time-varying signal input.\n2. **For each time $t$, compute the best projection $g^{(t)}$**:\n\n   The second box in the diagram shows that at any time $t$, we want to find a polynomial $g^{(t)}$ to approximate $f_{\\le t}$.\n\n   In the diagram, $g^{(t)}$ is drawn as a function curve approximating $f(t)$, but only considering the current time range $[0, t]$, and determining the best approximation according to the inner product space defined by the selected measure $\\mu^{(t)}$.\n\n3. **Coefficient Representation and Dynamics (ODE)**:\n\n   The third box shows that $g^{(t)}$ is written as a linear combination of orthogonal polynomial bases, yielding the coefficient vector $c(t)$.\n\n   At this point, the author's theoretical result shows that $c(t)$ will follow a set of linear differential equations (ODE):\n\n   $$\n   \\frac{d}{dt}c(t) = A(t)c(t) + B(t)f(t).\n   $$\n\n   This simplifies the problem into a computable dynamic system.\n\n   We no longer need to repeatedly minimize an error function, but instead update $c(t)$ over time.\n\n4. **Discretization and Online Update**:\n\n   Finally, since the actual data is a discrete sequence (e.g., $f_0, f_1, \\dots$), we discretize the ODE, leading to the state shown in the far-right diagram:\n\n   $$\n   c_{k+1} = A_k c_k + B_k f_k.\n   $$\n\n   This is a simple recurrence relation, similar to a special \"RNN-like\" structure. Given the previous time step's state $c_k$ and the current input $f_k$, we can update the state $c_{k+1}$.\n\n### HiPPO Application Examples\n\n\n> [!TIP]\n> \n> We’ve made it this far, but we’re only halfway through. Hang in there a bit longer!\n\n\nTo help us better understand how HiPPO can be applied in practice, the author provides two common \"memory types\" as examples: **Sliding Window (LegT) Measure** and **Exponential Decay (LagT) Measure**.\n\n- **1. Sliding Window (LegT) Measure**\n\n  The sliding window is an intuitive memory mechanism, commonly used in scenarios such as monitoring real-time video streams in a control room.\n\n  We are generally only concerned with events that occurred in the last few seconds, and older frames are no longer important. Therefore, we set a \"window length\" $\\theta$, storing only the most recent $\\theta$ seconds of video and constantly discarding older fragments. This is like sliding a fixed-length window along the timeline, only looking at the \"latest\" history inside this window.\n\n  Mathematically, this idea of \"uniform weighting over the interval [ $t-\\theta, t$ ]\" can be expressed using a measure:\n\n  $$\n  \\mu^{(t)}(x) = \\frac{1}{\\theta} I_{[t-\\theta,\\, t]}(x),\n  $$\n\n  where $I_{[t-\\theta, t]}$ is the indicator function, indicating that only time points within the interval $[t-\\theta, t]$ are considered.\n\n  For this measure, the HiPPO projection yields the time-invariant linear ODE used by LMU, with the signal leaving the left edge of the sliding window approximated from the current coefficients:\n\n  $$\n  \\frac{d}{dt} c(t) = -A c(t) + B f(t),\n  $$\n\n  where $A$ and $B$ have a clear closed-form solution. This update rule is the one used by the **Legendre Memory Unit (LMU)**.\n\n  LMU was originally motivated as a memory mechanism for spiking neural networks. HiPPO derives its update from sliding-window projection and clarifies the approximation required at the window's departing boundary. The finite-state update is not an exact sliding projection for every possible signal.\n\n- **2. Exponential Decay (LagT) Measure**\n\n  Another common scenario is when we want to consider the infinite past history, but the more distant information becomes less important.\n\n  Mathematically, this can be modeled by applying exponential decay to the weights of the history:\n\n  $$\n  \\mu^{(t)}(x) = e^{-(t-x)} I_{(-\\infty,t]}(x),\n  $$\n\n  which gives the most weight to recent time points, and gradually less weight to older ones, but never completely disappearing.\n\n  Similarly, using HiPPO, we can derive the corresponding ODE and linear recurrence update rules for this measure:\n\n  $$\n  A_{nk} =\n  \\begin{cases}\n  1 & \\text{if } n \\geq k \\\\\n  0 & \\text{if } n < k\n  \\end{cases}, \\quad B_n = 1.\n  $$\n\n  This set of formulas describes a \"long-term memory\" that \"gradually fades,\" which is very useful for many practical applications, such as finance, signal filtering, and weighted averaging.\n\n---\n\nThe author uses these two examples to demonstrate the versatility of the HiPPO framework. By simply changing the measure, we can generate different memory update rules with varying characteristics.\n\nSliding windows and exponential decay are just two typical examples of measures. Through the same derivation process, we can generate corresponding formulas for other special measures (such as Fourier or Chebyshev) or specific task requirements.\n\n### HiPPO Discretization\n\nIn the original definition, HiPPO starts with continuous time and describes the evolution of the memory state $c(t)$ over time through a set of ODEs. However, in practice, we often deal with discrete sequential data, such as daily stock prices, temperature readings every second, or image signals at each time step.\n\nTherefore, we need to discretize the continuous-time ODE, which means dividing time into small steps $\\Delta t$ and converting the continuous evolution equation into a discrete update formula, making it applicable to real sequential data.\n\nIn continuous time, we have\n\n$$\n\\frac{d}{dt} c(t) = u(t, c(t), f(t)),\n$$\n\nwhere $f(t)$ is the input function, $c(t)$ is the state vector, and $u$ describes how the state is updated.\n\nBy dividing time into small steps $\\Delta t$, at each time point $t = k \\Delta t$ (where $k$ is an integer), we update the state. Forward Euler below illustrates the idea; the paper's experiments use more stable bilinear or zero-order-hold discretization.\n\nThe basic Forward Euler discretization is:\n\n$$\nc(t+\\Delta t) = c(t) + \\Delta t \\cdot u(t, c(t), f(t)).\n$$\n\nIf we have a sequence of inputs $(f_0, f_1, f_2, \\dots)$, we can assume that $f_k = f(k \\cdot \\Delta t)$, and then use the above formula to update $c_k = c(k \\cdot \\Delta t)$ step by step.\n\nWith this discretization step, we can compute it in a \"single loop\" on a computer:\n\n- Starting from the initial state $c_0$,\n- Each time a new data point $f_k$ arrives, we update $c_{k+1} = c_k + \\Delta t \\cdot u(k\\Delta t, c_k, f_k)$.\n\nThis results in a discrete-time recurrence relation, eliminating the need to deal with continuous differential equations.\n\nIf observations are irregular or missing, the difference between successive timestamps can set each $\\Delta t$, followed by the chosen discretization update. This accounts for elapsed time but cannot automatically recover an unobserved value; the input between observations still needs an interpretation.\n\n### HiPPO Low-Degree Projection\n\nHiPPO approximates historical memory with polynomials. $N$ is the coefficient dimension, so the highest polynomial degree is $N-1$. A larger $N$ supplies more basis functions for complex histories.\n\nBut what happens if $N=1$, meaning that we use only a constant, degree-zero basis?\n\nTaking HiPPO-LagT (the exponential decay example) as an example, when $N=1$, the corresponding ODE becomes very simple, with $A=B=1$, and upon discretization, we get:\n\n$$\nc_{k+1} = (1-\\Delta t)c_k + \\Delta t f_k.\n$$\n\nThis formula can be interpreted as:\n\n- When $0\\le\\Delta t\\le1$, $(1-\\Delta t)$ acts like the proportion of old memory retained.\n- Under the same condition, $\\Delta t$ acts like the proportion of new information injected.\n\nIf we additionally make the input $f_k$ depend on the state and allow $\\Delta t$ to adapt to the input and state, we find that:\n\n- **Hey! This update rule is exactly the familiar Gated RNN!**\n\nThis lowest-order projection plus those input and step-size designs produces a gated-RNN-style update. A GRU still needs parallel units and its particular update functions; it is not merely the $N=1$ equation by itself.\n\nIt turns out that Gated RNNs can also be derived from the \"function approximation\" perspective, rather than being solely an empirically designed architecture.\n\n## Timescale Robustness of HiPPO\n\nIn the previous sections, we learned that HiPPO determines the characteristics of the memory mechanism by selecting an appropriate measure. The measures we explored earlier, such as Sliding Window (LegT) or Exponential Decay (LagT), rely on certain hyperparameters to represent time range or time intervals (for example, the window size $\\theta$ in LegT or $\\Delta t$ in LagT when discretized).\n\nHowever, in practice, we often cannot determine the optimal timescale in advance, and fixed timescales may not be applicable when the data distribution changes over time.\n\nTo address this issue, the author introduces a new measure in this section:\n\n- **Scaled Legendre (LegS)**\n\nThis measure aims to establish a memory mechanism that is robust to timescales, without requiring specific time hyperparameters.\n\n### Definition of the LegS Measure\n\n$$\n\\mu^{(t)}(x) = \\frac{1}{t} I_{[0,t]}(x).\n$$\n\nThis measure assigns uniform weights to the entire historical interval $[0,t]$, unlike the sliding window, which only focuses on the recent $\\theta$-length time interval, or exponential decay, which heavily weights the recent history and exponentially reduces the weight of distant history.\n\nLegS essentially dynamically expands the \"window\" over time, always including all history from $0$ to $t$.\n\nBased on the author's proofs, HiPPO-LegS has dynamic equations for both continuous and discrete time:\n\n- **Continuous-time differential equation:**\n\n  $$\n  \\frac{d}{dt} c(t) = -\\frac{1}{t} A c(t) + \\frac{1}{t} B f(t).\n  $$\n\n- **Discrete-time recurrence relation:**\n\n  $$\n  c_{k+1} = \\left(I - \\frac{A}{k}\\right) c_k + \\frac{1}{k} B f_k, \\quad k\\ge 1.\n  $$\n\nHere, $A \\in \\mathbb{R}^{N \\times N}$ and $B \\in \\mathbb{R}^{N \\times 1}$, with specific structures for $A$ and $B$ as follows:\n\n$$\nA_{nk} =\n\\begin{cases}\n\\sqrt{2n+1}\\sqrt{2k+1} & \\text{if } n > k \\\\[6pt]\nn + 1 & \\text{if } n = k \\\\[6pt]\n0 & \\text{if } n < k\n\\end{cases},\n\\quad\nB_n = \\sqrt{2n+1}.\n$$\n\nHere $I$ is the identity matrix. The discrete example is valid only for $k\\ge1$; initialization must handle the singularity at $t=0$ rather than divide by zero. These formulas avoid a fixed window-length hyperparameter, though state size and discretization still need choices.\n\n\n> [!TIP]\n> \n> To put it simply, the definition of $(A, B)$ describes the \"rule matrix\" (A) and the \"input vector\" (B) used in HiPPO-LegS to update the memory representation. They determine how the coefficients of the polynomial basis ($c_n(t)$) at each time step change based on the current state and the new input $f(t)$.\n> \n> The definition of $A_{nk}$ is divided into three cases:\n> \n> 1. **$n = k$ (Diagonal elements):**\n> \n>    $$\n>    A_{nn} = n + 1.\n>    $$\n> \n>    On the diagonal, each element $A_{nn}$ is $n+1$. Intuitively, this means that each \"coefficient of degree $n$\" affects its own update with a linear proportion ($n+1$). The higher the degree $n$, the stronger the scaling of the higher-degree polynomial coefficients during the update.\n> \n> 2. **$n > k$ (Below the diagonal):**\n> \n>    $$\n>    A_{nk} = \\sqrt{2n+1}\\sqrt{2k+1}.\n>    $$\n> \n>    When $n$ is greater than $k$, this element lies below the diagonal. It couples the higher-degree coefficient in row $n$ to the lower-degree coefficient in column $k$ with weight $\\sqrt{2n+1}\\sqrt{2k+1}$, a **product**, not a ratio.\n> \n> 3. **$n < k$ (Above the diagonal):**\n> \n>    $$\n>    A_{nk} = 0.\n>    $$\n> \n>    All elements above the diagonal are zero. Thus $A$ is lower triangular: the update of coefficient $n$ does not directly depend on coefficients $k>n$ through this matrix term. Information flows from lower to higher degree here.\n> \n> This structure ensures that during polynomial approximation, each coefficient update refers sequentially to the information of lower-degree coefficients, and the influence between degrees is orderly and controllable.\n> \n> ---\n> \n> Additionally, the structure of the $B$ vector is:\n> \n> $$\n> B_n = \\sqrt{2n+1}.\n> $$\n> \n> $B_n$ controls the influence of input $f(t)$ on the $n$-th degree polynomial coefficient.\n> \n> - As $n$ increases, the input coefficient $\\sqrt{2n+1}$ increases. The actual state response also depends on coupling through $A$, so $B_n$ alone does not show that higher-degree components respond more strongly.\n\n\n### Theoretical Properties and Advantages\n\nThe **LegS (Scaled Legendre)** measure proposed by HiPPO exhibits several advantages.\n\nIts key feature is assigning uniform weights to the entire historical interval $[0, t]$, which leads to \"timescale robustness,\" \"computational efficiency,\" \"stable gradient flow,\" and \"controlled approximation error.\"\n\n- **Timescale Robustness**\n\n  Since LegS does not rely on fixed time window sizes or step-size parameters, it naturally adapts to time stretching or compression. The author provides the following mathematical expression:\n\n  > For any $\\alpha > 0$, define $h(t) = f(\\alpha t)$, then\n  >\n  > $$\n  > hippo(h)(t) = hippo(f)(\\alpha t).\n  > $$\n\n  In other words, for any time scaling function $\\gamma: t \\mapsto \\alpha t$, we have:\n\n  $$\n  hippo(f \\circ \\gamma) = hippo(f) \\circ \\gamma.\n  $$\n\n  This means constant compression or stretching of the input time axis produces the same coefficient trajectory at a rescaled time. The coefficient values are not multiplied by $\\alpha$, and this statement does not cover arbitrary time warping.\n\n  In contrast, other measures (such as LegT or LagT) require additional window lengths $\\theta$ or step sizes $\\Delta t$ as hyperparameters, which are easily affected by changes in timescale and must be carefully adjusted in practice.\n\n- **Computational Efficiency**\n\n  After discretizing the continuous ODE, we face the issue of performing matrix-vector multiplications at each time step. Typically, this requires $O(N^2)$ computations.\n\n  However, for LegS, through a specific generalized bilinear transformation discretization method, each update step can be computed in $O(N)$ time:\n\n  > For the HiPPO-LegS discrete recurrence update\n  >\n  > $$\n  > c_{k+1} = \\left(I - \\frac{A}{k}\\right)c_k + \\frac{1}{k} B f_k, \\quad k\\ge1,\n  > $$\n  >\n  > under appropriate discretization conditions, each calculation can be completed in $O(N)$ time.\n\n  Due to the structure of $A$, the generalized bilinear discretizations analyzed in the paper permit $O(N)$ memory updates. The measured speed table below concerns the memory updater on one CPU core; it does not show a tenfold acceleration for an entire RNN or for every choice of $\\Delta t$.\n\n- **Gradient Flow**\n\n  RNNs often suffer from the \"vanishing gradient\" problem, which makes it difficult to capture long-term dependencies. For LegS, the author provides an analysis of the gradient flow, showing that it avoids exponential gradient decay:\n\n  > For any $t_0 < t_1$,\n  >\n  > $$\n  > \\left\\|\\frac{\\partial c(t_1)}{\\partial f(t_0)}\\right\\| = \\Theta\\left(\\frac{1}{t_1}\\right).\n  > $$\n\n  The analyzed continuous memory path decays like $1/t_1$ rather than exponentially. The gradient still decreases.\n\n- **Approximation Error Bounds**\n\n  Finally, the author discusses the theoretical accuracy of LegS in function approximation.\n\n  If $f$ is L-Lipschitz, then\n\n  $$\n  \\|f_{\\le t} - g^{(t)}\\| = O\\left(\\frac{tL}{\\sqrt{N}}\\right),\n  $$\n\n  and if $f$ has $k$-th bounded derivatives, then\n\n  $$\n  \\|f_{\\le t} - g^{(t)}\\| = O\\left(\\frac{t^k}{N^{k-\\frac{1}{2}}}\\right).\n  $$\n\n  Under the stated smoothness and derivative bounds, increasing the coefficient dimension $N$ lowers these error bounds. A fixed-dimensional state cannot exactly reconstruct every signal.\n\n## Discussion\n\nIn the experiments, the author used different HiPPO methods, including the previously introduced LegT, LagT, and LegS.\n\n- **LegT**: Requires a hyperparameter $\\theta$ to represent the sliding window length.\n\n  If the timescale of the data is known and set to an ideal value, LegT performs well. However, if $\\theta$ is not suitable (too large or too small), performance will significantly degrade.\n\n- **LagT**: Requires setting a step size hyperparameter $\\Delta t$, and its performance also depends on this choice.\n\n- **LegS**: Does not require such time-scale parameters and is theoretically more robust to different timescales.\n\nIn addition, the author designed a control group **Rand**, where random $A$ and $B$ matrices (properly normalized) are used in the update rule. By comparing Rand with other rigorously derived HiPPO dynamics, the importance of \"specific dynamic design\" is confirmed, as opposed to the effect of purely random matrices.\n\nThe HiPPO methods were also combined with MGU (Minimal Gated Unit, a simplified GRU) as the basic architecture. Additionally, various RNN variants designed for long-term dependencies, such as LSTM, GRU, expRNN, and LMU, were also compared.\n\nThe compared methods use the same hidden-state dimension; HiPPO variants additionally set memory dimension $N=d$. This makes their scales roughly comparable.\n\n### Long-Term Memory Benchmark Task\n\n\n\n![long-term-memory-benchmark](/content/papers/assets/1d8ecfa323942e41770f4191d4b5891dfbf88d450a0da7e5b95672d512c251a7.jpg)\n\n\n\nThe pMNIST task involves shuffling the pixel order of MNIST handwritten digits and then inputting the model pixel by pixel. Since the spatial adjacency of pixels is shuffled, the model must use long-term memory to retain and integrate information from the entire image for classification, making this task a true test of long-term dependency capability.\n\nThe left side of the table reports validation accuracy averaged over three random seeds; the right side lists test accuracy reported by this paper and earlier studies.\n\n- HiPPO-LegS reaches the highest validation accuracy in the left-side comparison, 98.34%; its reported test accuracy on the right is 98.3%.\n- LegT performed well when the window length $\\theta$ was set to the ideal value but suffered a significant performance drop when $\\theta$ was set incorrectly.\n- LagT also performed well with an appropriate $\\Delta t$.\n\nLMU had been the state-of-the-art (SOTA) for this task in RNN-based models, but LegS surpassed LMU and other sequence models, achieving a test accuracy of 98.3% on pMNIST, setting the \"strict sequence model\" true SOTA.\n\n\n> [!TIP]\n> \n> \"Strict sequence\" means that a model reads the long input step by step, without seeing future values at the current step. At time $t$ it may use $f_1, f_2, \\dots, f_t$, but not $f_{t+1}, f_{t+2}, \\dots$. RNNs, LSTMs, and GRUs naturally update along this time axis.\n> \n> The paper's right-side table also includes nonrecurrent models, such as Transformer and TCN, that can use a global receptive field over the sequence, so the authors compare them separately from stepwise RNNs. A causal mask restricts each position to the past, including during parallel training.\n\n\n### Timescale Robustness\n\nSequence models generally benefit from appropriate timescale priors, but these priors are often presented as additional hyperparameters and are not automatic solutions. For example:\n\n- In LSTM, the \"forget bias\" parameter requires adjustment to handle long-term dependency issues.\n- HiPPO-LagT and HiPPO-LegT require choosing an appropriate step size $\\Delta t$ or window length $\\theta$ during discretization.\n\nIn the previous section, ablation experiments were conducted, showing that if a sliding window mechanism like LegT does not have the correct $\\theta$, its performance drastically drops.\n\nIn the paper, the author used the **Character Trajectories** dataset as an experimental target.\n\n- [**[18.11] The UEA Multivariate Time Series Classification Archive**](https://arxiv.org/abs/1811.00075)\n\nThe task of this dataset is to classify characters drawn by handwriting strokes (coordinates collected at a fixed sampling rate). To simulate \"timescale shifts,\" the author considered two standard time series generation processes:\n\n1. **Sampling Rate Change**:\n   The training model's sampling rate is different from the testing sampling rate, resulting in different timescales between training and testing. Since the sequence length can vary, the model cannot directly infer the change in sampling rate from the sequence length.\n\n2. **Irregular Sampling or Missing Data**:\n   In this case, the time stamps of the sequence are not at fixed intervals. A simple transformation is performed: the time stamps in the testing phase are scaled. If the model can naturally handle multiple timescales, it will automatically adapt to the time expansion or compression of the test data.\n\n---\n\nHiPPO treats input data as continuous functions and processes observed samples through discretization. Supplying the elapsed time and an input interpretation between observations lets its update handle irregular sampling; missing values are not reconstructed automatically. Together with LegS's timescale equivariance, this motivates testing whether it adapts under those settings.\n\nIn this experiment, the author compared HiPPO-LegS with other models designed for irregular sampling and missing data, including:\n\n- **GRU-D**: Models the time intervals between observations using a decay mechanism.\n- **Neural ODE Models**: Use ODEs to simulate state evolution between observation points, naturally handling irregular sampling.\n\nThe results are shown in the table below:\n\n\n\n![time-scale-robustness](/content/papers/assets/548b53e12e900a583e64f02b94728f7546b41fa0a7c27a66bf73a1899f130252.jpg)\n\n\n\nWithout the timescale shift, all listed models exceed 95% accuracy. Under the tested shifts, several baselines fall sharply.\n\nHiPPO-LegS scores 88.8%–94.9% across the four shifted test settings in this table, above each corresponding baseline. That supports greater robustness in these particular evaluations; the shifts do still lower its accuracy.\n\n### Scalability Verification\n\n\n\n![function-approximation](/content/papers/assets/da5cf9efca254a5f9118d9980f42af2cbd41816b48cbf27d4be2b3f86b7a2e4c.jpg)\n\n\n\nIn this function-reconstruction experiment, the paper reports about 0.25 error for LSTM even at sequence length 1,000. This is not a universal upper limit on the sequence lengths LSTMs can process.\n\nTo test very long sequences, the authors sample a continuous-time, band-limited white-noise process and reconstruct a sequence of length $10^6$, allowing each method up to 256 memory units.\n\nAfter one million steps, LMU and HiPPO-LegS obtain reconstruction MSEs of 0.05 and 0.02, respectively, below the LSTM result of 0.25. This is lossy reconstruction, not exact recovery of every input value.\n\nThe next figure shows HiPPO-LegS tracking the input curve more closely in this setting, with larger LSTM reconstruction deviations:\n\n\n\n![scalability](/content/papers/assets/c0be95bf1a71708b07ff305ad66534a93fa3e6b6377ccbdf6899330c2fd374af.jpg)\n\n\n\nThe authors also measure the memory updater's practical speed, in addition to its $O(N)$ per-step complexity under the analyzed discretization:\n\nWith a C++ updater and PyTorch bindings on one CPU core, HiPPO-LegS processes 470,000 elements per second in this function-reconstruction benchmark; the LSTM and LMU updater rows show 35,000 and 41,000. These are memory-update throughput measurements, not speed for full classification models.\n\n## Conclusion\n\nThe HiPPO framework treats memory representation as an ongoing optimal function approximation problem over time, and by introducing a measure of time variation, it unifies and explains existing memory mechanisms.\n\nHiPPO-LegS, as an instance of the HiPPO framework, not only theorizes towards \"timescale robustness,\" but also practically demonstrates its effectiveness in handling long-term dependency problems with sequences up to millions of time steps and with high computational efficiency.\n\n\n> [!TIP]\n> \n> We’ve finally covered the full paper!\n> \n> HiPPO has also been applied to the S4 framework, leading to the development of the Mamba model. This shows the tremendous power of this theoretical framework in real-world applications.\n> \n> Finally, we highly recommend readers to read the original paper to gain a deeper understanding of HiPPO's potential.\n","imageSizes":{"/content/papers/assets/0edb34026ecfca3a51cdc9a978386d1f6a0aab6d88e101c0fd44d7a9db21462f.jpg":{"width":1440,"height":514},"/content/papers/assets/1d8ecfa323942e41770f4191d4b5891dfbf88d450a0da7e5b95672d512c251a7.jpg":{"width":1164,"height":616},"/content/papers/assets/548b53e12e900a583e64f02b94728f7546b41fa0a7c27a66bf73a1899f130252.jpg":{"width":1992,"height":350},"/content/papers/assets/da5cf9efca254a5f9118d9980f42af2cbd41816b48cbf27d4be2b3f86b7a2e4c.jpg":{"width":804,"height":320},"/content/papers/assets/c0be95bf1a71708b07ff305ad66534a93fa3e6b6377ccbdf6899330c2fd374af.jpg":{"width":2772,"height":999}}}
