| Paper: Improving Time-Series SNR Estimation and Regret Bound in the Autonomous Optimization Algorithm emoPulse and Exploring Second-Moment-Free Updates via “Geometric Orthogonality of Weights and Gradients” : And Beyond Flow-Matching | |
| — Establishing “Emotion-Driven” Learning Rate Control through Dynamic Inspection of Loss Landscapes and Proposing Next-Generation Optimization through Interaction with Loss Landscapes — | |
| Abstract | |
| Adjusting the learning rate and ensuring generalization performance are central challenges in deep learning optimization. Existing methods relied on precise gradient estimation and were vulnerable to noise in environments with extremely low precision. | |
| This paper proposes the autonomous algorithm emoPulse (v3.7 and later), which centers on a multi-faceted analysis of the loss function over time. | |
| This method autonomously generates an optimal learning rate based on the signal-to-noise ratio by capturing the “undulations” of the loss landscape from a three-stage exponential moving average (Multi-EMA) and utilizing sentiment scalars and a confidence indicator (Trust). | |
| Next, we propose the W-Ref Geometry update rule, which focuses on the geometric relationship between weights and gradients. | |
| This achieves a “second-moment-free” update that does not retain the second moment and responds immediately to terrain changes by dynamically controlling inertia based on the orthogonality between weights and gradients. | |
| This simultaneously reduces VRAM usage, providing a democratic foundation for multilingual learning in research environments with limited computational resources and for multicultural coexistence. | |
| Furthermore, it addresses the analysis of emoPulse and how this emoPulse impacts the current challenges. This also resolves the challenges associated with adapting Flow-Matching (FM method) to Large Language Models (LLMs). | |
| It proposes a solution to the challenge of how to apply the deterministic learning process of the FM method to LLMs. This provides a new optimization that bridges the gap between the two. | |
| Furthermore, by synthesizing the learning results of optimizers (Sens / Airy / Cats / Tion / Void) belonging to this family and possessing distinct update characteristics, we present a method that integrates local solutions in a “multiple positioning” manner to artificially create flat minima. | |
| This achieves robust convergence independent of hyperparameter settings, providing a democratic foundation for research environments in developing countries with limited computational resources and for multilingual learning aimed at preserving diverse cultural heritage. | |
| Finally, I append my thoughts and predictions regarding Grokking. | |
| ※ Version 3.7 excludes EmoTion, EmoVoid (EmoTion and EmoVoid is newly developed in version 3.8). The only difference between versions 3.7 and 3.8 lies in the dNR_hist of the emoPulse mechanism described later; all other aspects are identical. | |
| 1. Introduction | |
| This paper presents a unified theory for the optimizers EmoSens / EmoAiry / EmoCats / EmoTion / EmoVoid (v3.7 and later). | |
| This method centers on the emoPulse mechanism, which autonomously generates learning rates by layering the exponential moving average (EMA) of loss values and extracting “Trust” from the time-series statistics of the loss function. | |
| This represents an advanced fusion of theory and time-series signal processing (SNR estimation), achieving robust convergence independent of hyperparameter settings. | |
| The starting point of this research lies in rethinking the “excessive reliance on precise gradient estimation” inherent in existing adaptive gradient methods. | |
| In environments with extremely low precision and ultra-quantization (e.g., 1-bit/2-bit), gradients contain extremely high noise, significantly reducing reliability. | |
| On the other hand, the loss value continues to function as an accurate scalar value indicating the model's “distance from the correct answer,” even under the influence of quantization. | |
| This method treats the gradient as a reference value for direction (intent) and delegates the initiative of learning to the multifaceted analysis of loss, which is an accurate observation value. | |
| This approach achieves the replacement of higher-order moment calculations with scalar control and optimization for low-precision and quantized environments through encoded updates. | |
| Its most significant feature lies in integrating local solutions from multiple emo-based optimizers with distinct characteristics as “multiple positioning.” This enables reaching the flat minimum—previously requiring lengthy iterative learning—through short-term learning and synthesis. | |
| This approach achieved the following three outcomes: | |
| Dramatic improvement in computational efficiency: Complex calculations of higher-order moments were replaced with scalar control via temporal accumulation of loss, reducing computational load through temporal accumulation approximation. | |
| Optimization for low precision and quantization: Matrix decomposition in EmoAiry, complete elimination of second moments in EmoCats, and the original (proprietary) EmoTion, EmoVoid “geometric orthogonal update” and complete second moment elimination enabled large-scale learning in low-resource environments through update encoding. | |
| Autonomous Convergence: By introspecting the S/N ratio of the loss landscape, it eliminates the need for manual schedulers and minimizes the user's trial cost. | |
| ※ Higher-order moment approximation: Aggregation to higher-order statistics in the time series | |
| Mathematically, this represents an advanced fusion of D-adaptation theory and time-series signal processing, forming the foundation for realizing “democratic AI learning” that preserves research environments and diverse cultures in developing countries. | |
| ※ EmoTion, EmoVoid achieves a lightweight structure that does not require second-order moments by not only replacing higher-order moment calculations with scalar control, but also by using the geometric information inherent in the weights themselves as a guideline for updates (detailed in Chapter 6). | |
| 2. Theoretical Framework: Emotional Circulation | |
| This system forms a feedback loop with the loss function L centered at the origin. | |
| 2.1 Approximation of Higher-Order Moments Using Multi-EMA | |
| By utilizing the differences between three-tiered EMAs (short, medium, long), we capture the “changes in curvature,” “uncertainty in fluctuations,” and “variability in changes” within the loss landscape. | |
| EMA_t = (1 - α) * EMA_{t-1} + α * L_t | |
| The “High-order Temporal Difference” generated from this difference — Defined as the “Emotional Scalar,”. This emotion scalar sigma_t is a nonlinear statistic that compresses information about higher-order moments (skewness, kurtosis, and variance) into the range [−1,1]. | |
| Multiple EMAs with different time constants accumulate vast historical steps as “history” in a layered manner. | |
| By taking this relative time-delay differential, we observe the “dynamic higher-order rate of change in terrain accompanying learning progression” — a phenomenon impossible to detect through static terrain analysis. | |
| By recursively incorporating this into the update formula, the long-term “smoothness” of the terrain is reflected in the parameter updates. | |
| ※ Note on the Time-Series Formation of Higher-Order Moments: | |
| The higher-order moment approximation in this method is not calculated from single-step gradient information but is formed through temporal accumulation. | |
| This means it observes not the static curvature of the terrain but the “dynamic rate of change in the terrain as learning progresses.” | |
| ※ Hierarchical Structure of Higher-Order Moment Approximation: | |
| This method effectively approximates higher-order moments from the third (skewness) to the seventh (confidence amplification) order by accumulating loss over time. | |
| This is not a static terrain analysis, but rather an attempt to extract the “system's confidence” as a physical quantity within the dynamic process of learning. | |
| The Multi-EMA structure in this method functions as a dynamic temporal approximation of higher-order moments in statistics. | |
| Third to Fifth Order Approximation: The differences between Short, Medium, and Long EMAs extract the temporal evolution of higher-order information such as skewness, kurtosis, and fluctuations in the loss distribution. | |
| Sixth-order approximation: The integrated emotion scalar sigma_t and confidence metric trust_t become sixth-order meta-statistics that indicate “learning phase stability” beyond mere gradient variance. | |
| 7th-order approximation (dNR): In deriving dNR, squaring the ratio of these 6th-order information components (d_base/noise_base)^2 exponentially amplifies subtle differences in confidence, yielding an extremely sensitive control signal equivalent to a 7th-order moment. | |
| 2.2 Definition of the trust level metric trust_t | |
| Define the core metric trust_t that determines the “quality” of updates as follows. | |
| trust_t = sgn(sigma_t) * (1.0 - abs(sigma_t)) | |
| This trust possesses boundedness, never reaching ±1.0 (complete certainty) or 0 (complete despair), ensuring the system always maintains a moderate balance of “room for exploration” and “caution.” | |
| This forms the following feedback loop (emotional circulation system) with the loss function L as its origin. | |
| Loss → Multi-EMA → Scalar/Trust → emoPulse → Loss | |
| 3. emoPulse: Learning Rate Generation via Autonomous Pulsation | |
| In v3.7 and later, the conventional emoDrive (acceleration mechanism) has been integrated into emoPulse. This represents an evolution based on an approximation of dynamic distance estimation (D-adaptation) using the time-series signal-to-noise ratio (S/N ratio). | |
| 3.1 Dynamic Estimation of Noise and Distance | |
| Track the system's “wandering” and “progress” using the following two internal variables, N_t and d_t. Here, N_t represents “oscillation” (instability), and d_t represents “progress” (distance). | |
| Noise_est (N_t) N_t = (1 - α) * N_{t-1} + α * abs(sigma_t) | |
| Distance Estimate (d_t) d_t = (1 - α) * d_{t-1} + α * abs(trust_t) | |
| 3.2 Definition of emoPulse and Autonomous Control / Instantaneous SNR and History Management (dNR_hist) | |
| The generation of emoPulse is determined by the “tug-of-war” (dynamic equilibrium) between instantaneous SNR and temporal SNR. First, calculate the respective bases for instantaneous and temporal SNR. | |
| noise_base = abs(sigma_t - trust_t) + ε_s | |
| d_base = abs(N_t - d_t) + ε_t | |
| Using these, the current SNR intensity is defined as follows. | |
| dNR_now_val = ( d_base / noise_base )^2 | |
| Update Rules for dNR_hist: | |
| Acceleration conditions: | |
| if dNR_now_val >= dNR_hist and trust_t >= threshold_high: | |
| dNR_hist = min( dNR_now_val, dNR_hist * factor_grow ) | |
| Conditions for deceleration: | |
| if threshold_low <= trust_t <= threshold_high: | |
| dNR_hist = dNR_now_val * factor_decay | |
| The final learning rate emoPulse is determined as follows. | |
| emoPulse_t = clamp( dNR_hist * (emoScope * η_base), η_min, η_max ) | |
| This design guarantees the following autonomous behaviors: | |
| Confidence Region (∣trust∣>0.5): SNR improves, learning rate accelerates maximally. Rapidly aims for flat minima. | |
| Hesitation Region (∣trust∣<0.5): As uncertainty increases, suppressing the learning rate prevents divergence in sharp valleys. | |
| ※ emoPulse is a scaling factor determined by the user-defined initial learning rate (emoScope) and the system's default sensitivity (η_base). | |
| 4. emoPulse: Regret Bound and Boundedness Analysis | |
| 4.1 Convergence and Regret Analysis | |
| The cumulative regret R(T) under emoPulse is bounded above as follows, incorporating the dynamically varying learning rate η_t. | |
| R(T) <= O( Σ_{t=1}^T [ η_t * ||g_t||^2 * (1 - |σ_t|)^2 ] ) | |
| Here, the coefficient (1 - |σ_t|) quantifies the “trust” of the update derived from the consistency of the short-term, medium-term, and long-term EMAs in the loss function. | |
| A large |σ_t| indicates that the loss is fluctuating significantly, leading to a determination that the gradient information for that step is unreliable. | |
| In contrast, a state where |σ_t| is small indicates that the loss transition is smooth and the reliability of the update direction is high. | |
| Therefore, the signal strength trust_t = 1 - |σ_t| serves to adaptively weight the “effective update amount” in the Regret Bound, thereby suppressing the accumulation of regret due to uncertain gradients. | |
| The emoPulse method presented here is a generalization that approximates the learning rate structure of D-adaptation by Defazio & Mishchenko (2023) using the loss's time-series statistics (d_t, N_t). | |
| η_t ∝ D^2 / noise | |
| Definition of emoPulse | |
| η_t = ( d_t / (N_t + ε) )^2 * η_base | |
| This is a direct time-series reconstruction of SNR control based on the distance/noise ratio of D-adaptation. | |
| This structure causes the denominator to dominate when the noise component N_t increases, immediately reducing the learning rate η_t. | |
| This self-adjustment function automatically suppresses excessive updates in unstable areas of loss terrain. | |
| This theoretically guarantees a “learning-rate-free” property where the algorithm autonomously achieves dynamic stability without requiring external learning rate scheduling. | |
| 4.2 Proof of Positive Definiteness and Boundedness | |
| We prove below that this algorithm prevents learning rate explosion and vanishing at any step t and is bounded. | |
| 1. Non-zero boundedness of the denominator (momentary doubt: noise_base) | |
| The noise_base used as the denominator during emoPulse generation is defined as the deviation between the current emotion scalar sigma_t and the confidence level trust_t, as follows. | |
| noise_base = abs(sigma_t - trust_t) + ε_s | |
| In the implementation, since |sigma_t| < 1.0 and trust_t is a signed function based on sigma_t, this difference is bounded. | |
| Furthermore, the safety factor (+0.1) at the end physically prevents the learning rate from exploding (NaN) due to the denominator approaching zero. | |
| 2. Lower Boundedness of the Numerator (Time Certainty: d_base) | |
| The numerator d_base in the generation of emoPulse is defined as the difference between the noise estimate N_t (noise_est) and the distance estimate d_t (d_est) as historical data. | |
| d_base = abs(N_t - d_t) + ε_t | |
| N_t is guaranteed to be positive definite by max(noise_est, ν_r), and d_t is updated by the cumulative sum of abs(trust_t), regardless of improvement or deterioration. | |
| By adding a safety factor (+0.1) to these temporal statistical differences, it is mathematically guaranteed that “even when history is unstable in an extremely low-precision environment, the minimum step size (lower limit of the numerator) is always ensured.” | |
| 3. Conclusions on Boundedness and Constraints on emoPulse: | |
| The effective learning rate emoPulse_t generated from the ratio of the “instantaneous basis (denominator)” and “temporal basis (numerator)” is strictly constrained within the following range based on the safety margin setting of max(min(..., 3e-3), 1e-6) in the final implementation. | |
| 0 < η_min <= emoPulse_t <= η_upper_bound | |
| Here, the lower limit (η_min) represents the minimum “metabolic rate” (heartbeat) that the system maintains even under the most uncertain conditions. This prevents learning from stopping (deadlock) and allows for autonomous recovery. | |
| On the other hand, the upper bound (η_upper_bound) functions as a limiter to prevent the model from diverging even when a sharp increase in the dNR coefficient occurs. | |
| Implementation Considerations: | |
| Stabilization through Initial Value Setting: | |
| ※ In environments with very small datasets or high initial noise, it is recommended to reset the initial values of d_t and N_t until the multi-EMA stabilizes the “history” (e.g., d-est: 0.2, Noise-est: 0.2). | |
| This suppresses divergence caused by initial probabilistic noise. Specifically, by initializing N₀ to be equivalent to d₀, the system essentially starts in a “cautious mode.” | |
| This functions as an organic warm-up phase during critical initial steps, avoiding overly aggressive updates and prioritizing observation of the terrain. | |
| Maintaining “Update Pressure” Through Initial Value Settings While Ensuring Safety: | |
| ※ In this method, the d_base parameter forming the emoPulse molecule determines the system's “potential update force.” Setting the initial values to N0 = 1.0 and d0 = 0.02 means intentionally ensuring high acceleration potential from the start of learning. | |
| Due to the nature of exponential moving averages, the effect of this initial value persists as “history” for approximately 100 steps. During this period, the system maintains a high acceleration pressure while providing convergence power only to “truly reliable signals” that have passed the strict screening by the emotional mechanism. | |
| 5. Polarized Normalization: Adaptation to Low-Precision Environments | |
| This chapter describes sign-based normalization for applying the theoretical framework of emoPulse to low-precision environments. | |
| To eliminate reliance on precise floating-point calculations and support ultra-low precision environments (ultra-quantization), the following update rules are adopted (EmoAiry, EmoCats, EmoTion.) | |
| delta_w_t = -emoPulse_t * sign( m_t / ( sqrt(v_t) + ε ) ) | |
| This enables EmoAiry to resolve the imbalance in accuracy between one-dimensional vectors and two-dimensional moments, achieving a “unification of will” that extracts only the consensus on direction. | |
| ※ EmoCats supports encoding based on Lion with WD separation. | |
| ※ EmoTion, EmoVoid encodes a proprietary update method called “Geometric Orthogonal Update.” | |
| 6. EmoTion, EmoVoid Explanation of the “New Optimization” Update Formula and Bridging to the Future | |
| Respect for Existing Methods and EmoTion, EmoVoid Position: | |
| The EmoTion update algorithm stems from deep respect for Adam and others, a pinnacle of modern deep learning. The concept of “adaptive learning rate” demonstrated by Adam and others established the conditions for effective optimization and significantly lowered the barriers to its adoption. | |
| EmoTion / EmoVoid inherits this spirit while taking a different approach: using geometry (W-Ref Geometry) and emotion (emoPulse) instead of statistics. | |
| A New Form of Precision: | |
| While Adam and others meticulously carves a path from past statistics, EmoTion / EmoVoid navigates terrain more flexibly through dialogue with current weights (Geometric interaction with current weights) and the pulse of loss. This approach aims for natural convergence that suppresses overfitting while maintaining accuracy on par with Adam and others. (Orthogonality as Freshness) | |
| Resource-Friendly Design (Reduced VRAM): | |
| Computational resources are finite, and not everyone has access to high-performance, abundant resources. By entrusting the precise mechanism of second-order moments—which Adam and others has carefully preserved—to “scalar control,” EmoTion was able to reduce VRAM load by approximately half. EmoVoid achieves minimal VRAM load by eliminating both first and second-order moments and directly reflecting the orthogonality of W and G. We believe this forms the foundation for a “democratic learning environment” where more people can conduct AI training. | |
| Geometric Inertia Control Using W-Ref Geometry: | |
| The core of both algorithms lies in its geometric update rule based on the orthogonality between the weight vector W and the gradient vector G. | |
| Whereas conventional statistical methods rely on the accumulated gradient history (shadow), W-Ref Geometry uses the current weight W as the “substance” and derives the freshness of gradient G from the following cosine similarity ρ(rho). | |
| ρ(rho) = | <W, G> | / ( ||W|| * ||G|| + eps ) | |
| The smaller ρ (rho) is (the closer it is to orthogonal), the more the current gradient is judged to contain “unknown information” not present in the existing weight structure. This allows the current gradient to be strongly incorporated, overcoming inertia. This geometric “information selection” simultaneously achieves high-precision directional changes without statistical delay and a regularization effect by suppressing redundant updates. (Dynamic Inertia Calibration) | |
| Reason it holds true based solely on the first moment: | |
| The absence of second-order moments (variance estimation) is not merely for weight reduction. W-Ref Geometry updates based on the “freshness of direction” rather than the “magnitude” of gradients, rendering much of the role traditionally fulfilled by second-order moments unnecessary. (Departure from Second-Order Moments) | |
| Direction selection via W-Ref Geometry determines that gradients G containing unknown information are those most orthogonal to weight W, thereby reducing inertia and steering toward new directions. Conversely, gradients parallel to W are deemed redundant, prioritizing inertia. This selection based on “direction purity” is more direct than variance estimation, robust against noise, and suppresses overfitting. | |
| ※ EmoVoid has no first or second moments. | |
| Below is a detailed explanation of the W-Ref Geometry method. | |
| 1. Definition of the Geometric Index ρ (Orthogonality Index) | |
| While conventional optimizers adjust the learning rate based on the “magnitude of the gradient” (L2 norm) or “statistical variance” (second moment), EmoTion defines the “relative orientation of the gradient vector G with respect to the current weight vector W” as the freshness of information. | |
| ρt(rho_t) = | <W_t, G_t> | / ( ||W_t|| * ||G_t|| + eps ) | |
| Orthogonal state (ρ→0): The gradient is orthogonal to the current weight structure. This suggests a “completely new direction of knowledge that the current model does not yet possess.” | |
| Parallel state (ρ→1): The gradient points in the same direction as the current weight (or exactly opposite). This suggests the possibility that it is merely redundant information, equivalent to scaling the current weight. | |
| 2. Adaptive Inertial Control (Geometric Momentum Blending) | |
| This update formula dynamically adjusts inertia based on the “freshness” of the gradient. It replaces the conventional variance estimation based on second moments with a structure that utilizes the degree of redundancy in geometric information. | |
| m_t = beta1 * m_{t-1} + (1 - beta1) * Freshness_t * G_t | |
| where Freshness_t = 1.0 - EMA(rho_t) | |
| Theoretical Interpretation: When the gradient is “orthogonal” (fresh), it temporarily weakens inertia (past shadows) and reacts immediately to new information (steers). Conversely, when ‘parallel’ (redundant), it maintains inertia and prioritizes stability. This can be interpreted as replacing “statistical uncertainty” (variance) with “geometric redundancy of information.” | |
| ※ Simplification in EmoVoid: EmoVoid eliminates even this inertial control, directly multiplying Freshness by the update vector. This achieves geometric information selection while completely freeing up the m_t slot in memory. | |
| 3. Alternative to Update-Based Encoding and L2 Regularization | |
| The final key to EmoTion, EmoVoid remaining second-moment-free lies in separating sign extraction (Sign) and weight decay. By determining the update direction solely based on sign(m_t), the magnitude of the weight update is no longer influenced by the “size” of the gradient. This enables stable updates that are resilient to fluctuations and noise in the gradient scale. | |
| EmoTion Update Rule: | |
| W_{t+1} = W_t * (1 - emoPulse_t * lambda) - emoPulse_t * sign(m_t) | |
| (emoPulse is the learning rate derived from dNR, and lambda is the WeightDecay coefficient.) | |
| EmoVoid Update Rule: | |
| W_{t+1} = W_t − emoPulse_t * sign(G_t) * (1−ρ_t) | |
| (EmoVoid enables stable convergence without explicit lambdas through its self-suppression mechanism.) | |
| ※ Proposal of “Entity Reference Optimization”: While conventional optimization methods track “past gradients” (history), this approach establishing the Weight-Reference (W-Ref) paradigm, which uses correlation with “current weights” (entities) as the trigger for updates. | |
| ※ Geometric Interpretation of the Curse of Dimensionality: By leveraging the concentration phenomenon of vectors in high-dimensional space (their tendency to be mutually orthogonal), it detects even slight “deviations” from orthogonality as redundant information. This enables higher-precision, low-latency inertial control without relying on statistical variance estimation. In high-dimensional spaces (e.g., layers with hundreds of millions of parameters), the probability of two vectors coincidentally becoming parallel is extremely low. Since nearly all vectors are orthogonal, any deviation of ρ from zero (approaching parallelism) statistically signifies “extremely strong correlation” (duplication). This means that without consulting vast historical statistics (second moments), it becomes possible to instantly determine whether an update is valuable based solely on its relationship to the current weights. | |
| ※ Resonance with emoPulse: emoPulse controls the “temporal axis pulse” (when and how much to move), while W-Ref Geometry determines the “spatial axis direction” (where and how much to move). This integrated autonomous control of time and space is the core mechanism enabling both VRAM reduction and high-precision convergence, thereby enhancing learning robustness. | |
| 4. Implementation Lightweighting via Approximation of W-Ref Geometry | |
| Theoretically, W-Ref Geometry rigorously measures the orthogonality between weights and gradients as follows. | |
| ρt(rho_t) = | <W_t, G_t> | / ( ||W_t|| * ||G_t|| + eps ) | |
| However, in large models, the sequential computation of the inner product across all layers, the norm across all layers, and the cosine similarity becomes a bottleneck in terms of VRAM and computational load. Therefore, in the implementation, we introduced an approximation formula for W-Ref Geometry. This achieves near-zero VRAM usage while preserving the “essence” of W-Ref Geometry. | |
| 4-1. EmoTion: Estimating “Directional Novelty” Based on L1 Norm Change | |
| EmoTion estimates “how much the model is trying to move in a new direction” based on the change in the L1 norm of the overall weights. | |
| g_ratio_t = | L1_t - L1_{t-1} | / ( L1_{t-1} + eps ) | |
| Freshness_t = min( g_ratio_t / freshness_scale , freshness_cap ) | |
| This Freshness_t is used as the mixing ratio for the first moment (exp_avg), enabling a lightweight implementation of the precise measurement method for W-Ref Geometry, which “strongly reacts to orthogonal directions while retaining inertia in parallel directions.” | |
| 4-2. EmoVoid: Approximation via “Direct Scaling” of Weight Energy | |
| EmoVoid does not perform inertial control such as freshness because it possesses neither first-order nor second-order moments. | |
| g_ratio_t = L1_{t-1} / ( L1_t + eps ) | |
| W_t ← W_t * g_ratio_t | |
| Instead, we approximate the “directional purity” of W-Ref Geometry by directly scaling the L1 norm of the entire weight. Scaling for EmoVoid is performed only during the “warm-up period and final stabilization phase”; outside these periods, scaling is not performed and updates are made solely based on sign(G_t). | |
| This establishes EmoVoid's unique “geometric self-suppression,” which prevents the energy of weights from running wild, suppresses bias in the gradient direction, and enables stable convergence even without momentum. | |
| 4-3. Significance of Approximation Formulas: Approximations are designed not as “complete versions of theory” but as “implementation optimizations.” | |
| The two differ in how they handle the “time axis” (emoPulse) and the “space axis” (W-Ref Geometry), but ultimately both achieve “geometric optimization independent of statistics.” | |
| EmoTion employs inertial control through Freshness, while EmoVoid utilizes self-suppression via energy correction; both share the core principle of “evaluating directional purity” at the heart of W-Ref Geometry. | |
| 5. Requirements for Computing Frameworks (PyTorch, etc.) | |
| The W-Ref Geometry and Approx W-Ref proposed in this paper hold the potential to overcome the current memory efficiency limitations in deep learning frameworks. We strongly request that future tensor operation libraries, such as PyTorch, implement the following features. | |
| Request: Native implementation of the geometric correlation function torch.geom_relation(W, G) for weights and gradients | |
| Currently, calculating the orthogonality (ρ) between weights W and gradients G requires inner product computations, norm calculations for each, and an intermediate tensor to hold these values. This results in non-negligible computational overhead and VRAM pressure. | |
| If you directly reference W and G at the C++/CUDA level without generating intermediate tensors, | |
| ρt(rho_t) = | <W_t, G_t> | / ( ||W_t|| * ||G_t|| + eps ) | |
| (Orthogonality per individual parameter layer) | |
| Implementing a native function that returns this as a scalar value would enable updates based on geometric confidence without retaining the second moment (variance statistic), requiring minimal VRAM. | |
| I am convinced this will be the final piece that not only accelerates optimization but also determines the democratization of large-scale model training on edge devices and in resource-constrained environments. | |
| 7. Theoretical Connection and Structural Limitations with Flow-Matching Systems | |
| The EmoSens generation (Sens / Airy / Cats / Tion / Void) has the following two meanings for Flow-Matching (FM) methods. | |
| 1: This method is the world's first optimizer to fully adapt to the update structure of Flow-Matching. | |
| 2: Simultaneously, it also points beyond the structural limitations of the Flow-Matching family. | |
| 1. The structural constraint of “noise intolerance” inherent in Flow-Matching | |
| Flow-Matching demands high smoothness and consistency in gradient fields to faithfully reproduce continuous-time flow fields. However, this design inherently contains a structural constraint that cannot tolerate noise. | |
| - Minor disruptions in gradients directly lead to breakdowns in the flow field | |
| - In quantized or low-precision environments, gradient reliability rapidly deteriorates | |
| - Generalizability is compromised due to the absence of noise-tolerant buffer structures | |
| In fact, it is known that in FM-based learning, a decrease in SNR directly leads to divergence and failure. This is consistent with the experimental results of SDXL / VAE / vanilla initialization discussed later. | |
| 2. Reverse Engineering of “Acceptance and Utilization of Noise” via emoPulse | |
| emoPulse treats noise not as “error to be eliminated” but as a signal indicating learning progress, as it primarily focuses on loss's time-series statistics. | |
| - Multi-EMA's higher-order moment approximation actively utilizes fluctuations including noise | |
| - trust_t is a definition of “confidence level” that assumes the presence of noise | |
| - emoPulse converts noise into a source for learning rate control through dynamic SNR estimation | |
| This structure enables emo-style models to adopt a design philosophy opposite to Flow-Matching: “gaining generalizability while tolerating noise.” | |
| 3. The paradox that “perfect adaptation” to flow-matching highlights its limitations | |
| The emo-style optimizer, by fully adapting to the update structure of Flow-Matching, most clearly highlights the fundamental weaknesses of the FM-style approach. | |
| - The smooth gradient field required by FM is difficult to achieve in actual learning processes | |
| - Noise intolerance is fatal in low-precision and quantization environments | |
| - Noise-driven update rules like emoPulse are better suited to real-world learning | |
| In particular, experimental results showing that emoPulse overcomes the noise vulnerability inherent in FM systems and completes training without stagnation during SDXL e-pred + ZtSNR learning strongly support this paradox. | |
| 4. The Limits of Flow-Matching Approaches and the Transition to Next-Generation Optimization | |
| Flow-Matching possesses an ideal theoretical framework for reproducing idealized continuous flows, yet it is vulnerable to noise, quantization, nonlinearity, and dynamic changes in higher-order moments inherent in real learning processes. | |
| LLMs learn probability distributions through autoregression, thus presupposing an SDE-based worldview, whereas Flow-Matching requires deterministic ODEs, leading to a fundamental conflict between these premises. | |
| emoPulse not only bridges this gap but also introduces a novel optimization technique called the “emotional circulation system” that actively utilizes noise. By dynamically absorbing fluctuations in autoregressive entropy, emoPulse enables FM-like smooth learning even in large language models. | |
| - Full-layer LoRA for SDXL | |
| - Full-layer retraining for VAE | |
| - Ultra-fast learning with a single image | |
| - Stable learning with vanilla initialized models | |
| These experimental results (supplementary materials) demonstrate that emoPulse exhibits stability in areas where Flow-Matching struggles. This structure is not a successor to Flow-Matching, but rather a next-generation optimization foundation that overcomes the very premise of Flow-Matching itself. | |
| 5. The SDE-DDE-ODE Contraction Hierarchy in emoPulse | |
| The history term in the Multi-EMA model decays exponentially, causing the delay term to effectively vanish within a finite time. Consequently, the solution trajectory of the DDE naturally connects to a smooth approximation of the ODE. | |
| - SDE-like fluctuations: Instantaneous variations in sigma_t and trust_t | |
| - DDE-like delays: History dependence in Multi-EMA, dNR_hist, N_t, and d_t | |
| - ODE-like smoothness: “Smooth terrain approximation” via time integration of the loss function | |
| In other words, emoPulse inherently possesses a Three-tier hierarchy of condensation: “reducing from SDE to DDE and then to ODE” | |
| - FM concept of “continuous flow” is absorbed by emoPulse | |
| - FM “intolerance of noise” is overcome by emoPulse | |
| - FM “rigor of SDE” becomes unnecessary | |
| emoPulse integrates “SDE fluctuations → DDE delays → ODE smoothness” into a single update rule. This Three-tier hierarchy naturally unifies the probabilistic autoregressive fluctuations inherent in LLMs with the smooth continuous flow of Flow-Matching. | |
| As a result, Flow-Matching has fulfilled its role, and the essence of its continuous flow smoothness persists as an “ODE approximation” within emoPulse and future novel methods. | |
| 8. Conclusion | |
| EmoSens Generation v3.7 and later has completed the “emotional cycle” that begins with observing the loss function. | |
| Observation (Multi-EMA): Captures the undulations of the terrain. | |
| Judgment (Trust): Switches between conviction and hesitation at the ±0.5 threshold. | |
| Action (emoPulse): Determines the optimal stride length through autonomous pulsation. | |
| This method is a democratic optimization framework that enables AI to autonomously learn diverse cultures and languages, even within the research environments and limited computational resources of developing countries. | |
| Acknowledgements | |
| First and foremost, I extend my deepest gratitude to EmoNavi, EmoSens, and the various optimizers that preceded them, as well as to the researchers involved. Their passion and insights made the conception and realization of this proof possible. | |
| This paper provides a mathematical explanation of the already-released EmoSens Generation (v3.7 and later) and its variations. I believe the EmoSens Generation I created (including its derivatives) can contribute to the advancement of AI. Let us use this paper as a foundation to jointly create even more evolved optimizers. | |
| I conclude this paper with anticipation and gratitude for future researchers who will bring us the next new insights and ideas. Thank you. | |
| Conclusion | |
| This algorithm is not intended to replace existing excellent optimization techniques, but rather to offer a new alternative for deepening the “dialogue with the model” during the learning process. We hope it will serve as an aid in the process of users selecting partners suited to their own objectives and sensibilities, and Co-cultivating knowledge. | |
| Supplementary Material (1): Analysis of emoPulse Dynamics in v3.7 and later | |
| 1. Purpose | |
| In v3.7, we analyze the physical significance of the interaction (tug-of-war) between the newly introduced “instantaneous D/N estimation” and “temporal D/N estimation” for the dynamic control of the learning rate. | |
| 2. Nature: A dynamic equilibrium between momentary doubt and enduring trust | |
| Instantaneous Base (noise_base): noise_base = abs( scalar_t - trust_t ) + ε_s Measures the deviation between the “current emotion scalar (wave)” and the “current trust level”. When these do not match (the divergence is large), the system develops “strong doubts (momentary noise)” about the current state and increases the denominator. | |
| Time-based foundation (d_base): d_base = abs(noise_est_t - d_est_t) + ε_d Measures the difference between “noise as history (wave average)” and “confidence as history”. This represents the “confidence level for updates (temporal distance)” derived from past context. | |
| 3. Effect: Creation of Dynamic Rhythm | |
| Effect A: Immediate Braking During Sudden Changes When sudden loss changes cause the scalar and trust to diverge, the noise_base (denominator) becomes dominant. This allows the learning rate to be instantly reduced as an immediate judgment, even when the temporal history is still stable, thereby preventing divergence before it occurs. | |
| Effect B: During the stable phase, when self-accelerated learning progresses smoothly (scalar and trust are stable) and confidence as history (d_base) accumulates, the dNR coefficient maximizes output with a “squared” term. dNR_now_val = ( d_base / noise_base )^2 This naturally increases the “step size” in stable regions, accelerating convergence. | |
| Effect C: Stability Maintenance via History (dNR_hist) Even if the instantaneous dNR_now_val is high, setting a growth limit of dNR_hist * μ_g suppresses excessive acceleration. On the other hand, in unreliable areas, we continue cautious exploration by accumulating deceleration pressure at dNR_hist * μ_d. | |
| ※ The asymmetry of Effect C functions through selection based on d_base <= dNR_hist and trust >= 0.5. This mathematically models the “thump” of love and the “thump” of caution, accelerating LR within the scalar range of 0 to ±0.5. However, LR acceleration in the negative direction is excluded from the LR history growth. (Values above ±0.5 are unquestionably treated as crisis levels exceeding caution, causing LR deceleration.) LR acceleration in the negative direction of the scalar value represents acceleration trusting the “modified update direction.” — essentially functioning as “Accelerated Correction”. This inherits the emoDrive mechanism from the EmoNavi generation (emo-type 1st generation), which leverages the time difference between EMA and loss (EMA delay). (This research belongs to the EmoSens generation (emo-type 2nd generation)). | |
| |--Danger--|---Wary---|---Fine---|--Danger--| Emotion | |
| Sigma_t [Minus] |---(-)---0.5---(+)---0---(+)---0.5---(-)---| [Plus] | |
| |--Hist(-)-|-Hist(Non)|--Hist(+)-|--Hist(-)-| Reglet | |
| μ_g and μ_d： | |
| v3.7：[Acceleration:LR Growth Max 1.05x] / [Deceleration:LR Decay 0.98x] | |
| v3.8：[Acceleration:LR Growth Max 1.50x] / [Deceleration:LR Decay 0.80x] | |
| 4. Conclusions on Numerical Stability | |
| This design, which pits the difference between the “time axis (history)” and the “instant axis (present)” against each other, is not merely a matter of decay. The system autonomously “constantly recalculates the ratio of ‘Doubt’ (Noise) to ‘Certainty’ (Distance)”, enabling dynamic control akin to “heartbeats responding to terrain complexity”—something impossible with manual schedulers. | |
| ※ EmoTion, EmoVoid is an original model implemented in v3.8. | |
| ※ dNR_hist has different coefficients in v3.7 and v3.8; v3.8 is more aggressive, designed to produce larger fluctuations than v3.7. | |
| The “synthesis of flat minima through multiple positioning” described below is a hypothesis derived from intuition and experimentation. | |
| I hope this intuition will be refined into a rigorous mathematical proof by the next generation of researchers. | |
| Autonomous Flat-Minima Generation via multiple Positioning of Heterogeneous Optimizers | |
| －Proposal of a New Learning Method: Prediction of “Evolutionary Flat Minimum Formation” via Local Synthesis Using of Emo Systems－ | |
| 1. Purpose: To resolve the high cost associated with achieving flat minimization. | |
| With existing learning methods, | |
| ・A single optimizer | |
| ・Long hours of repetitive learning | |
| Progressing toward improved generalizability and achieving flat minimization has become established. | |
| This requires various resources, including computational resources, and is not an environment that anyone can implement. | |
| This proposal aims to fundamentally alter this high-cost structure by employing an emo-style optimizer. | |
| 2. Proposal: Don't “search” for flat minimalism—create it yourself. | |
| Emo-style models (EmoSens, EmoAiry, EmoCats, EmoTion, EmoVoid) share a common learning structure despite differing update mechanisms. When trained under identical conditions, they yield learning results with differences representing “local solutions from different directions.” | |
| Integrating these divergent learning outcomes constitutes a synthesis of local solutions, and we anticipate that this synthesis may broaden and flatten the local solutions. In other words, it may bring local solutions closer to flat minima or transform them into flat minima themselves. | |
| Acquiring these local solutions as full-layer LoRA and integrating them using synthesis methods such as TALL-Mask-Merge, | |
| ∨∨∨ → \___/ Composite image of local solutions | |
| (multiple local solutions) (Post-synthesis flattening) | |
| ・The “commonly low areas” of local solutions in multiple directions are emphasized. | |
| ・The sharp edges on multiple (sharp minima) cancel each other out | |
| ・As a result, a shape close to a flat valley bottom (flat minimum) is reconstructed. | |
| This treats the local solution as multiple positioning (multiple-axis positioning), | |
| “Instead of exploring Flat Minima” | |
| This is a new learning method that “creates flat minima” through synthesis. | |
| 3. Organization: This integration leads to accelerated learning. | |
| Concretizing the proposal: Rather than performing long-term training with full-depth LoRA, FFT (Full Fine-Tuning), etc., achieve the goal by conducting slightly shallower learning across multiple types and employing synthesis techniques such as TALL-Mask-Merge. This is expected to make it easier to achieve high-precision learning results even in resource-constrained scenarios. | |
| The specific implementation method for this proposal is as follows: | |
| ・Instead of performing long-term training with a single optimizer using all layers of LoRA or FFT, | |
| ・Conduct shallow learning separately using multiple emo variants, | |
| ・Then integrate the results using TALL-Mask-Merge. | |
| As a result, | |
| ・Without relying on lengthy training sessions | |
| ・Even in resource-constrained environments | |
| ・It is possible to obtain high-precision models approaching flat minimalist architecture | |
| 4. Conclusion: Integration of Heterogeneous Emotion-Driven Models (Emotional Ensemble) | |
| The multiple optimizers proposed in this study (Sens, Airy, Cats, Tion, Void) each inspect the loss landscape based on different mathematical foundations. The “Flat Minima Synthesis via multiple Positioning” proposed in this study integrates these learning results generated under identical conditions through mask merging (e.g., TALL-Mask-Merge). This approach enables the simultaneous acquisition of “structural stability” and “expressive refinement” that cannot be achieved by a single optimization algorithm. This is expected to become a new optimization paradigm that shifts the learning process in optimization from a temporal pursuit to a spatial, multi-faceted integration. | |
| 5. Supplementary: Trial Method for Full-Layer LoRA Integration | |
| The multiple models were integrated by combining their respective learning results into the original model, and this new multiple-model system was then merged back into the original model using TM-merge. | |
| Original Model (org) ≪= TM Integration ≪= Model S (Sens), Model A (Airy), Model C (Cats), Model T (Tion), Model V (Void) | |
| Instead of directly integrating with LoRA alone, we integrated it into the base model and then reduced these multiple models back to the base model using TM-merge. | |
| FFT predicts that simply merging the multiple models after FFT back to the original model via TM-merge will yield equivalent results. | |
| 6. Background of Diversity in Terrain Exploration via Heterogeneous Optimizers | |
| The multi-positioning proposed by this method actively leverages differences in exploration characteristics arising from variations in algorithm lineage. | |
| Statistical inheritance Group: | |
| EmoSens (Adam-type): Dense gradient estimation via first- and second-order moments | |
| EmoAiry (Adafactor-type): Low-memory, wide-area curvature approximation via matrix decomposition | |
| EmoCats (Lion-type): Robust search with high noise tolerance via sign extraction | |
| These achieve liberation from manual schedulers by incorporating time-series SNR control via emoPulse while inheriting the orthodox essence of existing optimization theory. | |
| Evolutionary Groups in Geometry: | |
| EmoVoid / EmoTion (W-Ref Type): Executes updates based on the "freshness" of purely geometric information—the orthogonality between weights and gradients—thereby bypassing traditional statistical accumulation. | |
| The True Nature of Loss-Saturated Learning Progress | |
| －Reflections on a Steady Decline with Minimal Stagnation－ | |
| In this method, it is commonly observed that the loss value rarely stagnates or saturates, generally continuing to decrease. Particularly, the loss value continues to decrease to about half the value of the first step, even raising doubts about when convergence will occur. However, the learning results remain unaffected by failures like overfitting, maintaining highly normal generalization performance. An intuitive understanding of this suggests the possibility that “the model is learning by treating the repair of the original model as a differential.” | |
| This is merely a hypothesis, and like the creation of the flat minimas mentioned earlier, we hope it will be refined into a rigorous mathematical proof by the next generation of researchers. | |
| Furthermore, the following guarantees that “as long as the loss value has amplitude, the beat (emoPulse) will not stop.” | |
| noise_base = abs(sigma_t - trust_t) + ε_s | |
| d_base = abs(N_t - d_t) + ε_t | |
| These ε_s and ε_t are precisely what generate continuous downward behavior free from stagnation, creating the driving force to explore flat minima. This can also be interpreted as convergence occurring when the difference in loss values disappears. Through this design, learning tests on the Simplenet (FashionMNIST) demonstrate reproducible results, confirming that loss values below 0.30 can be achieved within 10,000 steps. | |
| In experimental verification using SDXL, training with e-pred + ZtSNR—which was achievable with the previous generation EmoNavi and its variants—can also be performed with this EmoSens and its variants. This resolves issues regarding noise tolerance in Flow-Matching (FM) and sampler compatibility, while simultaneously addressing challenges like color gamut limitations, which were considered weaknesses of e-pred. which are considered weaknesses of e-pred. Training for 300 epochs using only about 10 training images completed without stagnation, and we successfully created a full-layer LoRA model showing no overfitting tendencies. | |
| Further extreme testing with a single image over 300 steps also completed without stagnation, confirming the learning results remained intact. | |
| Even under extreme learning settings, no breakdown occurs—we believe this is because updates are performed without accumulating noise. | |
| Fundamentally, noise is thought to arise from errors in weighting minute data points. We consider it crucial to prevent noise generation by appropriately updating minute data to protect and maintain valuable information. | |
| Furthermore, we performed full-layer training (both encoding and decoding) on the SDXL VAE. Previous VAE retraining efforts resulted in compromised consistency with the model, ultimately leading to degraded generation outcomes. However, we confirmed that the optimizer proposed in this study maintains this consistency without degradation. We believe this will enhance the reusability of the VAE and contribute to extending the model's operational lifespan. | |
| An investigation into extreme noise model training: We performed SDXL vanilla model initialization (weight initialization with random values) and conducted full-layer LoRA training using this as the base model. | |
| Under normal circumstances, training would diverge or produce NaN values within a few steps, leading to failure. However, the EmoSens generations each progressed through training and completed 1500 steps. | |
| This LoRA should have failed, yet it defied expectations and applied successfully to the pre-initialized SDXL vanilla model without breakdown. | |
| Surprisingly, since this LoRA was trained as a state prior to the vanilla model, it improved the continuity of horizons and ground lines—areas where the vanilla model struggles—and corrected positional shifts when crossing subjects (it is also applicable to derivative SDXL models with similar effects). | |
| This test confirms that the EmoSens generation possesses excellent robustness in terms of stability and safety. | |
| ※ This LoRA exhibited similar effects across multiple seeds, potentially demonstrating “regularizing behavior” that mitigates specific artifacts in SDXL. However, it remains inconclusive whether this effect stems from intentional learning or coincidental alignment. Please understand this solely as confirmation that learning progression remains stable under extreme conditions. | |
| Predictions about Grokking | |
| This study focused on the behavior of continuous loss value reduction with minimal stagnation and conducted various tests to verify its underlying factors. | |
| Specifically, as an extreme learning condition, we evaluated “how far safe and stable learning progress is possible using only a single image.” | |
| As a result, we observed no typical failures such as overfitting, collapse into a copying state, or interference with unrelated prompts, confirming extremely stable learning results. | |
| Based on these results, we predict that Grokking is a “stagnation phenomenon” arising from the combined effects of the following two factors. | |
| - The accumulation of noise learned during the training process increases inaccuracies requiring correction in the latter stages of training, causing the model's visibility to deteriorate rapidly (whiteout/blackout phenomenon) | |
| - In the latter stages of training—the phase most in need of correction—the scheduler and gradient statistics suppress learning rate (LR), causing LR to drop drastically. | |
| These two factors occurring simultaneously cause the model to lose its fundamental direction and fall into a prolonged stagnation period. In other words, Grokking is considered an avoidable phenomenon. | |
| Emo-style (EmoSens generation) The reason why Grokking can be avoided is clear. | |
| This method enables the following updates, thereby maintaining a clear field of view and preserving the driving force for continued learning. | |
| - Maintaining update accuracy and preventing noise accumulation | |
| - Autonomously securing the necessary learning rate even in the latter stages of training | |
| Even if visibility deteriorates, the entire emotional mechanism functions like a high-precision GPS, ensuring emoPulse's accurate heartbeat keeps moving forward. This allows one to naturally approach flat minima or global optima without experiencing Grokking. | |
| Grokking is often examined as an “unexplained delay generalization,” but as seen in the aforementioned SDXL training results, the essence of the Grokking phenomenon can be considered a stagnation caused by structural flaws within the algorithm itself. | |
| dNR detects signs of incorrect weighting and unorganized microdata, identifies inconsistencies with abstract structures, and corrects them. We believe that if microdata is handled correctly, generalized solutions will form more quickly. | |
| Future Challenges: Introduction of Adaptive Accuracy Assessment Using the 8th-Order Moment Approximation | |
| Looking ahead, we are considering introducing a “higher-order accuracy assessment mechanism” utilizing dNR cubed (equivalent to the 8th-order moment). | |
| This approach does not directly output the 8th-order information as emoPulse output (the emoPulse mechanism remains unchanged). Instead, it attempts to utilize this information as a meta-indicator to evaluate the “purity” of the current learning process. | |
| We anticipate this will enable earlier detection of overfitting signs in minimal datasets, pushing autonomous control accuracy to its limits. Alternatively, accuracy detection might be possible by analyzing differences between past and present dNR histories. | |
| However, this is an optional feature to be implemented as needed. Based on current validation test results, we judge there is no urgency to proceed. | |
| Perspectives on Mathematical Analysis | |
| Mathematically analyzing this research suggests it may be concluded that while employing an SDE approach, it exhibits ODE-like characteristics. This update rule via emoPulse incorporates both stochastic fluctuations and temporal smoothness, potentially possessing a unique structure positioned at the boundary between SDE and ODE. (Since the loss value is the result of learning, the method is expected to behave in an ODE-like manner as it derives from the final outcome). | |
| How the history formation via Multi-EMA and the transitions of internal variables might be interpreted in continuous time remains a vital challenge for future mathematical research. This paper indicates only the intuitive direction; the detailed formalization is left to future researchers for further development. | |
| ※ The process of the SDE-DDE-ODE contraction cascade described in this paper is a hypothesis rooted in physical intuition and experimental facts. The task of formalizing this transition with rigorous equations is an open invitation to the next generation of researchers. I believe that the true "beginning of dialogue with the model" lies in filling these gaps—discovering what new mathematical order lies hidden within the rhythmic interstices of emoPulse. | |
| References | |
| Kingma, D. P., & Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980. | |
| Reddi, S. J., Kale, S., & Kumar, S. (2019). On the Convergence of Adam and Beyond. ICLR. | |
| Defazio, A., & Mishchenko, K. (2023). Learning-Rate-Free Learning by D-Adaptation. ICML.) | |
| Orabona, F., & Tommasi, T. (2017). Training Deep Networks without Learning Rates Through Coin Betting. NeurIPS. | |
| Luo, L., Xiong, Y., & Liu, Y. (2019). Adaptive Gradient Methods with Dynamic Bound of Learning Rate. ICLR. | |
| Shazeer, N., & Stern, M. (2018). Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. ICML. | |
| Bernstein, J., Wang, Y. X., Azizzadenesheli, K., & Anandkumar, A. (2018). signSGD: Compressed Optimisation for Non-Convex Problems. ICML. | |
| Chen, S. B., et al. (2023). Symbolic Discovery of Optimization Algorithms. arXiv. | |
| Zeyuan Allen-Zhu. (2017). Natasha: Faster Non-Convex Optimization Than SGD. arXiv. |