Interactive Numerical Methods

Numerical Integration (Quadrature)

Euler's Method (1D)

Stiff Equations & Stability

A stiff differential equation has components that change on very different time scales. This poses a challenge for explicit solvers like Forward Euler. For the equation $y' = -20y$, the solution decays very rapidly. Watch what happens when the number of steps is too low (i.e., the step size $\Delta t$ is too large).

Explicit Euler (2D): Pendulum Dynamics

The standard (Explicit/Forward) Euler method applied to the pendulum system. The recurrence is: $\theta_{n+1} = \theta_n + \Delta t \cdot \omega_n$ and $\omega_{n+1} = \omega_n - \Delta t \cdot (g/L) \sin(\theta_n)$. Notice how the numerical solution spirals outwards, indicating that the system is gaining energy, which is physically incorrect.

Implicit Euler (2D): Pendulum Dynamics

The Backward (Implicit) Euler method uses future values to determine the step: $\theta_{n+1} = \theta_n + \Delta t \cdot \omega_{n+1}$ and $\omega_{n+1} = \omega_n - \Delta t \cdot (g/L) \sin(\theta_{n+1})$. This creates a non-linear system that must be solved at each step. The result is a trajectory that spirals inwards, indicating numerical energy dissipation, but it is much more stable than the explicit method.

Newton's Method for Optimization

Newton's method is a powerful root-finding algorithm that can be adapted for optimization. To find a local minimum or maximum of a function $f(x)$, we look for points where the derivative $f'(x) = 0$. Applying Newton's method to the derivative gives the following iterative update rule, which uses the second derivative (the Hessian):

This demo uses the function $f(x) = x^4 + x^3 - x^2 - x$. See how the initial guess affects which minimum the algorithm converges to. A poor initial guess (e.g., try $x_0 = 0$) can even lead to convergence to a maximum!

Regularization in Optimization

A major issue with the standard Newton's method arises when the second derivative, $f''(x_n)$, is negative or zero. A negative $f''(x_n)$ causes the algorithm to move towards a maximum (an ascent direction), which is the opposite of our goal. To fix this, we can add a non-negative constant, $\lambda$, to the denominator. This is a form of regularization.

This ensures the denominator is positive (for a sufficiently large $\lambda$), forcing the update step to be in a descent direction. Try starting with an initial guess of $x_0=0$. With $\lambda=0$, this is standard Newton's method, which will converge to the local maximum. Increase $\lambda$ to see how regularization pushes the algorithm away from the maximum and towards a true minimum.