Reference manual/Simulator
THE DOCUMENT SIMULATOR

See color in context.

Try your colors on a paper or table. Keep the content; change the emphasis.

White page · live preview
Paper title#000000

The title at the top of the paper.

Paragraph text#000000

The first paragraph in the introduction.

Paragraph background#FFFFFF

The highlighted paragraph in the analysis; its text stays black.

Equation#000000

The displayed update rule.

Select a color from any set. Enable a second color only when you want a blend.

Paper text previewFixed white page
OPTIMIZATION · RESEARCH EXAMPLE1

Stability of Adaptive Gradient Methods

Alex Chen and Morgan Rivera
Department of Applied Mathematics

Abstract

Adaptive gradient methods improve convergence by adjusting the step size to local variation. We study a simple update rule and show how stable step sizes reduce oscillation near a stationary point. Experiments compare predictive accuracy, optimization loss, and computational cost.

1. Introduction

Gradient-based optimization is a central component of modern statistical learning. A fixed learning rate is often sensitive to the scale of the objective, motivating methods that adapt the update to the observed gradient. We consider a smooth objective and analyze the resulting trajectory.

Let f:ℝd→ℝ denote the objective. Starting from an initial parameter vector, the method updates the estimate using a gradient evaluated at the current iterate.

2. Analysis

Our analysis separates the effect of the learning rate from the geometry of the objective. Under a bounded-gradient assumption, the adaptive rule controls the magnitude of each update and yields a stable sequence of iterates. This result motivates the empirical comparison in Table 1.

θt+1=θt−ηt∇f(θt)(1)

The step size determines how quickly the method responds to local curvature. A decreasing schedule balances early progress with stable behavior near a stationary point. The same update rule can be applied without changing the underlying objective.

Sample content for color comparison · 1

Screen colors approximate LaTeX output. The page stays white and unchanged text stays black.

LaTeX example
\documentclass{article}
\usepackage{xcolor}
\usepackage{amsmath,amssymb}

\title{\textcolor{black}{Stability of Adaptive Gradient Methods}}
\author{Alex Chen and Morgan Rivera\\Department of Applied Mathematics}
\date{}
\begin{document}
\color{black}
\maketitle
\begin{abstract}
Adaptive gradient methods improve convergence by adjusting the step size to local variation. We study a simple update rule and show how stable step sizes reduce oscillation near a stationary point. Experiments compare predictive accuracy, optimization loss, and computational cost.
\end{abstract}
\section{Introduction}
\textcolor{black}{Gradient-based optimization is a central component of modern statistical learning. A fixed learning rate is often sensitive to the scale of the objective, motivating methods that adapt the update to the observed gradient. We consider a smooth objective and analyze the resulting trajectory.}

Let $f:\mathbb{R}^d \to \mathbb{R}$ denote the objective. Starting from an initial parameter vector, the method updates the estimate using a gradient evaluated at the current iterate.

\section{Analysis}
\noindent\colorbox{white}{%
  \parbox{\dimexpr\linewidth-2\fboxsep}{Our analysis separates the effect of the learning rate from the geometry of the objective. Under a bounded-gradient assumption, the adaptive rule controls the magnitude of each update and yields a stable sequence of iterates. This result motivates the empirical comparison in Table 1.}%
}
\begin{equation}
  {\color{black}
  \theta_{t+1} = \theta_t - \eta_t \nabla f(\theta_t)}
\end{equation}
The step size determines how quickly the method responds to local curvature. A decreasing schedule balances early progress with stable behavior near a stationary point. The same update rule can be applied without changing the underlying objective.
\end{document}