Weights

Weights

Many OnlineStats are parameterized by a Weight that controls the influence of new observations. If the OnlineStat is capable of calculating the same result as a corresponding offline estimator, it will have a keyword argument weight. If the OnlineStat uses stochastic approximation, it will have a keyword argument rate.

Consider how weights affect the influence of the next observation on an online mean $\theta^{(t)}$, as many OnlineStats use updates of this form. A larger weight $\gamma_t$ puts higher influence on the new observation $x_t$:

\[\theta^{(t)} = (1-\gamma_t)\theta^{(t-1)} + \gamma_t x_t\]
Note

The values produced by a Weight must follow two rules:

  1. $\gamma_1 = 1$ (guarantees $\theta^{(1)} = x_1$)
  2. $\gamma_t \in (0, 1), \quad \forall t > 1$ (guarantees $\theta^{(t)}$ stays inside a convex space)

Weight Types

EqualWeight()

Equally weighted observations.

$γ(t) = 1 / t$

ExponentialWeight(λ::Float64)
ExponentialWeight(lookback::Int)

Exponentially weighted observations. The first weight is 1.0 and all else are λ = 2 / (lookback + 1).

$γ(t) = λ$

LearningRate(r = .6)

Slowly decreasing weight. Satisfies the standard stochastic approximation assumption $∑ γ(t) = ∞, ∑ γ(t)^2 < ∞$ if $r ∈ (.5, 1]$.

$γ(t) = inv(t ^ r)$

HarmonicWeight(a = 10.0)

Weight determined by harmonic series.

$γ(t) = a / (a + t - 1)$

McclainWeight(α = .1)

Weight which decreases into a constant.

$γ(t) = γ(t-1) / (1 + γ(t-1) - α)$

Weight wrappers

Bounded(w::Weight, λ::Float64)

Bound the weight by a constant.

$γ′(t) = max(γ(t), λ)$

Scaled(w::Weight, λ::Float64)

Scale a weight by a constant.

$γ′(t) = λ * γ(t)$

Custom Weighting

The Weight can be any callable object that receives the number of observations as its argument. For example:


julia> y = randn(100);


julia> fit!(Mean(weight = EqualWeight()), y)
Mean: n=100 | value=-0.0442521

julia> fit!(Mean(weight = inv), y)
Mean: n=100 | value=-0.0442521


julia> fit!(Mean(weight = ExponentialWeight(.01)), y)
Mean: n=100 | value=0.0454786

julia> fit!(Mean(weight = x -> x == 1 ? 1.0 : .01), y)
Mean: n=100 | value=0.0454786