Using Exponentially Weighted Moving Averages to navigate trade-offs in systematic trading

A big part of the job of the indie trader is data analysis. We’re always looking in the past data to validate (or more often, invalidate) a hypothesis about what might predict future returns.

And one could argue that recent data is more useful than past data, since it may reflect the current state of affairs more accurately. Or it might not… market data are inherently noisy, and so there’s also value in looking back further into the past and averaging out the effects of that noise.

For example, say I was interested in the correlation between SPY and TLT. What’s more useful to me? The long-term average correlation, or a more up-to-date estimation based on say three months of returns?

It really depends on how “sticky” you think your correlation estimate is – that is, is tomorrow’s correlation likely to be the same as today’s? But that’s another story. The point I really wanted to make was that the long-term estimate will change slowly over time and not react quickly to recent changes, while the short-term estimate will disregard the historical correlation in favour of what’s happened more recently.

And since markets are dominated by noise, those recent data points might be pretty meaningless. Or they might not. You don’t really know. In any event, there’s certainly a trade-off between stability of our estimates and their responsiveness to new data.

Let’s explore that trade-off.

We’ll start with a plot of TLT/SPY correlations measured over different windows:

The first thing we need to do is get some price data into our session.

An easy way to do that is to source the functions for getting data from Yahoo Finance that we shared in this blog post:

We can then calculate correlations between the returns of TLT and SPY over different time horizons.

We’ll calculate the average correlation over the entire data set, and rolling correlations on 500- and 90-day windows.

A few things stand out.

The estimate over the entire data set captures the average correlation. It shows that, on average, SPY and TLT were negatively correlated (~ -0.3).

But when you look at a rolling window of recent data, the correlation of SPY and TLT moves around all over the place. Sometimes they become very negatively correlated (~ -0.6), and sometimes it flips sign and they become very positively correlated.

You see why this is important… if you looked only at the average correlation, you might think that SPY and TLT make for a nice diversified portfolio. But there are clearly times when they become more correlated and lose that nice diversification effect.

But over what horizon should we measure correlation?

The shorter window (blue line) is very responsive to recent data – it changes quickly and abruptly.

The longer window (green line) is more stable. It changes more slowly and gently and within a smaller range.

Which is better?

Of course it all depends on what you’re trying to do. You can imagine that if you had some rule where you flipped out of a position if the correlation crossed some threshold that the shorter window would have you trading hyperactively. But if you used the longer window, you might not react in time to take meaningful action.

This tension between signal and noise is a fact of life when analysing financial data. But there are things we can do to try to have our cake and eat it too. Namely, to try to include data from the past in our analysis, while giving more credence to recent history.

One such tool for doing that is the Exponentially Weighted Moving Average (EWMA).

What is an EWMA?

Exponentially Weighted Moving Average (EWMA) is a type of moving average that gives more weight to recent data, making it more responsive to new information. The “exponentially weighted” part means that as you go back in time, the influence of past data on the average decreases exponentially – but never disappears entirely.

Here’s a simple breakdown:

1. Weighting: In a simple moving average, every data point in the period (like the last 10 days) gets the same weight. In an EWMA, the most recent data point gets the most weight, and each previous data point gets progressively less weight.

In the correlation example above, we did the equivalent of a simple moving average where each data point in the window got the same weight.

2. The Formula: The EWMA for a series Y at time t is calculated using the following formula:

   $EWM{A}_t=(1-\lambda )\times Y_t+\lambda \times EWM{A}_{t-1}EWMA\_t\; =\; (1\; –\; \backslash lambda)\; \backslash times\; Y\_t\; +\; \backslash lambda\; \backslash times\; EWMA\_\{t-1\}$

   Here:

   • EWMA_t is the exponentially weighted moving average at time t.
   • Y_t is the actual value at time t.
   • λ (lambda) is the smoothing factor, between 0 and 1. A higher lambda places less weight on recent data.
   • EWMA_{t-1} is the EWMA value from the previous period.

3. Lambda: The value of λ determines how quickly the weight of previous observations decreases. A small λ makes the average decay quickly, so it’s more sensitive to recent changes. A large λ makes it decay slowly, so it’s more stable and less sensitive to recent changes.

Note that the definition of λ I’ve used here is in line with Risk Metrics, but might be opposite to lambda’s usage in other applications (ie in other implementations a higher λ makes the series decay more quickly).

Implementing the EWMA in R

To implement the EWMA in R, we need to be aware of a few of its characteristics. Firstly, each value depends on the previous value – that’s the $EWM{A}_{t-1}EWMA\_\{t-1\}$ term. That means that we can’t vectorise the calculation of the EWMA, so unfortunately one of R’s very efficient computation methods is unavailable to us.

Secondly, early values of the EWMA will be very volatile – since we have little history in our series, the EWMA will be very dependent on a small number of values. It will take some number of observations for the EWMA to “settle down” and start behaving sensibly.

For that reason, we include a warmup parameter that defines how many observations to discard from the start of the series. In this approach, we replace those warmup values with NA.

Let’s take a look at this function in action. We’ll plot EWMAs and SMAs of the price of SPY and TLT:

You can see that the EWMA (green line) is more responsive than the SMA (blue). I think it does a better job of smoothing out the noise while being true to the underlying “trend” in the data.

Next, I want to make this ewma function a little more performant. I mentioned that since the next value of the EWMA depends on the previous one, we can’t vectorise this and must use a for loop. But loops are not efficient in R. This isn’t a big deal when we’re using daily prices as we are here, but if we wanted to use higher frequency data, or we had many EWMAs to calculate, we might want to speed things up a bit. For that, we can port our function to C++ using Rcpp.

Implemetning the EWMA in Rcpp

Implementing the EWMA in Rcpp is pretty simple. Our R function is fairly trivial, so we can translate it to C++ fairly easily.

To compile and make it available in our session, we can use the Rcpp::sourceCpp function and pass it the C++ function as one long string:

Let’s see if it looks the same as the R implementation:

Looks good! Let’s see what sort of speedup we got:

Our original base R function wasn’t that slow… it processed a series of ~5,200 data points in about 525 microseconds on average. So that’s totally fine for casual usage and won’t slow you down at all.

But if you’re calculating a lot of these things, then you’ll benefit from the ~18x speed-up we get with our Rcpp code, where the average was about 30 microseconds.

Exponentially weighted correlation

Let’s revisit our original example of estimating the correlation of SPY and TLT.

We saw that there’s a tension between reacting fast and reacting smoothly when we use normal moving windows to estimate correlations. Let’s see if we can use an EWMA to navigate this tension.

First, we need a function for calculating an exponentially weighted correlation. I’ll use the approach in Risk Metrics (Eq. 5.8, page 83).

In this approach, we:

• Estimate an EWMA covariance
• Estimate asset-wise EWMA variances and convert to standard deviations
• The EWMA correlation is then just the covariance divided by the product of the standard deviations

In this approach, we need an initialisation window over which we get our initial covariance and variance estimates. We then update those in an exponentially weighted fashion, and use them to calculate the exponentially weighted correlation.

Let’s test it out:

You can see that the EWMA correlation estimate is both responsive and arguably less jittery than the 90-day moving window correlation estimate.

Let’s zoom in to see this better:

In this approach, the decay effect from lamda is applied separately to both the covariance and the variances, not the correlation estimate itself. This implies an interesting possibility of using different decay rates on the variances and covariances, if you think that maybe they change at different rates. This might find application in position sizing, where maybe you want to respond to changes in volatility faster than changes in the co-movement of assets.

Here’s an example of how you might implement that and what a more responsive variance estimate would look like:

Summary

Exponentially weighted estimates can help traders navigate the trade-offs around stability and responsiveness of estimates of quantities such as correlation, covariance, and variance.

In this article, we implemented functions for:

• Calculating an EWMA of a vector
• Doing it faster using Rcpp
• Estimating an exponentially weighted correlation between two assets
• Doing it using different decay factors for the variance and covariance estimates.

Next time, I’ll show you how I estimate an exponentially weighted covariance matrix for a portfolio of assets. This is particularly useful if you need a stable yet responsive estimate of risk, for example if you were doing medium frequency portfolio optimisation.