The post How To Make A Kalman Filter<br> in R for Pairs Trading appeared first on Robot Wealth.

]]>If they did, pairs trading would be the easiest game in town. But the reality is that relationships are constantly evolving and changing. At some point, we’re forced to make uncertain decisions about how best to capture those changes.

One way to incorporate both uncertainty and dynamism in our decisions is to use the Kalman filter for parameter estimation.

The Kalman filter is a state space model for estimating an unknown (‘hidden’) variable using observations of related variables and models of those relationships. The Kalman filter is underpinned by Bayesian probability theory and enables an estimate of the hidden variable in the presence of noise.

There are plenty of tutorials online that describe the mathematics of the Kalman filter, so I won’t repeat those here (this article is a wonderful read). Instead, I’ll show you how to implement the Kalman filter framework to provide a * dynamic estimate of the hedge ratio in a pairs trading strategy*. I’ll provide just enough math as is necessary to follow the implementation.

To implement our Kalman filter, we need four variables:

- A vector of our observed variable
- A vector of our hidden variable
- A state transition model (which describes how the hidden variable evolves from one state to the next)
- An observation model (a matrix of coefficients for the other variable – we use a hedge coefficient and an intercept)

For our hedge ratio/pairs trading application, the observed variable is one of our price series \(p_1\) and the hidden variable is our hedge ratio, \(\beta\). The observed and hidden variables are related by the familiar spread equation: \[p_1 = \beta * p_2 + \epsilon\] where \(\epsilon\) is noise (in our pairs trading framework, we are essentially making bets on the mean reversion of \(\epsilon\)). In the Kalman framework, the other price series, \(p_2\) provides our observation model.

We also need to define a state transition model that describes the evolution of \(\beta\) from one time period to the next. If we assume that \(\beta\) follows a random walk, then our state transition model is simply \[\beta_t = \beta_{t-1} + \omega\]

Here’s the well-known iterative Kalman filter algorithm.

For every time step:

- Predict the next state of the hidden variable given the current state and the state transition model
- Update the state covariance prediction
- Predict the next value of the observed variable given the prediction for the hidden variable and the observation model
- Update the measured covariance prediction
- Calculate the error between the observed and predicted values of the observed variable
- Calculate the Kalman gain
- Update the estimate of the hidden variable
- Update the state covariance prediction

To start the iteration, we need initial values for the covariances of the measurement and state equations. Methods exist to estimate these from data, but for our purposes we will start with some values that result in a relatively slowly changing hedge ratio. To make the hedge ratio change faster, increase the values of

deltaand

Vein the R code below. The initial estimates of these values are as close to ‘parameters’ that we have in our Kalman filter framework.

Here’s some R code for implementing the Kalman filter.

The two price series used are daily adjusted closing prices for the “Hello world” of pairs trading: GLD and GDX (you can download the data at the end of this post).

First, read in and take a look at the data:

library(xts) path <- "C:/Path/To/Your/Data/" assets <- c("GLD", "GDX") df1 <- xts(read.zoo(paste0(path, assets[1], ".csv"), tz="EST", format="%Y-%m-%d", sep=",", header=TRUE)) df2 <- xts(read.zoo(paste0(path, assets[2], ".csv"), tz="EST", format="%Y-%m-%d", sep=",", header=TRUE)) xy <- merge(df1$Close, df2$Close, join="inner") colnames(xy) <- assets plot(xy, legend.loc=1)

Here’s what the data look like:

Looks OK at first glance.

Here’s the code for the iterative Kalman filter estimate of the hedge ratio:

x <- xy[, assets[1]] y <- xy[, assets[2]] x$int <- rep(1, nrow(x)) delta <- 0.0001 Vw <- delta/(1-delta)*diag(2) Ve <- 0.001 R <- matrix(rep(0, 4), nrow=2) P <- matrix(rep(0, 4), nrow=2) beta <- matrix(rep(0, nrow(y)*2), ncol=2) y_est <- rep(0, nrow(y)) e <- rep(0, nrow(y)) Q <- rep(0, nrow(y)) for(i in 1:nrow(y)) { if(i > 1) { beta[i, ] <- beta[i-1, ] # state transition R <- P + Vw # state cov prediction } y_est[i] <- x[i, ] %*% beta[i, ] # measurement prediction Q[i] <- x[i, ] %*% R %*% t(x[i, ]) + Ve # measurement variance prediction # error between observation of y and prediction e[i] <- y[i] - y_est[i] K <- R %*% t(x[i, ]) / Q[i] # Kalman gain # state update beta[i, ] <- beta[i, ] + K * e[i] P = R - K %*% x[i, ] %*% R } beta <- xts(beta, order.by=index(xy)) plot(beta[2:nrow(beta), 1], type='l', main = 'Kalman updated hedge ratio') plot(beta[2:nrow(beta), 2], type='l', main = 'Kalman updated intercept')

And here is the resulting plot of the dynamic hedge ratio:

The value of the Kalman filter is immediately apparent – you can see how drastically the hedge ratio changed over the years.

We could use that hedge ratio to construct our signals for a trading strategy, but we can actually use the other by-products of the Kalman filter framework to generate them directly *(hat tip to Ernie Chan for this one):*

The prediction error (

ein the code above) is equivalent to the deviation of the spread from its predicted value. Some simple trade logic could be to buy and sell our spread when this deviation is very negative and positive respectively.

We can relate the actual entry levels to the standard deviation of the prediction error. The Kalman routine also computes the standard deviation of the error term for us: it is simply the square root of

Qin the code above.

Here’s a plot of the trading signals at one standard deviation of the prediction error (we need to drop a few leading values as the Kalman filter takes a few steps to warm up):

# plot trade signals e <- xts(e, order.by=index(xy)) sqrtQ <- xts(sqrt(Q), order.by=index(xy)) signals <- merge(e, sqrtQ, -sqrtQ) colnames(signals) <- c("e", "sqrtQ", "negsqrtQ") plot(signals[3:length(index(signals))], ylab='e', main = 'Trade signals at one-standard deviation', col=c('blue', 'black', 'black'), lwd=c(1,2,2))

Cool! Looks OK, except the number of signals greatly diminishes in the latter half of the simulation period. Later, we might come back and investigate a more aggressive signal, but let’s press on for now.

At this point, we’ve got a time series of trade signals corresponding to the error term being greater than one standard deviation from its (estimated) mean. We could run a vectorised backtest by calculating positions corresponding to these signals, then determine the returns of holding those positions.

In fact, let’s do that next:

# vectorised backtest sig <- ifelse((signals[1:length(index(signals))]$e > signals[1:length(index(signals))]$sqrtQ) & (lag.xts(signals$e, 1) < lag.xts(signals$sqrtQ, 1)), -1, ifelse((signals[1:length(index(signals))]$e < signals[1:length(index(signals))]$negsqrtQ) & (lag.xts(signals$e, 1) > lag.xts(signals$negsqrtQ, 1)), 1, 0)) colnames(sig) <- "sig" ## trick for getting only the first signals sig[sig == 0] <- NA sig <- na.locf(sig) sig <- diff(sig)/2 plot(sig) ## simulate positions and pnl sim <- merge(lag.xts(sig,1), beta[, 1], x[, 1], y) colnames(sim) <- c("sig", "hedge", assets[1], assets[2]) sim$posX <- sim$sig * -1000 * sim$hedge sim$posY <- sim$sig * 1000 sim$posX[sim$posX == 0] <- NA sim$posX <- na.locf(sim$posX) sim$posY[sim$posY == 0] <- NA sim$posY <- na.locf(sim$posY) pnlX <- sim$posX * diff(sim[, assets[1]]) pnlY <- sim$posY * diff(sim[, assets[2]]) pnl <- pnlX + pnlY plot(cumsum(na.omit(pnl)), main="Cumulative PnL, $")

*Just a quick explanation of my hacky backtest…*

The ugly nested

`ifelse`

statement in line 2 creates a time series of trade signals where sells are represented as -1, buys as 1 and no signal as 0. The buy signal is the prediction error crossing under its -1 standard deviation from above; the sell signal is the prediction error crossing over its 1 standard deviation from below.The problem with this signal vector is that we can get consecutive sell signals and consecutive buy signals. We don’t want to muddy the waters by holding more than one position at a time, so we use a little trick in lines 7 – 10 to firstly replace any zeroes with

`NA`

, and then use the`na.locf`

function to fill forward the`NA`

values with the last real value. We then recover the original (non-consecutive) signals by taking the`diff`

and dividing by 2.If that seems odd, just write down on a piece of paper a few signals of -1, 1 and 0 in a column and perform on them the operations described. You’ll quickly see how this works.

Then, we calculate our positions in each asset according to our spread and signals, taking care to lag our signals so that we don’t introduce look-ahead bias. We’re trading 1,000 units of our spread per trade. Our estimated profit and loss is just the sum of the price differences multiplied by the positions in each asset.

Here’s the result:

Looks interesting!

But recall that our trading signals were few and far between in the latter half of the simulation? If we plot the signals, we see that we were actually holding the spread for well over a year at a time:

I doubt we’d want to trade the spread this way, so let’s make our signals more aggressive:

# more aggressive trade signals signals <- merge(e, .5*sqrtQ, -.5*sqrtQ) colnames(signals) <- c("e", "sqrtQ", "negsqrtQ") plot(signals[3:length(index(signals))], ylab='e', main = 'Trade signals at one-standard deviation', col=c('blue', 'black', 'black'), lwd=c(1,2,2))

Better! A smarter way to do this would probably be to adapt the trade level (or levels) to the recent volatility of the spread – I’ll leave that as an exercise for you.

These trade signals lead to this impressive and highly dubious equity curve:

Why is it dubious?

Well, you probably noticed that there are some pretty out-there assumptions in this backtest. To name the most obvious:

- We’re trading at the daily closing price with no market impact or slippage
- We’re trading for free

My gut feeling is that this would need a fair bit of work to cover costs of trading – but that gets tricky to assess without a more accurate simulation tool.

You can see that it’s a bit of a pain to backtest – particularly if you want to incorporate costs. To be fair, there are native R backtesting solutions that are more comprehensive than my quick-n-dirty vectorised version. But in my experience none of them lets you move quite as fast as the Zorro platform, which also allows you to go from backtest to live trading with almost the click of a button.

You can see that R makes it quite easy to incorporate an advanced algorithm* (well, at least I think it’s advanced; our clever readers probably disagree).* But tinkering with the strategy itself – for instance, incorporating costs, trading at multiple standard deviation levels, using a timed exit, or incorporating other trade filters – is a recipe for a headache, not to mention a whole world of unit testing and bug fixing.

On the other hand, Zorro makes tinkering with the trading aspects of the strategy easy. Want to get a good read on costs? That’s literally a line of code. Want to filter some trades based on volatility? Yeah, you might need two lines for that. What about trading the spread at say half a dozen levels and entering and exiting both on the way up and on the way down? OK, you might need four lines for that.

The downside with Zorro is that it would be pretty nightmarish implementing a Kalman filter in its native Lite-C code. But thanks to Zorro’s R bridge, I can use the R code for the Kalman filter that I’ve already written, with literally only a couple of minor tweaks. We can have the best of both worlds.

*Which leads to my next post…*

Next time I post, it will be to show you a basic pairs trading script in Zorro, using a more vanilla method of calculating the hedge ratio. After that, I’ll show you how to configure Zorro to talk to R and thus make use of the Kalman filter algorithm.

*I’d love to know if this series is interesting for you, and what else you’d like to read about on Robot Wealth. Let us know in the comments.
*

The post How To Make A Kalman Filter<br> in R for Pairs Trading appeared first on Robot Wealth.

]]>The post Pattern Recognition with the Frechet Distance appeared first on Robot Wealth.

]]>*ah, I see a blue star pattern on my chart… a good omen.*

The problem is that such an approach is *inherently subjective* since price action almost never matches perfectly with the idealized version of price patterns you see in every beginner’s guide to trading. It is up to you, the individual, to determine whether a particular chart formation matches closely enough with a particular pattern for it to be considered valid.

This is quite tricky! It’s *very difficult* to codify a trading system based on their use. By extension, it is difficult to test the efficacy of these patterns in an objective, evidence-based manner.

That won’t stop smart people from trying, though. An attempt to do just this was made by MIT’s Andrew Lo, Harry Mamaysky and Jian Wang back in 20001and, perhaps surprisingly, they found statistically significant evidence that *some* patterns provide useful incremental information in *some *markets.

Lo, Mamaysky and Wang were also generally enthusiastic about using automated detection methods. There are many possible approaches to pattern detection algorithms, of which Zorro implements one: a function that calculates the Frechet distance.

*So, l**et’s dive in and explore it!*

**The Frechet distance between two curves is a measure of their similarity —** it’s often described like so:

Suppose a man is walking his dog and that he is forced to walk on a particular path and his dog on another path. Both the man and the dog are allowed to control their speed independently but are not allowed to go backwards. Then, the Fréchet distance of the two paths is the minimal length of a leash that is necessary to keep man and dog joined throughout their walk.

*there are a lot more variables in the real world….*

If the Frechet distance between two curves is small, it follows that the curves are similar. Conversely, a large Frechet distance implies that the curves are not similar.

So, we can leverage the Frechet distance as a pattern detection algorithm by comparing sections of the price curve to a curve corresponding to a pattern of interest, for example, a triangle. A small Frechet distance implies that the section of the price curve that was analyzed is similar to the pre-defined pattern.

There have been a whole bunch of algorithms proposed over the years for calculating Frechet distance (it was first described in 1906), and Zorro implements a simple variant that enables the comparison of some part of a price curve with a known and previously described pattern. Zorro’s

frechet()function returns a number between approximately 0 and 80 that measures the similarity of the part of the price curve being analyzed and the pattern.

**Note that this is proportional to the inverse of the Frechet distance, in that a larger similarity measure implies a smaller Frechet distance. **

frechet()takes the following arguments:

- A series (usually asset prices) to be compared with a predefined pattern
- An integer
TimeFrame

which sets the number of price bars to use in the comparison, that is, the horizontal length of the pattern in the price curve (setting this to zero tells Zorro to simply use the same length as the predefined pattern). - A var,
Scale

, specifying the vertical size of the pattern in the price chart. Setting this to a negative number inverts the pattern. - An array of positive numbers specifying the shape of the pattern to be detected. The final value of the array must be zero, which is used by the algorithm to signal the termination of the pattern.

There are several complications and considerations to be aware of in setting these arguments, so let’s go through each of them in more detail, starting with the array specifying the shape of the pattern.

If we want to detect chart patterns, the first thing we need to define is the shape of that pattern. Note that in describing our pattern in an array, we only need to be concerned with its *shape. *We can deal with its *size* (both horizontally and vertically) using other

frechet()arguments. Therefore don’t focus too much on the absolute values of the numbers that describe the pattern – their relative values are much more important here.

To define a pattern in an array, think of an \(x, y\) coordinate plane. The array indexes are the \(x\) values; the numbers stored in each index are the corresponding \(y\) values. We then map our pattern as a series of \(x,y\) pairs.

Here’s an example for a triangle pattern:

Remembering that zero terminates the pattern, the corresponding array would consist of the numbers 1, 8, 2, 7, 3, 6, 4, 5, 0. We would define such an array as

var Triangle[9] = {1, 8, 2, 7, 3, 6, 4, 5, 0};

The obvious question that arises from this approach is how well does the algorithm detect patterns that we would consider a triangle, but which deviate from the idealized triangle shown above?

For example, what about asymmetry in the legs of the triangle? That is, what if the early legs take longer to complete than later legs? In the example above, all the legs take the same amount of time. What about triangles that don’t terminate at the apex?

By way of example, the following would probably fit the definition of a triangle:

But now our array would be given by 1, 3, 5, 8, 6, 4, 2, 5, 7, 6, 4, 3, 6, 0. That is,

var Triangle[14] = {1,3,5,8,6,4,2,5,7,6,4,3,6,0};

Would these two patterns return different Frechet similarities when applied to the same price curve?

The answer is yes. But, bear in mind that a pattern corresponding to the first triangle will still be *somewhat* similar to the second triangle. In practice, this means that in order to use this approach effectively, we would need to cover our bases and check for multiple variants of the intended pattern, perhaps using some sort of confirmation between different variations of the pattern. We’ll see in the example below how much variation we see in our similarity measure for different variants of the same intended pattern.

The

Scaleparameter controls the vertical height of the pattern being searched in the price curve. This is the same as stretching the predefined pattern in the direction of price in a price chart. In most cases, it will make sense to set the height of the pattern to the range of the price action over the period of interest. We do this automatically by setting the

Scaleparameter based on the maximum and minimum values of the price series over the time period of interest via Zorro’s

MaxVal()and

MinVal()functions:

Scale = MaxVal(Data, TimeFrame) - MinVal(Data, TimeFrame);

The

TimeFrameparameter controls the pattern’s horizontal length and corresponds to the number of bars over which to apply the pattern. This is the same as stretching the predefined pattern in the direction of time on a price chart. This parameter requires a little more thought because there are no hard and fast rules regarding how long these patterns should take to form. Again, we must deal with the inherent subjectivity of the method.

Rather than constraining our pattern detection algorithm to a single time period, why not simply look over multiple time periods?

We could do this by calling

frechet()within a

for()loop that increments the

TimeFrameparameter on every iteration, like so:

for(i=5;i<100;i+=5) { frechet(Price, i, 100*PIP, Triangle); }

This will search for a pattern called “Triangle” that is 100 pips high over multiple time ranges, from 5 bars to 100 bars. In practice, we don’t really need to cover every incremental number of bars (1,2,3,4 etc) because patterns evolving over similar time horizons will tend to return similar

frechet()values. For example, a pattern evolving over 10 bars will be similar to the same pattern evolving over 11 bars. In the example above, we increment our search length by 5 bars.

We are now in a position to start detecting patterns and analyzing their usefulness as trading signals.

The code below plots the Frechet similarity metric for our symmetric and asymmetric triangles, and their inverses, over a number of time horizons. We use the

strf()function to enable us to pass a variable (in this case, the integer

i) into a string (in this case, the name of the plot) so that we can plot the different Frechet similarities from within the

for()loop:

/* PLOT FRECHET SIMILARITY */ function run() { set(PLOTNOW); StartDate = 20150712; EndDate = 20150826; BarPeriod = 1440; LookBack = 100; asset("SPX"); vars Price = series(priceClose()); static var Tri_Sym[9] = {1,8,2,7,3,6,5,4,0}; static var Tri_Asym[14] = {1,3,5,8,6,4,2,5,7,6,4,3,6,0}; int i; for(i=10;i<=30;i+=10) { plot(strf("Tri_Sym_%d", i),frechet(Price, i, MaxVal(Price,i) - MinVal(Price,i), Tri_Sym),NEW,RED); plot(strf("Tri_Asym_%d", i),frechet(Price, i, MaxVal(Price,i) - MinVal(Price,i), Tri_Asym),0,BLUE); plot(strf("Tri_Sym_Inv_%d", i),frechet(Price, i, -(MaxVal(Price,i) - MinVal(Price,i)), Tri_Sym),NEW,BLACK); plot(strf("Tri_Asym_Inv_%d", i),frechet(Price, i, -(MaxVal(Price,i) - MinVal(Price,i)), Tri_Asym),0,GREEN); } PlotWidth = 800; PlotScale = 15; PlotHeight1 = 500; PlotHeight2 = 125; }

Here we zoom into an area of the S&P500 index that saw a fairly obvious triangle develop from early July through to mid-August 2015:

You can see that most variants of our Frechet algorithm detected the triangle at some point during its evolution. In particular, the asymmetric inversive triangle measured over 20 days did a particularly good job of recognizing the triangle, reaching a similarity score of approximately 50 as the triangle approached its apex.

Looking more closely at other regions will reveal that the algorithm is far from perfect, sometimes scoring patterns that we would rather exclude relatively highly. This makes it difficult to differentiate the “true” patterns on the basis of some threshold similarity score. To overcome that, we could perhaps continue to refine our pattern definitions or implement a series of confirmations from different variations, but that would get tedious fairly quickly.

Here’s an example of a simple trading strategy that looks for our asymmetric triangle pattern across several time horizons. Again using the

strf()function, we switch to a new

Algofor each time horizon. I read somewhere that gold is particularly prone to triangle formations, so we’ll use the GLD Exchange Traded Fund. I also read that triangles are allegedly indicators of a strong breakout in any direction.

*I don’t know whether this has any basis in fact, but let’s go with it for the purpose of the exercise.*

On the basis of the alleged triangle behavior, when one is detected we bracket the market at a quarter of the 20-day ATR. We leave our pending orders for a maximum of 10 days, but no longer than the time horizon used to detect the pattern. Likewise, we close our trades after a maximum of 20 days, but no longer than the time horizon.

There’s much more you could do here, for example cancelling the remaining pending order when the opposite one is executed.2The code uses Alpha Vantage’s API for getting the required GLD historical data, so you’ll need to set this up in your Zorro.ini file if you don’t already have this data.

/* FRECHET TRADING */ var threshold = 30; function run() { set(PLOTNOW); StartDate = 2007; EndDate = 2017; BarPeriod = 1440; AssetList = "AssetsIB"; if(is(INITRUN)) assetHistory("GLD", FROM_AV); asset("GLD"); vars Price = series(priceClose()); var Tri_Asym[14] = {1,3,5,8,6,4,2,5,7,6,4,3,6,0}; int i; for(i=10;i<=50;i+=10) { algo(strf("_%d_Asym", i)); if(frechet(Price, i, MaxVal(Price,i)-MinVal(Price,i), Tri_Asym) > threshold) { Entry = 0.25*ATR(20); EntryTime = min(i, 10); LifeTime = min(i, 20); if(NumOpenLong == 0) enterLong(); if(NumOpenShort == 0) enterShort(); } } PlotWidth = 800; PlotHeight1 = 500; PlotHeight2 = 125; }

Here’s the equity curve:

You’ll find that such a trading strategy is difficult to apply directly to a universe of potential assets.

The effect of randomness, combined with the difficulty in refining pattern definitions sufficiently, invites overfitting to the parameters of the

frechet()function, not to mention selection bias. You’ll also be faced with the decision to trade a pattern detected at multiple time horizons.

Maybe a system of confirmations from numerous pattern variations would help, but perhaps a more practical application for the trader interested in patterns is to use

frechet()to scan a universe of assets and issue email or SMS alerts, or display a message listing assets where a pattern was detected for further manual analysis. Maybe we’ll cover this in a future blog post if you’re interested, let me know in the comments below.

Here are some rather idealized patterns to get you started. I leave it up to you to experiment with various departures from their idealized forms.

var rectangle[5] = {1,2,1,2,0}; var cup[10] = {6,3,2,1,1,1,2,3,6,0}; var zigzag[5] = {1,7,2,8,0}; var headShldrs[17] = {1,2,3,3,3,4,5,6,6,5,4,3,3,3,2,1,0}; var cup[10] = { 6,3,2,1,1,1,2,3,6,0 }; var triangle_symmetric[9] = {1,8,2,7,3,6,5,4,0}; var triangle_assymetric[14] = {1,3,5,8,6,4,2,5,7,6,4,3,6,0};

**Want a more robust and profitable approach to trading? Gain a broader understanding of how we use algorithms to trade systematically and make our capital grow by downloading the free Algo Basics PDF below.**

The post Pattern Recognition with the Frechet Distance appeared first on Robot Wealth.

]]>The post Can you apply factors to<br> trade performance? appeared first on Robot Wealth.

]]>For instance, maybe you wonder if your strategy tends to do better when volatility is high?

In this case, you can get very binary feedback by, say, running backtests with and without a volatility filter.

But this can mask interesting insights that might surface if the relationship could be explored in more detail.

Zorro has some neat tools that allow us to associate data of interest with particular trading decisions, and then export that data for further analysis. Here’s how it works:

Zorro implements a `TRADE`

struct for holding information related to a particular position. This struct is a data container which holds information about each trade throughout the life of our simulation. We can also add our own data to this struct via the `TRADEVAR`

array, which we can populate with values associated with a particular trade.

Zorro stores this array, along with all the other information about each and every position, as members of the `TRADE`

struct. We can access the `TRADE`

struct members in two ways: inside a trade management function (TMF) and inside a trade enumeration loop.

Here’s an example of exporting the last estimated volatility at the time a position was entered, along with the return associated with that position *(this is a simple, long only moving average cross over strategy, data is loaded from Alpha Vantage):*

/* Example of exporting data from a Zorro simulation. */ #define VOL TradeVar[0] int recordVol(var volatility) { VOL = volatility; return 16; } function run() { set(PLOTNOW); StartDate = 2007; EndDate = 2019; BarPeriod = 1440; LookBack = 200; MaxLong = MaxShort = 1; string Name; while(Name = loop("AAPL", "MSFT", "GOOGL", "IBM", "MMM", "AMZN", "CAT", "CL")) { assetHistory(Name, FROM_AV); asset(Name); Spread = Commission = Slippage = 0; vars close = series(priceClose()); vars smaFast = series(SMA(close, 10)); vars smaSlow = series(SMA(close, 50)); var vol = Moment(series(ROCP(close, 1)), 50, 2); // rolling 50-period standard deviation of returns if(crossOver(smaFast, smaSlow)) { enterLong(recordVol, vol); } else if(crossUnder(smaFast, smaSlow)) { exitLong(); } plot("volatility", vol, NEW, BLUE); } if(is(EXITRUN)) { int count = 0; char line[100]; string filename = "Log\\vol.csv"; if(file_length(filename)) { printf("\nFound existing file. Deleting."); file_delete(filename); } printf("\n writing vol file..."); sprintf(line, "Asset, EntryDate, TradeReturn, EntryVol"); file_append(filename, line); for(closed_trades) { sprintf(line, "\n%s, %i, %.6f, %.5f", Asset, ymd(TradeDate), (-2*TradeIsShort+1)*(TradePriceClose-TradePriceOpen)/TradePriceOpen, VOL); file_append(filename, line); count++; } printf("\nTrades: %i", count); } }

The general pattern for accomplishing this is:

- Define a meaningful name for the element of the
`TradeVar`

that we’ll use to hold our volatility data (line 5) - Define a Trade Management Function to expose the
`TRADE`

struct and use it to assign our variable to our`TradeVar`

(lines 7-12). A return value of 16 tells Zorro to run the TMF only when the position is entered and exited. - Calculate the variable of interest in the Zorro script. Here we calculate the rolling 50-day standard deviation of returns (line 34).
- Pass the TMF and the variable of interest to Zorro’s
`enter`

function (line 38). - In the
`EXITRUN`

(the last thing Zorro does after finishing a simulation), loop through all the positions using a trade enumeration loop and write the details, along with the volatility calculated just prior to entry, to a csv file.

Running this script results in a small csv file being written to Zorro’s Log folder. A sample of the data looks like this:

Once we’ve got that data, we can easily read it into our favourite data analysis tool for a closer look. Here, I’ll read it into R and use the `tidyverse`

libraries to dig deeper. *(This will be very cursory. You could and should go a lot deeper if this were a serious strategy.)*

First, read the data in, and process it by adding a couple of columns that might be interesting:

library(ggplot2) library(tidyverse) # analysis of gap size and trade profit path <- "C:/Zorro/Log/" file <- "vol.csv" df <- read.csv(paste0(path, file), header=TRUE, stringsAsFactors=FALSE, strip.white=TRUE) # make some additional columns df$AbsTradeReturn <- abs(df$TradeReturn) df$Result <- factor(ifelse(df$TradeReturn>0, "win", "loss"))

If we `head`

the resulting data frame, we find that it looks like this:

head(df) # Asset EntryDate TradeReturn EntryVol AbsTradeReturn Result # 1 MSFT 20190906 -0.011359 0.00019 0.011359 loss # 2 MSFT 20190904 0.017583 0.00020 0.017583 win # 3 MSFT 20190829 -0.015059 0.00020 0.015059 loss # 4 CL 20190828 -0.010946 0.00017 0.010946 loss # 5 MSFT 20190819 -0.036269 0.00017 0.036269 loss # 6 GOOGL 20190712 0.052325 0.00023 0.052325 win

*Sweet! Looks like we’re in business!*

Now we can start to answer some interesting questions. First, is volatility at the time of entry related to the magnitude of the trade return? Intuitively we’d expect this to be the case, as higher volatility implies larger price swings and therefore larger absolute trade returns:

# is volatility related to the magnitude of the trade return? ggplot(data=df[, c("AbsTradeReturn", "EntryVol")], aes(x=EntryVol, y=AbsTradeReturn)) + geom_point(alpha=0.4) + geom_smooth(method="lm", se=TRUE)

Nice! Just what we’d expect to see.

Does this relationship hold for each individual asset that we traded?

# what about by asset? ggplot(data=df[, c("AbsTradeReturn", "EntryVol", "Asset")], aes(x=EntryVol, y=AbsTradeReturn)) + geom_point(alpha=0.4) + geom_smooth(method="lm", se=TRUE) + facet_wrap(~Asset)

Looks like the relationship generally holds at the asset level, but note that we have a small sample size so take the results with a grain of salt:

# note that we have a small sample size: df %>% group_by(Asset) %>% count() # Asset n # <chr> <int> # 1 AAPL 36 # 2 AMZN 30 # 3 CAT 38 # 4 CL 47 # 5 GOOGL 36 # 6 IBM 37 # 7 MMM 32 # 8 MSFT 37

Is volatility related to the actual trade return?

# is volatility related to the actual trade return? ggplot(data=df[, c("TradeReturn", "EntryVol")], aes(x=EntryVol, y=TradeReturn)) + geom_point(alpha=0.4) + geom_smooth(method="lm", se=TRUE)

Looks like it might be. But this was a long-only strategy that made money in a period where everything went up, so I wouldn’t read too much into this without controlling for that effect.

Is there a significant difference in the entry volatility for winning and losing trades?

# what's the spread of volatility for winning and losing trades? ggplot(data=df[, c("EntryVol", "Result")], aes(x=Result, y=EntryVol)) + geom_boxplot()

Finally, we can treat our volatility variable as a “factor” to which our trade returns are exposed. Is this factor useful in predicting trade returns?

First, we’ll need some functions for bucketing our trade results by factor quantile:

# factor functions get_factor_quantiles <- function(factor_df, n_quantiles = 5, q_type = 7) { n_assets <- factor_df %>% ungroup %>% select(Asset) %>% n_distinct() factor_df %>% mutate(rank = rank(factor, ties.method='first'), quantile = get_quantiles(factor, n_quantiles, q_type)) } get_quantiles <- function(factors, n_quantiles, q_type) { cut(factors, quantile(factors, seq(0,1,1/n_quantiles), type = q_type), FALSE, TRUE) }

If we bucket our results by factor quantile, do any buckets account for significantly more profit and loss? Are there any other interesting relationships?

# if we bucket the vol, do any buckets account for more profit/loss? factor_df <- df[, c("Asset", "EntryVol", "TradeReturn")] names(factor_df)[names(factor_df) == "EntryVol"] <- "factor" quantiles <- get_factor_quantiles(factor_df) r <- quantiles %>% group_by(quantile) %>% summarise(MeanTradeReturn=mean(TradeReturn)) ggplot(data=r, aes(quantile, MeanTradeReturn)) + geom_col() + ggtitle("Returns by volatility quantile")

Looks like there might be something to that fifth quantile (but of course beware the small sample size).

We can retrieve the cutoff value for the fifth quantile by sorting our factor and finding the value four-fifths the length of the resulting vector:

# fifth bin cutoff sorted <- sort(factor_df$factor) sorted[as.integer(4/5*length(sorted))] # [1] 0.00043

There you have it. This was a simple example of exporting potentially relevant data from a Zorro simulation and reading it into a data analysis package for further research.

How might you apply this approach to more serious strategies? What data do you think is potentially relevant? Tell us your thoughts in the comments.

*Want a broader understanding of algorithmic trading? See why it’s the only sustainable approach to profiting from the markets and how you can use it to your success inside the free Algo Basics PDF below….*

The post Can you apply factors to<br> trade performance? appeared first on Robot Wealth.

]]>The post Time is NOT the Enemy:<br> Grow Your Capital by Showing Up appeared first on Robot Wealth.

]]>This assumption can lead us down long and unnecessary rabbit holes and away from the more mundane fundamentals that account for 80% of our day-to-day trading decisions.

When you run a trading business you quickly get to the meat of *practical* market theory – the 80-90% that matters. From our experience, there are **two fundamental concepts** you need to know that are absolutely vital to profiting from the markets.

These concepts are:

- The Time Value of Money
- The Principle of No-Arbitrage

This post is the first in a series of *Quant Basics* where we’ll explore these fundamentals, as well as others*. *We’ll focus solely on the Time Value of Money today as it’s the cornerstone of any profitable trading approach…. including what we do here at Robot Wealth!

So down tools on those deep neural networks for a second. Let’s look at why this first concept is so important for your success as a retail trader.

This is easily the most fundamental concept in finance — let’s break it down.

Simply put, $100 today is worth more to you than $100 received in a year’s time. *Call me captain obvious.*

If you have $100 today you can do potentially valuable things with it now. You could start a business, or you could invest it in a financial asset like a share of a company. By doing this you expect to get positive returns on your $100 for taking on this risk, so it seems like a good idea that is likely to pay off over the *long-run*.

But what if you don’t have a* long-run?*

What if you are going to need that $100 in a year’s time and you want to make sure you preserve that capital, but you also want to put it to work to make more money in the meantime?

In that case, you can lend it to someone who has a slightly longer investment horizon than you. In exchange for them having the use of your money, they will pay you a small amount of (theoretically) guaranteed interest when they give you your money back.

This is what your bank does. When you deposit money in the bank, you’re really *lending* it to the bank. They take various well-diversified risks with that money and they pay interest to their depositors for the use of that money. If they manage things appropriately they will receive a return greater than the interest they pay to depositors – producing a profit.

This simple idea that money has a time value is fundamental to *all of finance. *

What’s in all this for you?

Well, if you have money now you can use it to make more money in the future if you know what to do with it.

*(Stuffing your money under the mattress is a poor choice when someone with ideas and plans is willing to pay you for the use of that money…)*

The Time Value of Money principle gives us a baseline for our risk-taking. If we can get a certain guaranteed yield in the bank then we should only be taking on extra *unguaranteed* risk if we are confident that our expected rewards for taking that risk are greater than the baseline amount we get from the bank.

**In essence, trading is risk-taking.**

As investors, we are rewarded for taking on certain long-term risks, which include buying assets that are sensitive to disappointment in economic growth, inflation and interest rates. Here are some risks associated with various financial products:

So to put our capital to work we get long assets which are exposed to these risks. Getting long lots of them dramatically reduces portfolio volatility through diversification – *which is a good idea!*

**We primarily do this by harvesting risk premia.**

In extremely simple and general terms:

- Buying stocks is a good idea
- Buying bonds is a good idea
- Buying real estate is a good idea
- Selling a little bit of volatility is a good idea.

**By getting those in your portfolio you buy yourself the maximum chance of making money under most conditions. This also adds a portfolio “tailwind” to your active strategies.**

Crucially, this also means that you’ll always be trading one way or another, even when your more active strategies inevitably stop working….which *will* happen.

We incorporate this in our trading at Robot Wealth. Our risk premia strategy has returned around 17% at a CAGR of 26% since going live 9 months ago:

We provide this strategy to our Bootcamp participants, too

You can read more about why and how we harvest risk premia here.

The good news is that there’s no need to make this approach more complicated than it has to be — **80% of success here is just showing up. **

The precise way you execute the above risk-taking matters a lot less than the fact that *you do it at all. *If you find technical details like volatility and covariance management daunting, know that this makes up just 20%.

The 80% is just buying the right assets and keeping hold of them – which is simple to understand and implement, and will put you in a strong position when you get involved in more active, riskier trades.

Harvesting risk premia is a simple way of taking risk to earn above-baseline interest on your capital, especially as a small-time retail trader. It’s the most sensible way to use the time value of money to your advantage, before getting involved in more active strategies. You just need to show up and have a long time horizon.

*That’s Quant Basic number one.*

**In the next Quant Basics post we’ll investigate the pricing efficiency of the markets, why it’s hard to trade given this efficiency, and how you can do it anyway!**

**In the meantime, you can learn more about the fundamentals of algo trading by downloading the free PDF below:**

The post Time is NOT the Enemy:<br> Grow Your Capital by Showing Up appeared first on Robot Wealth.

]]>The post A Quant’s Approach to Drawdown:<br> The Cold Blood Index appeared first on Robot Wealth.

]]>Specifically, they’d:

- do the best job possible of designing and building their trading strategy to be robust to a range of future market conditions
- chill out and let the strategy do its thing, understanding that drawdowns are business-as-usual
- go and look for other opportunities to trade.

*Of course, at some point, you have to retire strategies. Alpha doesn’t persist forever.*

In our own trading, we don’t systematise this decision process. We weigh up the evidence and make discretionary judgements. All things being equal we tend to allow things a lot of space to work out.

However, in this post, we review a systematic approach which can aid this decision making…

In particular, we concentrate on the following question: *“*

Let’s dive in and explore *The Cold Blood Index!*

Johan Lotter, the brains behind the Zorro development platform, proposed an empirical approach to the problem of reconciling backtest returns with live returns.

Put simply, his approach compares a drawdown experienced in live trading to the backtested equity curve, and he called this approach the **Cold Blood Index (CBI)**.

Apart from sounding like something you’d use to rank your favourite reptiles, we’re going to break down the CBI and find out what it can tell you about your drawdowns in live trading — especially when panic alarms are busy going off in your lizard brain.

You can see Johan’s blog post from 2015 for the original article.

*Let’s break it down….*

Paying homage to its creator, we’ll utilise Zorro to illustrate the CBI in action.

*Don’t worry if any of these details are hard to grasp. Follow the bigger picture and revisit the finer nuances later.*

Say you have been trading a strategy live for \(t\) days and are in a drawdown of length \(l\) and depth \(D\).

You want to know how this compares with the backtest. Most people will want to use this to decide whether their strategy is ‘broken’, but remember that *backtests are far from a 100% accurate representation of future performance. *So be careful how you use this thing.

The CBI is an estimate of the probability of experiencing the current drawdown if the strategy *hasn’t* deviated from its backtest.

- A high CBI indicates that the current drawdown is
*not*unexpected, meaning the strategy probably hasn’t deviated from its backtest. - A low CBI indicates that the system that produced the backtest is very unlikely to have produced the current drawdown – meaning the live strategy
*has*deviated from the backtest.

Our *null hypothesis* (default stance) is that the live strategy’s current drawdown could have been produced by the backtested strategy.

The CBI is the *p*-value used to evaluate the statistical test of our null hypothesis.

**More simply,** **the CBI is the probability that the current drawdown would be equal to (or more extreme than) its observed value if the strategy hadn’t deviated**.

Typically, we don’t have access to the entire population of data points, but only a sample or subset of the total population. So, statistical hypothesis testing is a process that tests claims about the population we DO have, on the basis of evidence gleaned from the sample.

Since we don’t have the full population we can never be *totally sure* of any conclusion we draw (just like virtually all things in trading), but we CAN accept or reject the claims on the basis of the strength of the evidence.

The strength of the evidence is encapsulated in the ** p-value. **But we also need a statistical test for calculating it.

**In this case, our statistical test is empirical in nature – it is derived directly from the backtest equity curve. **

Deriving an empirical statistical test involves calculating the distribution of the phenomenon we are testing – in this case, the depth of drawdowns of length \(l\) within a live trading period \(t\) – from the sample data (our backtest).

Then, we compare the phenomenon’s observed value (\(D\), the depth of our current drawdown of length \(l\)) with the distribution obtained from the sample data, deriving the *p*-value directly from the observed value’s position on the sample distribution.

Naturally, let’s start with the *worst-case scenario.*

Take the simple case where our trading time, \(t\) is the same as the length of our current drawdown, \(l\).

To calculate the empirical distribution, we simply take a window of length \(l\) and place it at the first period in the backtest balance curve.

That is, the window initially covers the backtest balance curve from the first bar to bar \(l\).

Then, we simply calculate the difference in balance across the window, \(G\), and record it. Then, we slide the window by one period at a time until we reach the end of the balance curve, calculating the change in balance across each window as we go.

At the completion of this process, we have a total of \(M\) values for balance changes across windows of length \(l\) (\(M\) is equal to the length of the backtest minus \(l\) plus one). Of these \(M\) values, \(N\) will show a greater drawdown than our current drawdown, \(D\). Then, the CBI, here denoted \(P\), is simply \[P = \frac{N}{M}\]

Here’s the code for calculating the CBI and plotting the empirical distribution from the backtest, for this special case where the strategy is underwater from the first day of live trading:

/* Cold Blood Index Special case of drawdown length equal to trade time That is, strategy underwater since inception */ int TradeDays = 60; // Days since live start and in drawdown var DrawDown = 20; // Current drawdown depth in account currency string BalanceFile = "Log\\simple_portfolio.dbl"; void Histogram(string Name,var Value,var Step,int Color) /* plots a histogram given a value and bin width */ { var Bucket = floor(Value/Step); plotBar(Name,Bucket,Step*Bucket,1,SUM+BARS+LBL2,Color); } void main() { var HistStep = 10; //bin width of histogram plotBar("Live Drawdown",DrawDown/HistStep,DrawDown,80,BARS|LBL2,BLACK); //mark current drawdown in histogram // import balance curve int CurveLength = file_length(BalanceFile)/sizeof(var); var *Balances = file_content(BalanceFile); // get number of samples int M = CurveLength - TradeDays + 1; // sliding window calculations var GMin=0, N=0; //define N as a var to prevent integer truncation in calculation of P int i; for(i=0; i<M; i++) { var G = Balances[i+TradeDays-1] - Balances[i]; if(G <= -DrawDown) N += 1.; if(G < GMin) GMin = G; Histogram("G", G, HistStep, RED); } var P = N/M; printf("\nTest period: %i days",CurveLength); printf("\nWorst test drawdown: %.f",-GMin); printf("\nSamples: %i\nSamples worse than observed: %i",M,(int)N); printf("\nCold Blood Index: %.1f%%",100*P); }

To use this script, you first need to save the profit and loss time series data from a backtest (also, the backtest will need to use the same money management approach as used in live trading).

Do that by setting Zorro’s

LOGFILEand

BALANCEparameters, which automatically save the backtest’s balance curve in Zorro’s log file.

Having saved the balance curves, in the CBI script above, make sure the string

BalanceFileis set correctly (line 11).

Here’s an example. Say we had been trading our strategy live and we were concerned about the performance of the *EUR/USD: rsi* component. It’s been trading for 60 days now, and that component is showing a drawdown of $20.

Plugging those values into the script above and loading that component’s backtested balance curve gives the following histogram:

The script also outputs to the Zorro GUI window some pertinent information. Firstly, that of 1,727 sample windows, 73 were worse than our observed drawdown. CBI is then calculated as \(73/1,727 = 0.04\), which may exceed some individuals’ threshold confidence level of 0.05 (remember a smaller CBI provides stronger evidence of a “broken” strategy). But this is somewhat arbitrary.

We can also run the CBI script with various values of

TradeDaysand

DrawDownto get an idea of what sort of drawdown would induce changes in the p-value.

The implementation of CBI above is for the special case where the strategy has been experiencing a drawdown since the first day of live trading.

Of course, this (hopefully) won’t always be the case!

**For drawdowns that come after some new equity high, calculation of the empirical distribution is a little tricker. **

Why?

Because we now have to consider the total trading time \(t\) as well as the drawdown time \(l\).

Reproducing this distribution of drawdowns faithfully would require traversing the balance curve using nested windows: an outer window of length \(t\) and an inner window of length \(l\) traversing the outer window, period by period, at every step of the outer window’s journey across the curve.

That is, for a backtest of length \(y\), we now have \((y-t+1)*(t-l+1)\) windows to evaluate.

Rather than perform that cumbersome operation, we can apply the same single rolling window process that was used in the simple case, combined with some combinatorial probability to arrive at a formula for CBI, which we denote \(P\), in terms of the previously defined parameters \(M, N, T\):

\[P = 1 – \frac{(M – N)!(M – T)!}{M!(M – N – T)!}\]

The obvious problem with this equation is that it potentially requires evaluation of factorials on the order of \((10^3)!\), which is approximately \(10^{2500}\), far exceeding the maximum range of

vartype variables.

To get around that inconvenience, we can take advantage of the relationships \[ln(1000 * 999 * 998 * … * 1) = ln(1000) + ln(999) + ln(998) + … + ln(1)\] and \[e^{ln(x)} = x\] to rewrite our combinatorial equation thus:

\[P = 1 – e^{x}\] where \[x = ln(\frac{(M – N)!(M – T)!}{M!(M – N – T)!}\] \[= ln((M – N)!) + ln((M – T)!) – ln(M!) – ln((M – N – T)!)\]

Now we can deal with those factorials using a function that recursively sums the logarithms of their constituent integers, which is much more tractable.

Here’s a script that verifies the equivalence of the two approaches for small integers, as well as its output:

var logsum(int n) { if(n <= 1) return 0; else return log(n)+logsum(n-1); } int factorial(int n) { if (n <= 1) return 1; if (n >= 10) { printf("%d is too big", n); return 0; } return n*factorial(n-1); } void main() { int M = 5; int N = 3; printf("\nEvaluate x = (%d-%d)!/%d!using\nfactorial and log transforms", M, N, M); printf("\nBy evaluating factorials directly,\nx = %f", (var)factorial(M-N)/factorial(M)); printf("\nBy evaluating sum of logs,\nx = %f", exp(logsum(M-N) - logsum(M))); } /* OUTPUT: Evaluate x = (5-3)!/5!using factorial and log transforms By evaluating factorials directly, x = 0.016667 By evaluating sum of logs, x = 0.016667 */

Here’s the script for the general case of the CBI (courtesy Johan Lotter):

/* Cold Blood Index General case of the CBI where DrawDownDays != TradeDays */ int TradeDays = 100; // Days since live start int DrawDownDays = 60; // Length of drawdown var DrawDown = 20; // Current drawdown depth in account currency string BalanceFile = "Log\\simple_portfolio.dbl"; var logsum(int n) { if(n <= 1) return 0; else return log(n)+logsum(n-1); } void main() { // import balance curve int CurveLength = file_length(BalanceFile)/sizeof(var); var *Balances = file_content(BalanceFile); // calculate parameters and check sufficient length int M = CurveLength - DrawDownDays + 1; int T = TradeDays - DrawDownDays + 1; if(T < 1 || M <= T) { printf("Not enough samples!"); return; } // sliding window calculations var GMin=0, N=0; //define N as a var to prevent integer truncation in calculation of P int i = 0; for(; i < M; i++) { var G = Balances[i+DrawDownDays-1] - Balances[i]; if(G <= -DrawDown) N += 1.; if(G < GMin) GMin = G; } var P; if(TradeDays > DrawDownDays) P = 1. - exp(logsum(M-N)+logsum(M-T)-logsum(M)-logsum(M-N-T)); else P = N/M; printf("\nTest period: %i days",CurveLength); printf("\nWorst test drawdown: %.f",-GMin); printf("\nM: %i N: %i T: %i",M,(int)N,T); printf("\nCold Blood Index: %.1f%%",100*P); }

Using the same drawdown length and depth as in the simple case, but now having traded for a total of 100 days, our new CBI value is 83%, which provides* no evidence* to suggest that the component has deteriorated.

For comparing drawdowns in live trading to those in a backtest, the CBI is useful. **But we can do better by incorporating statistical resampling techniques.**

Drawdown is a function of the *sequence* of winning and losing trades. However, a backtest represents just one realization of the numerous possible winning and losing sequences that could arise from a trading system with certain returns characteristics.

As such, the CBI presented above considers just one of many possible returns sequences that could arise from a given trading system.

So, we can make the CBI more robust by incorporating the algorithm into a Monte Carlo routine, such that many unique balance curves are created by randomly sampling the backtested trade results, and running the CBI algorithm separately on each curve.

The code for this Resampled Cold Blood Index is shown below, including calculation of the 5th, 50th and 95th percentiles of the resampled CBI values.

/* Resampled Cold Blood Index General case of the Resampled CBI where DrawDownDays != TradeDays */ int TradeDays = 100; // Days since live start int DrawDownDays = 60; // Length of drawdown var DrawDown = 20; // Current drawdown depth in account currency string BalanceFile = "Log\\simple_portfolio.dbl"; var logsum(int n) { if(n <= 1) return 0; else return log(n)+logsum(n-1); } void main() { int CurveLength = file_length(BalanceFile)/sizeof(var); printf("\nCurve Length: %d", CurveLength); var *Balances = file_content(BalanceFile); var P_array[5000]; int k; for (k=0; k<5000; k++) { var randomBalances[2000]; randomize(BOOTSTRAP, randomBalances, Balances, CurveLength); int M = CurveLength - DrawDownDays + 1; int T = TradeDays - DrawDownDays + 1; if(T < 1 || M <= T) { printf("Not enough samples!"); return; } var GMin=0., N=0.; int i=0; for(; i < M; i++) { var G = randomBalances[i+DrawDownDays-1] - randomBalances[i]; if(G <= -DrawDown) N += 1.; if(G < GMin) GMin = G; } var P; if(TradeDays > DrawDownDays) P = 1. - exp(logsum(M-N)+logsum(M-T)-logsum(M)-logsum(M-N-T)); else P = N/M; // printf("\nTest period: %i days",CurveLength); // printf("\nWorst test drawdown: %.f",-GMin); // printf("\nM: %i N: %i T: %i",M,(int)N,T); // printf("\nCold Blood Index: %.1f%%",100*P); P_array[k] = P; } var fifth_perc = Percentile(P_array, k, 5); var med = Percentile(P_array, k, 50); var ninetyfifth_perc = Percentile(P_array, k, 95); printf("\n5th percentile CBI: %.1f%%",100*fifth_perc); printf("\nMedian CBI: %.1f%%",100*med); printf("\n95th percentile CBI: %.1f%%",100*ninetyfifth_perc); }

Using the same drawdown length, drawdown depth and trade time as we evaluated in the single-balance curve example, we now find that our median resampled CBI is around 98%.

It turns out that the value obtained by only evaluating the backtest balance curve was closer to the 5th percentile (that is, the lower limit) of resampled values.

While this is not significant in this example (regardless of the method used, it is clear that the component has not deteriorated) this could represent valuable information if things were more extreme.

The usefulness of the resampled CBI declines for increasing backtest length, but it is easily implemented, comes at little additional compute time (thanks in part to Lite-C’s blistering speed), and provides additional insight into strategy deterioration by considering the random nature of individual trade results.

Note however that this method would break down in the case of a strategy that exhibited significant serially correlated returns, since resampling the backtest balance curve would destroy those relationships.

As interesting as this approach is, probably not….

In nearly all cases **this is giving the right answer to the wrong question:**

“Should I pull out of this strategy bdcause its live performance looks different to the backtest?”

To labour the points we made in the first post in this series, experienced traders know that it’s a bad idea to set performance expectations based on a backtest. Manageable deviations from that backtest performance usually won’t trigger any alarm bells that inspire interference with the strategy.

Instead, we make tea and look for other trades.

To succeed in trading you have to realise and accept the randomness and efficiency of the markets — part of which means sitting through rather uncomfortable drawdowns which won’t show up in R&D. Being disappointed by live performance vs your exciting backtest is very much the norm…. you just have to take it on the chin and trust the soundness of your development process. The markets are too efficient and chaotic to care about meeting our expectations.

*We call this “Embracing the Mayhem”.*

*Embracing the Mayhem *is just one of the 7 Trading Fundamentals we teach inside our Bootcamps. These Fundamentals show the approach we use to trade successfully with Robot Wealth, which is normally only learned after years of expensive trial, error and frustration.

**But you can skip all that — you can get access to a bunch of these Fundamental videos for free by entering your email below:**

The post A Quant’s Approach to Drawdown:<br> The Cold Blood Index appeared first on Robot Wealth.

]]>The post A Quant’s Approach to Drawdown: Part 1 appeared first on Robot Wealth.

]]>Systems go, let’s trade it.

Imagine this new strategy **enters a drawdown.…maybe a lengthy one….maybe from day one!**

How would you react to such a letdown?

A common response to a long or sharp drawdown is to defer to our self-preservative instincts and pull the strategy entirely. Maybe you’d compare your poor live performance to your promising backtest, turn red and frisbee your laptop out the window.

If you enjoy making money trading you’ll need to do better.

A relatively experienced systematic trader, who we’ll call *Jack the Quant,* would look at this drawdown scenario a bit differently.

**Yes I know Jack is actually a terribly drawn Napoleon Dynamite**

Anyway, Jack would rely on:

- an understanding (and acceptance) of the nature of the markets
- a sound systematic research process
- generally
*chilling the heck out*

We’re going to quickly talk about the latter two. You can learn about all three in depth here (or here for existing RW members).

Firstly, he’d go make some tea and relax.

Jack knows it would be trivial to show that a strategy with a true Sharpe ratio of 1.5 stands a reasonable chance of having a two-year drawdown. Jack also knows that the strategy he backtested to a Sharpe of 1.5 isn’t likely to deliver him that performance once live.

This implies that in order to decide whether the performance of Jack’s strategy is “unexpected”, he might have to *wait years. *

That’s assuming he has an accurate picture of what the “expected” performance might be – which he almost certainly *doesn’t* have.

Quite the conundrum, *isn’t it?*

This is why having a **sound research process to begin with** is so important to your success and sanity as a trader.

A sensible research process lets you gather enough evidence to make a smart bet that your strategy has a good chance of paying off in the long run, even if your strategy isn’t too hot out of the gates.

Satisfied with that evidence, Jack’s approach is to:

- size the strategy small
- chill out and let it do its thing for a couple of years
- in the meantime go hunting for other trades.

This might surprise most beginners who would assume a systematic trader would have some kind of automated process for decision making during drawdown. But, Jack’s approach stems from his understanding that trading is hard and that losing money sometimes is completely normal. Sometimes (or actually, most times), the solution is just to relax and keep moving.

Besides, Jack is smart and has structured his portfolio sensibly so that his risk premia tailwind acts as a buffer for when his alpha trades don’t go his way, which is *inevitable*. In the end, he still has trades on and is making money.

This is the less frenzied, more liveable way to run a trading operation.

So be like Jack — copy his well-rounded approach and keep your eyes open for new trades.

You bet!

Jack would down his tea quick smart and pull the pin if live trading demonstrated that his backtest assumptions had been violated so badly that his edge was destroyed. Maybe he was a tad optimistic in his execution assumptions during the research phase, and the live market showed that unexpected frictions are too much of a hurdle to surmount.

He might also pull out if the basis for the trade ceased to exist. For instance, say the trade was based on forced flows from ETF rebalancing. If the ETF Jack was trading changed its mandate in such a way that impacted those forced flows, he’d stop trading the strategy.

But that’s a pretty clear cut example. Often trading *isn’t clear cut at all*.

Say you were running a convergence trade betting that two related financial assets were linked through shared risk factors. It’s a complex question to decide if the dynamics of those risk factors have changed permanently.

In that case again, as in most cases, the best answer is to size it small, chill out, and find more stuff to trade.

Trading can be uncomfortable, you’ll often lose money before you make it….

….this is part of the game, and that’s OK! We just need to make more than we lose. We show you this inside our Bootcamps.

There’s always a temptation to compare live trading results with the backtest to decide whether or not to pull out. But this violates one of the most important tenets of successful systematic trading:

**Don’t use a backtest to define performance objectives!**

Think about it. You’ve spent days or weeks or longer researching and developing the strategy. Consider how many decision points you encountered along the way. Which universe of assets to trade. Whether to choose this parameter, that parameter, or both. No matter how careful and considered your development process, some amount of optimistic bias creeps in with each decision. This is a fact of life. Unavoidable. It’s part and parcel of strategy development. We *can’t not* introduce data mining bias. Which means that we can’t trust our backtest to set future performance objectives for us.

Don’t get me wrong, backtesting is probably the most powerful tool in your arsenal. The ability to acquire empirical evidence that an idea worked or not in the past is crucial. Backtesting gives you this ability.

Problems arise, however, when you expect live trading performance to match the backtest.

Again, sensible ol’ Jack knows this.

*Not entirely….*

There’s always the temptation to try and systematise decisions (they don’t call it systematic trading for nothing). The decision to pull a strategy based on live performance compared to a backtest is no exception. It’s comfortable to know that there’s an algorithm or a set of rules that’s got our back.

This is generally *not* a good idea.

Generally speaking, to repeat what we talked about earlier, a good systematic trader is likely to approach drawdown by:

- adopting a sound research process and sizing small – before drawdown ever occurs
- chilling out and letting the strategy go to work, even during said drawdown
- always looking for other opportunities to trade

Find yourself faced with a strategy that’s a bit underwater? Does it fly in the face of your backtest? Are you frustrated?

**It’s all part of the journey.**

*But all that said, maybe there IS some small utility in quantifying this approach *— we’ll investigate this in **part 2** of this series.

While we’re putting that together you can learn more about where exactly algo trading CAN help you trade more profitably by downloading the free PDF below….

The post A Quant’s Approach to Drawdown: Part 1 appeared first on Robot Wealth.

]]>The post Run Trading Like a Business, <br>Not a Board Game appeared first on Robot Wealth.

]]>“Why would I need to *recalibrate* to a practical approach? *What do you think I’m doing??”*

Maybe you’re doing fine, but many aspiring traders arrive at the Robot’s doorstep fantasising about hunting down that one perfect alpha, building complex ML models, mass data mining their way to bloodshot eyes, or some other sophisticated (and crucially, *theoretical*) means of untangling the markets with supreme intellectual capacity.

**Many aspiring traders won’t make money — spending years chasing their own tails over such low ROI pursuits.**

“Oh, is that a neural network?!”

Rather than focusing on a tried and tested approach, in practice, many aspiring traders follow the approaches with the *weakest history of success, *wrongly assuming that this is how you make money in the markets.

To make your life easier, we suggest you bookmark the fun distractions (at least until you have the luxury of being so experimental) in favour of the simpler, lower-hanging fruit that will grow your account.

Let’s see what this shift looks like.

Say you have *X* dollars of trading capital. As traders, what we’re trying to do is **maximise the chance of turning ****$X**** into more than ****$X****.**

This doesn’t seem like a revolutionary way of looking at things – but it seems to me that most traders aren’t focusing * ruthlessly *on this problem, certainly not at the expense of the fun intellectual challenges.

For instance, a* fun* approach might be to pull some EUR/USD exchange rate data or price data from the E-mini S&P futures, and mine for buy and sell rules that would have proved profitable in the past.

Another *fun* approach may be to try out a lot of indicators and rules in some backtesting software until something promising comes up. Maybe throw in an automated, cloud-based data mining setup. Hell, why not a genetic optimisation algorithm or neural network?

These are fun and exciting games to play, and I understand why traders gravitate towards them.

Individual retail traders tend to enjoy tackling complex problems. They also tend to assume that greater complexity equates to greater market potential. These two factors combined mean you’ll often walk right by the *more effective activities that will make your money grow.*

**At Robot Wealth our focus is using our time, resources and capital to get a return on our money – and so should you if you want to win.**

We teach this approach using several * fundamental truths* in our Bootcamps.

The one we’ll talk briefly about now is *Run it Like a Business — *because it’s the fundamental that becomes most immediately apparent when you make this shift from *fun -and-theoretical* to *money-making-and-practical* (you can get primers on the other 6 at the bottom of this post).

*Not that making money isn’t fun or anything…yeesh. *

When you shift focus to ruthlessly turning $x into >$x, a lot of real-world issues and logistical/infrastructure problems begin bubbling to the surface. Problems which hadn’t become relevant before, probably because growing your money by the most realistic, mundane means wasn’t your sole objective.

It’s like fantasising about buying your dream car. You imagine driving it, and the sense of accomplishment in attaining it. Yet part of that fantasy

likely doesn’tinclude the higher-octane fuel needs, pricey insurance policy and diligent maintenance required just to get it out of your driveway — these issues only present themselves once you’re in the driver’s seat, not in fantasy land.

Let’s think about what you need to run your ~~Lambo~~ systematic trading setup.

You need:

- capital to trade
- strategies to trade
- systems and infrastructure and processes for trading, performance reporting, reconciliation accounting, backtesting etc
- skills, knowledge and experience

Like any business, you need to set up your infrastructure and continuously refine and improve it. **Much of the results you get from your trading will come from how effectively you operate this infrastructure, not how secretive and special your alpha is.**

You have several resources in order to achieve this. First, and most obviously, is your **time. **

But, there are a lot of things to do! How will you split your time between:

- Researching ideas and developing trading strategies
- Development or setup of the other essential software and infrastructure
- The actual trading: monitoring, tweaking, and reconciling systems; executing semi-automated strategies etc
- Accounting and other admin work
- Reading about the markets and technologies and improving your knowledge and skills

Essentially, the work of an entire trading floor is in your hands. But don’t panic.

You’ll be relieved to know that you *don’t* need to make everything from scratch yourself and *you shouldn’t*. Leave that to the people who play *the fun games* at the expense of trading returns. You can buy software or pay others to make it for you.

Of course, there are some things you probably can’t do yourself even if you wanted to, such as streaming market data.

Okay, now that we’ve knocked those off, the task becomes….

How do you make the best use of your resources of time and money to improve your trading setup in order to maximise the return you make on your trading capital?

One of the main lessons we communicate in *Embrace the Mayhem* is that the market is very efficient and trading is hard.

So the prospect of trading in order to turn $X into more than $X should somewhat intimidate you. If you don’t find it intimidating then you are underestimating the efficiency of the market!

The path is straight and narrow, but your resources are finite, so **you need to be prioritising the highest ROI activities.**

High ROI activities include:

- Implementing new trading strategies within a proven framework. An example might be to implement a portfolio of pairs trades in the equity market.
- Scaling existing strategies to new instruments or markets. For example, porting the pair trading setup to a different international equity market.
- Well planned iterative research, set up in such a way that you can test and invalidate ideas quickly. This is the kind of research we show in the Bootcamps.

Low ROI activities include:

- Large scale data mining exercise or any research that involves a large amount of effort to be expended before the idea can be invalidated
- Looking for totally unique alpha ideas when you could be implementing simple trades within a proven framework
- Building your own backtesting platform. You probably think you’ll learn a ton doing this and you’re not wrong about that – but it’s going to suck a huge amount of your time on something that you can buy in cost-effectively.
- Building your own execution platform
- Re-writing stuff that already works in a different language.

Of course, there’s room for fun and creativity in every business. But, if you’re serious about trading, you want to prioritise the activities that are most effective and give yourself the *maximum chance of return* on your capital, time and expenses.

It’ll surprise you how few solo retail traders genuinely do this. Which is why most of them go round in circles for most of their trading careers without getting any closer to making real money.

I’ll repeat this point again because it’s so vital to your success as a trader:

**Many of the results you get from your trading will come from how effectively you operate and refine your trading business, focusing on high ROI activities, and not how secretive, complex and special your alpha is.**

You want to *Run it Like a Business.*

When you start thinking like this you’re going to look at your “business” and realise it’s not going to be very viable one on day 1.

*That’s okay. *

It’s going to have to be a labour of love to start with. Chances are you don’t have the capital, strategies, systems, skills, experience and other assets that you need to make this viable. Not today, anyway.

This is fine. Perfectly normal.

You *don’t* go from 0 to 100 on day 1. And you *don’t* need an exact step-by-step plan.

Work hard, concentrate on the fundamentals and keep your eyes open to the opportunities that will present themselves if you do this work well with clear eyes and an open heart.

Concentrate on:

- Growing your skills, intuition and experience. And building a library of trading frameworks, strategies, approaches and tools.
- Saving money to grow your trading capital
- Growing a network of other traders
- And prioritising effective high ROI work over vanity projects.

Real trading isn’t some board game or puzzle, it’s a **business** which needs multiples roles filled and processes refined.

You don’t have many resources as a solo trader, and you are competing with some of the best minds in the game. Why waste your time and energy on the stuff that won’t move you any closer to your goals?

Focus ruthlessly on turning $x in >$x (running it like a business) – the next problems for you to tackle will soon make themselves known.

*Run it Like a Business* is **just one** of the 7 Fundamentals we teach inside the first week of Algo Bootcamp, along with:

**and The (only 2) Ways to Make Money (we don’t have a nice picture for that one yet…)**

Want a walkthrough of **all of these?**

On the run-up to our next Algo Bootcamp release, you can discover the approach behind the above concepts — sent directly to you via email. **Simply join the Bootcamp waiting list below.**

The post Run Trading Like a Business, <br>Not a Board Game appeared first on Robot Wealth.

]]>The post Practical Pairs Trading appeared first on Robot Wealth.

]]>While you can, in theory, create mean reverting portfolios from as many instruments as you like, this post will largely focus on the simplest case: pairs trading.

Pairs trading involves buying and selling a portfolio consisting of two instruments. The instruments are linked in some way, for example they might be stocks from the same business sector, currencies exposed to similar laws of supply and demand, or other instruments exposed to the same or similar risk factors. We are typically long one instrument and short the other, making a bet that the value of this long-short portfolio (the spread) has deviated from its equilibrium value and will revert back towards that value.

One of the major attractions of pairs trading is that we can achieve market neutrality, or something close to it. Because the long and short positions offset each other, pairs trading can be somewhat immune to movements of the overall market, thus eliminating or reducing market risk – theoretically at least.

The fact that we can construct artificial spreads with mean-reverting properties is one of the major attractions of this style of trading. But there are some drawbacks too.

For starters, if a series was mean reverting in the past, it may not be mean reverting in the future. Constructed spreads typically mean revert when random, non-structural events affect the value of the components. A good spread combined with a good trading strategy will capture these small opportunities for profit consistently. On the other hand, when a *structural* *shift* occurs – such as a major revaluation of one asset but not the other – such a strategy will usually get burned quite badly.

How can we mitigate that risk? Predicting the breakdown of a spread is very difficult, but a sensible way to reduce the risk of this approach is to trade a diverse range of spreads. As with other types of trading, diversity tends to be your friend. I like to have an economic reason for the relationship that links the components, but to be totally honest, I’ve seen pairs trading work quite well between instruments that I couldn’t figure a relationship for. If you understand the relationship, you may be in a better position to judge when and why the spread might break down. Then again, you might not either. Breakdowns tend to happen suddenly and without a lot of warning.

Probably the best example of what not to do with a pairs trading strategy is the Long Term Capital Management meltdown. I won’t go into the details here, but there is plenty written about this incident and it makes a fascinating and informative case study.

When two or more non-stationary series can be combined to make a stationary series, the component series are said to be **cointegrated**. One of the challenges of pairs trading is to determine the coefficients that define this stationary combination. In pairs trading, that coefficient is called the hedge ratio, and it describes the amount of instrument B to purchase or sell for every unit of instrument A. The hedge ratio can refer to a dollar value of instrument B, or the number of units of instrument B, depending on the approach taken. Here, we will largely focus on the latter approach.

The Cointegrated Augmented Dickey Fuller (CADF) test finds a hedge ratio by running a linear regression between two series, forms a spread using that hedge ratio, then tests the stationarity of that spread. In the following examples, we use some R code that runs a linear regression between the price series of Exxon Mobil and Chevron to find a hedge ratio and then tests the resulting spread for stationarity.

First, download some data, and plot the resulting price series (here we use the Robot Wealth data pipeline, which is a tool we wrote for members for efficiently getting prices and other data from a variety of sources):

# Preliminaries library(urca) source("data_pipeline.R") prices <- load_data(c('XOM', 'CVX'), start='2014-01-01', end = '2017-01-01', source='av', return_data = 'Adjusted', save_indiv = TRUE) plot(prices, col=c('blue', 'red'), main = 'XOM and CVX') legend('bottomright', col=c('blue', 'red'), legend=c('XOM', 'CVX'), lty=1, bty='n')

Next, we create a scatter plot of our price series, which will indicate whether there is an underlying relationship between them, and fit a linear regression model using ordinary least squares:

# scatter plot of prices plot(coredata(prices[, 'XOM.Adjusted']), coredata(prices[, 'CVX.Adjusted']), col='blue', xlab='XOM', ylab='CVX') # linear regression of prices fit <- lm(coredata(prices[, 'CVX.Adjusted']) ~ coredata(prices[, 'XOM.Adjusted'])) summary(fit) ''' # output: # Call: # lm(formula = coredata(prices[, "CVX.Adjusted"]) ~ coredata(prices[, # "XOM.Adjusted"])) # # Residuals: # Min 1Q Median 3Q Max # -9.6122 -2.2196 -0.3546 2.4747 9.3366 # # Coefficients: # Estimate Std. Error t value Pr(>|t|) # (Intercept) -41.39747 1.88181 -22.00 <2e-16 *** # coredata(prices[, "XOM.Adjusted"]) 1.67875 0.02309 72.71 <2e-16 *** # --- # Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 # # Residual standard error: 3.865 on 754 degrees of freedom # Multiple R-squared: 0.8752, Adjusted R-squared: 0.875 # F-statistic: 5287 on 1 and 754 DF, p-value: < 2.2e-16 ''' clip(min(coredata(prices[, 'XOM.Adjusted'])), max(coredata(prices[, 'XOM.Adjusted'])), min(coredata(prices[, 'CVX.Adjusted'])), max(coredata(prices[, 'CVX.Adjusted']))) abline(fit$coefficients, col='red')

The line of best fit over the whole data set has a slope of 1.68, which we’ll use as our hedge ratio. Notice however while that line is the *global* line of best fit, there are clusters where this line isn’t the *locally* best fit. We’ll likely find that points in those clusters occur close to each other in time, which implies that a dynamic hedge ratio may be useful. We’ll return to this idea later, but for now, we’ll use the slope of the global line of best fit.

Next, we construct and plot a spread using the hedge ratio we found above:

# construct and plot spread hedge <- fit$coefficients[2] spread <- prices[, 'CVX.Adjusted'] - hedge*coredata(prices[, 'XOM.Adjusted']) plot(spread, main='CVX-XOM Spread', col='black')

The resulting spread is clearly much more mean-reverting than either of the underlying price series, but let’s test that observation using the ADF test:

# ADF TEST adf <- ur.df(spread, type = "drift", selectlags = 'AIC') summary(adf) ''' ############################################### # Augmented Dickey-Fuller Test Unit Root Test # ############################################### # # Test regression drift # # # Call: # lm(formula = z.diff ~ z.lag.1 + 1 + z.diff.lag) # # Residuals: # Min 1Q Median 3Q Max # -3.8638 -0.5122 0.0037 0.5102 7.0391 # # Coefficients: # Estimate Std. Error t value Pr(>|t|) # (Intercept) -1.081908 0.375640 -2.880 0.00409 ** # z.lag.1 -0.026377 0.009032 -2.920 0.00360 ** # z.diff.lag -0.023002 0.036596 -0.629 0.52984 # --- # Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 # # Residual standard error: 0.9464 on 751 degrees of freedom # Multiple R-squared: 0.01263, Adjusted R-squared: 0.01 # F-statistic: 4.803 on 2 and 751 DF, p-value: 0.008459 # # # Value of test-statistic is: -2.9204 4.3109 # # Critical values for test statistics: # 1pct 5pct 10pct # tau2 -3.43 -2.86 -2.57 # phi1 6.43 4.59 3.78 '''

Observe that the value of the test statistic is -2.92, which is significant at the 95% level. Thus we reject the null of a random walk and assume our series is stationary.

You might have noticed that since we used ordinary least squares (OLS) to find our hedge ratio, we will get a different result depending on which price series we use for the dependent (y) variable, and which one we choose for the independent (x) variable. The different hedge ratios are *not* simply the inverse of one another, as one might reasonably and intuitively expect. When using this approach, it is a good idea to test both spreads using the ADF test and choosing the one with the most negative test statistic, as it will be the more strongly mean reverting option (at least in the past).

As an alternative OLS, we can use total least squares (TLS), which accounts for the variance in both series, while OLS accounts for the variance in only one (this is why we get different hedge ratios depending on our choice of dependent variable). TLS is symmetrical, and will give the same result regardless of our choice of dependent variable. In practical terms, the hedge ratios obtained by OLS and TLS usually won’t differ greatly, but when they *do* differ, that difference is likely to be significant. So it is worth including the TLS approach in your analysis.

Implementing TLS in R is straightforward using principal components analysis (PCA). Here’s the syntax:

# TLS for computing hedge ratio pca <- princomp(~ coredata(prices[, 'CVX.Adjusted']) + coredata(prices[, 'XOM.Adjusted'])) tls_hedge <- pca$loadings[1, 1]/pca$loadings[2, 1] tls_spread <- prices[, 'CVX.Adjusted'] - tls_hedge*coredata(prices[, 'XOM.Adjusted']) plot(tls_spread, main='CVX-XOM Spread', col='black')

The TLS approach results in a hedge ratio of 1.86 and a spread that is not all that different from our OLS hedge ratio. If we flipped the dependent and independent variables, our hedge ratio would simply the inverse of the one we just calculated.

Let’s see if it results in a more significant ADF test result:

# ADF TEST adf <- ur.df(tls_spread, type = "drift", selectlags = 'AIC') summary(adf) ''' # output ############################################### # Augmented Dickey-Fuller Test Unit Root Test # ############################################### # # Test regression drift # # # Call: # lm(formula = z.diff ~ z.lag.1 + 1 + z.diff.lag) # # Residuals: # Min 1Q Median 3Q Max # -4.0533 -0.5579 0.0100 0.5905 7.3852 # # Coefficients: # Estimate Std. Error t value Pr(>|t|) # (Intercept) -1.779796 0.545639 -3.262 0.00116 ** # z.lag.1 -0.031884 0.009693 -3.289 0.00105 ** # z.diff.lag -0.022874 0.036562 -0.626 0.53176 # --- # Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 # # Residual standard error: 1.057 on 751 degrees of freedom # Multiple R-squared: 0.01577, Adjusted R-squared: 0.01315 # F-statistic: 6.017 on 2 and 751 DF, p-value: 0.002556 # # # Value of test-statistic is: -3.2894 5.4476 # # Critical values for test statistics: # 1pct 5pct 10pct # tau2 -3.43 -2.86 -2.57 # phi1 6.43 4.59 3.78 '''

Our test statistic is slightly more negative than that resulting from the spread constructed using OLS.

The speed at which our spread mean reverts has implications for its efficacy in a trading strategy. Faster mean reversion implies more excursions away from, and subsequent reversion back to the mean. One estimate of this quantity is the **half-life of mean reversion**, which is defined for a continuous mean reverting process as the average time it takes the process to revert half-way to the mean.

We can calculate the half-life of mean reversion of our spread using the following

half_life()function:

half_life <- function(series) { delta_P <- diff(series) mu <- mean(series) lag_P <- Lag(series) - mu model <- lm(delta_P ~ lag_P) lambda <- model$coefficients[2] H <- -log(2)/lambda return(H) } H <- half_life(tls_spread)

In this example, our half life of mean reversion is 21 days.

The Johansen test is another test for cointegration that generalizes to more than two variables. It is worth being familiar with in order to build mean reverting portfolios consisting of three or more instruments. Such a tool has the potential to greatly increase the universe of portfolios available and by extension the diversification we could potentially achieve. It’s also a data-miner’s paradise, with all the attendant pitfalls.

The

urcapackage implements the Johansen test via the

ca.jo()function. The test can be specified in several different ways, but here’s a sensible specification and the resulting output (it results in a model with a constant offset, but no drift) using just the CVX and XOM price series:

jt <- ca.jo(prices, type="trace", K=2, ecdet="const", spec="longrun") summary(jt) ''' output ###################### # Johansen-Procedure # ###################### Test type: trace statistic , without linear trend and constant in cointegration Eigenvalues (lambda): [1] 1.580429e-02 2.304504e-03 -3.139249e-20 Values of teststatistic and critical values of test: test 10pct 5pct 1pct r <= 1 | 1.74 7.52 9.24 12.97 r = 0 | 13.75 17.85 19.96 24.60 Eigenvectors, normalised to first column: (These are the cointegration relations) XOM.Adjusted.l2 CVX.Adjusted.l2 constant XOM.Adjusted.l2 1.0000000 1.000000 1.0000000 CVX.Adjusted.l2 -0.4730001 -1.350994 -0.7941696 constant -36.1562621 48.872924 -23.2604096 Weights W: (This is the loading matrix) XOM.Adjusted.l2 CVX.Adjusted.l2 constant XOM.Adjusted.d -0.03890275 0.003422549 -1.424199e-16 CVX.Adjusted.d -0.01169526 0.006491211 2.416743e-17 '''

**Here’s how to interpret the output of the Johansen test:**

Firstly, the test actually has more than one null hypothesis. The first null hypothesis is that there are no cointegrating relationships between the series, and is given by the

r = 0test statistic above.

The second null hypothesis is that there is one or less cointegrating relationships between the series. This hypothesis is given by the

r <= 1test statistic. If we had more price series, there would be further null hypotheses testing for less than 2, 3, 4, etc cointegrating relationships between the series.

If we can reject all these hypotheses, then we are left with the number of cointegrating relationships being equal to the number of price series.

In this case, we can’t reject *either* null hypothesis at even the 10% level! This would seem to indicate that our series don’t combine to produce a stationary spread, despite what we found earlier. **What’s going on here?**

This raises an important issue, so let’s spend some time exploring this before continuing.

First, the tests for finding statistically significant cointegrating relationships should not be applied mechanically without at least some understanding of the underlying process. It turns out that in the

ca.jo()implementation of Johansen, the critical values reported may be arbitrarily high thanks to both the minimum number of lags required (2) and statistical uncertainty associated with the test itself. This can result in unjustified acceptance of the null hypothesis. For this reason, we might prefer to use the CADF test, but that’s not an option with more than two variables.

In practice, I would be a little slow to reject a portfolio on the basis of a failed Johansen test, so long as there was a good reason for there to be a link between the component series, and we could create a promising backtest. In practice, you’ll find that dealing with a changing hedge ratio is more an issue than the statistical significance of the Johansen test. More on this in another post.

Returning to the output of the Johansen test, the reported eignevectors are our hedge ratios. In this case, for every unit of XOM, we hold an opposite position in CVX at a ratio of 0.47. To be consistent with our previously constructed spreads, taking the inverse gives us the hedge ratio in terms of units of XOM for every unit of CVX, which works out to be 2.13 in this case. Here’s how we extract that value and construct and plot the resulting spread:

# Johansen spread jo_hedge <- 1/jt@V[2, 1] jo_spread <- prices[, 'CVX.Adjusted'] + jo_hedge*coredata(prices[, 'XOM.Adjusted']) plot(jo_spread, main='CVX-XOM Spread', col='black')

For completeness, here’s an example of using the Johansen test on a portfolio of three instruments, adding Conoco-Philips (COP, another energy company) to our existing pair:

# 3-series johansen example portfolio <- load_data(c('XOM', 'CVX', 'COP'), start='2014-01-01', end='2017-01-01', source='yahoo', return_data = 'Adjusted', save_indiv = TRUE) plot(portfolio, col=c('blue', 'red', 'black'), main = 'XOM and CVX') legend('bottomright', col=c('blue', 'red', 'black'), legend=c('XOM', 'CVX', 'COP'), lty=1, bty='n') jt <- ca.jo(portfolio, type="eigen", K=2, ecdet="const", spec="longrun") summary(jt) ''' output: ###################### # Johansen-Procedure # ###################### Test type: maximal eigenvalue statistic (lambda max) , without linear trend and constant in cointegration Eigenvalues (lambda): [1] 1.696099e-02 6.538698e-03 2.249865e-03 -3.753129e-18 Values of teststatistic and critical values of test: test 10pct 5pct 1pct r <= 2 | 1.70 7.52 9.24 12.97 r <= 1 | 4.95 13.75 15.67 20.20 r = 0 | 12.90 19.77 22.00 26.81 Eigenvectors, normalised to first column: (These are the cointegration relations) XOM.Adjusted.l2 COP.Adjusted.l2 CVX.Adjusted.l2 constant XOM.Adjusted.l2 1.00000000 1.0000000 1.000000 1.0000000 COP.Adjusted.l2 0.05672729 -0.5610875 1.169714 0.3599442 CVX.Adjusted.l2 -0.49143408 -0.7078675 -1.764155 -1.2310314 constant -37.60449105 19.0946477 31.146100 9.2824197 Weights W: (This is the loading matrix) XOM.Adjusted.l2 COP.Adjusted.l2 CVX.Adjusted.l2 constant XOM.Adjusted.d -0.04560861 0.003903014 -0.001297751 -1.251443e-15 COP.Adjusted.d -0.01296028 0.003347451 -0.003035065 -5.016783e-16 CVX.Adjusted.d -0.02437064 0.009714253 -0.001897541 -5.504833e-16 ''' # Johansen spread jo_hedge <- jt@V[, 1] jo_spread <- portfolio%*%jo_hedge[1:3] jo_spread <- xts(jo_spread, order.by = index(portfolio)) plot(jo_spread, main='CVX-XOM-COP Spread', col='black')

While it may seem tempting to check thousands of instrument combinations for cointegrating relationships, be aware of these two significant issues:

- Doing this indiscriminately will incur significant amounts of data mining bias. Remember that if you run 100 statistical significance tests, 5 will pass at the 95% confidence level through chance alone.
- The Johansen test creates portfolios using all components. This can result in significant transaction costs, as for every instrument added, further brokerage and spread crossing costs are incurred. One alternative is to explore the use of sparse portfolios – see for example this paper.

In practice, I use statistical tests such as CADF and Johansen to help find potential hedge ratios for price series that I think have a good chance of generating a profit in a pairs trading strategy. Specifically, **I don’t pay a great deal of attention to the statistical significance of the results.** The markets are noisy, chaotic and dynamic, and something as precise as a test for statistical signficance proves to be a subotpimal basis for making trading decisions.

In practice, I’ve seen this play out over and over again, when spreads that failed a test of statistical signifiance generated positive returns out of sample, while spreads that passed with flying colours lost money hand over fist. Intuitively this makes a lot of sense – if trading was as easy as running tests for statistical significance, anyone with undergraduate-level math (or the ability to copy and paste code from the internet) would be raking money from the markets. Clearly, other skills and insights are needed.

In future posts, I’ll show you some backtesting tools for running experiments on pairs trading. Thanks for reading!

We just launched our new **FX Bootcamp**, where you can team up, build and trade a live retail algo FX portfolio from scratch in just 16 weeks. Not only will you end up with a diverse, robust portfolio by the end, but you will have gained the exact approach we’ve used to trade for a living for decades between us.

Interested?

**Enrolment to FX Bootcamp will close Midnight Friday July 12th. **Join your new team of retail traders and get started!

The post Practical Pairs Trading appeared first on Robot Wealth.

]]>The post Bond. Treasury Bond appeared first on Robot Wealth.

]]>The Federal Reserve publishes the yield-to-maturity of US Treasury bonds. However, the actual returns earned by investors are not publicly available. Nor are they readily and intuitively discerned from historical yields, since *“a bond’s return equals its yield only if its yield stays constant and if all coupons (cash payments) are reinvested at that same yield”* (Tuckman and Angel, 2013, p.95).

Recently, Laurens Swinkels of Erasmus University in the Netherlands estimated such a return series using publicly available data for US government bonds with 10-year maturity. The working paper accompanying the data is not yet available, but he has generously published his data in an Excel spreadsheet, including formulas for the return estimation process. You can find the data here.

One nice result of this data is that it enables the construction of a proxy for a bond price series, which can be thought of as representing a constant exposure to 10-year treasuries, and thus facilitate *ex-post *analysis of various timing models.

This idea of a constant exposure is much like an allocation to the ETF IEF or the ZN futures contract in that it is essentially a dynamic trading strategy to maintain a roughly constant time to maturity. However, IEF has only been around since 2002. ZN first traded back in 1982. Swinkels’ data set on the other hand goes back all the way to 1962, so offers the opportunity to explore bonds trading strategies over more than 65 years.

A treasury is a debt obligation issued by the US government to finance its spending. US Treasury securities are among the most liquid securities in the world and are perceived as one of the safest investments globally. In practical terms, this means that the demand for treasuries increases when something happens to scare investors. You can see this playing out for example in the recent price action of the TLT ETF, which holds long-duration treasuries:

Treasury bonds (also known as *notes*) entitle the holder to regular interest payments until *maturity*, at which time the principal, or the amount loaned in exchange for the bond, is returned. Treasury *bills, on the other hand,* have short maturities (one year or less) and only pay out at maturity.

The idea of “buying a single bond” is easy to understand. The coupons are just priced on a fixed rate and are never going to change. Say you buy a 10 year bond. It turns into a 9 year bond, then an 8 year, a 7 year, and eventually a 1 year, paying you a fixed amount at known intervals along the way. Then it matures and your principal is returned. Therefore, if you hold your bond to maturity you know exactly what yield you are going to get on the original capital you paid for it.

Maintaining an “exposure” to bonds is more nuanced. Conceptually, the value of the exposure’s cash flows changes based on current rate conditions and time to maturity of the actual securities held. And rate conditions can change differently at different points on the curve at the same time. This all gets quite complex – hence the value of Swinkels’ data set.

First, here’s the cumulative return from holding a constant exposure to bonds since 1962 (Column F from Swinkels’ data):

That’s a total return of about 4,400%.

Of course, one needs to take care interpreting this figure. This is the return you would have achieved if you could have maintained a constant time-to-maturity, re-invested all of your profits, and done so cost-free. So it’s not overly useful for estimating an investor’s actual returns, but it is useful for getting some insight into how US Treasuries, as an asset class, have performed over this time scale, particularly for comparison with other assets.

Before we get started exploring timing models, it bears stating that in this analysis, we’re not so much interested in actual returns, as there will necessarily be many assumptions baked into the analysis (like those mentioned above). But we are *really *interested in whether timing models can beat a benchmark under the same assumptions.

I’m not holding out a lot hope that we can do better than buy and hold, as bonds have done so well for so long. The reality of trying to time a bond exposure is that whenever you are “out” of bonds you are giving up exposure to the yield – that is, you forgo interest payments, which probably account for the majority of the total returns (although I haven’t verified this). In line with our Robot Wealth mantra to ‘trade humble’ I very much doubt the ability of any simple timing model to overcome this hurdle.

But you never know. In part 2 of this series, we’ll look at various factors in an attempt to outperform a buy and hold bond exposure. We’ll start by looking at the usual momentum and value factors, but if you have any ideas you’d like us to take a look at, let us know in the comments.

The post Bond. Treasury Bond appeared first on Robot Wealth.

]]>The post Shannon Entropy: A Genius Gambler’s Guide to Market Randomness appeared first on Robot Wealth.

]]>The purpose of this post is to scratch the surface of the markets from an information theoretic perspective, using tools developed by none other than the father of the digital age, Claude Shannon. Specifically, we’re going to tinker with the concept of Shannon Entropy.

Shannon (the man, not the entropy) was one of those annoying people that excels at everything he touches. Most notably, he was the first to describe the theory of electrical circuit design (in his Master’s thesis at the age of 21, no less). Later, around 1948, he discovered Information Theory, which leverages his unique-at-the-time understanding that computers could express numbers, words, pictures, even audio and video as strings of binary digits.

Not being one to let his genius go to waste, he and his buddy Ed Thorpe secretly used wearable computers to beat roulette in Vegas casinos by synchronising their computers with the spins of the wheel. They also beat the casinos by counting cards at Blackjack.

Between Information Theory and digital circuit design (and less so his gambling escapades), Shannon’s work essentially ushered in the digital world we find ourselves in today.

Measured in bits, Shannon Entropy is a measure of the information content of data, where *information content* refers more to what the data *could* contain, as opposed to what it *does* contain. In this context, information content is really about quantifying predictability, or conversely, randomness.

This concept of information is somewhat counter-intuitive. In everyday usage, we equate the word *information* with *meaning. *Information has some meaning, otherwise it’s not really information. In Shannon’s Information Theory, *information *relates to the effort or cost to describe some variable, and Shannon Entropy is the minimum number of bits that are needed to do so, on average.

This sounds a bit whacky, but will become clearer as we introduce some equations and examples. The key is to divorce the information theoretical definition of *information* from our everyday concept of *meaning*.

Say we have some random variable, like a coin toss. Your friend tosses the coin and hides the result from you. You can discern the outcome of any individual coin toss by asking just one binary (yes-no) question: *was the outcome a head *?3

The number of binary questions we need to ask to describe each outcome is one. Here and in the examples below, consider that ‘the cost of encapsulating information’ is analogous to ‘the number of questions required to describe a random variable’.

Next, consider a deck of cards with the jokers removed. Our friend shuffles the deck, draws a card and records its suit without showing us. Our friend then replaces the card, reshuffles the deck and repeats. Our friend’s record of the suits drawn from the deck is then a random variable with four equally probable outcomes. What is the minimum number of binary questions we must now ask, on average, to ascertain the suit of each draw?

Our first question could be *is the suit red?* Then, if the answer was *yes*, we might ask *is the suit a diamond? * Otherwise, we might ask *is the suit a spade? *If you think about it, regardless of the suit and the binary questions asked, we always need two questions to arrive at the correct answer.

Now consider a deck stacked with one suit and with some cards of the other suits removed. Say we have 26 hearts, 10 diamonds, 8 spades and 8 clubs. The outcome of a card draw is no longer completely random in the sense that the outcomes are no longer equally likely. We can use that knowledge to reduce the number of questions we need, on average, to arrive at the correct suit.

The probability of a card being a heart, diamond, spade or club is now 0.5, 0.19, 0.15 and 0.15 respectively. If our first question is *Is the suit a heart?* in 50% of cases, we will have arrived at the correct suit after a single question. If the answer is *No*, our next question would be *Is the suit a diamond?* Now, in almost 70% of cases, we will have our answer within two questions. In 30% of cases, we’ll need a third question.

Now, the average number of questions is simply the sum of the probabilities associated with each possible number of questions, that is

\[\frac{26}{52} * 1 + \frac{10}{52} * 2 + \frac{8+8}{52}*3 \approx 1.81\]

Recall that when all suits had an equal probability of occurrence, we always needed to ask two questions to ascertain any particular draw’s suit. This equal weight case corresponds to the system that *maximizes randomness and minimizes order*. When we add some order to the system by making one outcome more likely, we reduce the average number of questions to ascertain the suit, in this case to 1.81.2

The phenomenon we’ve just seen is analogous to Shannon Entropy, which measures the cost or effort required to describe a variable. Like the number of questions we need to arrive at the correct suit, Shannon Entropy decreases when order is imposed on a system and increases when the system is more random. **Entropy is maximized (and predictability minimized) when all outcomes are equally likely.**

Shannon Entropy, \(H\) is given by the following equation:

\[H = -\sum_{i=1}^np_i\log_2 p_i\]

Where \(n\) is the number of possible outcomes, and \(p_i\) is the probability of the \(i^{th}\) outcome occurring.

Why did Shannon choose to use the logarithm in his equation? Surely there are more intuitive ways to measure information and randomness? Certainly when I first looked at this equation, I wondered where it came from. It turns out that randomness is a tricky thing to quantify, and there are several approaches in addition to Shannon’s. The choice of measure is really informed by the properties we wish our measure to take on, rather than the properties of the phenomenon being measured. In this case, it is partially to do with how we might perceive the measure of information to change (for example, to double the number of possible states of a binary string, we simply add a bit, which is equivalent to incrementing the base 2 logarithm of the number of possible states).

More than that, using the logarithm also means that a very likely outcome does not contribute much to the randomness measure (since in this case the \(log_2 p_i\) term approaches zero), and that a very unlikely outcome also does not contribute much (since in this case the \(p_i\) term approaches zero). In addition, the additive property of logarithms simplifies combining entropies from different systems.

Zorro implements a Shannon Entropy indicator, but it’s tucked away in the *Indicators* section of the manual, one of dozens of functions listed on that page, and it’s easy to miss it.

Zorro’s is quite a clever implementation that works by converting a price curve into binary information: either the current value is higher than the previous one, or it is not. The function then detects and counts every combination of price changes in the curve of a given length. For example, we can check for patterns of two consecutive price changes, of which there are four possible binary combinations (up-up, down-down, up-down, down-up). Zorro then determines the relative frequency of these binary combinations, which are of course the empirically determined \(p\) values for use in the Shannon Entropy equation.

Once the \(p\)’s are known, Zorro simply implements the Shannon Entropy equation and returns the calculated value for \(H\), in bits. That means that the maximum \(H\), which corresponds to a perfectly random system, is equal to the pattern size. In our example of analyzing patterns of length 2, \(H = 2\) implies that all the patterns were equally likely to occur, and thus the system is purely random.

Of course, deviations from randomness are of interest to traders, because less randomness implies more predictability.

Before we dive into some examples, let’s take a look at the arguments of the

ShannonEntropy()function:

- The function’s first argument is a data series, usually a price curve. Remember, the function differences this series for us so we can simply supply raw prices.
- Next, we supply an integer time period over which to analyze the price curve for randomness.
- Finally, we supply the length of the patterns that we are interested in, from 2 to 8, remembering that there are \(2^x\) possible patterns, where \(x\) is the pattern length.

Here’s a simple implementation that plots the Shannon Entropy for the SPY ETF from 2000 to 2016 for all pattern sizes from 2 to 5 measured at the daily time scale (the script gets its data from the Alpha Vantage API; if you don’t have access, comment out lines 16-17 and select an asset from your Zorro GUI):

/* SHANNON ENTROPY */ function run() { set(PLOTNOW); StartDate = 2000; EndDate = 2016; BarPeriod = 1440; LookBack = 80; PlotHeight1 = 250; PlotHeight2 = 125; PlotWidth = 800; if(is(INITRUN)) assetHistory("SPY", FROM_AV); asset("SPY"); int period = LookBack; vars Closes = series((priceClose())); int patterns; for(patterns=2;patterns<=5;patterns++) { var H = ShannonEntropy(Closes, period, patterns); plot(strf("H_%d", patterns), H, NEW|BARS, BLUE); } }

And the resulting plot:

While the markets are clearly highly random, we can see regular departures from perfect randomness across multiple pattern sizes. That’s good news! If the markets aren’t completely random, then there is hope for us traders!

The next obvious question is how could we apply this measure of randomness in our trading? In reality, you’re unlikely to derive a trading strategy from the calculation of \(H\) in isolation (more on this below), but perhaps there is merit in applying it as an additional trade filter. For example, if you’ve found an edge in a particular market, perhaps it makes sense to apply it selectively during periods of predictability. Like all backwards-looking measures however, we need to consider that the past may not be like the future, and we need to be careful about optimizing the lookback period used in our analysis.

While the past may not be like the future, there is a principle related to entropy that may provide clues about the future state of a system. This principle states that complex systems tend to evolve so as to maximize entropy production under present constraints. This principle of maximum entropy has found application in physics, biology, statistics and other fields. Perhaps we can apply it to the markets too.

If the principle of maximum entropy does indeed apply to the markets, we would expect that given an existent series of price changes, the next value in the series should tend to maximize the entropy of the system. That means that if we know the market direction that maximizes its entropy, we have a clue as to which way the market is more likely to move. To test this, we can simulate possible future price movements and work out which scenarios tend to maximize the system’s entropy at any given time.

Below is some code to accomplish this. Firstly, we need to set the

PEEKflag (line 8), which enables the

price()functions to access future data. We calculate Shannon Entropy of our price series as before (lines 30-31). But this time, we need to create two additional arrays that correspond to the two possible entropy states at the next time period (line 19). That is, one array corresponds to an up move, and the other corresponds to a down move. We fill these arrays in a

for()loop where we copy across the existing series into our new arrays, starting from index 1 (lines 21-25). Then, we place either a higher or lower price compared to the current price at index 0 of each array (lines 26-27).

Now we’ve got arrays from which we can calculate the two possible entropy states at the next time period (remember that entropy is calculated from binary price patterns, therefore only the direction of the next move is important, not its magnitude). Then, we simply calculate the entropy of both states, and the one that returns the higher value is our prediction (lines 30-31).

I’m going to apply this idea to one of the most efficient (and therefore random) markets around: the foreign exchange markets. Rather than run a traditional backtest, in this case I just want to count the number of correct and incorrect forecasts made by our maximum entropy predictor, and sum the total number of pips that would be collected in the absence of the realities of trading (slippage, commission and the like). Lines 33-41 accomplish this.

Here’s the code:

/* SHANNON ENTROPY */ function run() { set(PLOTNOW|PEEK); StartDate = 2004; EndDate = 2016; BarPeriod = 1; LookBack = 100; int patterns = 2; int period = LookBack; vars Closes = series((priceClose())); // create possible future entropy states var up[501], down[501]; //set these large to enable experimenting with lookback - can't be set by variable int i; for(i=1;i<period+1;i++) { up[i] = Closes[i]; down[i] = Closes[i]; } up[0] = Closes[0] + 1; down[0] = Closes[0] - 1; // get entropy of next state var entropyUp = ShannonEntropy(up, period+1, patterns); var entropyDn = ShannonEntropy(down, period+1, patterns); // idealized backtest static int win, loss; static var winTot, lossTot; if(is(INITRUN)) win = loss = winTot = lossTot = 0; if(entropyUp > entropyDn and priceClose(-1) > priceClose(0)) {win++; winTot+=priceClose(-1)-priceClose(0);} else if(entropyUp < entropyDn and priceClose(-1) < priceClose(0)) {win++; winTot+=priceClose(0)-priceClose(-1);} else if(entropyUp > entropyDn and priceClose(-1) < priceClose(0)) {loss++; lossTot+=priceClose(0)-priceClose(-1);} else if(entropyUp < entropyDn and priceClose(-1) > priceClose(0)) {loss++; lossTot+=priceClose(-1)-priceClose(0);} if(is(EXITRUN)) printf("\n\n%W: %.2f%%\nWins: %d Losses: %d\nWinTot: %.0f LossTot: %.2f\nPips/Trade: %.1f", 100.*win/(win + loss), win, loss, winTot/PIP, lossTot/PIP, (winTot-lossTot)/PIP/(win+loss)); ColorUp = ColorDn = 0; plot("PipsWon", (winTot-lossTot)/PIP, MAIN|BARS, BLUE); PlotWidth = 1000; }

You can have some fun experimenting with this script. Try some different assets, different bar periods and different pattern sizes. You’ll see some interesting things in relation to randomness – some of which may go against the existing common wisdom.

As an example, when we run this script using 1-minute bars, we nearly always get a very slightly positive result, usually on the order of 51% correct predictions. The plot of pips collected for the major currency pairs from 2004-2016 are shown below (from left to right, top to bottom, AUD/USD, USD/JPY, USD/CAD, EUR/USD, NZD/USD, GBP/USD):

While there is very likely a tiny edge here, it is just that: tiny. In reality, you’ll never make money by predicting the direction of the next minute’s price change correctly 51% of the time! However, what if you could predict the direction over a longer time frame and improve the accuracy of your prediction? Could you make money from that?

Here’s an idea for extending our directional forecast based on maximum entropy to two time steps into the future. Now we have four possible scenarios (up-up, down-down, up-down, down-up) to assess. In the idealized backtester, we now go long when the two-ahead entropy is maximized by the up-up case, and short when it is maximized by the down-down case.

/* SHANNON MULTI-STEP AHEAD FORECAST */ function run() { set(PLOTNOW|PEEK); StartDate = 2010; EndDate = 2019; BarPeriod = 1; LookBack = 100; int patterns = 3; int period = LookBack; vars Closes = series((priceClose())); var upup[501], dndn[501], updn[501], dnup[501]; int i; for(i=1;i<period+1;i++) { upup[i] = Closes[i]; dndn[i] = Closes[i]; updn[i] = Closes[i]; dnup[i] = Closes[i]; } upup[1] = Closes[0] + 1; upup[0] = upup[1] + 1; dndn[1] = Closes[0] - 1; dndn[0] = dndn[1] - 1; updn[1] = Closes[0] + 1; updn[0] = updn[1] - 1; dnup[1] = Closes[0] - 1; dnup[0] = dnup[1] + 1; var entropyUpUp = ShannonEntropy(upup, period+2, patterns); var entropyDnDn = ShannonEntropy(dndn, period+2, patterns); var entropyUpDn = ShannonEntropy(updn, period+2, patterns); var entropyDnUp = ShannonEntropy(dnup, period+2, patterns); static int win, loss; static var winTot, lossTot; if(is(INITRUN)) win = loss = winTot = lossTot = 0; if(max(max(max(entropyUpUp, entropyDnDn), entropyUpDn), entropyDnUp) == entropyUpUp and priceClose(-2) > priceClose(0)) {win++; winTot+=priceClose(-2)-priceClose(0);} else if(max(max(max(entropyUpUp, entropyDnDn), entropyUpDn), entropyDnUp) == entropyDnDn and priceClose(-2) < priceClose(0)) {win++; winTot+=priceClose(0)-priceClose(-2);} else if(max(max(max(entropyUpUp, entropyDnDn), entropyUpDn), entropyDnUp) == entropyUpUp and priceClose(-2) < priceClose(0)) {loss++; lossTot+=priceClose(0)-priceClose(-2);} else if(max(max(max(entropyUpUp, entropyDnDn), entropyUpDn), entropyDnUp) == entropyDnDn and priceClose(-2) > priceClose(0)) {loss++; lossTot+=priceClose(-2)-priceClose(0);} if(is(EXITRUN)) printf("\n\n%W: %.2f%%\nWins: %d Losses: %d\nWinTot: %.0f LossTot: %.0f\nPips/Trade: %.1f", 100.*win/(win + loss), win, loss, winTot/PIP, lossTot/PIP, (winTot-lossTot)/PIP/(win+loss)); ColorUp = ColorDn = 0; plot("pips", (winTot-lossTot)/PIP, MAIN|BARS, BLUE); PlotWidth = 1000; }

And here’s a plot of the number of pips collected over the past few years, exlcuding trading frictions, on EUR/USD:

In this post, we looked at the markets from a slightly different perspective, through the lens of Information Theory. In particular, we saw how Shannon Entropy is a measure of the degree of order or predictability within a system, with increasing entropy corresponding to more randomness and maximum entropy occurring when all outcomes are equally likely. We saw that financial markets are highly random (in general displaying a Shannon Entropy close to that of a perfectly random system), but that they do depart from randomness regularly. They may even do so differently depending on the time horizon and granularity over which they are analyzed.

We also saw an attempt to use Shannon Entropy in a standalone trading system via the principle of maximum entropy production. While such a system does appear to have a small edge, in reality, it will be difficult to make a consistent trading system from these predictions alone as transaction costs will usually swamp the edge.

While this is all very interesting from an academic perspective, if you can think of a practical application to trading, we’d love to hear about it in the comments.

—————-

We recently released our new ebook **Embrace the Mayhem****. **

Embrace the Mayhem is an honest yet exciting guide to what really works as a retail trader — based on the skills and approach we currently use to trade full-time.

It is the result of all the hard-won lessons myself and James have learned from over 20 years in professional and retail markets.

All the costly mistakes and frustrating misguidance we’ve dealt with.

All the wins and losses we’ve celebrated or grieved over.

In an area brimming with deception and confusion, Embrace the Mayhem is the retail trader’s survival guide and playbook all in one.

**Find out more about Embrace the Mayhem and get it here.*******

***If you are already a Robot Wealth member, please get the ebook via this page (the same price, but you won’t unnecessarily open two accounts).**

The post Shannon Entropy: A Genius Gambler’s Guide to Market Randomness appeared first on Robot Wealth.

]]>