Why We Use Apache Beam For Our Systematic Trading Data Pipeline

In the world of Big Data, there are lots of tools and technologies to choose from. Choosing the “right” one depends on the things that you are building and the problems you are trying to solve. Trading firms have skilled teams that monitor and deploy data pipelines for their organisation and the technical overhead that …

Read more

Working with Tidy Financial Data in tidyr

Holding data in a tidy format works wonders for one’s productivity. Here we will explore the tidyr package, which is all about creating tidy data. In particular, let’s develop an understanding of the tidyr::pivot_longer and tidyr::pivot_wider functions for switching between different formats of tidy data. In this video, you’ll learn: What tidy data looks like …

Read more

How To Get Historical S&P 500 Constituents Data For Free

spx constituents historical mean return

In this post, we are going to construct snapshots of historic S&P 500 index constituents, from freely available data on the internet. Why? Well, one of the biggest challenges in looking for opportunities amongst a broad universe of stocks is choosing what stock “universe” to look at. One approach to dealing with this is to …

Read more

Fun with the Cryptocompare API

Cryptocompare is a platform providing data and insights on pretty much everything in the crypto-sphere, from market data for cryptocurrencies to comparisons of the various crytpo-exchanges, to recommendations for where to spend your crypto assets. The user-experience is quite pleasant, as you can see from the screenshot of their real-time coin comparison table: As nice …

Read more

We Love Free Data: Replacing Yahoo Finance Market Data

In keeping with our recent theme of providing useful tidbits of algo trading practicalities, here’s an elegant solution that resolves Yahoo’s unceremonious exit from the free financial data space. Regular readers would know that I use various tools in my algo trading stack, but the one I keep coming back to, particularly when I’m ready …

Read more

Solved: Errors Downloading Stock Price Data from Yahoo Finance

Recently, Yahoo Finance – a popular source of free end-of-day price data – made some changes to their server which wreaked a little havoc on anyone relying on it for their algos or simulations. Specifically, Yahoo Finance switched from HTTP to HTTPS and changed the data download URLs. No doubt this is a huge source of frustration, …

Read more