Crypto HFT Data Aggregator

github.com, @SpiralDevelopment

Great entry into collecting your own HFT crypto data. Solid python code, runs right out of the box. Includes connections for Binance, Bitmex, Okex, Bitfinex, Coinbase, Bitstamp, and Kraken.

A Simple Introduction to Copulas

YouTube, @DirtyQuant

An amazingly to-the-point explanation of copulas, and why they are useful. Includes sample python code. Video production skills are top notch, highly recommended.

Search New Tickers on Twitter, in <3 Seconds

Substack, @samchaaa

You know how to bookmark a search engine in Chrome as a hotkey, right? Go to chrome://settings/searchEngines > “Site search”, and click “Add”.

Add “t” (or your preference) as a shortcut, and paste in the twitter search URL with “%s” instead of your search term (https://twitter.com/search?q=%24%s&src=cashtag_click).

Now, all you have to do is hit: Ctrl + T, “T” (or your shortcut), Tab, “COIN”, Enter

I timed myself, and it takes consistently <3 seconds to enter and get search results.

Thanks for reading. Good luck and good trading.

Share

Hey readers!

Here’s a tool I use myself (for intraday trading) that will soon be written up on Medium. I want you to know about it before it gets published… let me know if you have any feedback!

Here’s the Google Sheet.

Here’s the repo (if you’d like to run it yourself).

All it does is download free tick data from https://datafeed.dukascopy.com/datafeed, run some calculations, and write to Google Sheets. It’s simple, but highly effective.

At the time of this writing, bitcoin is $16,000… up +40% in the last month. While I am late to the party, I want to implement a simple trend following system (so I don’t miss trends in the future). However, I’ve never developed trend following system before, so am starting absolutely from square one.

Read more

This article shows how you can make money more consistently, by standardizing your risk (distance and size) across any asset classes… using only Google Sheets.

Our goal is to standardize our risk, and make it instantly accessible and understandable to you, at a single glance.

Here’s the final product (scroll to the end for the HTML version):

Background: A while ago I was trying to standardize my risk more exactly across different instruments. I realized a 20 pip stop on GBPUSD is not the same monetary value as a 20 pip stop on USDJPY. Not only that, but the pairs have a different volatility… one could stop out out in hours, and the other in seconds. How do you go across asset classes, say futures (with variable contract sizes) or bitcoin? What about something denominated in a different currency, like DAX futures (denominated in Euro, calculating risk in US dollars)?

Given a reasonable percentage stop of 1–2%, how do we get a reasonable stop distance, for any instrument, taking into account volatility, contract size, and quote currency?

This article (part one) will be a guide for setting up the framework of the sheet, using a simple AVERAGE as a placeholder metric.

Exploring other more robust measures of market movement, and better risk parameters will be part two.

Finally, we’ll talk about percentage risk in part three (you can get distance and size right, but percentage wrong… and completely blow up).

Actual instructions bolded, everything else is commentary.

Step 0.) Set up Sheet

Create two tabs, name one “main” and the other “data”.

In “main”, set up your sheet like this:

Enter your own values in B1, B2, and =B1*B2 in B3.

Step 1.) Get data

To prototype this I used Google Sheets, with the GOOGLEFINANCE function. GOOGLEFINANCE offers daily data on stocks, forex, and a few cryptocurrencies. GOOGLEFINANCE does not have futures, but they do have a few indices. For assets not covered by GOOGLEFINANCE, we have a workaround to get the data from external sources.

Entering the following will give you the last 300 closes for EURUSD (just an example):

=GOOGLEFINANCE(“EURUSD”, “close”, TODAY()-300, TODAY())

GOOGLEFINANCE doesn’t have OHLC… so to get daily change, we take movement from close to close. Using the QUERY function, we shift the data by OFFSET 1 to remove header from our selection, and shift by OFFSET 2 and remove the header and shift by one day. Then we get the difference subtracting one QUERY from the other, inside ARRAYFORMULA:

=IFERROR(  ARRAYFORMULA(    QUERY(GOOGLEFINANCE("EURUSD","price",TODAY()-300,TODAY()), "SELECT Col2 OFFSET 2", 0)-    QUERY(GOOGLEFINANCE("EURUSD","price",TODAY()-300,TODAY()), "SELECT Col2 OFFSET 1", 0)  ))

Note GOOGLEFINANCE will give you the close prices over the weekend… leading to 0’s every 7 days.

Let’s quickly look at the MEAN and MEDIAN:

The median comes out 0? Must be those duplicate weekend closes. Also, the average looks abnormally small… 1.7 pips average daily movement for EURUSD. This is because the daily movements are coming out both negative and positive. I knew we said “quick-and-dirty”, but this will artificially understate our final number.

So, we filter out 0 and look at movements as absolute. We do this with IF(close2-close1 <> 0, ABS(close2-close1), “”), and wrapping it in an ARRAYFORMULA.

=IFERROR(   ARRAYFORMULA(     IF(        QUERY(GOOGLEFINANCE("EURUSD", "price", TODAY()-300, TODAY()), "SELECT Col2 OFFSET 2", 0)-        QUERY(GOOGLEFINANCE("EURUSD", "price", TODAY()-300, TODAY()), "SELECT Col2 OFFSET 1", 0) <> 0,        ABS(         QUERY(GOOGLEFINANCE("EURUSD", "price", TODAY()-300, TODAY()), "SELECT Col2 OFFSET 2", 0)-          QUERY(GOOGLEFINANCE("EURUSD", "price", TODAY()-300, TODAY()), "SELECT Col2 OFFSET 1", 0)       ),       ""     )   ) )

Looking at the results, this gives us a way less understated amount:

Original vs. new average: 1.7 pips / 37 pips

Original vs. new median: 0 pips / 28 pips

I would say the new method is much more robust; this roughly checks out to something I’d want to use.

Step 2.) Reshape the sheet

Originally, I called all of the data stuff inside the final metric formula. This is cumbersome and inefficient. Eventually, we’ll want to add many instruments and test out many metrics.

Let’s break our data into another tab, then put our final numbers in the main tab.

Go to “main”, and add the instruments you want in A6:A.

In cell B6, enter the following and drag all the way down:

=IFNA(GOOGLEFINANCE(A6))

This allows you to check if GOOGLEFINANCE has the data you want.

Note: There are some instruments you can’t immediately find on GOOGLEFINANCE, such as ES and NQ futures… however, if you search S&P 500 or NASDAQ you can find the symbol for the indices… “.IXIC” and “.INX”, which is available on GOOGLEFINANCE.

For anything you can’t find, you may be able to find externally (see Step 5 below).

Switch over to “data”. Put the following in cell A1:

=TRANSPOSE(main!A6:A)

Your list from main!A6:A should populate horizontally.

Enter the following in cell A2 and drag all the way to the right:

=IFERROR(  ARRAYFORMULA(    IF(       QUERY(GOOGLEFINANCE(A1, "price", TODAY()-300, TODAY()), "SELECT Col2 OFFSET 2", 0)-       QUERY(GOOGLEFINANCE(A1, "price", TODAY()-300, TODAY()), "SELECT Col2 OFFSET 1", 0) <> 0,       ABS(        QUERY(GOOGLEFINANCE(A1, "price", TODAY()-300, TODAY()), "SELECT Col2 OFFSET 2", 0)-         QUERY(GOOGLEFINANCE(A1, "price", TODAY()-300, TODAY()), "SELECT Col2 OFFSET 1", 0)      ),       ""    )  ))

*Looking at the data, you can see some instruments are leaving out the weekend gap, while others don’t (bitcoin). It looks like the ETF (SPY) doesn’t have the gaps, and GBPJPY doesn’t (although the other fx pairs do). I’ll leave this to you to figure out. This is the quick-and-dirty guide, after all.

Average the arrays in the main tab

Back in our main tab, “main”, we take each instrument name, and look up the array from tab “data”.

Enter the following in C6, and drag all the way down:

=IFNA(AVERAGE(ARRAYFORMULA(IF(data!$1:$1=A6, data!A$6:Z, ))), “”)

Step 3.) Standardizing for contract size, denominations, and pip/tick/point size

So now we have our framework in place. Ideally, we would be able to just divide our stop value (in $) by our average daily distance (also in $) to get trade size.

Unfortunately, that will only work for stocks and crypto. If you only trade stocks and crypto, you may skip this part.

For forex and futures, we will be dealing with different contract sizes and quote currencies. How do we standardize?

Break this down by adding three more columns to the right: “Denomination”, “Contract Size”, and “Daily Movement $”.

Denomination (or quote currency)

Paste the following in E6, and copy down:

=IF(  A6<>"",   IF(    LEN(A6) <> 6,    1,    IF(      RIGHT(A6, 3) = "USD",       1,       GOOGLEFINANCE(CONCATENATE(RIGHT(A6, 3), "USD"))    )  ),   "")

Here is what this means in plain English:

• If the length of the instrument string is not six, it is probably not forex. So let’s just use one dollar (1).

• If the length of the instrument is six, it’s probably forex or something else. Look at the last three letters of the string to get the quote currency.

• If the last three letters are “USD”, then it’s denominated in dollars anyway… so we just use 1.

• If it is denominated in anything else, we need to find the conversion from that currency to our account currency (the GOOGLEFINANCE(CONCAT()) term).

Could any of these rules potentially break? Sure — for example, it would not automatically cover ASX 200 futures denominated in Aussie dollars, and would incorrectly denominate those in US dollars. You may add conditions in the formula.

Also, for our purposes we are dealing with a USD-denominated account.

Contract size

Look up and enter contract size manually in F6 down.

Contract size you need to look up and enter manually (this could potentially be broken out into a formula, or scraping futures websites for contract specs… let me know if you do).

Here are some rules of thumb:

• Forex lots are in 100,000’s of the quote currency (the last three letters in the pair name)

• Futures vary… look them up.

• For stocks and crypto, one share is one share… so use 1.

Get the dollar value of average daily movement, completely standardized

Finally, to get the dollar value of the daily movement of your instrument, just multiply across.

Enter the following formula in G6, and copy down:

=IF(A6<>””, C6*E6*F6, )

And finally, to get the size, you divide your stop value in $ / the daily movement to get lot size.

Put the following in D6 and copy down:

=IF(A6<>””, $B$3/G6, “”)

Converting pips to round numbers

But wait, there’s more!

The purpose of this sheet is to look in one place, and instantly know the stop distance and trade size for your trade.

For anything not forex, we’re good to go, but for forex we could clean it up a bit to get round pip numbers.

Put the following in H6 and copy down:

=IF(  A6<>"",   IF(    G6=100000,    E6*IF(         RIGHT(A6, 3) = "JPY",          100,          10000       ),     E6  ),  "")

What this means in plain English:

• If your contract size is 100,000, you have a forex pair.

• If it is a forex pair:

• If the pair ends in “JPY”, multiply the price movement in decimals by 100 to get pips.

• If the pair ends in anything else, multiply times 10,000 to get pips.

• Otherwise, the instrument movement is in round dollar numbers, just return the distance column.

*Note this may not apply to exotics, check it out for yourself and let me know if you add any conditions successfully.

After this step, rename the column “Pips/Ticks/Points” to “Distance”, and name our new column “Pips/Ticks/Points”:

Step 4.) Styling

Now to make the sheet pretty. We want this sheet to instantly tell us stop distance and lot size. Any styling we can do to make it more readable, easier on the eyes, or more accessible helps that goal.

Drag the columns in the following order: “Instrument”, “Last”, “Pips/Ticks/Points”, and “Size”:

Select all, make font “Helvetica Neue”.

Center everything vertically and horizontally.

Set everything to font color “dark gray 3”.

Now we want to emphasize the three most important columns, “Instrument”, “Pips/Ticks/Points”, and “Size”.

Select A6:A, C6:C, and D6:D, set font color to “dark grey 2”, and set font size to 18.

Finally, select column “Size” (D:D).

Go to “Format” > “Conditional Formatting” and select “Color Scale”.

Under “Preview”, select “green to yellow to red”.

Switch the green with the red.

Select “light green 3”, “light yellow 2”, and “light red 2”.

Sorting

I like to sort just once*, after we get our preliminary results. It’s kind of nice to see things in order of volatility.

*Note: sorting will break down after you add external data. If you come up with an elegant solution to this, let me know.

In this example, my stuff is already sorted. So I added some instruments to show it out of order.

Select all rows A5 and down, hit “Filter”. Sort by “Size”, “A to Z”.

Note: This will rearrange your data in the data tab. It will take a while to reload, so use sparingly.

Now we’ve got our instruments, sorted by daily volatility, factoring in contract size and pip value. I’m starting to like this sheet.

Step 5.) How to add instruments with no data on GOOGLEFINANCE

Despite how freakin’ cool this sheet is thus far, there are a few instruments not listed on GOOGLEFINANCE.

For example, I’d like to see gold futures, oil futures, and DAX. For now we’ll just look at gold and oil.

Add the symbols “XAUUSD” and “/CL” in the tab “main”. These names are just placeholders, it doesn’t really matter what you put:

To work around this, you will need to go outside GOOGLEFINANCE and find the data somewhere else on the internet. Here, I used “myfxbook.com” for XAUUSD:

In “data”, replace GOOGLEFINANCE with IMPORTHTML (table number will differ between sources):

=IFERROR(  ARRAYFORMULA(    ABS(      QUERY(IMPORTHTML("https://www.myfxbook.com/en/forex-market/currencies/XAUUSD-historical-data", "table", 4), "SELECT Col4 OFFSET 2", 0)-      QUERY(IMPORTHTML("https://www.myfxbook.com/en/forex-market/currencies/XAUUSD-historical-data", "table", 4), "SELECT Col4 OFFSET 1", 0)    )  ))

*Note there is only 100 days of data from this source, making it less reliable (but more representative of recent volatility) than our 300 day averages.

Note that now sorting values in the main tab will break, as this formula is tied to Col I, returning the analytic results for XAUUSD under a different instrument’s symbol.

Potentially this could be fixed with a more elaborate formula, or making a separate tab for external data.

Do the same for /CL in tab “data”, with this URL from “Investing.com”:

=IFERROR(  ARRAYFORMULA(    ABS(      QUERY(IMPORTHTML("https://www.investing.com/commodities/crude-oil-historical-data", "table", 2), "SELECT Col4 OFFSET 2", 0)-      QUERY(IMPORTHTML("https://www.investing.com/commodities/crude-oil-historical-data", "table", 2), "SELECT Col4 OFFSET 1", 0)     )   ) )

Head back over to the tab “main”, and enter your contract sizes:

*Investing.com only shows past month of data.

**You could add length of data in “main”… just so you know the reliability and recency. For example, most data from GOOGLEFINANCE will come in around 299 samples. However, the external stuff could be limited to 100 or 30 samples. Actually, you could get external sources with MORE data than GOOGLEFINANCE, and that would be even better.

If all you wanted was the functional sheet, now you are good to go.

However, I do not actually use AVERAGE in my own calculations. There are a lot of ways to skin a cat, especially when the cat is making up risk models from scratch in Google Sheets. To be discussed in part 2.

Again, a disclaimer: we purposely haven’t discussed what to use as a percentage risk yet. This is just as important as distance and size in your risk management. To be discussed in part 3.

See the sheet results in HTML format here.

Anything I can write about to help you find success in trading? Tell me about it here: https://bit.ly/3mStNJG

Finally, if you would like the completed sheet to copy for yourself… get it $5 here. On top of that, I will send you any tools, quantitative studies, and algos I come up with in the future (not available to public).

Setting up AWS with Python with Alpaca Trade API

It is always challenging for a new quant trader to get an algorithm up and running live in the cloud. In my last post, I wrote an instruction for using PythonAnywhere.

This walkthrough focuses on the basics of setting up AWS with Python, the Alpaca Trade API, and running a basic algorithm.

Amazon Web Services (AWS) - Cloud Computing Services
Amazon Web Services offers reliable, scalable, and inexpensive cloud computing services. Free to join, pay only for…aws.amazon.com

Alpaca Documents
All about Alpaca.docs.alpaca.markets

AWS may seem daunting at first, but is well worth the time to get comfortable with. The benefits of AWS include billing by usage and a free tier of cloud services, meaning you can get up and running with zero upfront cost!

Let’s jump in with a fresh account.

Step 1.) Sign up for AWS.

https://aws.amazon.com/

AWS will ask you for a credit card and phone number to verify your account, but you do not need to spend a dime to get started.

Go to the console sign-in.

Step 2.) Launch EC2 instance.

When you sign in, “Launch a virtual machine” with EC2 on your front page. Click that.

(Or, search “EC2” and click the “Launch Instance” button.)

Click the “Amazon Linux 2 AMI” free tier eligible option.

Use the t2.micro free tier eligible and click “Review and Launch”.

You’ll get notified to set a security group if this is your first time create a new key pair and download the .pem file. You’ll need the path to this file later.

Click “Launch Instances”. Click the instance id link on the following page to go to your EC2 instances.

Step 3.) SSH to instance from local computer.

Your instance will take a bit to initialize, while it’s doing that, copy your public DNS.

Open a terminal window (I’m doing this from Mac OSX) and type the following:

ssh -i [path to your key here] ec2-user@[your public DNS here]

My example:

ssh -i demo_ec2_key.pem ec2-user@ec2–18–220–99–177.us-east-2.compute.amazonaws.com

(If you ever use a different image, your log-in may be ubuntu@[yourDNShere])

*Some troubleshooting may be required. In this case, my connection was timing out, so I searched “EC2 timeout” and found the solution. In my security group, I had to specify port 22 as inbound traffic. This was because I originally skipped the security group step above, by default your security group should include port 22 for inbound traffic.

How to fix your key permissions

The first time you try to connect, you’ll probably get “bad permissions” as a result:

All you have to do fix this is run:

sudo chmod 600 [path to your key]

Now run the command again. This screen means you did everything correctly:

Step 4.) Set up Alpaca Trade API.

Step 4A.) Install Python 3

Enter python -V to see what version of python you’re running. By default, I get 2.7. Run the following to install Python3.7:

sudo yum install python37

Step 4B.) Install Alpaca Trade API

pip3 install --user alpaca-trade-api

Now your EC2 instance should be good to go! Keep this window up for later.

Step 5.) Copy your algo over to EC2.

For this tutorial, we’ll be running the following sample algorithm. The strategy is a simple EMA crossover, checking a list of stocks every 1 minute.

Open a new terminal window, and run the following:

scp -i [path to your key] [path to your algo] ec2-user@[your public DNS]:/~

My example:

scp -i demo_ec2_key.pem apca_5min_ema.py ec2-user@ec2–18–220–99–177.us-east-2.compute.amazonaws.com:~/

Go over to your original EC2 window, and run ls to check if the upload worked.

You should see your algorithm listed in the directory. Test it by running python3 apca_5min_ema.py.

Step 7.) Closing your terminal without quitting your algo.

Start a new instance of screen

In order to keep our algo running without quitting when we disconnect, we can use a handy Linux command, screen. Go ahead and run it.

This will pop up a new, blank terminal. It’s actually another window of your terminal. Now run python3 apca_algo.py.

Hit CTRL + A + D to detach the screen and return to your normal terminal.

Now you can type screen -ls to see your process is still running.

Typing tail apca_log.log confirms this as well, seeing the results in our log file.

Hit CTRL + D to logout of EC2 altogether.

Log into your Alpaca account, and confirm your orders are being placed.

Reconnecting to your screen

To get back to your algo, log into EC2 with SSH.

Run screen -ls to see what screens are running. You should see something like:

There is a screen on:     3634.pts-0.ip-172–31–34–247 (Detached)1 Socket in /var/run/screen/S-ec2-user.

That long screen id is what you’ll type in next to reconnect:

screen -r 3634.pts-0.ip-172–31–34–247

Alternatively, if you just want to quit the screen you can use ps aux | grep apca_5min_algo.py to see the process ID:

ec2-user 3658 0.1 6.4 427400 65388 pts/1 S+ 22:14 0:00 python3 apca_5min_ema.pyec2-user 3726 0.0 0.1 119468 1040 pts/0 S+ 22:20 0:00 grep — color=auto apca_5min_ema.py

Then, you can use screen -XS [process id] quit to quit the screen immediately. My example:

screen -SX 3658 quit

Finally, confirm if you’ve quit your algo with screen -ls .

There you have it! This should give you a good starting point for spinning up new EC2 instances, navigating AWS, and managing your algorithms.

References:

Step 1: Launch an Amazon EC2 Instance - AWS Quick Start Guide
Quick start guide to launching an EC2 instance.docs.aws.amazon.com

Linux screen Command: Keep Processes Running Despite a Dropped Connection
I guess you all know this: you are connected to your server with SSH and in the middle of compiling some software (e.g…www.howtoforge.com

Technology and services are offered by AlpacaDB, Inc. Brokerage services are provided by Alpaca Securities LLC (alpaca.markets), member FINRA/SIPC. Alpaca Securities LLC is a wholly-owned subsidiary of AlpacaDB, Inc.

You can find us @AlpacaHQ, if you use twitter.

Follow Automation Generation, a Medium’s publication created for developers/makers in trading and fintech.

Using PythonAnywhere When Running an Algorithm Live in the Cloud

After first getting into quant trading and writing algos on your local machine, the biggest area of resistance for newbie quant traders is running an algorithm live in the cloud.

There are many cloud services on the market, including AWS, Google Cloud, Heroku, Digital Ocean, and PythonAnywhere. This walkthrough focuses on setting up PythonAnywhere and running a sample algorithm in Python and the Alpaca Trade API.

Host, run, and code Python in the cloud: PythonAnywhere
Host, run, and code Python in the cloud: PythonAnywherewww.pythonanywhere.com

Alpaca Documents
All about Alpaca.docs.alpaca.markets

PythonAnywhere has a free tier of accounts, but only with access to whitelisted URLs. To access the Alpaca endpoints, you will need a paid account ($5/month).

PythonAnywhere is a nice option because it gives you some manner of GUI for accessing consoles, files, notebooks, and even scheduling tasks that automatically restart if they go down (excellent for algo trading).

Let’s jump in with a fresh account.

Step 1.) Setting up PythonAnywhere.

Step A.) Make a new PythonAnywhere account.

https://www.pythonanywhere.com/

You’re welcomed with a dashboard like below.

Step B.) Install alpaca-trade-api.

Under “Consoles”, open a new Bash console.

Run the following:

pip3 install --user alpaca-trade-api

Step 2.) Set up algo.

Our sample algo for this walkthrough is a simple 5/20 period ema (exponential moving average) crossover on the 5-minute timeframe, checking every 1 minute for signals.

Upload the algo as a file, or go to “Open another file” on the dashboard.

Enter it as /home/yourusername/algoname.py

You can run this right from the terminal editor, and see if it works. If you’re missing any dependencies, install them in the Bash console.

If the algo runs, you can leave it up and check results in the log file.

Step 3.) Set as task.

Go to “Tasks”, and go down to “Always On Tasks”.

You can schedule your algo to automatically restart if it crashes by entering “python3.6 /home/yourusername/algoname.py”.

This works especially well with our sample algo, which dynamically finds the amount of time to sleep until next market open.

Checking our Alpaca account, using the below as an example, we can see it’s generating signals about every minute:

Please note that this example is for illustrative purposes only. Past performance is not indicative of future results.

Technology and services are offered by AlpacaDB, Inc. Brokerage services are provided by Alpaca Securities LLC (alpaca.markets), member FINRA/SIPC. Alpaca Securities LLC is a wholly-owned subsidiary of AlpacaDB, Inc.

You can find us @AlpacaHQ, if you use twitter.

Follow Automation Generation, a Medium’s publication created for developers/makers in trading and fintech.

Defining Technical Chart Patterns Programmatically

Ever wondered how to programmatically define technical patterns in price data?

At the fundamental level, technical patterns come from local minimum and maximum points in price. From there, the technical patterns may be defined by relative comparisons in these min/max points.

Let’s see if we could have played this by algorithmically identifying any inverse head & shoulders patterns!

Follow along with the notebook here.

samchaaa/alpaca_tech_screener
Contribute to samchaaa/alpaca_tech_screener development by creating an account on GitHub.github.com

The following code can easily be retooled to work as a screener, backtester, or trading algo, with any timeframe or patterns you define.

Disclaimer: this code is intended as a starting point for finding technical patterns, it is for educational purposes only. The framework for this code came from here.

An Empirical Algorithmic Evaluation of Technical Analysis
At a recent meeting of the Quantopian staff journal club, I presented a paper by Andrew Lo, Harry Mamaysky, and Jiang…www.quantopian.com

Step 1.) Read in data

I’m reading in data using the Alpaca API (which I’ll also use to place trades later).

I wrote this function to grab data beyond the one request limit of 2,000 minute bars. Later we’ll resample to our timeframe of choice.

We’ll resample data separately, in case we want to try out different timeframes later.

Step 2.) Find minima and maxima

For this step we’ll use a function from scipy’s signal processing library to find peaks in the data.

This code looks complicated, but the point is to return the integer index values with price, for each min/max point.

Let’s plot it with the resampled price data to visually confirm we’re on the right track.

Step 3.) Find patterns

To find patterns, we simply iterate over all our min max points, and find windows where the points meet some pattern criteria.

For example, an inverse head and shoulders can roughly be defined as:

C < A, B, D, E

A, E < B, D

To filter for head and shoulders with even necklines:

abs(B-D) < np.mean([B, D])*0.05

(The difference between the necklines must not be more than 5%.)

Here’s the code:

And a plot for visual confirmation:

As you can see, we are getting more patterns than we need. Our params (smoothing and window range) are too sensitive for this timeframe (60 minutes).

Step 4.) Reorganize and iterate to find best params

In order to find the best params, I reorganized my code into functions and iterated through multiple stocks, smoothing, and window parameters.

Run the above like so:

Now we can see how our timeframes, patterns, and params are playing out!

Step 5.) Go live!

To use this live, I made the following changes to screener():

def screener(stock_data, ema_list, window_list):        triggers = []        all_results = pd.DataFrame()        for stock in stock_data:        prices = stock_data[stock]                for ema_ in ema_list:            for window_ in window_list:                 max_min = get_max_min(prices, smoothing=ema_, window_range=window_)                pat = find_patterns(max_min)                                if len(pat) > 0:                    triggers.append(stock)                    return triggers

And ran like so:

stocklist = ['AA', 'AAL', 'AAPL', 'AMZN'] # Long list of stocks here

stock_data = get_stock_data(stocklist, 2)

resampled_stock_data = resample(stock_data, '360T')

ema_list = [5]window_list = [5]

results = screener(resampled_stock_data, ema_list, window_list)

for x in results:    api.submit_order(x, 100, 'buy', 'market', 'day')

Finding the right params for your pattern to play out may take experimentation. See the results() function in the notebook to confirm whether your patterns have a positive edge or not.

Technology and services are offered by AlpacaDB, Inc. Brokerage services are provided by Alpaca Securities LLC (alpaca.markets), member FINRA/SIPC. Alpaca Securities LLC is a wholly-owned subsidiary of AlpacaDB, Inc.

You can find us @AlpacaHQ, if you use twitter.

In this article, I will show how you can make “roll-your-own” visualizations, using the Alpaca API with Python.

See the final code on GitHub here.

samchaaa/alpaca_correlation_viz
Contribute to samchaaa/alpaca_correlation_viz development by creating an account on GitHub.github.com

Visualizing your portfolio correlation by heatmap in Python (jupyter notebook)

Step 1: Setup

For this tutorial, I used Python 3 in jupyter notebook, some basic libraries, and the Alpaca trade API. Of course, you’ll need an Alpaca account for the API key as well!

1. Get Python 3 + jupyter notebook. If you’re not setup with this already, just download Anaconda (it will install everything you need in one go).

2. Sign up for an Alpaca account. Make note of the API key you get when you sign up.

3. Install the Alpaca trade API for Python.

Once you have a fresh notebook started, here are the imports you’ll need:

import pandas as pd

import matplotlib.pyplot as plt

import seaborn as sns

import alpaca_trade_api as tradeapi

# Makes output higher resolution.

%config InlineBackend.figure_format = ‘retina’

To sign into your Alpaca account, initialize REST with the API key you got at signup:

# Paper trade log-in

api = tradeapi.REST(‘YOUR_API_KEY_HERE’,

                    ‘YOUR_SECRET_CODE_HERE’,

                    ‘https://paper-api.alpaca.markets')

*Note the endpoint (‘https://paper-api.alpaca.markets') will be different for live trading.

Step 2: Get account positions.

If you’ve already started paper trading and have positions on, you can call these with:

api.list_positions()

If you’re working with a fresh account, submit some fresh orders like so:

api.submit_order(‘SNAP’, 100, ‘buy’, ‘market’, ‘day’)

api.submit_order(‘SLV’, 100, ‘buy’, ‘market’, ‘day’)

api.submit_order(‘JNJ’, 100, ‘buy’, ‘market’, ‘day’)

api.submit_order(‘AAPL’, 10, ‘buy’, ‘market’, ‘day’)

api.submit_order(‘GLD’, 100, ‘buy’, ‘market’, ‘day’)

api.submit_order(‘GOOG’, 1, ‘buy’, ‘market’, ‘day’)

api.submit_order(‘CAT’, 100, ‘buy’, ‘market’, ‘day’)

api.submit_order(‘EWZ’, 10, ‘buy’, ‘market’, ‘day’)

To request our data, we just need a list of the symbol of each security:

pos_list = [x.symbol for x in api.list_positions()]

Step 3: Get data.

Alpaca has several methods of requesting data. Learn more here.

The short version is, if you sign up with a paper trading account, you may only access data from the IEX exchange.

If you have a live account, you may access IEX and full-volume data from Polygon. Polygon includes quotes, bars, fundamentals, and volume data (both historical and real-time) from all exchanges. This is extremely valuable!

This distinction is only important if you’re doing more demanding trading strategies. For our purposes, we only need the daily close of each security.

We’ll do this by requesting a “barset” object for each symbol:

def get_bars(symbol):

     data = api.get_barset(symbol, ‘day’, limit=1000)

     data = data.df[symbol][‘close’]

     return data

We’ll call this later, within our next step.

Step 4: Get correlations.

Here’s our code for getting all the correlations within our portfolio:

def correlation(equity_list):          df = pd.DataFrame()    equity_columns = []        # Get symbol history    for symbol in equity_list:           try:            symbol_df = get_bars(symbol)            df = pd.concat([df, symbol_df], axis=1)            equity_columns.append(symbol)        except:            print('Exception with {}'.format(symbol))                df.columns = equity_columns        # Get correlation and sort by sum    sum_corr = df.corr().sum().sort_values(ascending=True).index.values        return df[sum_corr].corr()

For each equity in our portfolio, we call get_bars(), then concatenate the results to our dataframe.

The results of this step alone would return a df like this:

(equity_columns collects the symbols we’ve successfully requested, so we don’t get a mismatch with the df later.)

Calling .corr() returns a dataframe with the correlation coefficient (Pearson’s correlation, by default) of each symbol. It looks like this:

Note if you’re not familiar with correlations: Correlation coefficients range from (-1, 1). A positive value means two things move together, and a negative value that they move opposite. A value close to zero is “weak”, meaning there is no correlation, and a value close to -1 or +1 is “strong”.

For better visual organization, sort symbols by total correlation* by summing the correlation df and sorting:

sum_corr = df.corr().sum().sort_values(ascending=True).index.values

This returns a list of symbols, from least to most correlated.

Calling df.corr() again with this list sorts it by summed correlation:

return df[sum_corr].corr()

Step 5: Plot heatmap.

Finally, the easiest step:

plt.figure(figsize=(13, 8))

sns.heatmap(correlation(pos_list), annot=True, cmap=”Greens”);

plt.figure() sets our figure size nice and big.

sns.heatmap() plots colors along a linear colormap. Calling annot adds numerical values, cmap defines your color scheme.

(I’m a fan of “RdYlGn_r”, “summer_r”, “Blues”, and “Greens”.)

Further Steps and Ideas

Thanks for reading! See the final code here.

samchaaa/alpaca_correlation_viz
Contribute to samchaaa/alpaca_correlation_viz development by creating an account on GitHub.github.com

• For each position, return the top three least correlated S&P100 stocks.

• Hedge 5% of each position with least correlated stocks.

• Find rolling correlation over time (divergence strategies!)

• Scale heatmap to position size and volatility.

Notes

*On sorting overall correlation by summing correlations:

Isn’t this disregarding data (due to +/- coefficients canceling each other out)?

In a way, yes. An alternative approach is summing the absolute values of the correlation coefficients:

# Get correlation and sort by sum

sum_corr = abs(df.corr()).sum().sort_values(ascending=True).index.values

df[sum_corr].corr()

However, this method is also disregarding data — the direction of the correlation coefficient.

Here is a comparison of results using both methods (with a larger portfolio):

As you can see, using the absolute method is visually smoother. For example, sorting normally (negative coefs cancelling positive), some assets such as EWZ or GS get moved further down the list, even though they have greater strengths of coefficients.

As another example, using the normal method SNAP gets sorted all the way to the left. It is the top least correlated. However, using the absolute method its strength of correlation is actually in the middle.

Overall the method you use should depend on which metric you care more about: strength of correlation, strength and direction, or having a visually smooth heatmap.

Resources

Alpaca Documentation — Web API

Alpaca Documentation — Getting Started Sample Algorithm

Alpaca Trade API Documentation

Why did I use Alpaca API?

As a trader, I’m really stoked about Alpaca.

At the time of this writing, no other platform offers live trading, backtesting, accurate market data, or commission-free trading — all in one place.

Unlike other trading platforms out there, Alpaca not only offers an API for backtesting and live trading, but also brokerage services through Alpaca Securities LLC, meaning they can fill your trades (for free!) in the same ecosystem.

The catch? You must be (at least barely) code-literate. Beside the online dashboard, Alpaca has no GUI features.

The benefits, however, are endless. Essentially Alpaca’s API is a tool for building whatever you want- backtesting, live trading, paper trading, or even entering trades from the command line or Google Sheets (!).

Technology and services are offered by AlpacaDB, Inc. Brokerage services are provided by Alpaca Securities LLC (alpaca.markets), member FINRA/SIPC. Alpaca Securities LLC is a wholly-owned subsidiary of AlpacaDB, Inc.

You can find us @AlpacaHQ, if you use twitter.

Follow Automation Generation, a Medium’s publication created for developers/makers in trading and fintech.

Ever wondered how to apply your machine learning/data science skills to algorithmic trading?

If you already know Python and basic statistical principles of data science (like train-test-split, over-fitting, etc), you’re already way ahead of the curve.

Translating machine learning models into trading algorithms is pretty simple, once you know some of the quirks of how data is handled and executed in these environments.

Let’s look at a super-basic machine learning model (adapted to QuantConnect from the Quantopian platform).

Go here to make an account for QuantConnect.

See the code, backtest, and stats here.

1. Get imports

import numpy as npfrom sklearn.ensemble import GradientBoostingRegressor

2. Understand the basic QuantConnect structure

Here’s a code skeleton of our algo:

class SimpleML(QCAlgorithm):

    def Initialize(self):          # Set settings here.        # Instantiate variables here.        # Schedule events here.

    def OnData(self, data):

        # Handle data live.                def scheduled_event(self):                # Scheduled event function.

In platforms like QuantConnect and Quantopian, algos are structured in classes. This is where you set your settings, instantiate variables, and schedule other functions you want to run.

3. Settings

Let’s fill out the def Initialize(self) part.

def Initialize(self):

    self.SetStartDate(2018,1,1)    self.SetEndDate(2018,9,1)    self.SetCash(100000)    self.AddEquity('AMZN', Resolution.Daily)            self.model = GradientBoostingRegressor()            self.lookback = 30    self.history_range = 200            self.X = []    self.y = []            self.Schedule.On(self.DateRules.WeekStart(), self.TimeRules.BeforeMarketClose('AMZN', minutes=10), Action(self.create_model))

    self.Schedule.On(self.DateRules.EveryDay(), self.TimeRules.AfterMarketOpen('AMZN', minutes=1), Action(self.trade))

• self.SetStartDate(): The start date of backtest (YYYY, MM, DD).

• self.SetEndDate(): The end date of backtest (YYYY, MM, DD).

• self.SetCash(100000): How much cash to trade.

• self.AddEquity(‘AMZN’, Resolution.Daily): Add any instruments we want data on, as well as the resolution. More on adding instruments in QuantConnect’s documentation.

• self.model: Instantiate our model.

• self.lookback: Number of past samples to train on.

• self.history_range: Range of history we’ll train on.

• self.X: Empty list for our features (in this case, past returns).

• self.y: Empty list for target (the next day’s return).

• self.Schedule.On(): This is how we call functions we’ve defined.

Note we’re calling our create_model function on the first trading day of the week, 10 minutes before close. Then we predict and trade off the model each day 10 minutes after open.

4. OnData

OnData handles live data for the algo. This is only used if your algo is executing on a lower timeframe (like HTF algorithms). We won’t need this because we’re only fitting and trading once per day.

def OnData(self, data):        pass

5. Write modeling function

This part should look familiar. The important thing here is knowing how to read in past price data.

def create_model(self):            recent_prices = self.History(['AMZN'], self.history_range)['close'].values

    price_changes = np.diff(recent_prices).tolist()            for i in range(self.history_range-self.lookback-1):        self.X.append(price_changes[i:i+self.lookback])        self.y.append(price_changes[i+self.lookback])                self.model.fit(self.X, self.y)

• recent_prices = self.History(): This is how we call in past data for whatever instruments we added at the top of the algo. This will return a dataframe of open, high, low, and close for bid and ask, plus volume over the timeframe you specify. Selecting [‘close’] selects just the close of each day. (Remember, we specified Resolution.Daily at the top.)

• price_changes = np.diff(recent_prices).to_list(): We’ll be predicting off the percentage price changes, from daily close to daily close. (If we wanted to be more statistically sound, we’d use the log returns, but let’s keep it simple for now.)

• for loop: For each day in our history, our X will be the number of days in our look back. The y will be the return for that day. We loop through the number of days in history, less the lookback (so we don’t run out of room!).

• Fit the model.

6. Write trade execution

Last part!

def trade(self):            if len(self.y) > self.lookback:                    recent_prices = self.History(['AMZN'], self.lookback+1)['close'].values        price_changes = np.diff(recent_prices)                    prediction = self.model.predict(price_changes.reshape(1, -1))                    if prediction > 0:            self.SetHoldings('AMZN', 1.0)        else:            self.SetHoldings('AMZN', 0)

• if len(self.y) > self.lookback: This is a check to make sure our y is at least the amount of our lookback.

• prediction: Predict using our model, feeding in the most recent price changes.

If our prediction is positive, we go all-in! If not, we’re flat.

Results:

Here’s a look at our results on the QuantConnect platform. While it beats the market, you would have made +72% buy-and-hold AMZN during this time.

Complete code:

import numpy as npfrom sklearn.ensemble import GradientBoostingRegressor# Adapted from https://www.quantopian.com/posts/simple-machine-learning-example-mk-iiclass SimpleML(QCAlgorithm):      def Initialize(self):          self.SetStartDate(2018,1,1)        self.SetEndDate(2018,9,1)        self.SetCash(100000)        self.AddEquity('AMZN', Resolution.Daily)                self.model = GradientBoostingRegressor()                self.lookback = 30        self.history_range = 200                self.X = []        self.y = []                self.Schedule.On(self.DateRules.WeekStart(), self.TimeRules.BeforeMarketClose('AMZN', minutes=10), Action(self.create_model))        self.Schedule.On(self.DateRules.EveryDay(), self.TimeRules.AfterMarketOpen('AMZN', minutes=1), Action(self.trade))    def OnData(self, data):        pass                def create_model(self):                recent_prices = self.History(['AMZN'], self.history_range)['close'].values        price_changes = np.diff(recent_prices).tolist()                for i in range(self.history_range-self.lookback-1):            self.X.append(price_changes[i:i+self.lookback])            self.y.append(price_changes[i+self.lookback])                    self.model.fit(self.X, self.y)        def trade(self):                if len(self.y) > self.lookback:                        recent_prices = self.History(['AMZN'], self.lookback+1)['close'].values            price_changes = np.diff(recent_prices)                        prediction = self.model.predict(price_changes.reshape(1, -1))                        if prediction > 0:                self.SetHoldings('AMZN', 1.0)            else:                self.SetHoldings('AMZN', 0)

Anything I can write about to help you find success in data science or trading? Tell me about it here: https://bit.ly/3mStNJG

# Plot two lines with different scales on the same plot

fig = plt.figure(figsize=(8, 5))

line_weight = 3alpha = .5

ax1 = fig.add_axes([0, 0, 1, 1])ax2 = fig.add_axes()

# This is the magic that joins the x-axisax2 = ax1.twinx()

lns1 = ax1.plot(wnv3['mosq'], color='blue', lw=line_weight, alpha=alpha, label='Mosquitos')lns2 = ax2.plot(wnv3['wnv'], color='orange', lw=line_weight, alpha=alpha, label='Westnile')

# Solution for having two legendsleg = lns1 + lns2labs = [l.get_label() for l in leg]ax1.legend(leg, labs, loc=0)

plt.title('Cumulative yearly mosquito & West Nile levels', fontsize=20)        plt.show()

Here’s a breakdown of how this works:

ax2 = ax1.twinx()

The magic of the graph is the .twinx() element, which makes the new axis share the old axes x-axis, but keeps an independent y-axis.

# Solution for having two legendsleg = lns1 + lns2labs = [l.get_label() for l in leg]ax1.legend(leg, labs, loc=0)

One difficulty with this is creating a legend with both labels. There is no default way to do this, and calling two .legends() will result in one legend being on top of the other.

One solution is to set different loc variables in .legend(), but this looks too annoying.

Thanks to this StackOverflow thread, we have the above solution to getting everything onto one legend.

Anything I can write about to help you find success in data science or trading? Tell me about it here: https://bit.ly/3mStNJG