Grid search example#
This is a grid search example, modified the default Bollinger bands example.
Set up searchable strategy parameter and then generate a matrix of all possible combinations of these for a grid search
Perform parallel run to multiple backtests at the same time using a thread pool
Compare results
For more details, see README.
Github notebook rendering issues#
If you are viewing this file on Github: The current Github online notebook viewer cannot render the interactive Plotly charts used in this notebook. To view the charts, you need to download run the notebook locally.
Set up#
Set up the parameters used in in this strategy backtest study.
Unlike in a fixed parameter strategy study, we define some of the strategy parameters as list of options
We later createa a grid search matrix which will go through all the combinations of these options
Convert normal strategy to grid test#
Convert relevant parameters with fixed values to list of values
Wrap
decide_trades
withgrid_search_worker
and addcombination.destructure()
linePrepare grid combinations with
prepare_grid_combinations
Run grid search with
perform_grid_search
Pro tip#
Start with a coarse grid search. To save time and computational resources, begin with a wide range of parameter values. Once you have identified promising regions, you can then perform a finer grid search to fine-tune the parameters.
[1]:
import datetime
from tradingstrategy.chain import ChainId
from tradingstrategy.timebucket import TimeBucket
from tradeexecutor.strategy.cycle import CycleDuration
from tradeexecutor.strategy.strategy_module import TradeRouting, ReserveCurrency
#
# Strategy properties
#
STRATEGY_NAME = "Bollinger bands example"
# How our trades are routed.
TRADE_ROUTING = TradeRouting.trader_joe_usdc
# How often the strategy performs the decide_trades cycle.
TRADING_STRATEGY_CYCLE = CycleDuration.cycle_4h
# Time bucket for our candles
CANDLE_TIME_BUCKET = TimeBucket.h4
# Candle time granularity we use to trigger stop loss checks
STOP_LOSS_TIME_BUCKET = TimeBucket.m15
# Strategy keeps its cash in USDC
RESERVE_CURRENCY = ReserveCurrency.usdc
# Which trading pair we are backtesting on
# (Might be different from the live trading pair)
# https://tradingstrategy.ai/trading-view/polygon/quickswap/eth-usdc
TRADING_PAIR = (ChainId.avalanche, "trader-joe", "WAVAX", "USDC")
# How much % of the cash to put on a single trade
POSITION_SIZE = 0.50
# Start with this amount of USD
INITIAL_DEPOSIT = 5_000
#
# Strategy inputs
#
# How many candles we load in the decide_trades() function for calculating indicators
LOOKBACK_WINDOW = 90
# How many candles we use to calculate the Relative Strength Indicator
RSI_LENGTH = 14
#
# Grid searched parameters
#
# Bollinger band's standard deviation options
#
STDDEV = [1.0, 1.5, 1.7, 2.0]
# RSI must be above this value to open a new position.
RSI_THRESHOLD = [55, 65, 75, 85]
# What's the moving average length in candles for Bollinger bands
MOVING_AVERAGE_LENGTH = [7, 14, 20]
# Backtest range
#
# Note that for this example notebook we deliberately choose a very short period,
# as the backtest completes faster, charts are more readable
# and tables shorter for the demostration.
#
START_AT = datetime.datetime(2022, 3, 1)
# Backtest range
END_AT = datetime.datetime(2022, 10, 1)
# Stop loss relative to the mid price during the time when the position is opened
#
# If the price drops below this level, trigger a stop loss
STOP_LOSS_PCT = 0.98
Strategy logic and trade decisions#
decide_trades()
function decide what trades to take.As this is a grid search example, we nest
decide_trades()
withingrid_search_worker()
.grid_search_worker()
is responsible for getting the combination inputs from the grid search manager and then setting updecide_trades()
with its dynamic parameters.We do not plot any visualisations like upper and lower band like we do in individual strategy analysis; plotting takes time and this information is not relevant for grid search analysis in this point.
*Note: We have lowercased the grid searched parameter names here. In most programming languages, uppercased variables are constant and lowercased dynamic. Because grid searcher parameters are now dynamic, we use snake case to notate them to differ them from constants.
[2]:
from typing import List, Dict
import pandas as pd
from pandas_ta import bbands
from pandas_ta.momentum import rsi
from tradingstrategy.universe import Universe
from tradeexecutor.state.trade import TradeExecution
from tradeexecutor.strategy.pricing_model import PricingModel
from tradeexecutor.strategy.pandas_trader.position_manager import PositionManager
from tradeexecutor.state.state import State
from tradeexecutor.backtest.grid_search import GridCombination, GridSearchResult
from tradeexecutor.backtest.grid_search import run_grid_search_backtest
from tradeexecutor.strategy.trading_strategy_universe import TradingStrategyUniverse
def grid_search_worker(
universe: TradingStrategyUniverse,
combination: GridCombination,
) -> GridSearchResult:
"""Run a backtest for a single grid combination."""
# Open grid search options as they are given in the setup later.
# The order here *must be* the same as given for prepare_grid_combinations()
rsi_threshold, stddev, moving_average_length = combination.destructure()
def decide_trades(
timestamp: pd.Timestamp,
universe: Universe,
state: State,
pricing_model: PricingModel,
cycle_debug_data: Dict) -> List[TradeExecution]:
# Trades generated in this cycle
trades = []
# We have only a single trading pair for this strategy.
pair = universe.pairs.get_single()
# Here we manipulate the pair trading fee.
#
# A live trading would happen on TraderJoe AVAX-USDC pool with 0.20% LP fee.
# https://traderjoexyz.com/avalanche/pool/v21/AVAX/0xb97ef9ef8734c71904d8002f8b6bc66dd9c48a6e/20
# But this pool was deployed only couple of weeks back, so we do not have backtesting history for it.
# Thus, we are backtesting with the old pair with 0.30% LP fee tier for more data.
# We then manually bump down the fee tier to reflect the live trading situation.
pair.fee = 0.0020
# How much cash we have in a hand
cash = state.portfolio.get_current_cash()
# Get OHLCV candles for our trading pair as Pandas Dataframe.
# We could have candles for multiple trading pairs in a different strategy,
# but this strategy only operates on single pair candle.
# We also limit our sample size to N latest candles to speed up calculations.
candles: pd.DataFrame = universe.candles.get_single_pair_data(timestamp, sample_count=LOOKBACK_WINDOW)
if len(candles) == 0:
# We are looking back so far in the history that the pair is not trading yet
return trades
# We have data for open, high, close, etc.
# We only operate using candle close values in this strategy.
close_prices = candles["close"]
price_latest = close_prices.iloc[-1]
# Create a position manager helper class that allows us easily to create
# opening/closing trades for different positions
position_manager = PositionManager(timestamp, universe, state, pricing_model)
# Calculate RSI for candle close
# https://tradingstrategy.ai/docs/programming/api/technical-analysis/momentum/help/pandas_ta.momentum.rsi.html#rsi
rsi_series = rsi(close_prices, length=RSI_LENGTH)
if rsi_series is None:
# Not enough data in the backtesting buffer yet
return trades
# Calculate Bollinger Bands with a 20-day SMA and 2 standard deviations using pandas_ta
# See documentation here https://tradingstrategy.ai/docs/programming/api/technical-analysis/volatility/help/pandas_ta.volatility.bbands.html#bbands
bollinger_bands = bbands(close_prices, length=moving_average_length, std=stddev)
if bollinger_bands is None:
# Not enough data in the backtesting buffer yet
return trades
# bbands() returns a dictionary of items with different name mangling
bb_upper = bollinger_bands[f"BBU_{moving_average_length}_{stddev}"]
bb_lower = bollinger_bands[f"BBL_{moving_average_length}_{stddev}"]
bb_mid = bollinger_bands[f"BBM_{moving_average_length}_{stddev}"] # Moving average
if not position_manager.is_any_open():
# No open positions, decide if BUY in this cycle.
# We buy if the price on the daily chart closes above the upper Bollinger Band.
if price_latest > bb_upper.iloc[-1] and rsi_series[-1] >= rsi_threshold:
buy_amount = cash * POSITION_SIZE
trades += position_manager.open_1x_long(
pair,
buy_amount,
stop_loss_pct=STOP_LOSS_PCT)
else:
# We have an open position, decide if SELL in this cycle.
# We close the position when the price closes below the 20-day moving average.
if price_latest < bb_mid.iloc[-1]:
trades += position_manager.close_all()
return trades
return run_grid_search_backtest(
combination,
decide_trades,
universe,
start_at=START_AT,
end_at=END_AT,
cycle_duration=TRADING_STRATEGY_CYCLE,
)
Defining the trading universe#
We create a trading universe with a single blockchain, single exchange and a single trading pair.
Trading Strategy framework supports complex strategies, spanning thousands of pairs and lending pools, but we are not interested in this example.
[3]:
import datetime
from tradingstrategy.client import Client
from tradeexecutor.strategy.trading_strategy_universe import load_pair_data_for_single_exchange, TradingStrategyUniverse
from tradeexecutor.strategy.execution_context import ExecutionContext
from tradeexecutor.strategy.universe_model import UniverseOptions
def create_single_pair_trading_universe(
ts: datetime.datetime,
client: Client,
execution_context: ExecutionContext,
universe_options: UniverseOptions,
) -> TradingStrategyUniverse:
# Fetch backtesting datasets from the server
dataset = load_pair_data_for_single_exchange(
client,
time_bucket=CANDLE_TIME_BUCKET,
pair_tickers=[TRADING_PAIR],
execution_context=execution_context,
universe_options=universe_options,
stop_loss_time_bucket=STOP_LOSS_TIME_BUCKET,
)
# Convert loaded data to a trading pair universe
universe = TradingStrategyUniverse.create_single_pair_universe(
dataset,
pair=TRADING_PAIR,
)
return universe
Set up the market data client#
The Trading Strategy market data client is the Python library responsible for managing the data feeds needed to run the backtest.None
We set up the market data client with an API key.
If you do not have an API key yet, you can register one.
[4]:
from tradingstrategy.client import Client
client = Client.create_jupyter_client()
Started Trading Strategy in Jupyter notebook environment, configuration is stored in /home/alex/.tradingstrategy
Load data#
This will pull JSONL data feed for the trading pair from Trading Strategy oracle node.
[5]:
from tradeexecutor.strategy.execution_context import ExecutionMode
from tradeexecutor.strategy.universe_model import UniverseOptions
universe = create_single_pair_trading_universe(
END_AT,
client,
ExecutionContext(mode=ExecutionMode.data_preload),
UniverseOptions()
)
print(f"We loaded {universe.universe.candles.get_candle_count():,} candles.")
We loaded 3,934 candles.
Prepare grid search#
We prepare the grid search
By definiting the strategy parameter search space
Setting up the file system folder where the strategy run results are stored. The framework caches the results on disk, so that if the grid search is aborted, it can later resume and long operation results are not lost.
If the grid search results already exist, the backtesting phase is skipped altogether and notebook directly jumps to analysis
To make the analysis easier to read, you should choose the parameter with the most options, or most dominating, to be the first one
Note: If you change the strategy logic, or fixed parameters, you need to clear the cache folder with clear_cached_results
setting:
combinations = prepare_grid_combinations(parameters, storage_folder, clear_cached_results=True)
Otherwise the grid searcher will use the existing stored results and does not rerun your tests with the changed code.
[6]:
from pathlib import Path
from tradeexecutor.backtest.grid_search import prepare_grid_combinations
# This is the path where we keep the result files around
storage_folder = Path(f"/tmp/{STRATEGY_NAME}")
parameters = {
"rsi_threshold": RSI_THRESHOLD,
"stddev": STDDEV,
"moving_average_length": MOVING_AVERAGE_LENGTH,
}
combinations = prepare_grid_combinations(parameters, storage_folder)
print(f"We prepared {len(combinations)} grid search combinations")
We prepared 48 grid search combinations
Run grid search#
Run backtest using giving trading universe and grid search worker function.
Currently this runs in a threaded mode, which is memory optimised but not throughput optimised, for Python
Increasing
max_workers
may not yield to speed gains
[7]:
from tradeexecutor.backtest.grid_search import perform_grid_search
grid_search_results = perform_grid_search(
grid_search_worker,
universe,
combinations,
max_workers=4,
)
Analysing the grid search results#
We examine the different grid search data.
[8]:
# Set Jupyter Notebook output mode parameters
from tradeexecutor.backtest.notebook import setup_charting_and_output
setup_charting_and_output()
# Print extension of our backtest
print(f"Grid search combinations available: {len(grid_search_results)}")
Grid search combinations available: 48
Grid combination table#
Show the result of all combinations in a single table.
[9]:
from tradeexecutor.analysis.grid_search import analyse_grid_search_result
from tradeexecutor.analysis.grid_search import visualise_table
table = analyse_grid_search_result(grid_search_results)
visualise_table(table)
Positions | Annualised return | Max drawdown | Sharpe | Sortino | Average position | Median position | |||
---|---|---|---|---|---|---|---|---|---|
rsi_threshold | stddev | moving_average_length | |||||||
55 | 1.000000 | 7 | 54 | -31.66% | -25.00% | -1.470000 | -2.110000 | -0.79% | -2.50% |
14 | 45 | -10.23% | -26.00% | -0.250000 | -0.380000 | -0.21% | -2.55% | ||
20 | 45 | -23.71% | -28.00% | -0.760000 | -1.110000 | -0.63% | -2.67% | ||
1.500000 | 7 | 41 | -20.09% | -22.00% | -0.930000 | -1.360000 | -0.60% | -2.54% | |
14 | 38 | -8.15% | -21.00% | -0.210000 | -0.330000 | -0.20% | -2.54% | ||
20 | 38 | -19.98% | -25.00% | -0.690000 | -1.010000 | -0.63% | -2.54% | ||
1.700000 | 7 | 34 | -11.99% | -15.00% | -0.560000 | -0.820000 | -0.39% | -2.54% | |
14 | 35 | -5.30% | -16.00% | -0.110000 | -0.170000 | -0.11% | -2.54% | ||
20 | 30 | -15.37% | -19.00% | -0.540000 | -0.800000 | -0.61% | -2.54% | ||
2.000000 | 7 | 22 | -25.95% | -17.00% | -2.290000 | -3.030000 | -1.58% | -2.57% | |
14 | 23 | 4.82% | -14.00% | 0.330000 | 0.510000 | 0.33% | -2.54% | ||
20 | 26 | -14.71% | -18.00% | -0.600000 | -0.900000 | -0.66% | -2.56% | ||
65 | 1.000000 | 7 | 27 | -17.89% | -16.00% | -1.030000 | -1.510000 | -0.82% | -2.44% |
14 | 21 | 5.98% | -12.00% | 0.380000 | 0.610000 | 0.40% | -2.62% | ||
20 | 17 | 4.72% | -8.00% | 0.320000 | 0.480000 | 0.37% | -2.42% | ||
1.500000 | 7 | 21 | -15.45% | -11.00% | -1.060000 | -1.540000 | -0.91% | -2.60% | |
14 | 18 | 13.18% | -12.00% | 0.710000 | 1.140000 | 0.89% | -2.61% | ||
20 | 16 | 7.09% | -8.00% | 0.420000 | 0.630000 | 0.56% | -2.42% | ||
1.700000 | 7 | 19 | -20.06% | -14.00% | -1.820000 | -2.440000 | -1.36% | -2.62% | |
14 | 17 | 19.76% | -9.00% | 1.000000 | 1.670000 | 1.33% | -2.60% | ||
20 | 14 | 11.52% | -8.00% | 0.610000 | 0.940000 | 0.98% | -2.51% | ||
2.000000 | 7 | 12 | -18.09% | -13.00% | -2.090000 | -2.650000 | -1.93% | -2.56% | |
14 | 13 | 6.28% | -9.00% | 0.440000 | 0.720000 | 0.64% | -2.62% | ||
20 | 12 | 12.37% | -9.00% | 0.680000 | 1.090000 | 1.21% | -2.44% | ||
75 | 1.000000 | 7 | 4 | nan% | nan% | nan | nan | nan% | nan% |
14 | 4 | nan% | nan% | nan | nan | nan% | nan% | ||
20 | 4 | nan% | nan% | nan | nan | nan% | nan% | ||
1.500000 | 7 | 4 | nan% | nan% | nan | nan | nan% | nan% | |
14 | 4 | nan% | nan% | nan | nan | nan% | nan% | ||
20 | 4 | nan% | nan% | nan | nan | nan% | nan% | ||
1.700000 | 7 | 3 | nan% | nan% | nan | nan | nan% | nan% | |
14 | 4 | nan% | nan% | nan | nan | nan% | nan% | ||
20 | 4 | nan% | nan% | nan | nan | nan% | nan% | ||
2.000000 | 7 | 3 | nan% | nan% | nan | nan | nan% | nan% | |
14 | 2 | nan% | nan% | nan | nan | nan% | nan% | ||
20 | 3 | nan% | nan% | nan | nan | nan% | nan% | ||
85 | 1.000000 | 7 | 0 | nan% | nan% | nan | nan | nan% | nan% |
14 | 0 | nan% | nan% | nan | nan | nan% | nan% | ||
20 | 0 | nan% | nan% | nan | nan | nan% | nan% | ||
1.500000 | 7 | 0 | nan% | nan% | nan | nan | nan% | nan% | |
14 | 0 | nan% | nan% | nan | nan | nan% | nan% | ||
20 | 0 | nan% | nan% | nan | nan | nan% | nan% | ||
1.700000 | 7 | 0 | nan% | nan% | nan | nan | nan% | nan% | |
14 | 0 | nan% | nan% | nan | nan | nan% | nan% | ||
20 | 0 | nan% | nan% | nan | nan | nan% | nan% | ||
2.000000 | 7 | 0 | nan% | nan% | nan | nan | nan% | nan% | |
14 | 0 | nan% | nan% | nan | nan | nan% | nan% | ||
20 | 0 | nan% | nan% | nan | nan | nan% | nan% |
Two dimensional heatmaps#
Heatmap allows see how one strategy performance metrics varies over the search space
The example has three parameter search space
We compare two of these parameters and lock up one parameter to a fixed value
[10]:
from tradeexecutor.analysis.grid_search import visualise_heatmap_2d
# Remove one extra axis by focusing only rsi_threshold=65
heatmap_data = table.xs(65, level="rsi_threshold")
# Draw heatmaps for two different metrics
fig = visualise_heatmap_2d(heatmap_data, "stddev", "moving_average_length", "Annualised return")
display(fig)