Avoid These 7 Options Backtesting Pitfalls

Backtesting is a simulation of an investment strategy in the past that seeks consistent trading performance in the future. It can help develop and scrutinize the performance of systematic options strategies. The complexity of options market can be simplified by defining rule-based strategies and observing the backtest outcomes.

This makes it an important tool, especially for options traders. However, there are some shortcomings investors need to be aware of. Here's how to understand these pitfalls, as well as solutions for avoiding them.

Pitfall 1: Over-reliance on backtesting

Starting to trade based only on a backtest is premature. More needs to be done. Many backtested strategies do not live up to expectations. They perform well in the backtest, but do not perform as well when implemented.

Solution

Testing out of sample and paper trading against live markets, and seeing consistency with backtest, provides more confidence in the strategy.

Pitfall 2: Overfitting the historical market data

As more strategies are tested on the same dataset, there is a greater risk of uncovering a lucky strategy, which lacks consistent performance out of sample or in actual trading.

Solution

Attempt to cross validate out of sample and with other similar tickers. Cross validation of the factors in the test by isolating these parameters and analyzing performance.

Pitfall 3: Data mining, correlation and causation

Using massive amounts of data and identifying spurious relationships will not be consistent when going forward and implementing a strategy. Finding causation is good but finding correlation instead of a causation will also create inconsistent results going forward.

Solution

Understand the source of alpha, understand the data and consider more stringent tests and thresholds for significance. An options strategy is trading based on predefined rules. The strategy can be based on fundamental, technical or options information, but needs to be quantifiable information that makes sense to the task at hand. Dependent variables need to have some reasonable relationship to dependent variables and have enough differentiated data for backtesting with accuracy and rigor.

The job of backtesting is to find opportunities that exist. These opportunities need to be understandable and justifiable. Here are some examples of opportunities that may exist:

• Psychological trading anomalies: Anomalies in the market may present themselves as a result of such fallacies of sunk costs or the disposition effect of holding stocks that have lost money and selling those that have been profitable.
• Asymmetry of information: Opportunities may exist from finding where people in the know tip their hand when they trade on asymmetric information. 
• Risk tolerance differences: Options may be over or underpriced for exogenous reasons like bank risk mitigation and may present an opportunity to those with differing risk preferences.
• Tax idiosyncrasies: Dividend preferences based on tax rules may present mispricing for those with different tax treatment.
• Barriers to entry: Some investments may be unavailable to traders. Information on the inner workings of ETFs might be understood better by various traders. Understanding and gaining access to such markets may be advantageous.

Pitfall 4: Hidden risks

Some options positions have nonlinear risk as the underlying moves against the position.

Solution

Know your risks by simulating a position and stress testing. Use appropriate slippage and commission assumptions. A major risk in backtesting is assuming where to trade. Especially in wide bid-ask spreads, trading based on mid-market will not be where the trade can be filled. Avoid wide markets in backtesting.

Pitfall 5: Over-focus on return

By narrowing in on annual returns, many other important statistics can be missed.

Solution

Use other statistics and temper return results:

• Sharpe ratios-- annual return divided by volatility
• Sortino ratios-- annual return divided by volatility when returns go against the position
• Drawdowns-- amount lost from peak to valley
• Win rates-- percent of winning trades to losing trades
• Best/worst month/year-- seeing outlying returns in various timeframes

These and more are important statistics to understand the strategy's behavior.

Pitfall 6: Selecting an inappropriate time period

Testing on a time when the market was in an environment that is different than are likely to repeat, will cause inconsistent and misleading expectations.

Solution

Make sure there is enough history to contain a representative set of environments. For example, market data should go back to 2007 to include the 2008 market downturn. Out of sample testing should also have a variety of environments.

Pitfall 7: Survival bias

Some options strategies rely on tickers that may not have been in existence or may have been delisted. Including delisted tickers can affect the results greatly.

Solution

Include delisted tickers in the backtest. Only trade indexes or ETFs that are likely to exist for a long time.

Final note

Avoiding pitfalls in backtesting can produce a higher probability of consistent outcomes when implementing trading strategies. A rules-based strategy can help the trader avoid behavioral biases and help take advantage of structural anomalies in the market. It is important to estimate and understand why the alpha exists and if the source is sustainable. Backtesting a rules based strategy that avoid pitfalls is a great tool for traders.

References

• Backtesting: A Practitioner’s Guide to Assessing Strategies and Avoiding Pitfalls by Olivier Sarfati, Citigroup presented at the 2015 Cboe Risk Management Conference.
• Backtesting and Forward Testing: The Importance of Correlation by Jean Folger in Investopedia March 10, 2018