Evaluating Profitability of Trading Strategies Using Hypothesis Testing

In the world of algorithmic trading, developing a strategy is only the first step. The next crucial step is to validate the profitability of the strategy. One popular method to evaluate the profitability is using statistical hypothesis testing. In this blog post, we will walk through how to use a one-sample t-test to evaluate the profitability of a trading strategy. If you're interested in understanding more about the concept of backtesting and the pitfalls of overfitting, you can check out my blog post on Backtesting and Overfitting.

Understanding the Basics

First, let's understand the data we're working with. To assess the performance of a trading strategy, we usually evaluate some performance metrics like the Sharpe Ratio, Sortino Ratio, or Profit Factor. However, the initial implementation of a strategy often needs optimization. A common problem here is overfitting, where the strategy is finely tuned to perform well on historical data but may not perform as well under future market conditions. Therefore, when optimizing a strategy, we need to consider all optimized instances and resist the temptation to cherry-pick the best ones. The best instances are usually not profitable in real market conditions due to their overfitting.

In this blog post, we will focus on the Profit Factor as our main metric for hypothesis testing. The Profit Factor is a measure of the effectiveness of a trading strategy, calculated as the gross profit divided by the gross loss for each trade.

Before we proceed, let's get familiar with some statistical concepts:

Mean: The average value, calculated by adding up all the numbers and dividing by the count of numbers.
Median: The middle value in a set of numbers.
Standard Deviation (Std): A measure of how spread out the numbers in the data set are. It quantifies the dispersion and variation in the data.

Statistical Hypothesis Testing

In our context, we will set up our hypothesis test as follows:

Null Hypothesis (H0): The mean profit factor is less than or equal to 1 (i.e., the strategy is not profitable).
Alternative Hypothesis (H1): The mean profit factor is greater than 1 (i.e., the strategy is profitable).

We will use a one-sample t-test for this purpose. This statistical test compares the mean of our sample data to a known value, in this case, 1. A t-test calculates a score (t-statistic) that expresses the difference between the sample mean and the known value, relative to the variability in the data. A larger t-statistic indicates a larger difference between the sample mean and the known value and a smaller p-value indicates stronger evidence against the null hypothesis.

Getting the Performance Metrics and Conducting the T-Test

Our focus here is on the cTrader cAlgo API, as it's the platform where I conduct my trading. One challenge with cTrader is that when you run an optimization, you cannot export the performance metrics into an external file. To overcome this, I've created a custom method that you can add to the OnStop method to calculate the profit factor and export it to an external file. So when you run an optimization, all profit factors from all instances will be exported there. To get this Code and add it to your cBot, follow this link Here.

The next step is to conduct the t-test. We can easily perform this using Python, R, or any statistical application. I created here an easy-to-use Application, where you can choose you analyze a CSV file that contains your performance metric. to Download this, follow the links Here.

After choosing the file containing your Performance Metrics of the Optimized strategy, choose your null Hypothesis Value and a Critical Value. usually, a critical value of 0.05 is enough. If we are analyzing Profit Factors, a value of 1 is enough.

You will see that the application will calculate the Mean, Median, Skewness, and Standard Deviation, and will conduct the T-Test and the Hypothesis Test according to our Null hypothesis and critical values. Here I used the profit factors of the Divergence Strategy based on my divergence indicator. As you see we were able to reject the null Hypothesis. So there is a high chance the strategy is profitable

This process allows us to rigorously test the profitability of our strategy, beyond simply looking at the profit factor of the best instance from optimization. By using this method, we can avoid overfitting and increase our confidence in the strategy's performance under future market conditions.

It is important to notice, that an optimized and overfitted Strategy will leave us with significant results but these results are worthless. I am sure you know the saying. Garbage in = Garbage out. To learn more about overfitting, read my Blogpost Here.

If you liked this post and it helped you in your trading, please support us and buy us a cup of coffee Here.

If you have any questions or critics, feel free to contact us at info@singalgolab.com