Trade System Evaluation
Original article by Donald W. Pendergast Jr.
AIQ Code by Richard Denning
Traders Studio Code by Richard Denning
AIQ Version:
The AIQ code for the reverse MACD crossover system in the article, “Trade System Evaluation” by Donald W. Pendergast, Jr., is shown below.
In the article the author uses Monte Carlo testing on a portfolio of the NASDAQ 100 stocks. The system suggested does not include a sorting algorithm to choose trades when there are more trades than we can take on any single day based on our capitalization and position sizing rules. Instead the author suggests that we randomly choose the trades that are to be taken using Monte Carlo analysis which samples trades without replacement. Also from the description in the article, it appears that the order of the trades is not randomized. In order to apply this type of Monte Carlo analysis to a system there are several characteristics of the system and assumptions re how the testing will proceed:
1) The system must generate more trades than can be taken when capitalization is applied and
2) Trades are not reordered and samples are taken without replacement and
3) A sorting algorithm is not part of the system
So if we had a system where there were no excess trades and all trades were taken or if our system uses a sorting algorithm to choose trades, then the Monte Carlo analysis would yield all the same results and would be of no analysis value. For the simple system used for illustration purposes, since the trades are entered next day at market, as a system developer, I would never randomly choose the trades but would add an algorithm, such as sorted rate of change or relative strength, to choose the trades to take. For systems that do not meet the three requirements above, it would be better to use the boot strap method of sampling where trades are replaced and hence can be chosen more than one time. By sampling with replacement, we could generate multiple equity curves with different total profit and drawdown amounts even with systems that take all trades generated.
This analysis might be useful for more complex systems that use stops to enter. In this case, we can’t use a sorting algorithm in backtesting because we don’t know which stop orders will be hit the next day after the set up conditions are true. In any case we can perform an analysis similar to the one run by the author using a free addin program called “TradeIt!” that is specifically designed to work with the AIQ software. In Figure 1, I used the addin program to generate 20 equity curves running the system on the NASDAQ 100 list of stocks. The equity curve graphic is helpful in visually seeing how much variance there is between the possible equity runs. Also the software produces a numerical report as show in Table 1.
Figure 1: Twenty equity curves from the reverse MACD system that were generated by randomly choosing which trades to take when there were more trades than could be taken.
Table 1: Numerical summary and statistical analysis of the 20 equity curves shown in Figure 1.
AIQ EDS Code for Trade System Evaluation:
Trd Sys Eval.EDS
Traders Studio Version:
The TradersStudio code for the reverse MACD crossover system in the article, “Trade System Evaluation” by Donald W. Pendergast, Jr., is shown below.
Although the author applies Monte Carlo testing on a portfolio of the NASDAQ 100 stocks, I wanted to try the Monte Carlo approach as it might apply to testing the reverse MACD system on futures. I loaded a portfolio of futures that included two of the more liquid markets from each of the futures sectors. I then ran the system and examined the results on a market by market basis. I should note that in the code provided, I modified the system so that it can trade long or short or both based the input “longOnly” (see notes at top of code). In my tests, I used both long and short trading combined.
Most of the futures markets I tested showed net losses for the test period from November 2002 to January 2009 with the default MACD parameters of 12,26,9. One of the markets that showed a profit was heating oil. I decided to work with just this market. With a single market system and with futures trading in general, we usually design the system to take all the trades that the system generates (after filtering), so the type of test that the author did is not applicable to futures testing.
A similar test using the Monte Carlo approach is to sample drawdowns for the purpose of setting a system profile and setting realistic expectations. I ran a parameter robustness test on heating oil by optimizing the three MACD parameters over the insample test period of November 2002 to January 2009. I concluded that the system was robust for heating oil because 93% of the optimization sets were profitable and because it passed other the parameter robustness tests. In normal system development, I would have run a walk forward test using TradersStudio’s builtin walk forward processor but this was omitted due to a shorter than normal publishing deadline. Instead I just used one of the better parameter sets and proceeded to run several Monte Carlo randomizations to give me an idea of the maximum drawdown. TradersStudio comes with a preprogrammed macro that does Monte Carlo analysis on drawdowns. In addition, any other type of sampling technique such as bootstrapping (sampling with replacement) can be programmed with the macro scripting language. I made a few modifications to the builtin macro script for Monte Carlo drawdown analysis so as to output additional statistics. The modified macro script code is not shown below but will be posted to the web sites listed below.
In Monte Carlo analysis, there is the question of how many runs are sufficient. To answer this question, I simply ran several tests with different run lengths as shown in Table 1. Twenty thousand runs are more than is needed but I ran the rest of the analysis with 20,000 as there is no harm in using too many runs.
Another important factor in correctly analyzing drawdown is the trade sampling chunk or serial correlation sample size. What we are doing in this type of analysis is to take a group of trades in the original order that they occurred and then reorganize the order of the chunks. If you do not sample chunks of trades, as is done in many Monte Carlo type analyses, you destroy any serial correlation that may exist in the trading system. Using an inappropriate chunk size may cause inaccurate results that are not reliable. To solve this problem, the macro that is within TradersStudio has a chunk or sample length variable that can be changed to suit the system being tested. If the system has a tendency for wins to follow wins then the sample size would need to be larger. The bottom half of Table 1 shows the effect of running the analysis with varying chunk or sample length using a run size of 20,000. In Figures 1, I show the drawdown distribution graph that results from using a sample (chunk) size of 20 and 20,000 runs. Additional graphs for other sample sizes will be posted to the web sites listed below. As the sample size is increased, the maximum drawdown at the various confidence levels starts to approach the drawdown from the original trades order. It appears that using too small of a sample (chunk) size will overstate the maximum drawdown and might cause one to reject a system that has potential. Further research is needed to determine the best method of finding the appropriate sample (chunk) size. The main benefit of this analysis is to provide a down to earth expectation of what the system might do in real trading re the risk of account drawdown and also to provide a shut down point should the maximum drawdown exceed the confidence level chosen.
Figure 1: Drawdown distribution using a sample (chunk) size of 20 with 20,000 random reorganizations of the equity curve for the reversed MACD system as applied to heating oil futures contract for the period November 2002 to January 2009.
Figure 2: Drawdown distribution same as Figure 1 except using a sample (chunk) size of 1 with 20,000 random reorganizations of the equity curve.
Figure 3: Drawdown distribution same as Figure 1 except using a sample (chunk) size of 30 with 20,000 random reorganizations of the equity curve.
Table 1: Resulting statics for a variety of Monte Carlo tests on the reversed MACD system as applied to heating oil futures contract for the period November 2002 to January 2009.
Traders Studio Monte Carlo Test for Trade System Evaluation:
MonteCarloTest2.zip
Traders Studio Code for Reversed MACD System:
Reversed_MACD_TSZ.zip





