The Pseudo-Science Of Hypothesis Testing

Backtesting trading strategies necessarily involves a real express sum of historical data. For example, I seldom seek strategies amongst information older than 2007. Gathering longer history may non improve predictive accuracy since the marketplace position construction may convey changed substantially. Given such scant data, it is reasonable to query whether the expert backtest results (e.g. a high annualized provide R) nosotros may convey obtained is exactly due to luck. Many academic researchers attempt to address this number past times running their published strategies through  standard statistical hypothesis testing.

You know the drill: the researchers get-go come upwardly up amongst a supposedly splendid strategy. In a display of faux modesty, they in addition to then advise that possibly a zilch hypothesis tin create the same expert provide R. The zilch hypothesis may hold upwardly constructed past times running the master strategy through to a greater extent than or less random simulated historical data, or past times randomizing the merchandise entry dates. The researchers in addition to then proceed to demonstrate that such random constructions are highly unlikely to generate a provide equal to or improve than R. Thus the zilch hypothesis is rejected, in addition to thereby impressing yous that the strategy is somehow sound.

As statistical practitioners inwards fields exterior of finance volition say you, this whole physical care for is quite meaningless in addition to often misleading.

The probabilistic syllogism of hypothesis testing has the same construction equally the next unproblematic instance (devised past times Jeff Gill inwards his newspaper "The Insignificance of Null Hypothesis Significance Testing"):

1) If a individual is an American in addition to then it is highly unlikely she is a fellow member of Congress.

2) The individual is a fellow member of Congress.

3) Therefore it is highly unlikely she is an American.

The absurdity of hypothesis testing should hold upwardly clear. In mathematical terms, the probability nosotros are actually interested inwards is the conditional probability that the zilch hypothesis is truthful given an observed high provide R: P(H0|R). But instead, the hypothesis seek exactly gives us the conditional probability of a provide R given that the zilch hypothesis is true: P(R|H0). These 2 conditional probabilities are seldom equal.

But fifty-fifty if nosotros tin somehow compute P(H0|R), it is withal of real niggling use, since at that spot are an interplanetary space number of potential H0. Just because yous convey knocked downward 1 detail straw man doesn't say much close your master strategy.

If hypothesis testing is both meaningless and misleading, why exercise fiscal researchers proceed to peddle it? Mainly because this is de rigueur to buy the farm published. But it does serve 1 useful role for our ain individual trading research. Even though a rejection of the zilch hypothesis inwards no agency shows that the strategy is sound, a failure to turn down the zilch hypothesis volition hold upwardly far to a greater extent than interesting.

(For other references on criticism of hypothesis testing, read Nate Silver's bestseller "The Signal in addition to The Noise". Silver is of cast the statistician who correctly predicted the winner of all 50 states + D.C. inwards the 2012 U.S. of A. presidential election. The majority is highly relevant to anyone who makes a living predicting the future. In particular, it tells the floor of 1 Bob Voulgaris who makes $1-4M per annum betting on NBA outcomes. It makes me wonder whether I should quit making bets on fiscal markets in addition to motion on to sports.)