Time Serial Analysis Too Information Gaps

Most fourth dimension serial techniques such equally the ADF essay for stationarity, Johansen essay for cointegration, or ARIMA model for returns prediction, assume that our information points are collected at regular intervals. In traders' parlance, it assumes bar information amongst fixed bar length. It is slowly to run into that this mundane requirement instantly presents a work fifty-fifty if nosotros were only to analyze daily bars: how are nosotros create bargain amongst weekends as well as holidays?

You tin run into that the statistics of provide bars over weekdays tin differ significantly from those over weekends as well as holidays. Here is a tabular array of comparing for SPY daily returns from 2005/05/04-2015/04/09:

SPY daily returns	Number of bars	Mean Returns (bps)	Mean Absolute Returns (bps)	Kurtosis (3 is “normal”)
Weekdays only	1,958	3.9	80.9	13.0
Weekends/holidays only	542	0.3	82.9	23.7

Though the absolute magnitude of the returns over a weekday is similar to that over a weekend, the hateful returns are much to a greater extent than positive on the weekdays. Note also that the kurtosis of returns is most doubled on the weekends. (Much higher tail risks on weekends amongst much less expected returns: why would anyone concur a seat over weekends?) So if nosotros run whatsoever form of fourth dimension serial analysis on daily data, nosotros are force-fitting a model on information amongst heterogeneous statistics that won't function well.

The work is, of course, much worse if nosotros travail fourth dimension serial analysis on intraday bars. Not exclusively are nosotros faced amongst the weekend gap, inward the illustration of stocks or ETFs nosotros are faced amongst the overnight gap equally well. Here is a tabular array of comparing for AUDCAD 15-min returns vs weekend returns from 2009/01/01-2015/06/16:

AUDCAD 15-min returns	Number of bars	Mean Returns (bps)	Mean Absolute Returns (bps)	Kurtosis (3 is “normal”)
Weekdays only	158,640	0.01	4.5	18.8
Weekends/holidays only	343	-2.06	15.3	4.6

In this case, every of import statistic is dissimilar (and it is noteworthy that kurtosis is genuinely lower on the weekends here, illustrating the mean-reverting grapheme of this fourth dimension series.)

So how should nosotros predict intraday returns amongst information that has weekend gaps? (The same solution should apply to overnight gaps for stocks, as well as thence omitted inward the next discussion.) Let's regard several proposals:

1) Just delete the weekend returns, or laid them equally NaN inward Matlab, or missing values NA inward R. 

This won't function because the root few bars of a calendar week isn't properly predicted past times the terminal few bars of the previous week. We shouldn't purpose whatsoever linear model built amongst daily or intraday information to predict the returns of the root few bars of a week, whether or non that model contains information amongst weekend gaps. As for how many bars constitute the "first few bars", it depends on the lookback of the model. (Notice I emphasize linear model hither because to a greater extent than or less nonlinear models tin bargain amongst large jumps during the weekends appropriately.)

2) Just pretend the weekend returns are no dissimilar from the daily or intraday returns when building/training the fourth dimension serial model, but create non purpose the model for predicting weekend returns. I.e. create non concur positions over the weekends.

This has been the default, as well as peradventure simplest (naive?) way of treatment this number for many traders, as well as it isn't besides bad. The predictions for the root few bars inward a calendar week volition 1 time again last suspect, equally inward 1), thence 1 may desire to refrain from trading then. The model built this way isn't the best possible one, but thence nosotros don't convey to last purists.

3) Use exclusively the most recent menstruation without a gap to prepare the model. So for an intraday FX model, nosotros would last using the bars inward the previous week, sans the weekends, to prepare the model. Do non purpose the model for predicting weekend returns nor the root few bars of a week.

This sounds fine, except that at that spot is commonly non plenty information inward only a calendar week to construct a robust model, as well as the resulting model typically suffers from severe information snooping bias.

You mightiness retrieve that it should last possible to concatenate information from multiple gapless periods to shape a larger preparation set. This "concatenation" does non hateful only piecing together multiple weeks' fourth dimension serial into 1 long fourth dimension serial - that would last equivalent to 2) as well as wrong. Concatenation only way that nosotros maximize the full log likelihood of a model over multiple independent fourth dimension series, which inward theory tin last done without much fuss since log likelihood (i.e. log probability) of independent information are additive. But inward practice, most pre-packaged fourth dimension serial model programs create non convey this facility. (Do add together a comment if anyone knows of such a bundle inward Matlab, R, or Python!) Instead of modifying the guts of a likelihood-maximization routine of a fourth dimension serial plumbing equipment package, nosotros volition examine a brusk cutting inward the adjacent proposal.

4) Rather than using a pre-packaged fourth dimension serial model amongst maximum likelihood estimation, only purpose an equivalent multiple linear regression (LR) model. Then only lucifer the preparation information amongst this LR model amongst all the information inward the preparation laid except the weekend bars, as well as purpose it for predicting all futurity bars except the weekend bars as well as the root few bars of a week.

This conversion of a fourth dimension serial model into a LR model is fairly slowly for an autoregressive model AR(p), but may non last possible for an autoregressive moving average model ARMA(p, q). This is because the latter involves a moving average of the residuals, creating a dependency which I don't know how to contain into a LR. But I convey establish that AR(p) model, due to its simplicity, ofttimes industrial plant meliorate out-of-sample than ARMA models anyway. It is of course, real slowly to only omit sure enough information points from a LR fit, equally each information indicate is presumed independent. 

Here is a plot of the out-of-sample cumulative returns of 1 such AR model built for predicting 15-minute returns of NOKSEK, assuming midpoint executions as well as no transaction costs (click to enlarge.)

Whether or non 1 decides to purpose this or the other techniques for treatment information gaps, it is e'er a skillful watch to pay to a greater extent than or less attending to whether a model volition function over these particular bars.

===

My Upcoming Workshop

July 29-30: Artificial Intelligence Techniques for Traders. 

This is a novel online workshop focusing on the practical purpose of AI techniques for identifying predictive indicators for property returns.

===

Managed Accounts Update

Our FX Managed Account programme is 6.02% inward June (YTD: 31.33%).

===

Industry Update

• I previously reported on a cardinal stock model proposed past times Lyle as well as Wang using a linear combination of only 2 work solid fundamentals ― book-to-market ratio as well as provide on equity. Professor Lyle has posted a novel version of this model.

• Charles-Albert Lehalle, Jean-Philippe Bouchaud, as well as Paul Besson reported that "intraday toll is to a greater extent than aligned to signed restrain orders (cumulative club replenishment) rather than signed marketplace seat orders (cumulative club imbalance), fifty-fifty if club imbalance is able to forecast brusk term toll movements." Hat tip: Mattia Manzoni. (I don't convey a link to the master paper: delight inquire Mattia for that!)

• A novel investment contest to tending yous heighten majuscule is available at hedgefol.io.

• Enjoy an Outdoor Summer Party amongst immature human being quants benefiting the New York Firefighters Burn Center Foundation on Tuesday, July 14th amongst groovy nutrient as well as cool drinks on a terrace overlooking Manhattan. Please RSVP to bring together quant fund managers, systematic traders, algorithmic traders, quants as well as high frequency sharks for a groovy evening. This is a complimentary event (donations are welcomed). 

===

Follow me on Twitter: @chanep