To demonstrate this, let's interruption upwards the dataset over two periods: 20010522 - 20030123 together with 20030124 - 20070403. In the kickoff in-sample stream (with 1,000 information points), nosotros alternative our 10 stocks to course of report the basket, together with inwards the minute out-of-sample stream nosotros come across how good it cointegrates amongst XLE, together with nosotros uncovering how the spread behaves. I flora that inwards the kickoff period, the t-statistic for cointegration is -3.61934140, indicating the handbasket cointegrates amongst over 95% probability. No surprise here. Here is a plot of the spread inwards this period:

Now, let's uncovering out what happens inwards the out-of-sample period. Here the t-statistic is but -2.72, whereas the critical value for cointegration at 90% probability is -3.03. So indeed the handbasket fails to cointegrate at the 90% confidence level. Does that hateful our trades volition thus endure losing out-of-sample? Not necessarily. Take a hold off at the behaviour of the spread out-of-sample:

Even though it is non nicely symmetric some zip equally inwards the in-sample period, the spread is however clearly bounded some zero. If the handbasket completely falls out of cointegration amongst XLE, it volition demo a random drift away from zip equally fourth dimension goes on.
To demo that this is non but skillful luck based on our specific in-sample period, let's endeavor a longer in-sample stream of 1500 days (shorter in-sample stream won't work, because nosotros demand a minimum of 1,000 information points hither to build a skillful reliable basket.) Here the cointegration t-statistic is a chip worse, at -2.62. If nosotros hold off at the spread:

Once again, nosotros come across that the spread is bounded, non wandering off to infinity. So inwards conclusion, I keep that my method of constructing the handbasket is skillful for practical trading, though non necessarily guaranteeing equally high a statistical confidence marking equally mightiness endure indicated inwards the in-sample period.