The file drawer problem
Most tested trading ideas fail. The problem is that the failures rarely get written down. They go in a drawer, and only the survivors are shown, which makes the survivors look far better than they are.
Academic research has the same issue. McLean and Pontiff (Journal of Finance, 2016) found that many published stock market patterns weakened after they were published, part of it because some had been found by searching rather than by a real effect. Keeping the failures visible is the simplest guard against fooling yourself.
What is in the research log
Every study goes into the research log, newest first. Each one carries a plain-language question, what was tested, the result and the caveats, plus one of three labels:
- Verified: held up on later, unseen data. It is not a claim that the idea will make money.
- Verified (no edge): tested cleanly, no usable effect.
- Failed / retracted: did not hold up, or was withdrawn, with the reason.
Three retractions, in plain words
These are already public on the research page:
- Machine-learning stock rankings. An early version let future data leak into the test, a look-ahead bug. The study was retracted and superseded by corrected runs.
- Completed chart patterns over one to six months. The apparent result came from a handful of extreme winners, not a broad effect. Withdrawn as a skew artifact.
- The old system's chart-pattern alerts. Compared with a fair placebo, the alerts showed no edge, so the earlier headline result was replaced. It stays in the log so the earlier claim can be checked.
Why most backtested strategies fail
- Too many tries. Test enough versions of an idea and one will look good by chance. SageFusion discounts the best version for the number of tries before it is labelled.
- Look-ahead bias. Using data that was not known at the time. Tests use the date data was actually recorded.
- Survivorship bias. Using only companies that still exist. Each affected study says so.
- Costs. Many small edges vanish after commissions and slippage, so score studies are judged after trading costs.
- No fair comparison. Making money in a rising market is not the test. Our guide to testing a signal against random explains why.
Who the log is for
For investors, the log is a way to see how an idea was tested before trusting it. For research teams at firms, it is a way to audit the method, failures included, before licensing any data. See stock research data that shows what failed.