The Knowable Future
Essential Reading: Forecastability and the Information Theory of Prediction
1. Foundations of predictive information and limits
Kolmogoroff, A. (1941). "Interpolation und Extrapolation von stationären zufälligen Folgen." Izvestiya Akademii Nauk SSSR, Seriya Matematicheskaya, 5(1), 3–14.
Shannon, C. E. (1948). "A mathematical theory of communication." Bell System Technical Journal, 27(3), 379–423; 27(4), 623–656.
Wiener, N. (1949). Extrapolation, Interpolation, and Smoothing of Stationary Time Series, with Engineering Applications. The Technology Press of MIT and John Wiley & Sons.
Jaynes, E. T. (1957). "Information theory and statistical mechanics." Physical Review, 106(4), 620–630.
Khinchin, A. Ya. (1957). Mathematical Foundations of Information Theory. Dover Publications.
Rényi, A. (1961). "On measures of entropy and information." In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1 (pp. 547–561). University of California Press.
Fano, R. M. (1961). Transmission of Information: A Statistical Theory of Communications. MIT Press.
Solomonoff, R. J. (1964). "A formal theory of inductive inference. Part I." Information and Control, 7(1), 1–22.
Solomonoff, R. J. (1964). "A formal theory of inductive inference. Part II." Information and Control, 7(2), 224–254.
Kolmogorov, A. N. (1965). "Three approaches to the quantitative definition of information." Problems of Information Transmission, 1(1), 1–7.
Martin-Löf, P. (1966). "The definition of random sequences." Information and Control, 9(6), 602–619.
Chaitin, G. J. (1966). "On the length of programs for computing finite binary sequences." Journal of the ACM, 13(4), 547–569.
Akaike, H. (1974). "A new look at the statistical model identification." IEEE Transactions on Automatic Control, 19(6), 716–723.
Rissanen, J. (1978). "Modeling by shortest data description." Automatica, 14(5), 465–471.
Schwarz, G. (1978). "Estimating the dimension of a model." The Annals of Statistics, 6(2), 461–464.
Rissanen, J. (1989). Stochastic Complexity in Statistical Inquiry. World Scientific.
Cover, T. M. & Thomas, J. A. (2006). Elements of Information Theory (2nd ed.). Wiley-Interscience.
Grünwald, P. D. (2007). The Minimum Description Length Principle. MIT Press.
Li, M. & Vitányi, P. (2019). An Introduction to Kolmogorov Complexity and Its Applications (4th ed.). Springer.
2. Predictability in dynamical and physical systems
Kolmogorov, A. N. (1958). "A new metric invariant of transitive dynamical systems and automorphisms of Lebesgue spaces." Doklady Akademii Nauk SSSR, 119(5), 861–864.
Sinai, Ya. G. (1959). "On the notion of entropy of a dynamical system." Doklady Akademii Nauk SSSR, 124(4), 768–771.
Lorenz, E. N. (1963). "Deterministic nonperiodic flow." Journal of the Atmospheric Sciences, 20(2), 130–141.
Lorenz, E. N. (1969). "The predictability of a flow which possesses many scales of motion." Tellus, 21(3), 289–307.
Pesin, Ya. B. (1977). "Characteristic Lyapunov exponents and smooth ergodic theory." Russian Mathematical Surveys, 32(4), 55–114.
Packard, N. H., Crutchfield, J. P., Farmer, J. D. & Shaw, R. S. (1980). "Geometry from a time series." Physical Review Letters, 45(9), 712–716.
Takens, F. (1981). "Detecting strange attractors in turbulence." In D. Rand & L.-S. Young (Eds.), Dynamical Systems and Turbulence, Warwick 1980 (Lecture Notes in Mathematics, Vol. 898, pp. 366–381). Springer.
Brudno, A. A. (1983). "Entropy and the complexity of the trajectories of a dynamical system." Transactions of the Moscow Mathematical Society, 44, 127–151.
Grassberger, P. & Procaccia, I. (1983). "Measuring the strangeness of strange attractors." Physica D: Nonlinear Phenomena, 9(1–2), 189–208.
Wolf, A., Swift, J. B., Swinney, H. L. & Vastano, J. A. (1985). "Determining Lyapunov exponents from a time series." Physica D: Nonlinear Phenomena, 16(3), 285–317.
Fraser, A. M. & Swinney, H. L. (1986). "Independent coordinates for strange attractors from mutual information." Physical Review A, 33(2), 1134–1140.
Sugihara, G. & May, R. M. (1990). "Nonlinear forecasting as a way of distinguishing chaos from measurement error in time series." Nature, 344(6268), 734–741.
Boffetta, G., Cencini, M., Falcioni, M. & Vulpiani, A. (2002). "Predictability: a way to characterize complexity." Physics Reports, 356(6), 367–474.
DelSole, T. (2004). "Predictability and information theory. Part I: Measures of predictability." Journal of the Atmospheric Sciences, 61(20), 2425–2440.
DelSole, T. & Tippett, M. K. (2007). "Predictability: recent insights from information theory." Reviews of Geophysics, 45(4), RG4002.
DelSole, T. & Tippett, M. K. (2009). "Average predictability time. Part I: Theory." Journal of the Atmospheric Sciences, 66(5), 1172–1187.
Pennekamp, F., et al. (2019). "The intrinsic predictability of ecological time series and its potential to guide forecasting." Ecological Monographs, 89(2), e01359.
3. Information dynamics in time series
Grassberger, P. (1986). "Toward a quantitative theory of self-generated complexity." International Journal of Theoretical Physics, 25(9), 907–938.
Crutchfield, J. P. & Young, K. (1989). "Inferring statistical complexity." Physical Review Letters, 63(2), 105–108.
Massey, J. L. (1990). "Causality, feedback and directed information." Proceedings of the 1990 International Symposium on Information Theory and Its Applications (ISITA-90), 303–305.
Pincus, S. M. (1991). "Approximate entropy as a measure of system complexity." Proceedings of the National Academy of Sciences, 88(6), 2297–2301.
Feder, M., Merhav, N. & Gutman, M. (1992). "Universal prediction of individual sequences." IEEE Transactions on Information Theory, 38(4), 1258–1270.
Tishby, N., Pereira, F. C. & Bialek, W. (1999). "The information bottleneck method." Proceedings of the 37th Annual Allerton Conference on Communication, Control and Computing, 368–377.
Richman, J. S. & Moorman, J. R. (2000). "Physiological time-series analysis using approximate entropy and sample entropy." American Journal of Physiology–Heart and Circulatory Physiology, 278(6), H2039–H2049.
Schreiber, T. (2000). "Measuring information transfer." Physical Review Letters, 85(2), 461–464.
Bialek, W., Nemenman, I. & Tishby, N. (2001). "Predictability, complexity, and learning." Neural Computation, 13(11), 2409–2463.
Bandt, C. & Pompe, B. (2002). "Permutation entropy: a natural complexity measure for time series." Physical Review Letters, 88(17), 174102.
Costa, M., Goldberger, A. L. & Peng, C.-K. (2002). "Multiscale entropy analysis of complex physiologic time series." Physical Review Letters, 89(6), 068102.
Crutchfield, J. P. & Feldman, D. P. (2003). "Regularities unseen, randomness observed: levels of entropy convergence." Chaos, 13(1), 25–54.
Barnett, L., Barrett, A. B. & Seth, A. K. (2009). "Granger causality and transfer entropy are equivalent for Gaussian variables." Physical Review Letters, 103(23), 238701.
Song, C., Qu, Z., Blumm, N. & Barabási, A.-L. (2010). "Limits of predictability in human mobility." Science, 327(5968), 1018–1021.
4. Estimation of entropy and mutual information
Ziv, J. & Lempel, A. (1977). "A universal algorithm for sequential data compression." IEEE Transactions on Information Theory, 23(3), 337–343.
Ziv, J. & Lempel, A. (1978). "Compression of individual sequences via variable-rate coding." IEEE Transactions on Information Theory, 24(5), 530–536.
Kaspar, F. & Schuster, H. G. (1987). "Easily calculable measure for the complexity of spatiotemporal patterns." Physical Review A, 36(2), 842–848.
Kozachenko, L. F. & Leonenko, N. N. (1987). "Sample estimate of the entropy of a random vector." Problems of Information Transmission, 23(2), 95–101.
Grassberger, P. (1988). "Finite sample corrections to entropy and dimension estimates." Physics Letters A, 128(6–7), 369–373.
Kontoyiannis, I., Algoet, P. H., Suhov, Yu. M. & Wyner, A. J. (1998). "Nonparametric entropy estimation for stationary processes and random fields, with applications to English text." IEEE Transactions on Information Theory, 44(3), 1319–1327.
Darbellay, G. A. & Vajda, I. (1999). "Estimation of the information by an adaptive partitioning of the observation space." IEEE Transactions on Information Theory, 45(4), 1315–1321.
Nemenman, I., Shafee, F. & Bialek, W. (2002). "Entropy and inference, revisited." In Advances in Neural Information Processing Systems 14 (pp. 471–478). MIT Press.
Paninski, L. (2003). "Estimation of entropy and mutual information." Neural Computation, 15(6), 1191–1253.
Kraskov, A., Stögbauer, H. & Grassberger, P. (2004). "Estimating mutual information." Physical Review E, 69(6), 066138. Erratum: Physical Review E, 83(1), 019903 (2011).
Barnett, L. & Seth, A. K. (2014). "The MVGC multivariate Granger causality toolbox: a new approach to Granger-causal inference." Journal of Neuroscience Methods, 223, 50–68.
Wibral, M., Vicente, R. & Lindner, M. (2014). "Transfer entropy in neuroscience." In M. Wibral, R. Vicente & J. T. Lizier (Eds.), Directed Information Measures in Neuroscience (pp. 3–36). Springer.
Gao, S., Ver Steeg, G. & Galstyan, A. (2015). "Efficient estimation of mutual information for strongly dependent variables." Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 38, 277–286.
Belghazi, M. I., Baratin, A., Rajeswar, S., Ozair, S., Bengio, Y., Courville, A. & Hjelm, R. D. (2018). "Mutual information neural estimation." Proceedings of the 35th International Conference on Machine Learning (ICML), PMLR 80, 531–540.
Holmes, C. M. & Nemenman, I. (2019). "Estimation of mutual information for real-valued data with error bars and controlled bias." Physical Review E, 100(2), 022404.
Poole, B., Ozair, S., van den Oord, A., Alemi, A. A. & Tucker, G. (2019). "On variational bounds of mutual information." Proceedings of the 36th International Conference on Machine Learning (ICML), PMLR 97, 5171–5180.
McAllester, D. & Stratos, K. (2020). "Formal limitations on the measurement of mutual information." Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 108, 875–884.
Czyż, P., Grabowski, F., Vogt, J. E., Beerenwinkel, N. & Marx, A. (2023). "Beyond normal: on the evaluation of mutual information estimators." Advances in Neural Information Processing Systems 36 (NeurIPS).
5. Forecastability: prior definitions and operational measures
Catt, P. M. (2009). "Forecastability: insights from physics, graphical decomposition, and information theory." Foresight: The International Journal of Applied Forecasting, Issue 13, 24–33.
Boylan, J. E. (2009). "Toward a more precise definition of forecastability." Foresight: The International Journal of Applied Forecasting, Issue 13, 34–40.
Goerg, G. M. (2013). "Forecastable component analysis." Proceedings of the 30th International Conference on Machine Learning (ICML), PMLR 28(2), 64–72.
Catt, P. M. (2014). "Entropy as an a priori indicator of forecastability." SSRN Working Paper 6235738.
Hill, A. V., Zhang, W. & Burch, G. F. (2015). "Forecasting the forecastability quotient for inventory management." International Journal of Forecasting, 31(3), 651–663.
Gardner, E. S., Jr. & Acar, Y. (2016). "The forecastability quotient reconsidered." International Journal of Forecasting, 32(4), 1208–1211.
Fulcher, B. D. & Jones, N. S. (2017). "hctsa: a computational framework for automated time-series phenotyping using massive feature extraction." Cell Systems, 5(5), 527–531.e3.
Kang, Y., Hyndman, R. J. & Smith-Miles, K. (2017). "Visualising forecasting algorithm performance using time series instance spaces." International Journal of Forecasting, 33(2), 345–358.
Lubba, C. H., Sethi, S. S., Knaute, P., Schultz, S. R., Fulcher, B. D. & Jones, N. S. (2019). "catch22: CAnonical Time-series CHaracteristics: selected through highly comparative time-series analysis." Data Mining and Knowledge Discovery, 33(6), 1821–1852.
Amigo, G., Díaz-Pachón, D. A., Marks, R. J. & Baylis, C. (2023). "Algorithmic information forecastability." arXiv preprint arXiv:2304.10752.
Wang, R., Klee, S. & Roos, A. (2025). "Time series forecastability measures." arXiv preprint arXiv:2507.13556.
Catt, P. M. (2026). "Forecastability profiles: horizon-specific limits to prediction before model selection." Research Square preprint; under review, Discover Analytics.
Catt, P. M. (2026). "Horizon-resolved forecastability of time series via auto mutual information." Submitted, Scientific Reports; preprint arXiv:2601.10006.
Li, Q., Zhang, X., Wang, S., Peng, T. & Wei, J. (2026). "FAME: forecastability-aware mixture of experts for heterogeneous time series forecasting." arXiv preprint arXiv:2606.08896.
6. Benchmarks and forecast evaluation
Hyndman, R. J. & Koehler, A. B. (2006). "Another look at measures of forecast accuracy." International Journal of Forecasting, 22(4), 679–688.
Makridakis, S., Spiliotis, E. & Assimakopoulos, V. (2020). "The M4 Competition: 100,000 time series and 61 forecasting methods." International Journal of Forecasting, 36(1), 54–74.
Godahewa, R., Bergmeir, C., Webb, G. I., Hyndman, R. J. & Montero-Manso, P. (2021). "Monash time series forecasting archive." Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1.
Petropoulos, F., et al. (2022). "Forecasting: theory and practice." International Journal of Forecasting, 38(3), 705–871.
Hewamalage, H., Ackermann, K. & Bergmeir, C. (2023). "Forecast evaluation for data scientists: common pitfalls and best practices." Data Mining and Knowledge Discovery, 37(2), 788–832.
​