The Rise of Time-Series Foundation Models for Data Analysis and Forecasting

Time series forecasting plays a vital role in crucial decision-making processes across various industries such as retail, finance, manufacturing, and healthcare. However, compared to domains like natural language processing and image recognition, the integration of advanced artificial intelligence (AI) techniques into time series forecasting has been relatively slow. Although foundational AI has made significant progress in areas like natural language processing and image recognition, its impact on time series forecasting has been limited until recently. Nevertheless, there is now an increasing momentum in the development of foundational models specifically tailored for time series forecasting. This article aims to delve into the evolving landscape of foundational AI for time series forecasting, exploring the recent advancements in this domain. However, before delving into these advancements, let’s briefly introduce time series forecasting and its applications in various industries.

Time Series Forecasting and Applications

Time series data refers to a sequence of data points collected or recorded at regular time intervals. This type of data is prevalent across various domains, such as economics, weather, health, and more. Each data point in a time series is time-stamped, and the sequence is often used to analyze trends, patterns, and seasonal variations over time.

Time series forecasting involves using historical data to predict future values in the series. It is a critical method in statistics and machine learning that helps in making informed decisions based on past patterns. Forecasting can be as simple as projecting the same growth rate into the future or as complex as using AI models to predict future trends based on intricate patterns and external factors.

Some applications of time series forecasting are as follows:

• Financial Markets: In finance, time series forecasting is used to predict stock prices, exchange rates, and market trends. Investors and analysts use historical data to forecast future movements and make trading decisions.
• Weather Forecasting: Meteorological departments use time series data to predict weather conditions. By analyzing past weather data, they forecast future weather patterns, helping in planning and decision-making for agriculture, travel, and disaster management.
• Sales and Marketing: Businesses utilize time series forecasting to predict future sales, demand, and consumer behavior. This helps in inventory management, setting sales targets, and developing marketing strategies.
• Energy Sector: Energy companies forecast demand and supply to optimize production and distribution. Time series forecasting helps in predicting energy consumption patterns, enabling efficient energy management and planning.
• Healthcare: In the healthcare sector, time series forecasting is used to predict disease outbreaks, patient admissions, and medical inventory requirements. This assists in healthcare planning, resource allocation, and policy making.

Foundation Time Series Models

Foundational AI models are extensive, pre-trained models that form the basis for various artificial intelligence applications. They are trained on large and diverse datasets, enabling them to discern patterns, connections, and structures within the data. The term “foundational” refers to their capacity for being fine-tuned or modified for tasks or domains with minimal additional training. In the context of time-series forecasting, these models are constructed similarly to large language models (LLMs), utilizing transformer architectures. Like LLMs, they are trained to predict the subsequent or missing element in a data sequence. However, unlike LLMs, which process text as subwords through transformer layers, foundational time-series models treat sequences of continuous time points as tokens, allowing them to sequentially process time-series data.

Recently, various foundational models have been developed for time series data. With better understanding and choosing the appropriate foundational model, we can more effectively and efficiently leverage their capabilities. In the subsequent sections, we will explore the different foundational models available for time series data analysis.

• TimesFM: Developed by Google Research, TimesFM is a decoder-only foundational model with 200 million parameters. The model is trained on a dataset of 100 billion real-world time points, encompassing both synthetic and actual data from varied sources such as Google Trends and Wikipedia Pageviews. TimesFM is capable of zero-shot forecasting in multiple sectors, including retail, finance, manufacturing, healthcare, and the natural sciences, across different time granularities. Google intends to release TimesFM on its Google Cloud Vertex AI platform, providing its sophisticated forecasting features to external clients.
• Lag-Llama: Created by researchers from the Université de Montréal, Mila-Québec AI Institute, and McGill University, Lag-Llama is a foundational model designed for univariate probabilistic time series forecasting. Build on the foundation of Llama, the model employs a decoder-only transformer architecture which uses variable sizes time lags and time resolutions for forecasting. The model is trained on diverse time series datasets from several sources across six different groups including energy, transportation, economics, nature, air quality and cloud operations. The model is conveniently accessible through the Huggingface library.
• Moirai: Developed by Salesforce AI Research, Moirai is a foundational time series model designed for universal forecasting. Moirai is trained on the Large-scale Open Time Series Archive (LOTSA) dataset, which contains 27 billion observations from nine distinct domains, making it the largest collection of open time series datasets. This diverse dataset allows Moirai to learn from a wide range of time series data, enabling it to handle different forecasting tasks. Moirai uses multiple patch size projection layers to capture temporal patterns across various frequencies. An important aspect of Moirai is to use any-variate attention mechanism, allowing forecasts across any number of variables. The code, model weights, and data associated with Moirai are available in the GitHub repository called “uni2ts“

• Chronos: Developed by Amazon, Chronos is a collection of pre-trained probabilistic models for time series forecasting. Built on the T5 transformer architecture, the models use a vocabulary of 4096 tokens and have varying parameters, ranging from 8 million to 710 million. Chronos is pretrained on a vast array of public and synthetic data generated from Gaussian processes. Chronos differs from TimesFM in that it is an encoder-decoder model, which enables the extraction of encoder embeddings from time series data. Chronos can be easily integrated into a Python environment and accessed via its API.

• Moment: Developed collaboratively by Carnegie Mellon University and the University of Pennsylvania, Moment is a family of open-source foundational time series models. It utilizes variations of T5 architectures, including small, base, and large versions, with the base model incorporating approximately 125 million parameters. The model undergoes pre-training on the extensive “Time-series Pile,” a diverse collection of public time-series data spanning various domains. Unlike many other foundational models, MOMENT is pre-trained on a wide spectrum of tasks, enhancing its effectiveness in applications such as forecasting, classification, anomaly detection, and imputation. The complete Python repository and Jupyter notebook code are publicly accessible for utilizing the model.

The Bottom Line

Time series forecasting is a crucial tool across various domains, from finance to healthcare, enabling informed decision-making based on historical patterns. Advanced foundational models like TimesFM, Chronos, Moment, Lag-Llama, and Moirai offer sophisticated capabilities, leveraging transformer architectures and diverse training datasets for accurate forecasting and analysis. These models provide a glimpse into the future of time series analysis, empowering businesses and researchers with powerful tools to navigate complex data landscapes effectively.