This edition first published 2019
© 2019 John Wiley & Sons, Inc.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law. Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions.

The right of Ruey S. Tsay and Rong Chen to be identified as the authors of this work has been asserted in accordance with law.

Registered Office
John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA

Editorial Office
John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA

For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com.

Wiley also publishes its books in a variety of electronic formats and by print-on-demand. Some content that appears in standard print versions of this book may not be available in other formats.

Limit of Liability/Disclaimer of Warranty
While the publisher and authors have used their best efforts in preparing this work, they make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives, written sales materials or promotional statements for this work. The fact that an organization, website, or product is referred to in this work as a citation and/or potential source of further information does not mean that the publisher and authors endorse the information or services the organization, website, or product may provide or recommendations it may make. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

Library of Congress Cataloging-in-Publication Data
Names: Tsay, Ruey S., 1951- author. | Chen, Rong, 1963- author.
Title: Nonlinear time series analysis / by Ruey S. Tsay and Rong Chen.
Description: Hoboken, NJ : John Wiley & Sons, 2019. | Series: Wiley series in probability and statistics | Includes index. |
Identifiers: LCCN 2018009385 (print) | LCCN 2018031564 (ebook) | ISBN 9781119264064 (pdf) | ISBN 9781119264071 (epub) | ISBN 9781119264057 (cloth)
Subjects: LCSH: Time-series analysis. | Nonlinear theories.
Classification: LCC QA280 (ebook) | LCC QA280 .T733 2019 (print) | DDC 519.5/5–dc23
LC record available at https://lccn.loc.gov/2018009385

Cover Design: Wiley
Cover Image: Background: © gremlin/iStockphoto;
Graphs: Courtesy of the author Ruey S. Tsay and Rong Chen

CONTENTS

Preface

Chapter 1: Why Should We Care About Nonlinearity?

1.1 Some Basic Concepts
1.2 Linear Time Series
1.3 Examples of Nonlinear Time Series
1.4 Nonlinearity Tests
1. 1.4.1 Nonparametric Tests
2. 1.4.2 Parametric Tests
1.5 Exercises
References

Chapter 2: Univariate Parametric Nonlinear Models

2.1 A General Formulation
1. 2.1.1 Probability Structure
2.2 Threshold Autoregressive Models
1. 2.2.1 A Two-regime TAR Model
2. 2.2.2 Properties of Two-regime TAR(1) Models
3. 2.2.3 Multiple-regime TAR Models
4. 2.2.4 Estimation of TAR Models
5. 2.2.5 TAR Modeling
6. 2.2.6 Examples
7. 2.2.7 Predictions of TAR Models
2.3 Markov Switching Models
1. 2.3.1 Properties of Markov Switching Models
2. 2.3.2 Statistical Inference of the State Variable
  1. 2.3.2.1 Filtering State Probabilities
  2. 2.3.2.2 Smoothing State Probabilities
3. 2.3.3 Estimation of Markov Switching Models
  1. 2.3.3.1 The States are Known
  2. 2.3.3.2 The States are Unknown
  3. 2.3.3.3 Sampling the Unknown Transition Matrix
4. 2.3.4 Selecting the Number of States
5. 2.3.5 Prediction of Markov Switching Models
6. 2.3.6 Examples
2.4 Smooth Transition Autoregressive Models
2.5 Time-varying Coefficient Models
1. 2.5.1 Functional Coefficient AR Models
2. 2.5.2 Time-varying Coefficient AR Models
2.6 Appendix: Markov Chains
2.7 Exercises
References

Chapter 3: Univariate Nonparametric Models

3.1 Kernel Smoothing
3.2 Local Conditional Mean
3.3 Local Polynomial Fitting
3.4 Splines
1. 3.4.1 Cubic and B-Splines
2. 3.4.2 Smoothing Splines
3.5 Wavelet Smoothing
1. 3.5.1 Wavelets
2. 3.5.2 The Wavelet Transform
3. 3.5.3 Thresholding and Smoothing
3.6 Nonlinear Additive Models
3.7 Index Model and Sliced Inverse Regression
3.8 Exercises
References

Chapter 4: Neural Networks, Deep Learning, and Tree-based Methods

4.1 Neural Networks
1. 4.1.1 Estimation or Training of Neural Networks
2. 4.1.2 An Example
4.2 Deep Learning
1. 4.2.1 Deep Belief Nets
2. 4.2.2 Demonstration
4.3 Tree-based Methods
1. 4.3.1 Decision Trees
  1. 4.3.1.1 Regression Tree
  2. 4.3.1.2 Tree Pruning
  3. 4.3.1.3 Classification Tree
  4. 4.3.1.4 Bagging
2. 4.3.2 Random Forests
4.4 Exercises
References

Chapter 5: Analysis of Non-Gaussian Time Series

5.1 Generalized Linear Time Series Models
1. 5.1.1 Count Data and GLARMA Models
5.2 Autoregressive Conditional Mean Models
5.3 Martingalized GARMA Models
5.4 Volatility Models
5.5 Functional Time Series
1. 5.5.1 Convolution FAR models
2. 5.5.2 Estimation of CFAR Models
3. 5.5.3 Fitted Values and Approximate Residuals
4. 5.5.4 Prediction
5. 5.5.5 Asymptotic Properties
6. 5.5.6 Application
5.6 Appendix: Discrete Distributions for Count Data
5.7 Exercises
References

Chapter 6: State Space Models

6.1 A General Model and Statistical Inference
6.2 Selected Examples
1. 6.2.1 Linear Time Series Models
2. 6.2.2 Time Series With Observational Noises
3. 6.2.3 Time-varying Coefficient Models
4. 6.2.4 Target Tracking
5. 6.2.5 Signal Processing in Communications
6. 6.2.6 Dynamic Factor Models
7. 6.2.7 Functional and Distributional Time Series
8. 6.2.8 Markov Regime Switching Models
9. 6.2.9 Stochastic Volatility Models
10. 6.2.10 Non-Gaussian Time Series
11. 6.2.11 Mixed Frequency Models
12. 6.2.12 Other Applications
6.3 Linear Gaussian State Space Models
1. 6.3.1 Filtering and the Kalman Filter
2. 6.3.2 Evaluating the likelihood function
3. 6.3.3 Smoothing
4. 6.3.4 Prediction and Missing Data
5. 6.3.5 Sequential Processing
6. 6.3.6 Examples and R Demonstrations
6.4 Exercises
References

Chapter 7: Nonlinear State Space Models

7.1 Linear and Gaussian Approximations
1. 7.1.1 Kalman Filter for Linear Non-Gaussian Systems
2. 7.1.2 Extended Kalman Filters for Nonlinear Systems
3. 7.1.3 Gaussian Sum Filters
4. 7.1.4 The Unscented Kalman Filter
5. 7.1.5 Ensemble Kalman Filters
6. 7.1.6 Examples and R implementations
7.2 Hidden Markov Models
1. 7.2.1 Filtering
2. 7.2.2 Smoothing
3. 7.2.3 The Most Likely State Path: the Viterbi Algorithm
4. 7.2.4 Parameter Estimation: the Baum–Welch Algorithm
5. 7.2.5 HMM Examples and R Implementation
7.3 Exercises
References

Chapter 8: Sequential Monte Carlo

8.1 A Brief Overview of Monte Carlo Methods
1. 8.1.1 General Methods of Generating Random Samples
2. 8.1.2 Variance Reduction Methods
3. 8.1.3 Importance Sampling
4. 8.1.4 Markov Chain Monte Carlo
8.2 The SMC Framework
8.3 Design Issue I: Propagation
1. 8.3.1 Proposal Distributions
2. 8.3.2 Delay Strategy (Lookahead)
8.4 Design Issue II: Resampling
1. 8.4.1 The Priority Score
2. 8.4.2 Choice of Sampling Methods in Resampling
3. 8.4.3 Resampling Schedule
4. 8.4.4 Benefits of Resampling
8.5 Design Issue III: Inference
8.6 Design Issue IV: Marginalization and the Mixture Kalman Filter
1. 8.6.1 Conditional Dynamic Linear Models
2. 8.6.2 Mixture Kalman Filters
8.7 Smoothing with SMC
1. 8.7.1 Simple Weighting Approach
2. 8.7.2 Weight Marginalization Approach
3. 8.7.3 Two-filter Sampling
8.8 Parameter Estimation with SMC
1. 8.8.1 Maximum Likelihood Estimation
2. 8.8.2 Bayesian Parameter Estimation
3. 8.8.3 Varying Parameter Approach
8.9 Implementation Considerations
8.10 Examples and R Implementation
1. 8.10.1 R Implementation of SMC: Generic SMC and Resampling Methods
  1. 8.10.1.1 Generic R Code for SMC Implementation
  2. 8.10.1.2 R Code for Resampling
2. 8.10.2 Tracking in a Clutter Environment
3. 8.10.3 Bearing-only Tracking with Passive Sonar
4. 8.10.4 Stochastic Volatility Models
5. 8.10.5 Fading Channels as Conditional Dynamic Linear Models
8.11 Exercises
References

Index

EULA

List of Illustrations

Chapter 1

Figure 1.1
Figure 1.2
Figure 1.3
Figure 1.4
Figure 1.5
Figure 1.6
Figure 1.7
Figure 1.8
Figure 1.9
Figure 1.10
Figure 1.11
Figure 1.12
Figure 1.13
Figure 1.14
Figure 1.15
Figure 1.16

Chapter 2

Figure 2.1
Figure 2.2
Figure 2.3
Figure 2.4
Figure 2.5
Figure 2.6
Figure 2.7
Figure 2.8
Figure 2.9
Figure 2.10
Figure 2.11
Figure 2.12
Figure 2.13
Figure 2.14
Figure 2.15
Figure 2.16
Figure 2.17
Figure 2.18
Figure 2.19
Figure 2.20
Figure 2.21
Figure 2.22
Figure 2.23
Figure 2.24
Figure 2.25
Figure 2.26
Figure 2.27
Figure 2.28
Figure 2.29
Figure 2.30

Chapter 3

Figure 3.1
Figure 3.2
Figure 3.3
Figure 3.4
Figure 3.5
Figure 3.6
Figure 3.7
Figure 3.8
Figure 3.9
Figure 3.10
Figure 3.11
Figure 3.12
Figure 3.13
Figure 3.14
Figure 3.15
Figure 3.16
Figure 3.17
Figure 3.18
Figure 3.19
Figure 3.20
Figure 3.21
Figure 3.22
Figure 3.23
Figure 3.24
Figure 3.25
Figure 3.26
Figure 3.27
Figure 3.28

Chapter 4

Figure 4.1
Figure 4.2
Figure 4.3
Figure 4.4
Figure 4.5
Figure 4.6
Figure 4.7
Figure 4.8
Figure 4.9
Figure 4.10
Figure 4.11
Figure 4.12
Figure 4.13
Figure 4.14
Figure 4.15
Figure 4.16
Figure 4.17

Chapter 5

Figure 5.1
Figure 5.2
Figure 5.3
Figure 5.4
Figure 5.5
Figure 5.6
Figure 5.7
Figure 5.8
Figure 5.9
Figure 5.10
Figure 5.11
Figure 5.12
Figure 5.13
Figure 5.14
Figure 5.15
Figure 5.16

Chapter 6

Figure 6.1
Figure 6.2
Figure 6.3
Figure 6.4
Figure 6.5
Figure 6.6
Figure 6.7
Figure 6.8
Figure 6.9
Figure 6.10
Figure 6.11
Figure 6.12
Figure 6.13
Figure 6.14
Figure 6.15
Figure 6.16
Figure 6.17
Figure 6.18
Figure 6.19
Figure 6.20
Figure 6.21
Figure 6.22
Figure 6.23
Figure 6.24
Figure 6.25

Chapter 7

Figure 7.1
Figure 7.2
Figure 7.3
Figure 7.4
Figure 7.5
Figure 7.6
Figure 7.7
Figure 7.8
Figure 7.9
Figure 7.10
Figure 7.11
Figure 7.12
Figure 7.13
Figure 7.14
Figure 7.15
Figure 7.16
Figure 7.17

Chapter 8

Figure 8.1
Figure 8.2
Figure 8.3
Figure 8.4
Figure 8.5
Figure 8.6
Figure 8.7
Figure 8.8
Figure 8.9
Figure 8.10
Figure 8.11
Figure 8.12
Figure 8.13
Figure 8.14
Figure 8.15
Figure 8.16
Figure 8.17
Figure 8.18
Figure 8.19
Figure 8.20
Figure 8.21
Figure 8.22
Figure 8.23
Figure 8.24
Figure 8.25
Figure 8.26
Figure 8.27
Figure 8.28
Figure 8.29
Figure 8.30

Preface

Time series analysis is concerned with understanding the dynamic dependence of real-world phenomena and has a long history. Much of the work in time series analysis focuses on linear models, even though the real world is not linear. One may argue that linear models can provide good approximations in many applications, but there are cases in which a nonlinear model can shed light far beyond where linear models can. The goal of this book is to introduce some simple yet useful nonlinear models, to consider situations in which nonlinear models can make significant contributions, to study basic properties of nonlinear models, and to demonstrate the use of nonlinear models in practice. Real examples from various scientific fields are used throughout the book for demonstration.

The literature on nonlinear time series analysis is enormous. It is too much to expect that a single book can cover all the topics and all recent developments. The topics and models discussed in this book reflect our preferences and personal experience. For the topics discussed, we try to provide a comprehensive treatment. Our emphasis is on application, but important theoretical justifications are also provided. All the demonstrations are carried out using R packages and a companion NTS package for the book has also been developed to facilitate data analysis. In some cases, a command in the NTS package simply provides an interface between the users and a function in another R package. In other cases, we developed commands that make analysis discussed in the book more user friendly. All data sets used in this book are either in the public domain or available from the book’s web page.

The book starts with some examples demonstrating the use of nonlinear time series models and the contributions a nonlinear model can provide. Chapter 1 also discusses various statistics for detecting nonlinearity in an observed time series. We hope that the chapter can convince readers that it is worthwhile pursuing nonlinear modeling in analyzing time series data when nonlinearity is detected. In Chapter 2 we introduce some well-known nonlinear time series models available in the literature. The models discussed include the threshold autoregressive models, the Markov switching models, the smooth transition autoregressive models, and time-varying coefficient models. The process of building those nonlinear models is also addressed. Real examples are used to show the features and applicability of the models introduced. In Chapter 3 we introduce some nonparametric methods and discuss their applications in modeling nonlinear time series. The methods discussed include kernel smoothing, local polynomials, splines, and wavelets. We then consider nonlinear additive models, index models, and sliced inverse regression. Chapter 4 describes neural networks, deep learning, tree-based methods, and random forests. These topics are highly relevant in the current big-data environment, and we illustrate applications of these methods with real examples. In Chapter 5 we discuss methods and models for modeling non-Gaussian time series such as time series of count data, volatility models, and functional time series analysis. Poisson, negative binomial, and double Poisson distributions are used for count data. The chapter extends the idea of generalized linear models to generalized linear autoregressive and moving-average models. For functional time series, we focus on the class of convolution functional autoregressive models and employ sieve estimation with B-splines basis functions to approximate the true underlying convolution functions.

The book then turns to general (nonlinear) state space models (SSMs) in Chapter 6. Several models discussed in the previous chapters become special cases of this general SSM. In addition, some new nonlinear models are introduced under the SSM framework, including targeting tracking, among others. We then discuss methods for filtering, smoothing, prediction, and maximum likelihood estimation of the linear and Gaussian SSM via the Kalman filter. Special attention is paid to the linear Gaussian SSM as it is the foundation for further developments and the model can provide good approximations in many applications. Again, real examples are used to demonstrate various applications of SSMs. Chapter 7 is a continuation of Chapter 6. It introduces various extensions of the Kalman filter, including extended, unscented, and ensemble Kalman filters. The chapter then focuses on hidden Markov models (HMMs) to which the Markov switching model belongs. Filtering and estimation of HMMs are discussed in detail and real examples are used to demonstrate the applications. In Chapter 8 we introduce a general framework of sequential Monte Carlo methods that is designed to analyze nonlinear and non-Gaussian SSM. Some of the methods discussed are also referred to as particle filters in the literature. Implementation issues are discussed in detail and several applications are used for demonstration. We do not discuss multivariate nonlinear time series, even though many of the models and methods discussed can be generalized.

Some exercises are given in each chapter so that readers can practice empirical analysis and learn applications of the models and methods discussed in the book. Most of the exercises use real data so that there exist no true models, but good approximate models can always be found by using the methods discussed in the chapter.

Finally, we would like to express our sincere thanks to our friends, colleagues, and students who helped us in various ways during our research in nonlinear models and in preparing this book. In particular, Xialu Liu provided R code and valuable help in the analysis of convolution functional time series and Chencheng Cai provided R code of optimized parallel implementation of likelihood function evaluation. Daniel Peña provided valuable comments on the original draft. William Gonzalo Rojas and Yimeng Shi read over multiple draft chapters and pointed out various typos. Howell Tong encouraged us in pursuing research in nonlinear time series and K.S. Chan engaged in various discussions over the years. Last but not least, we would like to thank our families for their unconditional support throughout our careers. Their love and encouragement are the main source of our energy and motivation. The book would not have been written without all the support we have received.

The web page of the book is http://faculty.chicagobooth.edu/ruey.tsay/ teaching/nts (for data sets) and www.wiley.com/go/tsay/nonlineartimeseries (for instructors).

R.S.T. Chicago, IL

R.C. Princeton, NJ

November 2017