Deep Learning in Signal Processing

(This quick personal note are written by hints from this online paper.)

Classical or traditional signal processing are usually done by modeling.

• Time domain analysis: Evolution of the signal over time. Common features are like width, heights of the time steps, statistics and etc.
• Frequency or time-frequency domain analysis: In many applications, time variation is more obvious after frequency transformations. Fourier analysis and wavelets are the most common transforms.
• Time warping: Sometime the signal shape shrink or expand in time, then the time scale usually causes the dissimilarity of signals. One well-known solution to compensate time shrink or expansion is called dynamic time warping.
• SARIMAX: Seasonal AutoRegressive Integrated Moving Average with eXogenous regressors model is very useful when there are seasonal variations in time series. ARMA model works fine when the time series does not change seasonal. Extension to ARMA model which has been shown to be successful is SARIMAX. The library statsmodels in Python is a reliable implementation of many of these statistical models. In all statistical models, like all modeling techniques, if the model really fits to the application or in another word, if the actual real world observation has as statistical model, then the interpretation and prediction usually works fine. If the actual real word phenomena does not follow any mathematical model or its model is much more complex to the existing and usable models, then modeling does not give a good solution.
  • Learning resources: Most of the books or resources have some problems, such as too mathematical, just introductory, too comprehensive, too old, relying on R, etc. For example one of the most recommended resources is “Time Series Analysis” by “James D. Hamilton”. This books is very old, too comprehensive and it doesn’t contain recent methods like SARIMAX. It is good as a fundamental textbook. One of the good resources which is quick, precise enough and recent is “Time Series Analysis by State Space Methods” by “J. Durbin and S.J. Koopman”. This book is a graduate level self-reading book which just goes to the interesting topics and it is not very old. The only disadvantages of the book is that it doesn’t contain any implementation for example in Python and the number exercises are not adequate.
• Decomposition to trend, seasonal and residual part is another method to interpret or forecast time series.
• Nonlinear dynamics: This is again a modeling technique, but instead of statistics, they used differential equations; from ordinary to system of partial stochastic differential equations. Also, when the numerical methods are used for solution, this method can be a powerful tool in many applications. Initial and boundary conditions are usually important for differential equations (DE).
• Machine learning (ML): Very complex non-linear multi-input multi-output regressive functions can be built by ML from training data. ML has shown to be very effective when there are enough data for pattern learning or extraction. A field of ML which uses neural network is called Deep Learning (DL).

The fundamental principle in many time series analysis method is that the output of a specific time is not just depend on the current input, but also previous inputs. How long of previous time is application specific.

In DL, recurrent neural network (RNN) and its predecessor convolutional neural network (CNN) are the most common well-known regressive systems that can learn signal patterns.