How do I remove spikes from data?

For various reasons data captured in the real world often contains spikes that will give erroneous results when analysed. The DATS software package provides various ways of editing and to remove spikes from data. Let us consider a real life case history.

despik1 — Fig. 1 : A sine wave with two obvious spikes before and after ‘de-spiking’

A large aerospace manufacturer gathers data from flight tests. They collect up to thirty channels of data and a test run can last for several hours. The resulting datasets are several megabytes long. Unfortunately, the method of recording the data and the noisy environment of an aircraft mean that the data often contains spikes.

despik2 — Fig. 2 : A slightly more subtle example of a pair of spikes in some narrow-band random data

Fortunately for the user, DATS provides methods of removing these spikes. First, and most obvious, would be to simply view the data, identify the spikes and edit the data values by hand. This is easily achieved in DATS by identifying a spike, switching to ‘Table view’, double clicking on the offending value and simply typing a new value. Another manual method would be to graphically edit with a linear or haversine replacement.

Manual methods are good for a small amount of data and a small number of spikes. However, our user has several megabytes of data and an unknown number of spikes. To modify this data by hand would be very time consuming, not to mention very boring and error prone.

For a more automated approach DATS has a purpose written ‘de-spike’ analysis. This analysis uses a sophisticated algorithm to search for and replace spikes. All the user has to do is load in the raw data, run the ‘de-spike’ analysis from the DATS Analysis menu and, ‘Hey, Presto’ the spikes are gone. The data can now be analysed in the normal way.

To further automate the process the user can then create a DATS worksheet to run the ‘de-spike’ as part of their normal analysis.

Bio
Latest Posts

Dr Colin Mercer

Founder / Chief Signal Processing Analyst (Retired) at Prosig

Dr Colin Mercer was formerly at the Institute of Sound and Vibration Research (ISVR), University of Southampton where he founded the Data Analysis Centre. He then went on to found Prosig in 1977. Colin retired as Chief Signal Processing Analyst at Prosig in December 2016. He is a Chartered Engineer and a Fellow of the British Computer Society.