You are currently viewing Methods To Remove Spikes From Data

Methods To Remove Spikes From Data

For various reasons data captured in the real world often contains spikes that will give erroneous results when analysed. The DATS software package provides various ways of editing and to remove spikes from data. Let us consider a real life case history.

Fig. 1 : A sine wave with two obvious spikes before and after ‘de-spiking’

A large aerospace manufacturer gathers data from flight tests. They collect up to thirty channels of data and a test run can last for several hours. The resulting datasets are several megabytes long. Unfortunately, the method of recording the data and the noisy environment of an aircraft mean that the data often contains spikes.

Fig. 2 : A slightly more subtle example of a pair of spikes in some narrow-band random data

Fortunately for the user, DATS provides methods of removing these spikes. First, and most obvious, would be to simply view the data, identify the spikes and edit the data values by hand. This is easily achieved in DATS by identifying a spike, switching to ‘Table view’, double clicking on the offending value and simply typing a new value. Another manual method would be to graphically edit with a linear or haversine replacement.

Manual methods are good for a small amount of data and a small number of spikes. However, our user has several megabytes of data and an unknown number of spikes. To modify this data by hand would be very time consuming, not to mention very boring and error prone.

For a more automated approach DATS has a purpose written ‘de-spike’ analysis. This analysis uses a sophisticated algorithm to search for and replace spikes. All the user has to do is load in the raw data, run the ‘de-spike’ analysis from the DATS Analysis menu and, ‘Hey, Presto’ the spikes are gone. The data can now be analysed in the normal way.

To further automate the process the user can then create a DATS worksheet to run the ‘de-spike’ as part of their normal analysis.

The following two tabs change content below.

Dr Colin Mercer

Founder / Chief Signal Processing Analyst (Retired) at Prosig
Dr Colin Mercer was formerly at the Institute of Sound and Vibration Research (ISVR), University of Southampton where he founded the Data Analysis Centre. He then went on to found Prosig in 1977. Colin was MD and Chairman of Prosig for many years. He retired as Chief Signal Processing Analyst at Prosig in December 2016. He is a Chartered Engineer and a Fellow of the British Computer Society.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.