What Can I Do About Spikes In My Data?
Whether you call them spikes, glitches, anomalies or data dropouts, these phenomena have been a problem to engineers ever since they started recording data. There are any number of reasons…
Interpolation Versus Resampling To Increase The Sample Rate
These are two different techniques aimed at different objectives. First consider a simple sinewave that has been sampled close to the Nyquist frequency (sample rate/2).
Visually this looks very pointy. We will examine it using a filter based interpolation and a classical curve fitting procedure to obtain a better representation.
Cleaning Up Data
When we have a very noisy signal with a large number of spikes and signal bursts then if all else fails try Median Filtering. This is a technique often used in cleaning up pictures. The operation is almost childishly simple in concept but we will save the details until we have examined an example.
Data Smoothing : RC Filtering And Exponential Averaging
[Updated 12th March 2013]
What are RC Filtering and Exponential Averaging and how do they differ? The answer to the second part of the question is that they are the same process! If one comes from an electronics background then RC Filtering (or RC Smoothing) is the usual expression. On the other hand an approach based on time series statistics has the name Exponential Averaging, or to use the full name Exponential Weighted Moving Average. This is also variously known as EWMA or EMA. (more…)
Spectrum Smoothing : Why and How?
Sometimes data has spikes which are clearly artefacts of the processing or are due to some other external source. One is used to seeing these on time series but in some cases there are unrepresentative “spikes” in the frequency analysed data. Here we discuss how we can use spectrum smoothing to alleviate the problem. An example spectrum is shown below.
Don’t Let Spikes Spoil Your Data
In many real-world applications it is impossible to avoid “spikes” or “dropouts” in data that we record. Many people assume that these only cause problems with their data if they become obvious. This is not always the case. Consider the following two time histories.
Methods To Remove Spikes From Data
For various reasons data captured in the real world often contains spikes that will give erroneous results when analysed. The DATS software package provides various ways of editing and to remove spikes from data. Let us consider a real life case history.
