British Astronomical Association
Supporting amateur astronomers since 1890

Secondary menu

Main menu

Home Forums Spectroscopy
Terms of use

Filtering Spectra

11 posts / 0 new
Last post
andrew.j.smith1905's picture
Online
Last seen: 13 min 33 sec ago
Joined: 02/12/2016 - 12:22
Filtering Spectra

I have been investigating using wavelets and multi spectral analysis to denoise my flare star spectra which are necessarily noisy.

The attached show the results using Matlab Wavelet Analyzer. The residuals show what looks like pure noise showing how effective it can be. This is from the HIP25953 flare on 2019-12-14.

I think it show the potential of cleaning the data before making measurements. Regards Andrew

Xilman's picture
Offline
Last seen: 6 hours 20 min ago
Joined: 24/03/2018 - 15:17
Smoothing

Fascinating!

Have you tried good old Savitzky-Golay smoothing? It was specifically designed for smoothing (and differentiating) spectra. If so, how does it compare with the wavelet approach?

andrew.j.smith1905's picture
Online
Last seen: 13 min 33 sec ago
Joined: 02/12/2016 - 12:22
Smoothing

No as I had not heard about it. I will investigate. What I like about the multi spectral approach is you can clearly see what is happening at the various levels.

Regards Andrew 

Robin Leadbeater's picture
Offline
Last seen: 7 hours 11 min ago
Joined: 05/03/2014 - 00:50
residuals

I like that the residuals follow the expected shape, noisy at the ends of the spectrum where the sensitivity is lower. It gives confidence that you are removing noise, not signal

andrew.j.smith1905's picture
Online
Last seen: 13 min 33 sec ago
Joined: 02/12/2016 - 12:22
Smoothing

Had a look at the S-G smoothing and like many filtering approaches it requires you to set parameters which I can find no obvious method of fixing other than by eyeballing the results.

While you need to set the number of layers in the wavelet multi spectral approach it uses noise modelling to set the thresholds on the filters at each layer. This together with the visualisation makes it simpler for me to assess the results.

It does seem to be the modern approach, see Astronomical Image and Data Analysis  by Starck and Murtagh.

Regards Andrew 

Xilman's picture
Offline
Last seen: 6 hours 20 min ago
Joined: 24/03/2018 - 15:17
S-G

Two standard approaches, one lazy and the other more rigorous.

Lazy: guess the width of the signal peaks and set the smoothing window to be around 3 times that. Use degree 2 or 3 polynomials unless the window is very large (over 20 say) when you could choose degree 4 or 5.

Rigorous: Compute residuals for a range of window widths and polynomial degrees. Perform statistical tests on them and choose the filter which best passes your acceptance requirements of high noise to signal ratio (that way round because you are trying to remove all the signal from the noise).  The Durbin-Watson statistic, which measures autocorrelatIon at lag 1, is a simple test which tends to work well, but a fuller autocorrelation at a variety of lags might be more appropriate.

Wikipedia has good articles on S-G, D-W and several other tests for signal in the presence of noise. Easy to use implementations are all over the place. I tend to use R as it is free and portable, unlike Matlab and Mathematica.

Xilman's picture
Offline
Last seen: 6 hours 20 min ago
Joined: 24/03/2018 - 15:17
Auto-optimizing Savitzky-Golay smoothing

Thanks for prompting me to re-visit the S-G algorithm. As a result I found https://cran.r-project.org/web/packages/ADPF/ADPF.pdf

which describes an implementation which automatically optimizes the polynomial order.  The window width still needs to be chosen by hand.

andrew.j.smith1905's picture
Online
Last seen: 13 min 33 sec ago
Joined: 02/12/2016 - 12:22
S-G

Thanks Paul, I have sent you the data I used in the example as requested by email and I look forward to what you discover. I looked at the Wiki on S-G and will look up D-W.

I am looking for something I can reliably program as I can have a hundred or more spectra per flare to process. The noise profile changes with observing conditions e.g.moon light so as simple and reliable as possible is good.

I used Matlab as I had it from a previous project not tried R but will look at it.

Regards Andrew

andrew.j.smith1905's picture
Online
Last seen: 13 min 33 sec ago
Joined: 02/12/2016 - 12:22
DN Gem

Hi Robin, I had a play with you DN Gem spectra. Filtering confirms the lines you picked out. The noise residuals look good.

Robin Leadbeater's picture
Offline
Last seen: 7 hours 11 min ago
Joined: 05/03/2014 - 00:50
Wavelets in Visual Spec

Sorry Andrew, I missed this

Impressive.  I have just updated my copy of Visual Spec and I see Valerie has added a wavelets feature

andrew.j.smith1905's picture
Online
Last seen: 13 min 33 sec ago
Joined: 02/12/2016 - 12:22
Visual Spec

Yes Robin, I have used it. Valerie, was kind enough to respond to my questions on what some of the numbers were. It gives similar results but you have to watch the extremes of the data set as when you have low frequency wavelets you need a way of managing the boundary. Visual Spec clearly does this in a simple way and you can get artifacts.

Regards Andrew