British Astronomical Association
Supporting amateur astronomers since 1890

Secondary menu

Main menu

Home Forums Variable Stars
Terms of use

Introducing MetroPSF - a program for ensemble photometry

36 posts / 0 new
Last post
maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
Introducing MetroPSF - a program for ensemble photometry

Hi All,

I would like to share a small program I wrote, currently called MetroPSF for the lack of better name, for differential ensemble PSF-fitting photometry. Initially wrote a small script to double-check the results I was getting through Astrometrica for the SN 2021gmj, however, thought it may be useful for public, so releasing the source code. It's a prototype, so the code quality is pretty low, but the results it produces should be ready for analysis. I would appreciate any feedback and suggestions. I thought I'd check if this is of interest to public at all by making a preliminary "alpha" release.

The READ ME file with a brief description and installation instructions as well as the latest version of the program can be found here:

http://trafyx.com/?p=2669

The installation process may seem a bit strange, but it all boils down to installing Python interpreter which is pretty much automatic process, and using pip to install libraries, which is also automatic. This takes a few minutes on both Windows and Unix-like platforms.

MetroPSF works with VizieR and Astrometry.net for retrieving catalogs and solving, and the READ ME file describes a quick use case to familiarize with the interface. The screen shot of the program in action is below - on NGC 3310 with SN 2021gmj.

Maxim

astroshot's picture
Offline
Last seen: 21 hours 19 min ago
Joined: 11/09/2014 - 21:37
installation error

Hi Maxim,

I downloaded the program, extracted it to a directory on my C:\ drive and installed Python

(I admit to not knowing anything about Python)

However, I'm unclear how to run "pip install -r requirements.txt"

Is that via the command prompt? It just gave an error 'pip is not recognised...'

Regards,

Michael.

astroshot's picture
Offline
Last seen: 21 hours 19 min ago
Joined: 11/09/2014 - 21:37
installation error

Ignore that. Got it going now.

Even though I installed python, it didn't seem to register properly as a program.

I had to go to the Microsoft store to download and install it from there.

Strange!

nickjames's picture
Offline
Last seen: 21 min 22 sec ago
Joined: 12/10/2013 - 18:16
I get the same problem trying

I get the same problem trying to install 3.9.4 direct from the Python website into a fresh Win 10 20H2 instance in a VM. Trying to run pip from Powershell gives a "command not recognised" error. Installing from the MS Store worked. The rest of the install then ran smoothly although it did moan about 'wheel' not being installed as per the attached screenshot and has various warnings (see screenshot). I'm assuming these are benign. I'll try out the program now.

Attachments: 
maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
Michael,

Michael,

Glad it works. Windows is doing these constant changes - head goes round. Any feedback is much appreciated. Let me know if you'll have any questions. Version 0.12 is already in the making with the ability to subtract VSX stars from the ensemble and report VSX sources nearby.

Maxim

astroshot's picture
Offline
Last seen: 21 hours 19 min ago
Joined: 11/09/2014 - 21:37
not opening images

I can get the program to run now, but when I open an image, only the FITS header loads in the lower panel - the image itself does not load.

I note that the Command Prompt window states the following:

WARNING: Astrometry.net API key not found in configuration file [astroquery.astrometry_net.core]

WARNING: You need to manually edit the configuration file and add it [astroquery.astrometry_net.core]

WARNING: You may also register it for this session with AstrometryNet.key = 'XXXXXXXX' [astroquery.astrometry_net.core]

Exception in Tkinter callback

Traceback (most recent call last):

  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1264.0_x64__qbz5n2kfra8p0\lib\tkinter\__init__.py", line 1892, in __call__

    return self.func(*args)

  File "C:\metropsf\metropsf.py", line 593, in update_histogram_high

    self.update_display()

  File "C:\metropsf\metropsf.py", line 502, in update_display

    self.display_image()

  File "C:\metropsf\metropsf.py", line 91, in display_image

    generate_FITS_thumbnail(self.histogram_slider_low, self.histogram_slider_high, self.zoom_level)

  File "C:\metropsf\metropsf.py", line 57, in generate_FITS_thumbnail

    generated_image = Image.fromarray(image_data)

NameError: name 'image_data' is not defined

maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
Very strange. If you could

Very strange. If you could send me the FITS to maxim.usatov at bcsatellite.net I could take a look what's up with it. So far it did load all FITS files fine on both of my FreeBSD and Windows computers coming from different remote telescopes.

nickjames's picture
Offline
Last seen: 21 min 22 sec ago
Joined: 12/10/2013 - 18:16
Yes, I get the same on Win 10

Yes, I get the same on Win 10. FITS data but no image. See the attached screenshot. FITS file is attached. Note that it is a gzip compressed fits but I can only attach files with an extension .fits so you may need to change that to fit.gz to get it to work.

nickjames's picture
Offline
Last seen: 21 min 22 sec ago
Joined: 12/10/2013 - 18:16
I've just converted the FITS

I've just converted the FITS that I was using from int16 to float32 and it now works. I have an image. I'll play around a bit more.

astroshot's picture
Offline
Last seen: 21 hours 19 min ago
Joined: 11/09/2014 - 21:37
I've just converted the FITS

Yes, that works for me now as well. Thanks Nick!

@Maxim: Is there any chance we would get this program to open 8-fits files directly please?

I have a little project in mind involving images from the Global Meteor Network pi camera.

Thanks,

Michael.

maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
Sorry, for some reason

Sorry, for some reason replied in the wrong thread. This version should open Nick's FITS fine:

maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
Nick, Michael, thank you very

Nick, Michael, thank you very much. This one should open Nick's FITS fine:

http://trafyx.com/wp-content/uploads/2021/04/metropsf-0.12.zip

nickjames's picture
Offline
Last seen: 21 min 22 sec ago
Joined: 12/10/2013 - 18:16
Maxim - Thanks that works

Maxim - Thanks that works fine, see attachment. I'll have a look through your documentation later and will do some experiments on known fields.

Attachments: 
maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
Nick, great! Please don't

Nick, great! Please don't hesitate to submit feature requests. I am trying to feel where to take this next. Perhaps, command-line processing... Definitely need a way to save all the settings.

astroshot's picture
Offline
Last seen: 21 hours 19 min ago
Joined: 11/09/2014 - 21:37
error

To clarify my above point, I get a long error message just when the program opens.

The error is contained in a separate window to the main application.

Also occurs with V0.12 shared above.

Anyone else get it?

WARNING: Astrometry.net API key not found in configuration file [astroquery.astrometry_net.core]

WARNING: You need to manually edit the configuration file and add it [astroquery.astrometry_net.core]

WARNING: You may also register it for this session with AstrometryNet.key = 'XXXXXXXX' [astroquery.astrometry_net.core]

Exception in Tkinter callback

Traceback (most recent call last):

  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1264.0_x64__qbz5n2kfra8p0\lib\tkinter\__init__.py", line 1892, in __call__

    return self.func(*args)

  File "C:\metropsf\metropsf.py", line 665, in update_histogram_high

    self.update_display()

  File "C:\metropsf\metropsf.py", line 560, in update_display

    self.display_image()

  File "C:\metropsf\metropsf.py", line 93, in display_image

    generate_FITS_thumbnail(self.histogram_slider_low, self.histogram_slider_high, self.zoom_level)

  File "C:\metropsf\metropsf.py", line 56, in generate_FITS_thumbnail

    converted_data = image_data.astype(float)

NameError: name 'image_data' is not defined

maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
For the time being just

For the time being just ignore those. I'll clean things up soon. As long as the program itself works, it shouldn't impact on anything.

nickjames's picture
Offline
Last seen: 21 min 22 sec ago
Joined: 12/10/2013 - 18:16
Michael - Yes, it seems to

Michael - Yes, it seems to work fine without the astrometry.net key although my files are already platesolved. Here's an example of the V1391 Cas field with the fit using V.

Maxim - Could you add Gaia DR2 G to your list of catalogues? Also, the image contrast stretch doesn't seem to work very well for me.  

Attachments: 
maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
Nick, here is 0.13 with Gaia

Nick, here is 0.13 with Gaia DR2 and non-linear image stretching for better views:
http://trafyx.com/wp-content/uploads/2021/04/metropsf-0.13.zip
Updated user guide PDF inside. Please let me know if you need anything else.

Maxim

nickjames's picture
Offline
Last seen: 21 min 22 sec ago
Joined: 12/10/2013 - 18:16
Thanks. I get an error "630

Thanks. I get an error "630 match_mag" when computing the regression model. now.

maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
Nick, looks like it cannot

Nick, looks like it cannot find the magnitudes of comparison stars in the ensemble. I've just went all the way - opening a 16-bit FITS file (ugc7513_000.fits), doing Iteratively Subtracted PSF Photometry, then set catalog to Gaia DR2, CCD Filter to G manually, ran Get Comparison Stars, then Find Regression Model and finally was able to get differential magnitudes by mouse-clicking on the sources. Try removing the old .phot file and executing the same steps. I'm going out of town for a couple of days but hopefully will be able to assist with delays.

Max

nickjames's picture
Offline
Last seen: 21 min 22 sec ago
Joined: 12/10/2013 - 18:16
Maxim - Thanks, I'm almost

Maxim - Thanks, I'm almost there. I had forgotten the "Get comparison stars" step! Now I get a fit. The only problem remaining is when I click on the object the flux/magnitude etc are all zero. What am I doing wrong?

One other question - Am I right that the ensemble fit graph is upside down, i.e. the brightest magnitudes are in the lower left corner. Not a problem but I would just like to understand.  

Attachments: 
maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
Nick, do you mind sending me

Nick, do you mind sending me the FITS file with this data? You are likely clicking on sources that didn't have any photometry fits at all, i.e. appearing without circles surrounding them. This means that the IRAFStarFinder procedure didn't detect a point source at this location. You should get a full photometry result if you click on a source with circle surrounding it. I initially thought that it could be that some of your sources are below the default 10-sigma detection limit, but I can see that some of the brighter sources weren't detected as well, so we probably need to introduce another adjustment to the algorithm. Maybe something up with roundness. Would like to play with this FITS data to resolve this.  

On the ensemble fit - yes, you are correct, the brightest magnitudes are in the lower left corner. Do you think we should invert the axes?

nickjames's picture
Offline
Last seen: 21 min 22 sec ago
Joined: 12/10/2013 - 18:16
Attached. It is a gzipped

Attached. It is a gzipped FITS.

Attachments: 
maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
Hi Nick,

Hi Nick,

Fortunately no additional setting are required - I wanted to keep the program with as little settings as possible to avoid overcomplicating it. Looks like the PSFs are sharper in your image than from the telescopes I typically use, so try setting Lower Bound for Sharpness to 0.2 and redoing photometry. It detects most of the sources in your image this way. If you want to go a little deeper, lower the Star Detection Threshold from 10 to, say, 5 sigmas. This detects pretty much all the sources down to SNR < 1, except for a faint extended source under and to the left of the galaxy.

I will next add a feature to save settings so you wouldn't have to manually adjust things each time. I think it would be beneficial for user to save their settings once and reuse them.

Please let me know if it works.

Maxim

nickjames's picture
Offline
Last seen: 21 min 22 sec ago
Joined: 12/10/2013 - 18:16
Thanks. That works very well.

Thanks. That works very well. I get a pretty good ensemble fit against Gaia G and a magnitude for SN 2021hpr of 14.21. This compares to 14.29 using my aperture photometry tool, presumably affected by the galaxy background. I think the SNR you quote is too low though. I get 168, you have 27. SNR is a difficult thing to calculate correctly. How do you do it?

Attachments: 
maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
Nick, I am using SNR = Peak

Nick, I am using SNR = Peak Flux at the centroid / Median Sky Background Level, where Peak Flux is taken from the raw FITS data, and not estimated from the photometric fit, and the sky background is taken over the entire image. Do we need any adjustments here? I wonder how 168 was derived.

Note I was able to improve the uncertainty by limiting the ensemble range to 13-16 mag - see attachment.

maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
Nick, I am using SNR = Peak

Deleted. Replied in wrong thread.

nickjames's picture
Offline
Last seen: 21 min 22 sec ago
Joined: 12/10/2013 - 18:16
SNR calculation

There is an old discussion on calculating SNR here. Basically I ended up calculating all of the non-photon sources directly by measuring the RMS value in the sky estimation. To calculate the photon, sqrt(N), noise you need to know the camera gain. I've implemented this in an automated aperture photometry tool that I use and it seems reasonable to me. 

maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
Nick, many thanks for the

Nick, many thanks for the information. With PSF photometry the sky annuli are not used at all, so implementing this approach directly is not possible, unless we introduce a sky annulus setting. But then, if we do this, wouldn't such uncertainty estimate be fundamentally detached from the photometry algorithm used?

Fortunately, I have found that the iterative PSF photometry algorithm implementation in photutils provides flux uncertainties as output. Here is the link to version 0.14 of the program that reports total error = sqrt(flux_unc^2 + fit_unc^2) instead of relying on an external SNR estimate:

http://trafyx.com/wp-content/uploads/2021/04/metropsf-0.14.zip

I don't think I have the necessary expertise to develop a new method to estimate uncertainties with PSF fitting photometry. It should be safe to use  the flux fit uncertainty provided by the underlying algorithm. I wonder if you agree with this approach and if you could give this version a try.

Maxim

nickjames's picture
Offline
Last seen: 21 min 22 sec ago
Joined: 12/10/2013 - 18:16
Maxim,

Maxim,

Just to let you know that I've been using the program but haven't had a chance to look at the SNR details yet. I'll do that when I get time.

Nick.

Grant Privett's picture
Offline
Last seen: 19 hours 47 min ago
Joined: 28/12/2014 - 18:30
Clarifications

Hi there,

Not quite sure I understand what your code does.

Differential photometry using Poissonian/Gaussian profile and then using gaia DR2 to get the magnitude of the reference star?

How does the fitted result differ from what you would get using DAOPHOT?

Also, do you set a value that allows stars that are nearing saturation to be ignored?

Is the star to be measured denoted by hand or by RA/Dec somehow?

It does look like code people would find useful.

maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
Hi Grant,

Hi Grant,

Yes, fitting à la DAOPHOT, then deriving differential magnitudes via linear regression fit to an ensemble obtained from a VizieR catalog. You can select the catalog you need. The result shouldn't be very much different from DAOPHOT, albeit I'm using its implementation in Astropy's photutils. So, MetroPSF is a convenient GUI to photutils with some add-ons like linear regression fitting, plate solving, source matching, etc.

You click on the source you'd like to measure in the window. Here is a short write-up on functionality, check the Quick Start section here:
http://trafyx.com/wp-content/uploads/2021/04/metropsf-3.pdf

Near-saturation are not ignored, but you can choose Gaussian sigma weighting for the ensemble, so PSF fits with high sigmas which generally appear with oversaturated sources, will be assigned less weight in the fit. Would you recommend to introduce "hard cut-off" for oversaturated stars?

Max

P.S. Grant, thinking this over again, an obvious manual solution to avoid oversaturation is to limit source magnitudes in the ensemble, which is possible in MetroPSF already. This can be combined with sigma weighting as well.

Grant Privett's picture
Offline
Last seen: 19 hours 47 min ago
Joined: 28/12/2014 - 18:30
Refinement

I think the version I use is a standard conversion of the DAOFIND routine to Python. I use it to provide the positions of the stars and then do photometry on them.

After that I compare my coordinates with those of stars in Gaia DR2 and generate the matches. So I then have measured flux versus catalogue mag.

I then fit a linear regression and recursively remove the outliers. I'm fairly sure I found that more successful than using weightings. I had expected extreme colour index stars to cause problems too but that had a relatively minor impact.

I think in my process any star with a peak brightness >50,000 was excluded from the linear regression. As you say setting a magnitude limit should work but the count was easy.

maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
Grant, which criteria you are

Grant, which criteria you are using to remove outliers from the ensemble?

Grant Privett's picture
Offline
Last seen: 19 hours 47 min ago
Joined: 28/12/2014 - 18:30
Criteria

I look at which point is furthest from the generated fit and remove that.

Then I recalculate the fit and repeat until a decent regression coefficient is achieved and/or errors are below a threshold and sufficient stars remain. 

Its not an ideal approach and has trouble when there are few stars in the scene, but the results I got suggested it was doing quite a good job - certainly much better than not culling the outliers. It may not make a huge difference generally, but its easy to code up and test so possibly worth a look as a potential refinement.

Had expected stars with the most extreme colour indices would be causing the data points far from the curve, but while they were not great they were not always the worst outliers.

Could probably dig out the code if you wanted it. I was using Gaia DR2 and cooled sensor data captured using a normal Silicon CCD.

maximusatov's picture
Offline
Last seen: 10 hours 22 min ago
Joined: 13/02/2019 - 10:58
Grant, many thanks for this

Grant, many thanks for this suggestion. Simple and effective. I have just implemented this, will appear in version 0.15.