British Astronomical Association
Supporting amateur astronomers since 1890

Secondary menu

Main menu

Home Forums Variable Stars
Terms of use

Introducing MetroPSF - a program for ensemble photometry

46 posts / 0 new
Last post
maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Introducing MetroPSF - a program for ensemble photometry

Hi All,

I would like to share a small program I wrote, currently called MetroPSF for the lack of better name, for differential ensemble PSF-fitting photometry. Initially wrote a small script to double-check the results I was getting through Astrometrica for the SN 2021gmj, however, thought it may be useful for public, so releasing the source code. It's a prototype, so the code quality is pretty low, but the results it produces should be ready for analysis. I would appreciate any feedback and suggestions. I thought I'd check if this is of interest to public at all by making a preliminary "alpha" release.

The READ ME file with a brief description and installation instructions as well as the latest version of the program can be found here:

http://trafyx.com/?p=2669

The installation process may seem a bit strange, but it all boils down to installing Python interpreter which is pretty much automatic process, and using pip to install libraries, which is also automatic. This takes a few minutes on both Windows and Unix-like platforms.

MetroPSF works with VizieR and Astrometry.net for retrieving catalogs and solving, and the READ ME file describes a quick use case to familiarize with the interface. The screen shot of the program in action is below - on NGC 3310 with SN 2021gmj.

Maxim

astroshot's picture
Offline
Last seen: 5 hours 20 min ago
Joined: 11/09/2014 - 21:37
installation error

Hi Maxim,

I downloaded the program, extracted it to a directory on my C:\ drive and installed Python

(I admit to not knowing anything about Python)

However, I'm unclear how to run "pip install -r requirements.txt"

Is that via the command prompt? It just gave an error 'pip is not recognised...'

Regards,

Michael.

astroshot's picture
Offline
Last seen: 5 hours 20 min ago
Joined: 11/09/2014 - 21:37
installation error

Ignore that. Got it going now.

Even though I installed python, it didn't seem to register properly as a program.

I had to go to the Microsoft store to download and install it from there.

Strange!

nickjames's picture
Offline
Last seen: 2 hours 40 min ago
Joined: 12/10/2013 - 18:16
I get the same problem trying

I get the same problem trying to install 3.9.4 direct from the Python website into a fresh Win 10 20H2 instance in a VM. Trying to run pip from Powershell gives a "command not recognised" error. Installing from the MS Store worked. The rest of the install then ran smoothly although it did moan about 'wheel' not being installed as per the attached screenshot and has various warnings (see screenshot). I'm assuming these are benign. I'll try out the program now.

Attachments: 
maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Michael,

Michael,

Glad it works. Windows is doing these constant changes - head goes round. Any feedback is much appreciated. Let me know if you'll have any questions. Version 0.12 is already in the making with the ability to subtract VSX stars from the ensemble and report VSX sources nearby.

Maxim

astroshot's picture
Offline
Last seen: 5 hours 20 min ago
Joined: 11/09/2014 - 21:37
not opening images

I can get the program to run now, but when I open an image, only the FITS header loads in the lower panel - the image itself does not load.

I note that the Command Prompt window states the following:

WARNING: Astrometry.net API key not found in configuration file [astroquery.astrometry_net.core]

WARNING: You need to manually edit the configuration file and add it [astroquery.astrometry_net.core]

WARNING: You may also register it for this session with AstrometryNet.key = 'XXXXXXXX' [astroquery.astrometry_net.core]

Exception in Tkinter callback

Traceback (most recent call last):

  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1264.0_x64__qbz5n2kfra8p0\lib\tkinter\__init__.py", line 1892, in __call__

    return self.func(*args)

  File "C:\metropsf\metropsf.py", line 593, in update_histogram_high

    self.update_display()

  File "C:\metropsf\metropsf.py", line 502, in update_display

    self.display_image()

  File "C:\metropsf\metropsf.py", line 91, in display_image

    generate_FITS_thumbnail(self.histogram_slider_low, self.histogram_slider_high, self.zoom_level)

  File "C:\metropsf\metropsf.py", line 57, in generate_FITS_thumbnail

    generated_image = Image.fromarray(image_data)

NameError: name 'image_data' is not defined

maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Very strange. If you could

Very strange. If you could send me the FITS to maxim.usatov at bcsatellite.net I could take a look what's up with it. So far it did load all FITS files fine on both of my FreeBSD and Windows computers coming from different remote telescopes.

nickjames's picture
Offline
Last seen: 2 hours 40 min ago
Joined: 12/10/2013 - 18:16
Yes, I get the same on Win 10

Yes, I get the same on Win 10. FITS data but no image. See the attached screenshot. FITS file is attached. Note that it is a gzip compressed fits but I can only attach files with an extension .fits so you may need to change that to fit.gz to get it to work.

nickjames's picture
Offline
Last seen: 2 hours 40 min ago
Joined: 12/10/2013 - 18:16
I've just converted the FITS

I've just converted the FITS that I was using from int16 to float32 and it now works. I have an image. I'll play around a bit more.

astroshot's picture
Offline
Last seen: 5 hours 20 min ago
Joined: 11/09/2014 - 21:37
I've just converted the FITS

Yes, that works for me now as well. Thanks Nick!

@Maxim: Is there any chance we would get this program to open 8-fits files directly please?

I have a little project in mind involving images from the Global Meteor Network pi camera.

Thanks,

Michael.

maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Sorry, for some reason

Sorry, for some reason replied in the wrong thread. This version should open Nick's FITS fine:

maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Nick, Michael, thank you very

Nick, Michael, thank you very much. This one should open Nick's FITS fine:

http://trafyx.com/wp-content/uploads/2021/04/metropsf-0.12.zip

nickjames's picture
Offline
Last seen: 2 hours 40 min ago
Joined: 12/10/2013 - 18:16
Maxim - Thanks that works

Maxim - Thanks that works fine, see attachment. I'll have a look through your documentation later and will do some experiments on known fields.

Attachments: 
maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Nick, great! Please don't

Nick, great! Please don't hesitate to submit feature requests. I am trying to feel where to take this next. Perhaps, command-line processing... Definitely need a way to save all the settings.

astroshot's picture
Offline
Last seen: 5 hours 20 min ago
Joined: 11/09/2014 - 21:37
error

To clarify my above point, I get a long error message just when the program opens.

The error is contained in a separate window to the main application.

Also occurs with V0.12 shared above.

Anyone else get it?

WARNING: Astrometry.net API key not found in configuration file [astroquery.astrometry_net.core]

WARNING: You need to manually edit the configuration file and add it [astroquery.astrometry_net.core]

WARNING: You may also register it for this session with AstrometryNet.key = 'XXXXXXXX' [astroquery.astrometry_net.core]

Exception in Tkinter callback

Traceback (most recent call last):

  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1264.0_x64__qbz5n2kfra8p0\lib\tkinter\__init__.py", line 1892, in __call__

    return self.func(*args)

  File "C:\metropsf\metropsf.py", line 665, in update_histogram_high

    self.update_display()

  File "C:\metropsf\metropsf.py", line 560, in update_display

    self.display_image()

  File "C:\metropsf\metropsf.py", line 93, in display_image

    generate_FITS_thumbnail(self.histogram_slider_low, self.histogram_slider_high, self.zoom_level)

  File "C:\metropsf\metropsf.py", line 56, in generate_FITS_thumbnail

    converted_data = image_data.astype(float)

NameError: name 'image_data' is not defined

maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
For the time being just

For the time being just ignore those. I'll clean things up soon. As long as the program itself works, it shouldn't impact on anything.

nickjames's picture
Offline
Last seen: 2 hours 40 min ago
Joined: 12/10/2013 - 18:16
Michael - Yes, it seems to

Michael - Yes, it seems to work fine without the astrometry.net key although my files are already platesolved. Here's an example of the V1391 Cas field with the fit using V.

Maxim - Could you add Gaia DR2 G to your list of catalogues? Also, the image contrast stretch doesn't seem to work very well for me.  

Attachments: 
maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Nick, here is 0.13 with Gaia

Nick, here is 0.13 with Gaia DR2 and non-linear image stretching for better views:
http://trafyx.com/wp-content/uploads/2021/04/metropsf-0.13.zip
Updated user guide PDF inside. Please let me know if you need anything else.

Maxim

nickjames's picture
Offline
Last seen: 2 hours 40 min ago
Joined: 12/10/2013 - 18:16
Thanks. I get an error "630

Thanks. I get an error "630 match_mag" when computing the regression model. now.

maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Nick, looks like it cannot

Nick, looks like it cannot find the magnitudes of comparison stars in the ensemble. I've just went all the way - opening a 16-bit FITS file (ugc7513_000.fits), doing Iteratively Subtracted PSF Photometry, then set catalog to Gaia DR2, CCD Filter to G manually, ran Get Comparison Stars, then Find Regression Model and finally was able to get differential magnitudes by mouse-clicking on the sources. Try removing the old .phot file and executing the same steps. I'm going out of town for a couple of days but hopefully will be able to assist with delays.

Max

nickjames's picture
Offline
Last seen: 2 hours 40 min ago
Joined: 12/10/2013 - 18:16
Maxim - Thanks, I'm almost

Maxim - Thanks, I'm almost there. I had forgotten the "Get comparison stars" step! Now I get a fit. The only problem remaining is when I click on the object the flux/magnitude etc are all zero. What am I doing wrong?

One other question - Am I right that the ensemble fit graph is upside down, i.e. the brightest magnitudes are in the lower left corner. Not a problem but I would just like to understand.  

Attachments: 
maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Nick, do you mind sending me

Nick, do you mind sending me the FITS file with this data? You are likely clicking on sources that didn't have any photometry fits at all, i.e. appearing without circles surrounding them. This means that the IRAFStarFinder procedure didn't detect a point source at this location. You should get a full photometry result if you click on a source with circle surrounding it. I initially thought that it could be that some of your sources are below the default 10-sigma detection limit, but I can see that some of the brighter sources weren't detected as well, so we probably need to introduce another adjustment to the algorithm. Maybe something up with roundness. Would like to play with this FITS data to resolve this.  

On the ensemble fit - yes, you are correct, the brightest magnitudes are in the lower left corner. Do you think we should invert the axes?

nickjames's picture
Offline
Last seen: 2 hours 40 min ago
Joined: 12/10/2013 - 18:16
Attached. It is a gzipped

Attached. It is a gzipped FITS.

Attachments: 
maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Hi Nick,

Hi Nick,

Fortunately no additional setting are required - I wanted to keep the program with as little settings as possible to avoid overcomplicating it. Looks like the PSFs are sharper in your image than from the telescopes I typically use, so try setting Lower Bound for Sharpness to 0.2 and redoing photometry. It detects most of the sources in your image this way. If you want to go a little deeper, lower the Star Detection Threshold from 10 to, say, 5 sigmas. This detects pretty much all the sources down to SNR < 1, except for a faint extended source under and to the left of the galaxy.

I will next add a feature to save settings so you wouldn't have to manually adjust things each time. I think it would be beneficial for user to save their settings once and reuse them.

Please let me know if it works.

Maxim

nickjames's picture
Offline
Last seen: 2 hours 40 min ago
Joined: 12/10/2013 - 18:16
Thanks. That works very well.

Thanks. That works very well. I get a pretty good ensemble fit against Gaia G and a magnitude for SN 2021hpr of 14.21. This compares to 14.29 using my aperture photometry tool, presumably affected by the galaxy background. I think the SNR you quote is too low though. I get 168, you have 27. SNR is a difficult thing to calculate correctly. How do you do it?

Attachments: 
maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Nick, I am using SNR = Peak

Nick, I am using SNR = Peak Flux at the centroid / Median Sky Background Level, where Peak Flux is taken from the raw FITS data, and not estimated from the photometric fit, and the sky background is taken over the entire image. Do we need any adjustments here? I wonder how 168 was derived.

Note I was able to improve the uncertainty by limiting the ensemble range to 13-16 mag - see attachment.

maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Nick, I am using SNR = Peak

Deleted. Replied in wrong thread.

nickjames's picture
Offline
Last seen: 2 hours 40 min ago
Joined: 12/10/2013 - 18:16
SNR calculation

There is an old discussion on calculating SNR here. Basically I ended up calculating all of the non-photon sources directly by measuring the RMS value in the sky estimation. To calculate the photon, sqrt(N), noise you need to know the camera gain. I've implemented this in an automated aperture photometry tool that I use and it seems reasonable to me. 

maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Nick, many thanks for the

Nick, many thanks for the information. With PSF photometry the sky annuli are not used at all, so implementing this approach directly is not possible, unless we introduce a sky annulus setting. But then, if we do this, wouldn't such uncertainty estimate be fundamentally detached from the photometry algorithm used?

Fortunately, I have found that the iterative PSF photometry algorithm implementation in photutils provides flux uncertainties as output. Here is the link to version 0.14 of the program that reports total error = sqrt(flux_unc^2 + fit_unc^2) instead of relying on an external SNR estimate:

http://trafyx.com/wp-content/uploads/2021/04/metropsf-0.14.zip

I don't think I have the necessary expertise to develop a new method to estimate uncertainties with PSF fitting photometry. It should be safe to use  the flux fit uncertainty provided by the underlying algorithm. I wonder if you agree with this approach and if you could give this version a try.

Maxim

nickjames's picture
Offline
Last seen: 2 hours 40 min ago
Joined: 12/10/2013 - 18:16
Maxim,

Maxim,

Just to let you know that I've been using the program but haven't had a chance to look at the SNR details yet. I'll do that when I get time.

Nick.

Grant Privett's picture
Offline
Last seen: 4 hours 53 min ago
Joined: 28/12/2014 - 18:30
Clarifications

Hi there,

Not quite sure I understand what your code does.

Differential photometry using Poissonian/Gaussian profile and then using gaia DR2 to get the magnitude of the reference star?

How does the fitted result differ from what you would get using DAOPHOT?

Also, do you set a value that allows stars that are nearing saturation to be ignored?

Is the star to be measured denoted by hand or by RA/Dec somehow?

It does look like code people would find useful.

maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Hi Grant,

Hi Grant,

Yes, fitting à la DAOPHOT, then deriving differential magnitudes via linear regression fit to an ensemble obtained from a VizieR catalog. You can select the catalog you need. The result shouldn't be very much different from DAOPHOT, albeit I'm using its implementation in Astropy's photutils. So, MetroPSF is a convenient GUI to photutils with some add-ons like linear regression fitting, plate solving, source matching, etc.

You click on the source you'd like to measure in the window. Here is a short write-up on functionality, check the Quick Start section here:
http://trafyx.com/wp-content/uploads/2021/04/metropsf-3.pdf

Near-saturation are not ignored, but you can choose Gaussian sigma weighting for the ensemble, so PSF fits with high sigmas which generally appear with oversaturated sources, will be assigned less weight in the fit. Would you recommend to introduce "hard cut-off" for oversaturated stars?

Max

P.S. Grant, thinking this over again, an obvious manual solution to avoid oversaturation is to limit source magnitudes in the ensemble, which is possible in MetroPSF already. This can be combined with sigma weighting as well.

Grant Privett's picture
Offline
Last seen: 4 hours 53 min ago
Joined: 28/12/2014 - 18:30
Refinement

I think the version I use is a standard conversion of the DAOFIND routine to Python. I use it to provide the positions of the stars and then do photometry on them.

After that I compare my coordinates with those of stars in Gaia DR2 and generate the matches. So I then have measured flux versus catalogue mag.

I then fit a linear regression and recursively remove the outliers. I'm fairly sure I found that more successful than using weightings. I had expected extreme colour index stars to cause problems too but that had a relatively minor impact.

I think in my process any star with a peak brightness >50,000 was excluded from the linear regression. As you say setting a magnitude limit should work but the count was easy.

maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Grant, which criteria you are

Grant, which criteria you are using to remove outliers from the ensemble?

Grant Privett's picture
Offline
Last seen: 4 hours 53 min ago
Joined: 28/12/2014 - 18:30
Criteria

I look at which point is furthest from the generated fit and remove that.

Then I recalculate the fit and repeat until a decent regression coefficient is achieved and/or errors are below a threshold and sufficient stars remain. 

Its not an ideal approach and has trouble when there are few stars in the scene, but the results I got suggested it was doing quite a good job - certainly much better than not culling the outliers. It may not make a huge difference generally, but its easy to code up and test so possibly worth a look as a potential refinement.

Had expected stars with the most extreme colour indices would be causing the data points far from the curve, but while they were not great they were not always the worst outliers.

Could probably dig out the code if you wanted it. I was using Gaia DR2 and cooled sensor data captured using a normal Silicon CCD.

maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Grant, many thanks for this

Grant, many thanks for this suggestion. Simple and effective. I have just implemented this, will appear in version 0.15.

maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
New version - 0.15

Happy to announce the latest version of MetroPSF, 0.15. Lots of new stuff:

• Aperture photometry.
This is similar to common aperture photometry algorithms found in other software except that a 2D model of the background is subtracted from the image, so sky annuli are not used.

• AAVSO report generation.
MetroPSF can now generate AAVSO-compatible report files.

• Automatic reporting on all VSX sources in the image.
MetroPSF can generate reports on all VSX sources found in the image. It cycles through each VSX variable and builds a report file. Although I would not recommend to do automatic reports and submit them blindly (unless you know what exactly you are doing) this is helpful when you need to process multiple FITS files to create a report on a single object or multiple objects with, say, similar magnitudes.

• Batch processing of all FITS files in a folder.
Now it is possible to do photometric series, as MetroPSF can process multiple FITS files in a folder at once...

• Light curve generation from BAA reports.
...and display a light curve from all the report files generated.

• Added ability to remove outliers from the fit - by ensemble limit and maximum separation.
Much more precise photometry, as you can set it up to automatically exclude comparison stars farther than R arcseconds from the target, or that are not fitting well into the comparison ensemble, e.g. limit ensemble to N stars.

Convenience changes:
• Reporting number of ensemble stars on the linear regression fit plot.
• Object name in the left panel is now automatically set from VSX catalog for matching sources.
• Display next VSX source.
Can easily cycle through all known variables in your image.

• Mouse click now reports time in ISO UTC format along with photometry result.
To copy-paste in forums and discussions.

Other changes:
• Increased Astrometry.Net default solve timeout from 120 to 360 s.
• Linear regression fit error is now based on standard deviation.
• MetroPSF now writes a log to metropsf.log.
• Astrometry.net URL and API key setting in the interface.
• PSF fitting algorithm setting.
• FITS crop option.

Latest manual: http://trafyx.com/wp-content/uploads/2021/05/metropsf.pdf
MetroPSF program and manual: http://trafyx.com/wp-content/uploads/2021/05/metropsf-0.15.zip

Max

nickjames's picture
Offline
Last seen: 2 hours 40 min ago
Joined: 12/10/2013 - 18:16
Hi Max,

Hi Max,

I've downloaded 0.15 and am running it under Windows 10. It does the PSF photometry but then fails when I try to download catalogue stars:

694 'charmap' codec can't encode character '\u03b1' in position 25: character maps to <undefined>

Any idea what this might be?

Nick.

maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Whoops. Forgot to specify

Whoops. Forgot to specify encoding for the log file which obviously produces an error on Windows. Please try this fixed version:

http://trafyx.com/wp-content/uploads/2021/05/metropsf-0.15.zip

nickjames's picture
Offline
Last seen: 2 hours 40 min ago
Joined: 12/10/2013 - 18:16
Max, Thanks. That works and

Max, Thanks. That works and this looks really powerful, particularly the ability to measure multiple variables in a field and generate automatic reports. My only remaining comment is that I get strange timestamps:

ES UMa, -4713-11-24 12:00:00.000 UTC, G = 10.91 ± 0.11

-4713 is something like JD = 0. The FITS file I'm measuring includes the record:

DATE-OBS= '2021-05-25T22:53:18' / Start time of stacked exposure

Nick.

maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Thanks, Nick. With those

Thanks, Nick. With those automatic reports one has to be careful, as you might need an ensemble covering a wide range of magnitudes which will raise uncertainties. Perhaps, we need to add an option to fine-tune the range of magnitudes in the ensemble per each target automatically during automatic reporting? 

On the timestamps: I've just tested a few FITS on my Windows 10 machine and don't see any issues. Could you please send me the FITS file for testing? Very odd.

Max

nickjames's picture
Offline
Last seen: 2 hours 40 min ago
Joined: 12/10/2013 - 18:16
Here is an example file which

Here is an example file which gives the JD zero date: http://www.nickdjames.com/Transfers/ngc6888_n_solved.fit.gz

maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Nick, I took exposure start

Nick, I took exposure start from the "JD" field in the FITS, never looked at DATE-OBS. MetroPSF now looks at DATE-OBS as well. I have uploaded the fixed 0.15 here:

http://trafyx.com/wp-content/uploads/2021/05/metropsf-0.15.zip

Thanks for pointing this out.

Max

Americo Watkins's picture
Offline
Last seen: 1 day 5 hours ago
Joined: 11/03/2014 - 13:12
Max,

Max,

It appears you have developed an interesting and useful piece of software.  Does it have the capability to automatically perform  photometry on  a time sequence series of images, if so can it follow the motion of the asteroid or comet through out the sequence.  It's the lack of this facility with Astrometrica that I miss when producing photometry on a time series run of images.  Having to do photometry one at a time of a run of what could be several hours is time consuming.

Best wishes, Eric

maximusatov's picture
Offline
Last seen: 3 weeks 6 days ago
Joined: 13/02/2019 - 10:58
Thank you, Eric. It can

Thank you, Eric. It can process the whole folder of input images and report in batch, but reports are generated only for objects from the AAVSO VSX database. I've never tried it on comets. If you could send me a series of FITS with a comet - three is enough - I could try to see if I can get it reporting.

Americo Watkins's picture
Offline
Last seen: 1 day 5 hours ago
Joined: 11/03/2014 - 13:12
Max,

Max,

Though I mentioned comets asteroids may be more appropriate in my case,  I'll dig out examples of both.

Thanks,

Eric