Difference between revisions of "LST1Analysis MVA"

From my_wiki
Jump to: navigation, search
(→‎MVA Analysis pages)
 
(83 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
== MVA Analysis pages ==
 +
* [https://www.lst1.iac.es/wiki/index.php/BLLacAnalysis_MVA BLLacAnalysis_MVA]
 +
* [https://www.lst1.iac.es/wiki/index.php/NovaAnalysis_MVA NovaAnalysis_MVA]
 +
* [https://www.lst1.iac.es/wiki/index.php/AGNAnalysis_MVA AGNAnalysis_MVA]
 +
 
== Location of LST1 data at PIC ==
 
== Location of LST1 data at PIC ==
  
Line 11: Line 16:
  
 
* Directory: /fefs/onsite/data <br/>Use cp01/cp02 to copy out data, never the login machine
 
* Directory: /fefs/onsite/data <br/>Use cp01/cp02 to copy out data, never the login machine
 +
 +
* Latest IRF <br/>/fefs/aswg/data/mc/IRF/20200629_prod5_trans_80/zenith_20deg/south_pointing/20210923_v0.7.5_prod5_trans_80_dynamic_cleaning/off0.4deg/irf_20210923_v075_prod5_trans_80_dynamic_cleaning_gamma_point-like_off04deg.fits.gz
 +
 +
== LST1 School 2022 ==
 +
 +
https://www.lst1.iac.es/wiki/index.php/LST1School2021_MVA
  
 
== LST1 DL2/DL3 Data  ==
 
== LST1 DL2/DL3 Data  ==
Line 22: Line 33:
 
** dl2_LST-1.Run029*h5
 
** dl2_LST-1.Run029*h5
 
** dl2_merged.h5
 
** dl2_merged.h5
 +
 +
 +
* Check DL2 data: RF models & config
 +
<pre style="color: blue">
 +
cd /fefs/aswg/data/real/DL2/20220206/v0.8.4/tailcut84/log
 +
 +
cat sequencer_v0.8.4.cfg
 +
 +
** CONFIGURATION:
 +
 +
dl1ab_config: /fefs/aswg/lstosa/lstchain_config_v083_tailcut84.json
 +
# In principle both config files should be the same, but in case
 +
# the MC DL2 uses a different config file, it can be set here.
 +
# For testing purposes
 +
dl2_config: /fefs/aswg/lstosa/lstchain_config_v083_tailcut84.json
 +
 +
** MODELS:
 +
rf_models: /fefs/aswg/workspace/abelardo.moralejo/Crab_tuned_MC_20211231/MC/DL1b_tuned/training/models
 +
 +
</pre>
  
 
== DL2toDL3 conversion ==
 
== DL2toDL3 conversion ==
−
General talk on the LST1 DL3: [https://indico.cta-observatory.org/event/3501/contributions/29581/attachments/19810/27478/DL3_Tools_and_Spectral_analysis_of_Crab_data_with_gammapy.pdf DL3_Tools_and_Spectral_analysis_of_Crab_data_with_gammapy.pdf]
+
General talk on the LST1 DL3 Tools by Chaitanya Priyadarshi: [https://indico.cta-observatory.org/event/3501/contributions/29581/attachments/19810/27478/DL3_Tools_and_Spectral_analysis_of_Crab_data_with_gammapy.pdf DL3_Tools_and_Spectral_analysis_of_Crab_data_with_gammapy.pdf]
  
 
=== DL2toDL3 Tools ===
 
=== DL2toDL3 Tools ===
Line 31: Line 62:
 
* <font style="color: blue"> lstchain_create_irf_file</font>: create IRFs from the MC file
 
* <font style="color: blue"> lstchain_create_irf_file</font>: create IRFs from the MC file
 
** The IRFs can be generated using only the gamma MC file, to produce Effective Area, Energy Dispersion <br/>(and PSF for Full Enclosure IRFs, ie. without any theta cuts).  
 
** The IRFs can be generated using only the gamma MC file, to produce Effective Area, Energy Dispersion <br/>(and PSF for Full Enclosure IRFs, ie. without any theta cuts).  
−
** Proton and Electron MC are used to create the Background IRF. For creating point-like IRFs, you have to pass <br>the --point-like flag in the command-line, as by default the Tools generates a Full Enclosure IRF.
+
** Proton and Electron MC are used to create the Background IRF.  
−
* <font style="color: blue"> lstchain_create_dl3_index_files</font>: index the DL3 files as per [https://gamma-astro-data-formats.readthedocs.io/en/latest/ GADF] and using [https://cta-observatory.github.io/ctapipe/ctapipe_api/tools/index.html ctapipe tools]
+
** For creating point-like IRFs, you have to pass <br>the --point-like flag in the command-line, as by default the Tools generates a Full Enclosure IRF.
 
* <font style="color: blue"> lstchain_create_dl3_file</font>: create DL3 files
 
* <font style="color: blue"> lstchain_create_dl3_file</font>: create DL3 files
 +
* <font style="color: blue"> lstchain_create_dl3_index_files</font>: generate HDU and OBS Index files of the DL3 files created. <br/>The DL3 files are indexed as per [https://gamma-astro-data-formats.readthedocs.io/en/latest/ GADF] and using [https://cta-observatory.github.io/ctapipe/ctapipe_api/tools/index.html ctapipe tools]
 +
 +
Command options can be checked: lstchain_create_irf_file --help-all
  
 
=== DL2 Cuts ===
 
=== DL2 Cuts ===
Line 41: Line 75:
 
* allowed_tels [1]
 
* allowed_tels [1]
  
−
Making cuts methods:
+
Cut Methods:  
−
# --config /path/to/config.json  
+
# '''--fixed-gh-cut 0.9'''
−
# --fixed-gh-cut 0.9
+
# '''--config /path/to/config.json''' <br/>Example json config files for the DL3 tools are [https://github.com/cta-observatory/cta-lstchain/blob/master/docs/examples here] <br/> The DL3 JSON config file ([https://github.com/cta-observatory/cta-lstchain/blob/master/docs/examples/dl3_tool_config.json dl3_tool_config.json]) is a subset of the IRF config file ([https://github.com/cta-observatory/cta-lstchain/blob/master/docs/examples/irf_tool_config.json irf_tool_config.json]). <br/>One can '''use the IRF config file for both tools''' to ensure that the same filter cuts for both MC and real data are used.
  
−
Example json config files for the DL3 tools are [https://github.com/cta-observatory/cta-lstchain/blob/master/docs/examples here] <br/> The DL3 JSON config file ([https://github.com/cta-observatory/cta-lstchain/blob/master/docs/examples/dl3_tool_config.json dl3_tool_config.json]) is a subset of the IRF config file ([https://github.com/cta-observatory/cta-lstchain/blob/master/docs/examples/irf_tool_config.json irf_tool_config.json]). <br/>One can '''use the IRF config file for both tools''' to ensure that the same filter cuts for both MC and real data are used.
+
'''NOTE''': the fixed_theta_cut option is not available in lstchain_create_dl3_file, so it is better to use the JSON option to be consistent with the IRF generation
  
−
=== Example commands ===
+
=== Example DL2toDL3 commands ===
  
 
<pre style="color: blue">
 
<pre style="color: blue">
Line 57: Line 91:
 
         --fixed-theta-cut 0.2
 
         --fixed-theta-cut 0.2
 
         --irf-obs-time 50
 
         --irf-obs-time 50
 +
 +
lstchain_create_dl3_file
 +
        -d /path/to/DL2_data_file.h5
 +
        -o /path/to/DL3/file/
 +
        --input-irf /path/to/irf.fits.gz
 +
        --source-name Crab
 +
        --source-ra 83.633deg
 +
        --source-dec 22.01deg
 +
        --fixed-gh-cut 0.9
 +
 +
lstchain_create_dl3_index_files
 +
        -d /path/to/DL3/files/
 +
        -o /path/to/DL3/index/files
 +
        -p dl3*[run_1-run_n]*.fits.gz
 
</pre>
 
</pre>
  
 +
==== Example DL2toDL3 commands - MVA specific ====
 
<pre>
 
<pre>
−
lstchain_create_irf_files -g /Users/monicava/LST1_DATA/Abelardo/dl2_merged.h5 -o /Users/monicava/LST1_DATA/Abelardo/irf.gh01.fits.gz --point-like --fixed-gh-cut 0.1 --fixed-theta-cut 0.2 --irf-obs-time 50
+
lstchain_create_irf_files -g /Users/monicava/LST1_DATA/Abelardo/dl2_merged.h5 -o /Users/monicava/LST1_DATA/Abelardo/irf.gh01.fits.gz --point-like --config irf_tool_config_gh01.json --irf-obs-time 50
−
lstchain_create_irf_files -g /Users/monicava/LST1_DATA/Abelardo/dl2_merged.h5 -o /Users/monicava/LST1_DATA/Abelardo/irf.gh02.fits.gz --point-like --fixed-gh-cut 0.2 --fixed-theta-cut 0.2 --irf-obs-time 50
+
lstchain_create_dl3_file -d /Users/monicava/LST1_DATA/Abelardo/dl2_LST-1.Run02967.h5 -o /Users/monicava/LST1_DATA/Abelardo/DL3_gh01 --input-irf /Users/monicava/LST1_DATA/Abelardo/irf.gh01.fits.gz --source-name Crab --source-ra 83.633deg --source-dec 22.01deg --config irf_tool_config_gh01.json
−
lstchain_create_irf_files -g /Users/monicava/LST1_DATA/Abelardo/dl2_merged.h5 -o /Users/monicava/LST1_DATA/Abelardo/irf.gh03.fits.gz --point-like --fixed-gh-cut 0.3 --fixed-theta-cut 0.2 --irf-obs-time 50
+
lstchain_create_dl3_index_files -d /Users/monicava/LST1_DATA/Abelardo/DL3_gh01
−
lstchain_create_irf_files -g /Users/monicava/LST1_DATA/Abelardo/dl2_merged.h5 -o /Users/monicava/LST1_DATA/Abelardo/irf.gh04.fits.gz --point-like --fixed-gh-cut 0.4 --fixed-theta-cut 0.2 --irf-obs-time 50
 
−
lstchain_create_irf_files -g /Users/monicava/LST1_DATA/Abelardo/dl2_merged.h5 -o /Users/monicava/LST1_DATA/Abelardo/irf.gh05.fits.gz --point-like --fixed-gh-cut 0.5 --fixed-theta-cut 0.2 --irf-obs-time 50
 
−
lstchain_create_irf_files -g /Users/monicava/LST1_DATA/Abelardo/dl2_merged.h5 -o /Users/monicava/LST1_DATA/Abelardo/irf.gh06.fits.gz --point-like --fixed-gh-cut 0.6 --fixed-theta-cut 0.2 --irf-obs-time 50
 
−
lstchain_create_irf_files -g /Users/monicava/LST1_DATA/Abelardo/dl2_merged.h5 -o /Users/monicava/LST1_DATA/Abelardo/irf.gh07.fits.gz --point-like --fixed-gh-cut 0.7 --fixed-theta-cut 0.2 --irf-obs-time 50
 
−
lstchain_create_irf_files -g /Users/monicava/LST1_DATA/Abelardo/dl2_merged.h5 -o /Users/monicava/LST1_DATA/Abelardo/irf.gh08.fits.gz --point-like --fixed-gh-cut 0.8 --fixed-theta-cut 0.2 --irf-obs-time 50
 
−
lstchain_create_irf_files -g /Users/monicava/LST1_DATA/Abelardo/dl2_merged.h5 -o /Users/monicava/LST1_DATA/Abelardo/irf.gh09.fits.gz --point-like --fixed-gh-cut 0.9 --fixed-theta-cut 0.2 --irf-obs-time 50
 
 
</pre>
 
</pre>
  
 
== Run summary info/data-check ==
 
== Run summary info/data-check ==
 
* [http://sagan.gae.ucm.es/LST1/ Run-summary web in GAE] <br/> LST1 account
 
* [http://sagan.gae.ucm.es/LST1/ Run-summary web in GAE] <br/> LST1 account
 +
 +
* Seiya's Crab Run list 2021[https://docs.google.com/spreadsheets/d/1S5wcgqldMN3Tt-51IseJOaRnKbDvh1QTc7fgUrCB-mc/edit#gid=2023820644 LST_RUN_CATALOG]
 +
 +
* Data check: [https://www.lst1.iac.es/datacheck/observation_overview/ DATACHECK]
  
 
== LST1 data formats ==
 
== LST1 data formats ==
−
El contenido de los ficheros se puede ver con:
+
File content can be obtained as follows:
−
 
 
 
<pre style="color: blue">
 
<pre style="color: blue">
 
from lstchain.io import get_dataset_keys
 
from lstchain.io import get_dataset_keys
 
from lstchain.io.io import dl1_params_lstcam_key, dl2_params_lstcam_key, dl1_images_lstcam_key
 
from lstchain.io.io import dl1_params_lstcam_key, dl2_params_lstcam_key, dl1_images_lstcam_key
−
file = "/fefs/aswg/data/real/DL2/20210903/v0.7.3/tailcut84/dl2_LST-1.Run05998.0000.h5"
+
dl2_file = "/fefs/aswg/data/real/DL2/20210903/v0.7.3/tailcut84/dl2_LST-1.Run05998.0000.h5"
 
import pandas as pd
 
import pandas as pd
−
get_dataset_keys(file)
+
get_dataset_keys(dl2_file)
−
 
+
</pre>
 +
<pre>
 
Out[]:  
 
Out[]:  
 
['configuration/instrument/subarray/layout',
 
['configuration/instrument/subarray/layout',
Line 97: Line 144:
 
  'dl1/event/telescope/monitoring/pedestal',
 
  'dl1/event/telescope/monitoring/pedestal',
 
  'dl2/event/telescope/parameters/LST_LSTCam']
 
  'dl2/event/telescope/parameters/LST_LSTCam']
−
 
+
</pre>
 +
<pre style="color: blue">
 
dl1_file = "/fefs/aswg/data/real/DL1/20210903/v0.7.3/tailcut84/dl1_LST-1.Run05998.0000.h5"
 
dl1_file = "/fefs/aswg/data/real/DL1/20210903/v0.7.3/tailcut84/dl1_LST-1.Run05998.0000.h5"
 
get_dataset_keys(dl1_file)
 
get_dataset_keys(dl1_file)
−
 
+
</pre>
 +
<pre>
 
Out[]:  
 
Out[]:  
 
['configuration/instrument/subarray/layout',
 
['configuration/instrument/subarray/layout',
Line 116: Line 165:
 
  'dl1/event/telescope/parameters/LST_LSTCam']
 
  'dl1/event/telescope/parameters/LST_LSTCam']
 
</pre>
 
</pre>
 +
Data is in the tables:
 +
* 'dl1/event/telescope/image/LST_LSTCam' (camera calibrated images)
 +
* 'dl1/event/telescope/parameters/LST_LSTCam' (Hillas parameters, timestamps, pointing, etc)
 +
In case of DL2 only the table parameters 'dl2/event/telescope/parameters/LST_LSTCam' are available, which is a copy of the DL1 table with the reconstructed parameters added
  
−
De ahí, los datos como tal están en las tablas 'dl1/event/telescope/image/LST_LSTCam' (imágenes calibradas de la cámara) y 'dl1/event/telescope/parameters/LST_LSTCam' (parámetros de Hillas, timestamps, pointing, etc)
+
To access the parameter tables in DL1 and DL2:
−
En el caso de dl2 solamente se tiene la tabla de parámetros 'dl2/event/telescope/parameters/LST_LSTCam' , que es una copia de la tabla dl1 con los parámetros reconstruidos añadidos.
 
−
 
 
−
Para acceder a la tabla de parámetros tanto en dl1 como en dl2:
 
 
<pre style="color: blue">
 
<pre style="color: blue">
 
dl2_dataframe = pd.read_hdf(file, key=dl2_params_lstcam_key)
 
dl2_dataframe = pd.read_hdf(file, key=dl2_params_lstcam_key)
 
dl2_dataframe
 
dl2_dataframe
−
 
+
</pre>
 +
<pre>
 
Out[]:  
 
Out[]:  
 
       obs_id  event_id      intensity  log_intensity        x        y        r      phi    length  ...  reco_energy  reco_disp_dx  reco_disp_dy  reco_src_x  reco_src_y  reco_alt  reco_az  reco_type  gammaness
 
       obs_id  event_id      intensity  log_intensity        x        y        r      phi    length  ...  reco_energy  reco_disp_dx  reco_disp_dy  reco_src_x  reco_src_y  reco_alt  reco_az  reco_type  gammaness
Line 140: Line 191:
  
 
[28276 rows x 51 columns]
 
[28276 rows x 51 columns]
−
 
 
</pre>
 
</pre>
−
 
+
Accesing images is more complicated and requires the tables module
−
 
 
−
Acceder a las imágenes es algo más complicado y requiere el módulo tables
 
−
 
 
 
<pre style="color: blue">
 
<pre style="color: blue">
 
from astropy.table import Table
 
from astropy.table import Table
 
images = Table.read(dl1_file, path=dl1_images_lstcam_key)['image']
 
images = Table.read(dl1_file, path=dl1_images_lstcam_key)['image']
 
images
 
images
−
 
+
</pre>
 +
<pre>
 
Out[]:  
 
Out[]:  
 
<Column name='image' dtype='float32' shape=(1855,) length=53000>
 
<Column name='image' dtype='float32' shape=(1855,) length=53000>
Line 179: Line 227:
 
                       ...
 
                       ...
 
</pre>
 
</pre>
 +
for each of the 53000 events, a calibrated image of the camera is obtained (array of 1855 pixels)
  
−
para cada uno de los 53000 eventos, se tiene una imagen calibrada de la cámara (array de 1855 pixels)
+
The rest of the tables in the files are metadata (info about the camera, optics, calibration, etc)
−
 
 
−
el resto de tablas en los ficheros son metadatos (info sobre cámara, óptica, calibración, etc)
 
  
 
== Software tool installation ==
 
== Software tool installation ==
Line 204: Line 251:
 
conda install -c conda-forge -n base mamba
 
conda install -c conda-forge -n base mamba
 
mamba env create -f environment.yml
 
mamba env create -f environment.yml
−
 
 
</pre>
 
</pre>
  
Line 236: Line 282:
 
=== Gammapy ===
 
=== Gammapy ===
  
 +
<pre style="color: blue">
 +
curl -O https://gammapy.org/download/install/gammapy-0.18.2-environment.yml
 +
conda env create -f gammapy-0.18.2-environment.yml
 +
conda activate gammapy-0.18.2
 +
gammapy info
 +
 +
##### Installing local copy
 +
git clone  -b v0.18.2 https://github.com/gammapy/gammapy
 +
cd gammapy
 +
pip install -e .
 +
 +
</pre>
 +
Installation in Ubuntu
 
<pre style="color: red">
 
<pre style="color: red">
−
FIX UBUNTU
 
 
conda create -n gammapyenv python=3.7
 
conda create -n gammapyenv python=3.7
 
conda activate gammapyenv
 
conda activate gammapyenv
 
conda install gammapy=0.18.2 sherpa=4.12.0
 
conda install gammapy=0.18.2 sherpa=4.12.0
 
conda install jupyter ipython jupyterlab pandas healpy iminuit naima emcee corner parfice
 
conda install jupyter ipython jupyterlab pandas healpy iminuit naima emcee corner parfice
−
gammapy info
 
−
 
−
</pre>
 
−
 
−
<pre style="color: blue">
 
−
curl -O https://gammapy.org/download/install/gammapy-0.18.2-environment.yml
 
−
conda env create -f gammapy-0.18.2-environment.yml
 
−
conda activate gammapy-0.18.2
 
 
gammapy info
 
gammapy info
 
</pre>
 
</pre>
Line 331: Line 381:
  
 
== Extra ==
 
== Extra ==
 +
 +
=== Conda useful commands ===
 +
 +
<pre>
 +
conda remove --name gammapy_default --all
 +
conda list -n gammapy-0.18.2
 +
</pre>
 +
 
=== MAGIC DL3 data ===
 
=== MAGIC DL3 data ===
 
https://github.com/open-gamma-ray-astro/joint-crab/tree/master/data/magic
 
https://github.com/open-gamma-ray-astro/joint-crab/tree/master/data/magic
 +
 +
https://gitlab.pic.es/magic_dl3/magic_dl3_sw_school_2021/-/tree/master/data/magic
 +
 +
=== Notebook examples ===
 +
https://github.com/rlopezcoto/intro-iact-analysis

Latest revision as of 10:13, 16 February 2022

MVA Analysis pages[edit]

Location of LST1 data at PIC[edit]

Use CTA observatory LDAP account (eg: monica.vazquez)

Location of LST1 data at La Palma[edit]

  • Directory: /fefs/onsite/data
    Use cp01/cp02 to copy out data, never the login machine
  • Latest IRF
    /fefs/aswg/data/mc/IRF/20200629_prod5_trans_80/zenith_20deg/south_pointing/20210923_v0.7.5_prod5_trans_80_dynamic_cleaning/off0.4deg/irf_20210923_v075_prod5_trans_80_dynamic_cleaning_gamma_point-like_off04deg.fits.gz

LST1 School 2022[edit]

https://www.lst1.iac.es/wiki/index.php/LST1School2021_MVA

LST1 DL2/DL3 Data[edit]

  • Crab DL3
    Location: /fefs/aswg/workspace/daniel.morcuende/data/real/DL3/20201120/v0.7.3/tailcut84/
    • dl3_LST-1.RunXXX.fits.gz
    • hdu-index.fits.gz
    • obs-index.fits.gz
  • Crab DL2 with dynamic cleaning & MC (tuned NSB & PSF - ICRC version)
    Location: /fefs/aswg/workspace/abelardo.moralejo/Crab_tuned_MC_20210924/20201120/merged/DL2/
    • dl2_LST-1.Run029*h5
    • dl2_merged.h5


  • Check DL2 data: RF models & config
cd /fefs/aswg/data/real/DL2/20220206/v0.8.4/tailcut84/log

cat sequencer_v0.8.4.cfg 

** CONFIGURATION:

dl1ab_config: /fefs/aswg/lstosa/lstchain_config_v083_tailcut84.json
# In principle both config files should be the same, but in case
# the MC DL2 uses a different config file, it can be set here.
# For testing purposes
dl2_config: /fefs/aswg/lstosa/lstchain_config_v083_tailcut84.json

** MODELS:
rf_models: /fefs/aswg/workspace/abelardo.moralejo/Crab_tuned_MC_20211231/MC/DL1b_tuned/training/models

DL2toDL3 conversion[edit]

General talk on the LST1 DL3 Tools by Chaitanya Priyadarshi: DL3_Tools_and_Spectral_analysis_of_Crab_data_with_gammapy.pdf

DL2toDL3 Tools[edit]

Tools are located in lstchain/tools:

  • lstchain_create_irf_file: create IRFs from the MC file
    • The IRFs can be generated using only the gamma MC file, to produce Effective Area, Energy Dispersion
      (and PSF for Full Enclosure IRFs, ie. without any theta cuts).
    • Proton and Electron MC are used to create the Background IRF.
    • For creating point-like IRFs, you have to pass
      the --point-like flag in the command-line, as by default the Tools generates a Full Enclosure IRF.
  • lstchain_create_dl3_file: create DL3 files
  • lstchain_create_dl3_index_files: generate HDU and OBS Index files of the DL3 files created.
    The DL3 files are indexed as per GADF and using ctapipe tools

Command options can be checked: lstchain_create_irf_file --help-all

DL2 Cuts[edit]

Fixed cut variables & default values (see here)

  • fixed_gh_cut [0.6]
  • fixed_theta_cut [0.2]
  • allowed_tels [1]

Cut Methods:

  1. --fixed-gh-cut 0.9
  2. --config /path/to/config.json
    Example json config files for the DL3 tools are here
    The DL3 JSON config file (dl3_tool_config.json) is a subset of the IRF config file (irf_tool_config.json).
    One can use the IRF config file for both tools to ensure that the same filter cuts for both MC and real data are used.

NOTE: the fixed_theta_cut option is not available in lstchain_create_dl3_file, so it is better to use the JSON option to be consistent with the IRF generation

Example DL2toDL3 commands[edit]

lstchain_create_irf_files
        -g /path/to/DL2_MC_gamma_file.h5
        -o /path/to/irf.fits.gz
        --point-like (Only for point_like IRFs)
        --fixed-gh-cut 0.9
        --fixed-theta-cut 0.2
        --irf-obs-time 50

lstchain_create_dl3_file
        -d /path/to/DL2_data_file.h5
        -o /path/to/DL3/file/
        --input-irf /path/to/irf.fits.gz
        --source-name Crab
        --source-ra 83.633deg
        --source-dec 22.01deg
        --fixed-gh-cut 0.9

lstchain_create_dl3_index_files
        -d /path/to/DL3/files/
        -o /path/to/DL3/index/files
        -p dl3*[run_1-run_n]*.fits.gz

Example DL2toDL3 commands - MVA specific[edit]

lstchain_create_irf_files -g /Users/monicava/LST1_DATA/Abelardo/dl2_merged.h5 -o /Users/monicava/LST1_DATA/Abelardo/irf.gh01.fits.gz --point-like --config irf_tool_config_gh01.json --irf-obs-time 50
lstchain_create_dl3_file -d /Users/monicava/LST1_DATA/Abelardo/dl2_LST-1.Run02967.h5 -o /Users/monicava/LST1_DATA/Abelardo/DL3_gh01 --input-irf /Users/monicava/LST1_DATA/Abelardo/irf.gh01.fits.gz --source-name Crab --source-ra 83.633deg --source-dec 22.01deg --config irf_tool_config_gh01.json
lstchain_create_dl3_index_files -d /Users/monicava/LST1_DATA/Abelardo/DL3_gh01

Run summary info/data-check[edit]

LST1 data formats[edit]

File content can be obtained as follows: 

from lstchain.io import get_dataset_keys
from lstchain.io.io import dl1_params_lstcam_key, dl2_params_lstcam_key, dl1_images_lstcam_key
dl2_file = "/fefs/aswg/data/real/DL2/20210903/v0.7.3/tailcut84/dl2_LST-1.Run05998.0000.h5"
import pandas as pd
get_dataset_keys(dl2_file)

Out[]: 
['configuration/instrument/subarray/layout',
 'configuration/instrument/subarray/layout.__table_column_meta__',
 'configuration/instrument/telescope/camera/geometry_LSTCam',
 'configuration/instrument/telescope/camera/geometry_LSTCam.__table_column_meta__',
 'configuration/instrument/telescope/camera/readout_LSTCam',
 'configuration/instrument/telescope/camera/readout_LSTCam.__table_column_meta__',
 'configuration/instrument/telescope/optics',
 'configuration/instrument/telescope/optics.__table_column_meta__',
 'dl1/event/telescope/monitoring/calibration',
 'dl1/event/telescope/monitoring/flatfield',
 'dl1/event/telescope/monitoring/pedestal',
 'dl2/event/telescope/parameters/LST_LSTCam']

dl1_file = "/fefs/aswg/data/real/DL1/20210903/v0.7.3/tailcut84/dl1_LST-1.Run05998.0000.h5"
get_dataset_keys(dl1_file)

Out[]: 
['configuration/instrument/subarray/layout',
 'configuration/instrument/subarray/layout.__table_column_meta__',
 'configuration/instrument/telescope/camera/geometry_LSTCam',
 'configuration/instrument/telescope/camera/geometry_LSTCam.__table_column_meta__',
 'configuration/instrument/telescope/camera/readout_LSTCam',
 'configuration/instrument/telescope/camera/readout_LSTCam.__table_column_meta__',
 'configuration/instrument/telescope/optics',
 'configuration/instrument/telescope/optics.__table_column_meta__',
 'dl1/event/telescope/image/LST_LSTCam',
 'dl1/event/telescope/monitoring/calibration',
 'dl1/event/telescope/monitoring/flatfield',
 'dl1/event/telescope/monitoring/pedestal',
 'dl1/event/telescope/parameters/LST_LSTCam']

Data is in the tables:

  • 'dl1/event/telescope/image/LST_LSTCam' (camera calibrated images)
  • 'dl1/event/telescope/parameters/LST_LSTCam' (Hillas parameters, timestamps, pointing, etc)

In case of DL2 only the table parameters 'dl2/event/telescope/parameters/LST_LSTCam' are available, which is a copy of the DL1 table with the reconstructed parameters added

To access the parameter tables in DL1 and DL2: 

dl2_dataframe = pd.read_hdf(file, key=dl2_params_lstcam_key)
dl2_dataframe

Out[]: 
       obs_id  event_id      intensity  log_intensity         x         y         r       phi    length  ...  reco_energy  reco_disp_dx  reco_disp_dy  reco_src_x  reco_src_y  reco_alt   reco_az  reco_type  gammaness
0        5998         1    5529.521235       3.742688 -0.032178 -0.020007  0.037890 -2.585328  1.168153  ...    12.028581     -1.395368     -0.322723   -1.427546   -0.342730  1.084904  0.228667        101   0.025000
1        5998         4      61.694788       1.790248  0.093973  0.365420  0.377309  1.319085  0.045765  ...     0.021868      0.033724      0.019880    0.127698    0.385300  1.140388  0.287830        101   0.056333
2        5998         5     131.579536       2.119188  0.242024  0.147060  0.283200  0.546006  0.123974  ...     0.114741      0.246228     -0.281001    0.488253   -0.133941  1.153445  0.243046        101   0.188333
3        5998         6      37.598133       1.575166  0.405366 -0.546641  0.680542 -0.932721  0.072361  ...     0.062491      0.011559     -0.049816    0.416924   -0.596457  1.150419  0.202634        101   0.290833
4        5998         7      75.891508       1.880193 -0.129859 -0.524562  0.540397 -1.813474  0.078325  ...     0.033458      0.010284      0.027173   -0.119575   -0.497389  1.131426  0.213075        101   0.165000
...       ...       ...            ...            ...       ...       ...       ...       ...       ...  ...          ...           ...           ...         ...         ...       ...       ...        ...        ...
28271    5998     52989      89.536526       1.952000 -0.775412  0.466715  0.905034  2.599783  0.061538  ...     0.043203     -0.004094      0.009934   -0.779506    0.476649  1.108024  0.292202        101   0.165000
28272    5998     52990     113.433699       2.054742 -0.015224 -0.558476  0.558683 -1.598049  0.072026  ...     0.034507     -0.150657      0.088593   -0.165881   -0.469883  1.129934  0.214743        101   0.091000
28273    5998     52991    3053.237697       3.484761  0.621141  0.404424  0.741198  0.577148  0.319016  ...     0.218918      0.064154     -0.829353    0.685294   -0.424929  1.160371  0.216046        101   0.010000
28274    5998     52992  132592.328588       5.122518  0.000555  0.000571  0.000796  0.799391  1.163453  ...    30.287504     -0.286239      0.399654   -0.285684    0.400225  1.125740  0.287283        101   0.178036
28275    5998     52997      74.290200       1.870932 -0.595926  0.781502  0.982788  2.222275  0.097937  ...     0.292780      0.711934      0.310702    0.116007    1.092203  1.138651  0.347324        101   0.442667

[28276 rows x 51 columns]

Accesing images is more complicated and requires the tables module 

from astropy.table import Table
images = Table.read(dl1_file, path=dl1_images_lstcam_key)['image']
images

Out[]: 
<Column name='image' dtype='float32' shape=(1855,) length=53000>
    8.701056 .. 7.1084614
   3.7857776 .. 1.2068052
   3.8285189 .. 0.9846253
   4.5408783 .. 5.1921587
     4.754586 .. 1.026284
  0.7226335 .. 0.09590531
   3.4738295 .. 2.0677526
 2.8597107 .. -0.43177217
 -0.20343336 .. 3.8313065
   1.3372487 .. 1.5678478
    3.2301373 .. 2.428795
   3.4438455 .. 4.2895527
    3.871757 .. 1.4984165
  -0.5311186 .. 1.8316865
    2.717239 .. 4.3192797
  1.3362556 .. 0.48472026
 0.52366954 .. 0.26254028
 1.7769248 .. -0.06901258
   2.6032615 .. 3.2203112
   1.7484305 .. 1.8331614
   4.7545853 .. 1.7067102
   2.2185874 .. 1.2590411
-0.43089166 .. 0.29031274
   2.7479038 .. 1.5956203
                      ...

for each of the 53000 events, a calibrated image of the camera is obtained (array of 1855 pixels)

The rest of the tables in the files are metadata (info about the camera, optics, calibration, etc)

Software tool installation[edit]

LSTCHAIN[edit]

Donwload LST analysis software cta-lstchain (wrapper that uses centrally supported ctapipe routines) 

git clone https://github.com/cta-observatory/cta-lstchain.git
cd cta-lstchain

LSTCHAIN_VER=0.7.2
wget https://raw.githubusercontent.com/cta-observatory/cta-lstchain/v$LSTCHAIN_VER/environment.yml
conda env create -n lst -f environment.yml
conda activate lst
pip install lstchain==$LSTCHAIN_VER
rm environment.yml

In case of problems with conda env try mamba:
conda install -c conda-forge -n base mamba
mamba env create -f environment.yml

Files needed to analyze golden Run 442 (20190527):

  • Pedestal run: pedestal_file_run446_0000.fits
  • Low level DRS4 calibration: calibration.hdf5

Run directly jupyter notebook[edit]

cd cta-lstchain/notebooks
jupyter notebook
-> select Analyze_real_muon_data.ipynb

Convert example jupyter notebook to python[edit]

cd cta-lstchain/notebooks
jupyter nbconvert --to python Analyze_real_muon_data.ipynb

Fix path of data, pedestal and calibration and run! 

Lines to fix:
source = event_source(input_url="../data/LST-1.4.Run00442.0001.fits.fz",max_events=None)
pedestal_path="../pedestal/pedestal_file_run446_0000.fits",
with HDF5TableReader('../calibration/calibration.hdf5') as h5_table:

python Analyze_real_muon_data.py

Gammapy[edit]

curl -O https://gammapy.org/download/install/gammapy-0.18.2-environment.yml
conda env create -f gammapy-0.18.2-environment.yml
conda activate gammapy-0.18.2
gammapy info

##### Installing local copy
git clone  -b v0.18.2 https://github.com/gammapy/gammapy
cd gammapy
pip install -e .

Installation in Ubuntu 

conda create -n gammapyenv python=3.7
conda activate gammapyenv
conda install gammapy=0.18.2 sherpa=4.12.0
conda install jupyter ipython jupyterlab pandas healpy iminuit naima emcee corner parfice
gammapy info

Ctools[edit]

Installation MAC 

conda config --append channels conda-forge
conda config --append channels cta-observatory
conda create -n ctathai python=3.8
conda activate ctathai
conda install ctools
conda install matplotlib
conda install jupyterlab
export CALDB=/path/where/you/installed/IRF/caldb
export HESSDATA=/path/where/you/installed/HESS/data

Installation UBUNTU 

conda create -n ctathai python=3.7
conda activate ctathai
conda install -c cta-observatory ctools=1.7.4
conda install matplotlib
conda install jupyterlab
export CALDB=/path/where/you/installed/IRF/caldb
export HESSDATA=/path/where/you/installed/HESS/data


Commands: 

ctobssim  edisp=yes
fv events.fits
ctskymap 
ds9 skymap.fits 
ctbin
fv cntcube.fits 
ctexpcube
ctpsfcube
ctbkgcube
ctlike
csresmap
csresspec components=yes
$CTOOLS/share/examples/python/show_residuals.py resspec.fits

Fermipy[edit]

conda config --append channels fermi
conda create -n fermiset -c conda-forge -c fermi fermitools=2.0.8 python=3.7 clhep=2.4.4.1
conda activate fermiset
conda install fermipy

CTApipe[edit]

git clone https://github.com/cta-observatory/ctapipe.git
cd ctapipe

CTAPIPE_VER=0.10.5
wget https://raw.githubusercontent.com/cta-observatory/ctapipe/v$CTAPIPE_VER/environment.yml
conda env create -n cta -f environment.yml
conda activate cta
conda install -c conda-forge ctapipe=$CTAPIPE_VER

Ubuntu 20.04 issues[edit]

  • Mars/root does not work so gcc needs to be downgrade to version 5: gcc downgrade
  • System Program Problem Detected at reboot:
sudo gedit /etc/default/apport 
enabled=0
  • Problem installing ctools with conda, modify ~/.condarc
channel_priority: flexible


Extra[edit]

Conda useful commands[edit]

conda remove --name gammapy_default --all
conda list -n gammapy-0.18.2

MAGIC DL3 data[edit]

https://github.com/open-gamma-ray-astro/joint-crab/tree/master/data/magic

https://gitlab.pic.es/magic_dl3/magic_dl3_sw_school_2021/-/tree/master/data/magic

Notebook examples[edit]

https://github.com/rlopezcoto/intro-iact-analysis

Retrieved from "https://www.lst1.iac.es/wiki/index.php?title=LST1Analysis_MVA&oldid=3897"