Difference between revisions of "LST1Analysis MVA"

From my_wiki
Jump to: navigation, search
(→‎Location of LST1 data at PIC)
(→‎Convert example jupyter notebook to python)
Line 71: Line 71:
 
</pre>
 
</pre>
  
 +
== LST1 data formats ==
 +
El contenido de los ficheros se puede ver con:
 +
<pre style="color: blue">
 +
In [1]: from lstchain.io import get_dataset_keys
 +
 +
In [2]: from lstchain.io.io import dl1_params_lstcam_key, dl2_params_lstcam_key, dl1_images_lstcam_key
 +
 +
In [3]: file = "/fefs/aswg/data/real/DL2/20210903/v0.7.3/tailcut84/dl2_LST-1.Run05998.0000.h5"
 +
 +
In [4]: import pandas as pd
 +
 +
In [5]: get_dataset_keys(file)
 +
Out[5]:
 +
['configuration/instrument/subarray/layout',
 +
'configuration/instrument/subarray/layout.__table_column_meta__',
 +
'configuration/instrument/telescope/camera/geometry_LSTCam',
 +
'configuration/instrument/telescope/camera/geometry_LSTCam.__table_column_meta__',
 +
'configuration/instrument/telescope/camera/readout_LSTCam',
 +
'configuration/instrument/telescope/camera/readout_LSTCam.__table_column_meta__',
 +
'configuration/instrument/telescope/optics',
 +
'configuration/instrument/telescope/optics.__table_column_meta__',
 +
'dl1/event/telescope/monitoring/calibration',
 +
'dl1/event/telescope/monitoring/flatfield',
 +
'dl1/event/telescope/monitoring/pedestal',
 +
'dl2/event/telescope/parameters/LST_LSTCam']
 +
 +
In [6]: dl1_file = "/fefs/aswg/data/real/DL1/20210903/v0.7.3/tailcut84/dl1_LST-1.Run05998.0000.h5"
 +
 +
In [7]: get_dataset_keys(dl1_file)
 +
Out[7]:
 +
['configuration/instrument/subarray/layout',
 +
'configuration/instrument/subarray/layout.__table_column_meta__',
 +
'configuration/instrument/telescope/camera/geometry_LSTCam',
 +
'configuration/instrument/telescope/camera/geometry_LSTCam.__table_column_meta__',
 +
'configuration/instrument/telescope/camera/readout_LSTCam',
 +
'configuration/instrument/telescope/camera/readout_LSTCam.__table_column_meta__',
 +
'configuration/instrument/telescope/optics',
 +
'configuration/instrument/telescope/optics.__table_column_meta__',
 +
'dl1/event/telescope/image/LST_LSTCam',
 +
'dl1/event/telescope/monitoring/calibration',
 +
'dl1/event/telescope/monitoring/flatfield',
 +
'dl1/event/telescope/monitoring/pedestal',
 +
'dl1/event/telescope/parameters/LST_LSTCam']
 +
</pre>
 +
 +
De ahí, los datos como tal están en las tablas 'dl1/event/telescope/image/LST_LSTCam' (imágenes calibradas de la cámara) y 'dl1/event/telescope/parameters/LST_LSTCam' (parámetros de Hillas, timestamps, pointing, etc)
 +
En el caso de dl2 solamente se tiene la tabla de parámetros 'dl2/event/telescope/parameters/LST_LSTCam' , que es una copia de la tabla dl1 con los parámetros reconstruidos añadidos.
 +
 +
Para acceder a la tabla de parámetros tanto en dl1 como en dl2:
 +
<pre style="color: blue">
 +
In [8]: dl2_dataframe = pd.read_hdf(file, key=dl2_params_lstcam_key)
 +
 +
In [9]: dl2_dataframe
 +
Out[9]:
 +
      obs_id  event_id      intensity  log_intensity        x        y        r      phi    length  ...  reco_energy  reco_disp_dx  reco_disp_dy  reco_src_x  reco_src_y  reco_alt  reco_az  reco_type  gammaness
 +
0        5998        1    5529.521235      3.742688 -0.032178 -0.020007  0.037890 -2.585328  1.168153  ...    12.028581    -1.395368    -0.322723  -1.427546  -0.342730  1.084904  0.228667        101  0.025000
 +
1        5998        4      61.694788      1.790248  0.093973  0.365420  0.377309  1.319085  0.045765  ...    0.021868      0.033724      0.019880    0.127698    0.385300  1.140388  0.287830        101  0.056333
 +
2        5998        5    131.579536      2.119188  0.242024  0.147060  0.283200  0.546006  0.123974  ...    0.114741      0.246228    -0.281001    0.488253  -0.133941  1.153445  0.243046        101  0.188333
 +
3        5998        6      37.598133      1.575166  0.405366 -0.546641  0.680542 -0.932721  0.072361  ...    0.062491      0.011559    -0.049816    0.416924  -0.596457  1.150419  0.202634        101  0.290833
 +
4        5998        7      75.891508      1.880193 -0.129859 -0.524562  0.540397 -1.813474  0.078325  ...    0.033458      0.010284      0.027173  -0.119575  -0.497389  1.131426  0.213075        101  0.165000
 +
...      ...      ...            ...            ...      ...      ...      ...      ...      ...  ...          ...          ...          ...        ...        ...      ...      ...        ...        ...
 +
28271    5998    52989      89.536526      1.952000 -0.775412  0.466715  0.905034  2.599783  0.061538  ...    0.043203    -0.004094      0.009934  -0.779506    0.476649  1.108024  0.292202        101  0.165000
 +
28272    5998    52990    113.433699      2.054742 -0.015224 -0.558476  0.558683 -1.598049  0.072026  ...    0.034507    -0.150657      0.088593  -0.165881  -0.469883  1.129934  0.214743        101  0.091000
 +
28273    5998    52991    3053.237697      3.484761  0.621141  0.404424  0.741198  0.577148  0.319016  ...    0.218918      0.064154    -0.829353    0.685294  -0.424929  1.160371  0.216046        101  0.010000
 +
28274    5998    52992  132592.328588      5.122518  0.000555  0.000571  0.000796  0.799391  1.163453  ...    30.287504    -0.286239      0.399654  -0.285684    0.400225  1.125740  0.287283        101  0.178036
 +
28275    5998    52997      74.290200      1.870932 -0.595926  0.781502  0.982788  2.222275  0.097937  ...    0.292780      0.711934      0.310702    0.116007    1.092203  1.138651  0.347324        101  0.442667
 +
 +
[28276 rows x 51 columns]
 +
 +
12:08
 +
acceder a las imágenes es algo más complicado y requiere el módulo tables  o from astropy.table import Table :
 +
 +
In [11]: images = Table.read(dl1_file, path=dl1_images_lstcam_key)['image']
 +
 +
In [12]: images
 +
Out[12]:
 +
<Column name='image' dtype='float32' shape=(1855,) length=53000>
 +
    8.701056 .. 7.1084614
 +
  3.7857776 .. 1.2068052
 +
  3.8285189 .. 0.9846253
 +
  4.5408783 .. 5.1921587
 +
    4.754586 .. 1.026284
 +
  0.7226335 .. 0.09590531
 +
  3.4738295 .. 2.0677526
 +
2.8597107 .. -0.43177217
 +
-0.20343336 .. 3.8313065
 +
  1.3372487 .. 1.5678478
 +
    3.2301373 .. 2.428795
 +
  3.4438455 .. 4.2895527
 +
    3.871757 .. 1.4984165
 +
  -0.5311186 .. 1.8316865
 +
    2.717239 .. 4.3192797
 +
  1.3362556 .. 0.48472026
 +
0.52366954 .. 0.26254028
 +
1.7769248 .. -0.06901258
 +
  2.6032615 .. 3.2203112
 +
  1.7484305 .. 1.8331614
 +
  4.7545853 .. 1.7067102
 +
  2.2185874 .. 1.2590411
 +
-0.43089166 .. 0.29031274
 +
  2.7479038 .. 1.5956203
 +
                      ...
 +
</pre>
  
 
== Gammapy ==
 
== Gammapy ==

Revision as of 11:27, 7 October 2021

Location of LST1 data at PIC

Location of LST1 at La Palma

  • Directory: /fefs/onsite/data
    Use cp01/cp02 to copy out data, never the login machine

Run summary info/data-check

CTApipe

git clone https://github.com/cta-observatory/ctapipe.git
cd ctapipe

CTAPIPE_VER=0.10.5
wget https://raw.githubusercontent.com/cta-observatory/ctapipe/v$CTAPIPE_VER/environment.yml
conda env create -n cta -f environment.yml
conda activate cta
conda install -c conda-forge ctapipe=$CTAPIPE_VER

Analysis of LST1 data

Donwload LST analysis software cta-lstchain (wrapper that uses centrally supported ctapipe routines) 

git clone https://github.com/cta-observatory/cta-lstchain.git
cd cta-lstchain

LSTCHAIN_VER=0.7.2
wget https://raw.githubusercontent.com/cta-observatory/cta-lstchain/v$LSTCHAIN_VER/environment.yml
conda env create -n lst -f environment.yml
conda activate lst
pip install lstchain==$LSTCHAIN_VER
rm environment.yml

In case of problems with conda env try mamba:
conda install -c conda-forge -n base mamba
mamba env create -f environment.yml

Files needed to analyze golden Run 442 (20190527):

  • Pedestal run: pedestal_file_run446_0000.fits
  • Low level DRS4 calibration: calibration.hdf5

Run directly jupyter notebook

cd cta-lstchain/notebooks
jupyter notebook
-> select Analyze_real_muon_data.ipynb

Convert example jupyter notebook to python

cd cta-lstchain/notebooks
jupyter nbconvert --to python Analyze_real_muon_data.ipynb

Fix path of data, pedestal and calibration and run! 

Lines to fix:
source = event_source(input_url="../data/LST-1.4.Run00442.0001.fits.fz",max_events=None)
pedestal_path="../pedestal/pedestal_file_run446_0000.fits",
with HDF5TableReader('../calibration/calibration.hdf5') as h5_table:

python Analyze_real_muon_data.py

LST1 data formats

El contenido de los ficheros se puede ver con: 

In [1]: from lstchain.io import get_dataset_keys

In [2]: from lstchain.io.io import dl1_params_lstcam_key, dl2_params_lstcam_key, dl1_images_lstcam_key

In [3]: file = "/fefs/aswg/data/real/DL2/20210903/v0.7.3/tailcut84/dl2_LST-1.Run05998.0000.h5"

In [4]: import pandas as pd

In [5]: get_dataset_keys(file)
Out[5]: 
['configuration/instrument/subarray/layout',
 'configuration/instrument/subarray/layout.__table_column_meta__',
 'configuration/instrument/telescope/camera/geometry_LSTCam',
 'configuration/instrument/telescope/camera/geometry_LSTCam.__table_column_meta__',
 'configuration/instrument/telescope/camera/readout_LSTCam',
 'configuration/instrument/telescope/camera/readout_LSTCam.__table_column_meta__',
 'configuration/instrument/telescope/optics',
 'configuration/instrument/telescope/optics.__table_column_meta__',
 'dl1/event/telescope/monitoring/calibration',
 'dl1/event/telescope/monitoring/flatfield',
 'dl1/event/telescope/monitoring/pedestal',
 'dl2/event/telescope/parameters/LST_LSTCam']

In [6]: dl1_file = "/fefs/aswg/data/real/DL1/20210903/v0.7.3/tailcut84/dl1_LST-1.Run05998.0000.h5"

In [7]: get_dataset_keys(dl1_file)
Out[7]: 
['configuration/instrument/subarray/layout',
 'configuration/instrument/subarray/layout.__table_column_meta__',
 'configuration/instrument/telescope/camera/geometry_LSTCam',
 'configuration/instrument/telescope/camera/geometry_LSTCam.__table_column_meta__',
 'configuration/instrument/telescope/camera/readout_LSTCam',
 'configuration/instrument/telescope/camera/readout_LSTCam.__table_column_meta__',
 'configuration/instrument/telescope/optics',
 'configuration/instrument/telescope/optics.__table_column_meta__',
 'dl1/event/telescope/image/LST_LSTCam',
 'dl1/event/telescope/monitoring/calibration',
 'dl1/event/telescope/monitoring/flatfield',
 'dl1/event/telescope/monitoring/pedestal',
 'dl1/event/telescope/parameters/LST_LSTCam']

De ahí, los datos como tal están en las tablas 'dl1/event/telescope/image/LST_LSTCam' (imágenes calibradas de la cámara) y 'dl1/event/telescope/parameters/LST_LSTCam' (parámetros de Hillas, timestamps, pointing, etc) En el caso de dl2 solamente se tiene la tabla de parámetros 'dl2/event/telescope/parameters/LST_LSTCam' , que es una copia de la tabla dl1 con los parámetros reconstruidos añadidos.

Para acceder a la tabla de parámetros tanto en dl1 como en dl2: 

In [8]: dl2_dataframe = pd.read_hdf(file, key=dl2_params_lstcam_key)

In [9]: dl2_dataframe
Out[9]: 
       obs_id  event_id      intensity  log_intensity         x         y         r       phi    length  ...  reco_energy  reco_disp_dx  reco_disp_dy  reco_src_x  reco_src_y  reco_alt   reco_az  reco_type  gammaness
0        5998         1    5529.521235       3.742688 -0.032178 -0.020007  0.037890 -2.585328  1.168153  ...    12.028581     -1.395368     -0.322723   -1.427546   -0.342730  1.084904  0.228667        101   0.025000
1        5998         4      61.694788       1.790248  0.093973  0.365420  0.377309  1.319085  0.045765  ...     0.021868      0.033724      0.019880    0.127698    0.385300  1.140388  0.287830        101   0.056333
2        5998         5     131.579536       2.119188  0.242024  0.147060  0.283200  0.546006  0.123974  ...     0.114741      0.246228     -0.281001    0.488253   -0.133941  1.153445  0.243046        101   0.188333
3        5998         6      37.598133       1.575166  0.405366 -0.546641  0.680542 -0.932721  0.072361  ...     0.062491      0.011559     -0.049816    0.416924   -0.596457  1.150419  0.202634        101   0.290833
4        5998         7      75.891508       1.880193 -0.129859 -0.524562  0.540397 -1.813474  0.078325  ...     0.033458      0.010284      0.027173   -0.119575   -0.497389  1.131426  0.213075        101   0.165000
...       ...       ...            ...            ...       ...       ...       ...       ...       ...  ...          ...           ...           ...         ...         ...       ...       ...        ...        ...
28271    5998     52989      89.536526       1.952000 -0.775412  0.466715  0.905034  2.599783  0.061538  ...     0.043203     -0.004094      0.009934   -0.779506    0.476649  1.108024  0.292202        101   0.165000
28272    5998     52990     113.433699       2.054742 -0.015224 -0.558476  0.558683 -1.598049  0.072026  ...     0.034507     -0.150657      0.088593   -0.165881   -0.469883  1.129934  0.214743        101   0.091000
28273    5998     52991    3053.237697       3.484761  0.621141  0.404424  0.741198  0.577148  0.319016  ...     0.218918      0.064154     -0.829353    0.685294   -0.424929  1.160371  0.216046        101   0.010000
28274    5998     52992  132592.328588       5.122518  0.000555  0.000571  0.000796  0.799391  1.163453  ...    30.287504     -0.286239      0.399654   -0.285684    0.400225  1.125740  0.287283        101   0.178036
28275    5998     52997      74.290200       1.870932 -0.595926  0.781502  0.982788  2.222275  0.097937  ...     0.292780      0.711934      0.310702    0.116007    1.092203  1.138651  0.347324        101   0.442667

[28276 rows x 51 columns]

12:08
acceder a las imágenes es algo más complicado y requiere el módulo tables  o from astropy.table import Table :

In [11]: images = Table.read(dl1_file, path=dl1_images_lstcam_key)['image']

In [12]: images
Out[12]: 
<Column name='image' dtype='float32' shape=(1855,) length=53000>
    8.701056 .. 7.1084614
   3.7857776 .. 1.2068052
   3.8285189 .. 0.9846253
   4.5408783 .. 5.1921587
     4.754586 .. 1.026284
  0.7226335 .. 0.09590531
   3.4738295 .. 2.0677526
 2.8597107 .. -0.43177217
 -0.20343336 .. 3.8313065
   1.3372487 .. 1.5678478
    3.2301373 .. 2.428795
   3.4438455 .. 4.2895527
    3.871757 .. 1.4984165
  -0.5311186 .. 1.8316865
    2.717239 .. 4.3192797
  1.3362556 .. 0.48472026
 0.52366954 .. 0.26254028
 1.7769248 .. -0.06901258
   2.6032615 .. 3.2203112
   1.7484305 .. 1.8331614
   4.7545853 .. 1.7067102
   2.2185874 .. 1.2590411
-0.43089166 .. 0.29031274
   2.7479038 .. 1.5956203
                      ...

Gammapy

FIX UBUNTU
conda create -n gammapyenv python=3.7
conda activate gammapyenv
conda install gammapy=0.18.2 sherpa=4.12.0
conda install jupyter ipython jupyterlab pandas healpy iminuit naima emcee corner parfice
gammapy info


curl -O https://gammapy.org/download/install/gammapy-0.18.2-environment.yml
conda env create -f gammapy-0.18.2-environment.yml
conda activate gammapy-0.18.2
gammapy info

Ctools

Installation MAC 

conda config --append channels conda-forge
conda config --append channels cta-observatory
conda create -n ctathai python=3.8
conda activate ctathai
conda install ctools
conda install matplotlib
conda install jupyterlab
export CALDB=/path/where/you/installed/IRF/caldb
export HESSDATA=/path/where/you/installed/HESS/data

Installation UBUNTU 

conda create -n ctathai python=3.7
conda activate ctathai
conda install -c cta-observatory ctools=1.7.4
conda install matplotlib
conda install jupyterlab
export CALDB=/path/where/you/installed/IRF/caldb
export HESSDATA=/path/where/you/installed/HESS/data


Commands: 

ctobssim  edisp=yes
fv events.fits
ctskymap 
ds9 skymap.fits 
ctbin
fv cntcube.fits 
ctexpcube
ctpsfcube
ctbkgcube
ctlike
csresmap
csresspec components=yes
$CTOOLS/share/examples/python/show_residuals.py resspec.fits

Fermipy

conda config --append channels fermi
conda create -n fermiset -c conda-forge -c fermi fermitools=2.0.8 python=3.7 clhep=2.4.4.1
conda activate fermiset
conda install fermipy

Ubuntu 20.04 issues

  • Mars/root does not work so gcc needs to be downgrade to version 5: gcc downgrade
  • System Program Problem Detected at reboot:
sudo gedit /etc/default/apport 
enabled=0
  • Problem installing ctools with conda, modify ~/.condarc
channel_priority: flexible
Retrieved from "https://www.lst1.iac.es/wiki/index.php?title=LST1Analysis_MVA&oldid=3258"