Difference between revisions of "LST1Analysis MVA"

From my_wiki
Jump to: navigation, search
(→‎Location of LST1 data at PIC)
Line 15: Line 15:
 
* [http://sagan.gae.ucm.es/LST1/ Run-summary web in GAE] <br/> LST1 account
 
* [http://sagan.gae.ucm.es/LST1/ Run-summary web in GAE] <br/> LST1 account
  
−
== CTApipe ==
 
−
<pre style="color: blue">
 
−
git clone https://github.com/cta-observatory/ctapipe.git
 
−
cd ctapipe
 
  
−
CTAPIPE_VER=0.10.5
 
−
wget https://raw.githubusercontent.com/cta-observatory/ctapipe/v$CTAPIPE_VER/environment.yml
 
−
conda env create -n cta -f environment.yml
 
−
conda activate cta
 
−
conda install -c conda-forge ctapipe=$CTAPIPE_VER
 
−
</pre>
 
  
 
== Analysis of LST1 data ==
 
== Analysis of LST1 data ==
Line 253: Line 243:
 
conda activate fermiset
 
conda activate fermiset
 
conda install fermipy
 
conda install fermipy
 +
</pre>
 +
 +
== CTApipe ==
 +
<pre style="color: blue">
 +
git clone https://github.com/cta-observatory/ctapipe.git
 +
cd ctapipe
 +
 +
CTAPIPE_VER=0.10.5
 +
wget https://raw.githubusercontent.com/cta-observatory/ctapipe/v$CTAPIPE_VER/environment.yml
 +
conda env create -n cta -f environment.yml
 +
conda activate cta
 +
conda install -c conda-forge ctapipe=$CTAPIPE_VER
 
</pre>
 
</pre>
  

Revision as of 11:40, 7 October 2021

Location of LST1 data at PIC

Use CTA observatory LDAP account (monica.vazquez)

Location of LST1 at La Palma

  • Directory: /fefs/onsite/data
    Use cp01/cp02 to copy out data, never the login machine

Run summary info/data-check


Analysis of LST1 data

Donwload LST analysis software cta-lstchain (wrapper that uses centrally supported ctapipe routines) 

git clone https://github.com/cta-observatory/cta-lstchain.git
cd cta-lstchain

LSTCHAIN_VER=0.7.2
wget https://raw.githubusercontent.com/cta-observatory/cta-lstchain/v$LSTCHAIN_VER/environment.yml
conda env create -n lst -f environment.yml
conda activate lst
pip install lstchain==$LSTCHAIN_VER
rm environment.yml

In case of problems with conda env try mamba:
conda install -c conda-forge -n base mamba
mamba env create -f environment.yml

Files needed to analyze golden Run 442 (20190527):

  • Pedestal run: pedestal_file_run446_0000.fits
  • Low level DRS4 calibration: calibration.hdf5

Run directly jupyter notebook

cd cta-lstchain/notebooks
jupyter notebook
-> select Analyze_real_muon_data.ipynb

Convert example jupyter notebook to python

cd cta-lstchain/notebooks
jupyter nbconvert --to python Analyze_real_muon_data.ipynb

Fix path of data, pedestal and calibration and run! 

Lines to fix:
source = event_source(input_url="../data/LST-1.4.Run00442.0001.fits.fz",max_events=None)
pedestal_path="../pedestal/pedestal_file_run446_0000.fits",
with HDF5TableReader('../calibration/calibration.hdf5') as h5_table:

python Analyze_real_muon_data.py

LST1 data formats

El contenido de los ficheros se puede ver con: 

In [1]: from lstchain.io import get_dataset_keys

In [2]: from lstchain.io.io import dl1_params_lstcam_key, dl2_params_lstcam_key, dl1_images_lstcam_key

In [3]: file = "/fefs/aswg/data/real/DL2/20210903/v0.7.3/tailcut84/dl2_LST-1.Run05998.0000.h5"

In [4]: import pandas as pd

In [5]: get_dataset_keys(file)
Out[5]: 
['configuration/instrument/subarray/layout',
 'configuration/instrument/subarray/layout.__table_column_meta__',
 'configuration/instrument/telescope/camera/geometry_LSTCam',
 'configuration/instrument/telescope/camera/geometry_LSTCam.__table_column_meta__',
 'configuration/instrument/telescope/camera/readout_LSTCam',
 'configuration/instrument/telescope/camera/readout_LSTCam.__table_column_meta__',
 'configuration/instrument/telescope/optics',
 'configuration/instrument/telescope/optics.__table_column_meta__',
 'dl1/event/telescope/monitoring/calibration',
 'dl1/event/telescope/monitoring/flatfield',
 'dl1/event/telescope/monitoring/pedestal',
 'dl2/event/telescope/parameters/LST_LSTCam']

In [6]: dl1_file = "/fefs/aswg/data/real/DL1/20210903/v0.7.3/tailcut84/dl1_LST-1.Run05998.0000.h5"

In [7]: get_dataset_keys(dl1_file)
Out[7]: 
['configuration/instrument/subarray/layout',
 'configuration/instrument/subarray/layout.__table_column_meta__',
 'configuration/instrument/telescope/camera/geometry_LSTCam',
 'configuration/instrument/telescope/camera/geometry_LSTCam.__table_column_meta__',
 'configuration/instrument/telescope/camera/readout_LSTCam',
 'configuration/instrument/telescope/camera/readout_LSTCam.__table_column_meta__',
 'configuration/instrument/telescope/optics',
 'configuration/instrument/telescope/optics.__table_column_meta__',
 'dl1/event/telescope/image/LST_LSTCam',
 'dl1/event/telescope/monitoring/calibration',
 'dl1/event/telescope/monitoring/flatfield',
 'dl1/event/telescope/monitoring/pedestal',
 'dl1/event/telescope/parameters/LST_LSTCam']

De ahí, los datos como tal están en las tablas 'dl1/event/telescope/image/LST_LSTCam' (imágenes calibradas de la cámara) y 'dl1/event/telescope/parameters/LST_LSTCam' (parámetros de Hillas, timestamps, pointing, etc) En el caso de dl2 solamente se tiene la tabla de parámetros 'dl2/event/telescope/parameters/LST_LSTCam' , que es una copia de la tabla dl1 con los parámetros reconstruidos añadidos.

Para acceder a la tabla de parámetros tanto en dl1 como en dl2: 

In [8]: dl2_dataframe = pd.read_hdf(file, key=dl2_params_lstcam_key)

In [9]: dl2_dataframe
Out[9]: 
       obs_id  event_id      intensity  log_intensity         x         y         r       phi    length  ...  reco_energy  reco_disp_dx  reco_disp_dy  reco_src_x  reco_src_y  reco_alt   reco_az  reco_type  gammaness
0        5998         1    5529.521235       3.742688 -0.032178 -0.020007  0.037890 -2.585328  1.168153  ...    12.028581     -1.395368     -0.322723   -1.427546   -0.342730  1.084904  0.228667        101   0.025000
1        5998         4      61.694788       1.790248  0.093973  0.365420  0.377309  1.319085  0.045765  ...     0.021868      0.033724      0.019880    0.127698    0.385300  1.140388  0.287830        101   0.056333
2        5998         5     131.579536       2.119188  0.242024  0.147060  0.283200  0.546006  0.123974  ...     0.114741      0.246228     -0.281001    0.488253   -0.133941  1.153445  0.243046        101   0.188333
3        5998         6      37.598133       1.575166  0.405366 -0.546641  0.680542 -0.932721  0.072361  ...     0.062491      0.011559     -0.049816    0.416924   -0.596457  1.150419  0.202634        101   0.290833
4        5998         7      75.891508       1.880193 -0.129859 -0.524562  0.540397 -1.813474  0.078325  ...     0.033458      0.010284      0.027173   -0.119575   -0.497389  1.131426  0.213075        101   0.165000
...       ...       ...            ...            ...       ...       ...       ...       ...       ...  ...          ...           ...           ...         ...         ...       ...       ...        ...        ...
28271    5998     52989      89.536526       1.952000 -0.775412  0.466715  0.905034  2.599783  0.061538  ...     0.043203     -0.004094      0.009934   -0.779506    0.476649  1.108024  0.292202        101   0.165000
28272    5998     52990     113.433699       2.054742 -0.015224 -0.558476  0.558683 -1.598049  0.072026  ...     0.034507     -0.150657      0.088593   -0.165881   -0.469883  1.129934  0.214743        101   0.091000
28273    5998     52991    3053.237697       3.484761  0.621141  0.404424  0.741198  0.577148  0.319016  ...     0.218918      0.064154     -0.829353    0.685294   -0.424929  1.160371  0.216046        101   0.010000
28274    5998     52992  132592.328588       5.122518  0.000555  0.000571  0.000796  0.799391  1.163453  ...    30.287504     -0.286239      0.399654   -0.285684    0.400225  1.125740  0.287283        101   0.178036
28275    5998     52997      74.290200       1.870932 -0.595926  0.781502  0.982788  2.222275  0.097937  ...     0.292780      0.711934      0.310702    0.116007    1.092203  1.138651  0.347324        101   0.442667

[28276 rows x 51 columns]


Acceder a las imágenes es algo más complicado y requiere el módulo tables o from astropy.table import Table : 

In [11]: images = Table.read(dl1_file, path=dl1_images_lstcam_key)['image']

In [12]: images
Out[12]: 
<Column name='image' dtype='float32' shape=(1855,) length=53000>
    8.701056 .. 7.1084614
   3.7857776 .. 1.2068052
   3.8285189 .. 0.9846253
   4.5408783 .. 5.1921587
     4.754586 .. 1.026284
  0.7226335 .. 0.09590531
   3.4738295 .. 2.0677526
 2.8597107 .. -0.43177217
 -0.20343336 .. 3.8313065
   1.3372487 .. 1.5678478
    3.2301373 .. 2.428795
   3.4438455 .. 4.2895527
    3.871757 .. 1.4984165
  -0.5311186 .. 1.8316865
    2.717239 .. 4.3192797
  1.3362556 .. 0.48472026
 0.52366954 .. 0.26254028
 1.7769248 .. -0.06901258
   2.6032615 .. 3.2203112
   1.7484305 .. 1.8331614
   4.7545853 .. 1.7067102
   2.2185874 .. 1.2590411
-0.43089166 .. 0.29031274
   2.7479038 .. 1.5956203
                      ...

para cada uno de los 53000 eventos, se tiene una imagen calibrada de la cámara (array de 1855 pixels)

el resto de tablas en los ficheros son metadatos (info sobre cámara, óptica, calibración, etc)

Gammapy

FIX UBUNTU
conda create -n gammapyenv python=3.7
conda activate gammapyenv
conda install gammapy=0.18.2 sherpa=4.12.0
conda install jupyter ipython jupyterlab pandas healpy iminuit naima emcee corner parfice
gammapy info


curl -O https://gammapy.org/download/install/gammapy-0.18.2-environment.yml
conda env create -f gammapy-0.18.2-environment.yml
conda activate gammapy-0.18.2
gammapy info

Ctools

Installation MAC 

conda config --append channels conda-forge
conda config --append channels cta-observatory
conda create -n ctathai python=3.8
conda activate ctathai
conda install ctools
conda install matplotlib
conda install jupyterlab
export CALDB=/path/where/you/installed/IRF/caldb
export HESSDATA=/path/where/you/installed/HESS/data

Installation UBUNTU 

conda create -n ctathai python=3.7
conda activate ctathai
conda install -c cta-observatory ctools=1.7.4
conda install matplotlib
conda install jupyterlab
export CALDB=/path/where/you/installed/IRF/caldb
export HESSDATA=/path/where/you/installed/HESS/data


Commands: 

ctobssim  edisp=yes
fv events.fits
ctskymap 
ds9 skymap.fits 
ctbin
fv cntcube.fits 
ctexpcube
ctpsfcube
ctbkgcube
ctlike
csresmap
csresspec components=yes
$CTOOLS/share/examples/python/show_residuals.py resspec.fits

Fermipy

conda config --append channels fermi
conda create -n fermiset -c conda-forge -c fermi fermitools=2.0.8 python=3.7 clhep=2.4.4.1
conda activate fermiset
conda install fermipy

CTApipe

git clone https://github.com/cta-observatory/ctapipe.git
cd ctapipe

CTAPIPE_VER=0.10.5
wget https://raw.githubusercontent.com/cta-observatory/ctapipe/v$CTAPIPE_VER/environment.yml
conda env create -n cta -f environment.yml
conda activate cta
conda install -c conda-forge ctapipe=$CTAPIPE_VER

Ubuntu 20.04 issues

  • Mars/root does not work so gcc needs to be downgrade to version 5: gcc downgrade
  • System Program Problem Detected at reboot:
sudo gedit /etc/default/apport 
enabled=0
  • Problem installing ctools with conda, modify ~/.condarc
channel_priority: flexible
Retrieved from "https://www.lst1.iac.es/wiki/index.php?title=LST1Analysis_MVA&oldid=3262"