LST1Analysis MVA2

LST1 Analysis after Software School 2024

MVA Analysis pages

• OP313_MVA

Creating DL3 in the La Palma IT cluster

School scripts/notebooks: /fefs/aswg/workspace/analysis-school-2024/cta-lstchain

NOTE: the home directory has limited quota. Should use /fefs/aswg/workspace/monica.vazquez for the output instead!

Open remote connection

ssh cp03

source /fefs/aswg/software/conda/etc/profile.d/conda.sh
conda activate lstchain-v0.10.7
jupyter notebook --no-browser

Link for browser in printout:
http://localhost:88XX/?token=YYYYYYYYYYYY

In another terminal:
ssh -NfL localhost:88XX:localhost:88XX monica.vazquez@cp03

Select good quality data

Run notebook:
/home/monica.vazquez/LST1SCHOOL2024/notebooks/data_quality.ipynb

data_quality notebook settings to select crab data taken in the night 2023/01/17:

files = glob.glob("/fefs/aswg/data/real/OSA/DL1DataCheck_LongTerm/v0.10/2023????/DL1_datacheck_2023*.h5")
source_coordinates = SkyCoord.from_name("Crab Nebula")
max_zenith = 35 * u.deg
first_date = 20230117
last_date  = 20230117

To check the most recent version of all the data use this location:

/fefs/aswg/data/real/OSA/DL1DataCheck_LongTerm/night_wise/all/

Copy the output good run list:

/fefs/aswg/data/real/DL1/20230117/v0.10/tailcut84/dl1_LST-1.Run11699.h5
/fefs/aswg/data/real/DL1/20230117/v0.10/tailcut84/dl1_LST-1.Run11700.h5
/fefs/aswg/data/real/DL1/20230117/v0.10/tailcut84/dl1_LST-1.Run11701.h5
/fefs/aswg/data/real/DL1/20230117/v0.10/tailcut84/dl1_LST-1.Run11702.h5
/fefs/aswg/data/real/DL1/20230117/v0.10/tailcut84/dl1_LST-1.Run11703.h5

to /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts/file_list.txt

DL1 to DL2

cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts
./launch_dl1_dl2.sh /fefs/aswg/data/models/AllSky/20230927_v0.10.4_crab_tuned/dec_2276 /fefs/aswg/data/models/AllSky/20230927_v0.10.4_crab_tuned/dec_2276/lstchain_config_2023-09-27.json
# check batch job status
squeue –u monica.vazquez
# check they completed well
sacct -u monica.vazquez

Processing v0.9 DL1 data

Process v0.9 DL1 files as if they were v0.10. Use lstchain v0.10 for the analysis, and v0.10 versions of the MC products (RF models and DL2).

Recommended MC trainings

Model files are located in: /fefs/aswg/data/models/AllSky/

Recommended trainings
Source	Model	json
Crab Nebula tuned	20230927_v0.10.4_crab_tuned	lstchain_config_2023-09-27.json
Standard Dark complete production	20230901_v0.10.4_allsky_base_prod	lstchain_config_2023-09-01.json
Standard Dark recent/incomplete!	20240131_allsky_v0.10.5_all_dec_base	lstchain_config_2024-01-31.json

Exploration of DL2 data

Run notebook:
/home/monica.vazquez/LST1SCHOOL2024/notebooks/explore_DL2.ipynb

IRF creation

Use the test MC of the same declination line used in the DL2 step

cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts
cp /fefs/aswg/workspace/analysis-school-2024/cta-lstchain/docs/examples/irf_dl3_tool_config.json .

The default json has energy dependent gammaness and theta containment cuts of 70%

"gh_efficiency": 0.7
"theta_containment": 0.7

cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts
mkdir IRF
DL2MC=/fefs/aswg/data/mc/DL2/AllSky/20230927_v0.10.4_crab_tuned/
./launch_irf_creation.sh $DL2MC/TestingDataset/dec_2276 ./irf_dl3_tool_config.json gh_eff_0.7_th_cont_0.7

DL2 to DL3

cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts
cp /fefs/aswg/workspace/analysis-school-2024/cta-lstchain/docs/examples/irf_dl3_tool_config.json .
IRFDIR=IRF/20230927_v0.10.4_crab_tuned/dec_2276/gh_eff_0.7_th_cont_0.7
./launch_dl2_dl3.sh ./irf_dl3_tool_config.json ./DL2 $IRFDIR Crab\ Nebula

(if source name has a blank space you have to escape it, using a backslash)

Create DL3 index files

cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts/DL3
lstchain_create_dl3_index_files -d ./DL3 -p dl3\*.fits.gz

In case the DL3 are unzipped, redo index file creation command:

gzip -d ./DL3/dl3\*.fits.gz
lstchain_create_dl3_index_files --overwrite -d ./DL3 -p dl3\*.fits

Output files to be used together with DL3 in gammapy:

hdu-index.fits.gz and obs-index.fits.gz

Running DL3 gammapy analysis locally

#mamba create -c conda-forge -n lstchain-v0.10.7 python=3.11 lstchain=0.10.7 

conda activate lstchain-v0.10.7
#transfer the DL3 files to laptop
scp cp03:/home/monica.vazquez/LST1SCHOOL2024/helpful_scripts/DL3/*fits* /Users/monicava/Documents/CODE/LST1ANALYSIS/DL3/Crab_20230117

Run locally the notebooks:

• Theta2: plot_theta2_from_dl3.ipynb

Change/check:

base_dir = Path("/Users/monicava/Documents/CODE/LST1ANALYSIS/DL3")
label="Crab_20230117" 
dl3_path = base_dir / label

target_position = SkyCoord.from_name("Crab Nebula")

• SED & LC: post_DL3_analysis.ipynb

Change/check:

base_dir = Path("/Users/monicava/Documents/CODE/LST1ANALYSIS/DL3")
label="Crab_20230117" 
dl3_path = base_dir / label

filename = dl3_path / "dl3_LST-1.Run11699.fits"

target_position = SkyCoord.from_name("Crab", frame='icrs')

DL3 cuts

The fill_X_cut is used by the main pyirf function pyirf.cuts.calculate_percentile_cut to give a "fill value" for bins, where the number of events is lower than min_event_p_en_bin (100 by default). Here X can be for gammaness, theta or alpha. In principle, yes, one should change both max_X_cut and fill_X_cut appropriately. However, in lstchain, we have also added a "rolling" feature on the cut values for such bins, update_fill_cuts, which takes the values in the nearest filled bin The default consideration of 100 events, was tested for the energy axes with 5 energy bins per decade. So far, it seems reasonable for any analyses

By number of events you mean MC events with which to compute the cut for the requested efficiency, right? Yes, for each energy bin Indeed for the bins with too short MC stats I think adopting the closest (in E) bin with good stats is reasonable

SLURM COMMANDS

squeue -t RUNNING 
scontrol show jobid XXX