Difference between revisions of "LST1Analysis MVA2"
(→DL3 cuts) |
(→DL3 cuts) |
||
Line 192: | Line 192: | ||
== DL3 cuts == | == DL3 cuts == | ||
− | [https://pyirf.readthedocs.io/en/latest/api/pyirf.cuts.calculate_percentile_cut.html#pyirf.cuts.calculate_percentile_cut pyirf.cuts.calculate_percentile_cut] | + | The '''fill_X_cut''' is used by the main pyirf function [https://pyirf.readthedocs.io/en/latest/api/pyirf.cuts.calculate_percentile_cut.html#pyirf.cuts.calculate_percentile_cut pyirf.cuts.calculate_percentile_cut] to give a "fill value" for bins, where the number of events is lower than min_event_p_en_bin (100 by default). Here X can be for gammaness, theta or alpha. |
+ | In principle, yes, one should change both max_X_cut and fill_X_cut appropriately. However, in lstchain, we have also added a "rolling" feature on the cut values for such bins, update_fill_cuts, which takes the values in the nearest filled bin |
Revision as of 10:58, 14 March 2024
LST1 Analysis after Software School 2024
Contents
MVA Analysis pages
Creating DL3 in the La Palma IT cluster
School scripts/notebooks: /fefs/aswg/workspace/analysis-school-2024/cta-lstchain
NOTE: the home directory has limited quota. Should use /fefs/aswg/workspace/monica.vazquez for the output instead!
Open remote connection
ssh cp03 source /fefs/aswg/software/conda/etc/profile.d/conda.sh conda activate lstchain-v0.10.7 jupyter notebook --no-browser Link for browser in printout: http://localhost:88XX/?token=YYYYYYYYYYYY In another terminal: ssh -NfL localhost:88XX:localhost:88XX monica.vazquez@cp03
Select good quality data
Run notebook: /home/monica.vazquez/LST1SCHOOL2024/notebooks/data_quality.ipynb
data_quality notebook settings to select crab data taken in the night 2023/01/17:
files = glob.glob("/fefs/aswg/data/real/OSA/DL1DataCheck_LongTerm/v0.10/2023????/DL1_datacheck_2023*.h5") source_coordinates = SkyCoord.from_name("Crab Nebula") max_zenith = 35 * u.deg first_date = 20230117 last_date = 20230117
To check the most recent version of all the data use this location:
/fefs/aswg/data/real/OSA/DL1DataCheck_LongTerm/night_wise/all/
Copy the output good run list:
/fefs/aswg/data/real/DL1/20230117/v0.10/tailcut84/dl1_LST-1.Run11699.h5 /fefs/aswg/data/real/DL1/20230117/v0.10/tailcut84/dl1_LST-1.Run11700.h5 /fefs/aswg/data/real/DL1/20230117/v0.10/tailcut84/dl1_LST-1.Run11701.h5 /fefs/aswg/data/real/DL1/20230117/v0.10/tailcut84/dl1_LST-1.Run11702.h5 /fefs/aswg/data/real/DL1/20230117/v0.10/tailcut84/dl1_LST-1.Run11703.h5
to /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts/file_list.txt
DL1 to DL2
cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts ./launch_dl1_dl2.sh /fefs/aswg/data/models/AllSky/20230927_v0.10.4_crab_tuned/dec_2276 /fefs/aswg/data/models/AllSky/20230927_v0.10.4_crab_tuned/dec_2276/lstchain_config_2023-09-27.json # check batch job status squeue –u monica.vazquez # check they completed well sacct -u monica.vazquez
Processing v0.9 DL1 data
Process v0.9 DL1 files as if they were v0.10. Use lstchain v0.10 for the analysis, and v0.10 versions of the MC products (RF models and DL2).
Recommended MC trainings
Model files are located in: /fefs/aswg/data/models/AllSky/
Recommended trainings | ||
---|---|---|
Source |
Model |
json |
Crab Nebula tuned | 20230927_v0.10.4_crab_tuned | lstchain_config_2023-09-27.json |
Standard Dark complete production | 20230901_v0.10.4_allsky_base_prod | lstchain_config_2023-09-01.json |
Standard Dark recent/incomplete! | 20240131_allsky_v0.10.5_all_dec_base | lstchain_config_2024-01-31.json |
Exploration of DL2 data
Run notebook: /home/monica.vazquez/LST1SCHOOL2024/notebooks/explore_DL2.ipynb
IRF creation
Use the test MC of the same declination line used in the DL2 step
cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts cp /fefs/aswg/workspace/analysis-school-2024/cta-lstchain/docs/examples/irf_dl3_tool_config.json .
The default json has energy dependent gammaness and theta containment cuts of 70%
"gh_efficiency": 0.7 "theta_containment": 0.7
cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts mkdir IRF DL2MC=/fefs/aswg/data/mc/DL2/AllSky/20230927_v0.10.4_crab_tuned/ ./launch_irf_creation.sh $DL2MC/TestingDataset/dec_2276 ./irf_dl3_tool_config.json gh_eff_0.7_th_cont_0.7
DL2 to DL3
cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts cp /fefs/aswg/workspace/analysis-school-2024/cta-lstchain/docs/examples/irf_dl3_tool_config.json . IRFDIR=IRF/20230927_v0.10.4_crab_tuned/dec_2276/gh_eff_0.7_th_cont_0.7 ./launch_dl2_dl3.sh ./irf_dl3_tool_config.json ./DL2 $IRFDIR Crab\ Nebula
(if source name has a blank space you have to escape it, using a backslash)
Create DL3 index files
cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts/DL3 lstchain_create_dl3_index_files -d ./DL3 -p dl3\*.fits.gz
In case the DL3 are unzipped, redo index file creation command:
gzip -d ./DL3/dl3\*.fits.gz lstchain_create_dl3_index_files --overwrite -d ./DL3 -p dl3\*.fits
Output files to be used together with DL3 in gammapy:
hdu-index.fits.gz and obs-index.fits.gz
Running DL3 gammapy analysis locally
#mamba create -c conda-forge -n lstchain-v0.10.7 python=3.11 lstchain=0.10.7 conda activate lstchain-v0.10.7 #transfer the DL3 files to laptop scp cp03:/home/monica.vazquez/LST1SCHOOL2024/helpful_scripts/DL3/*fits* /Users/monicava/Documents/CODE/LST1ANALYSIS/DL3/Crab_20230117
Run locally the notebooks:
- Theta2: plot_theta2_from_dl3.ipynb
Change/check:
base_dir = Path("/Users/monicava/Documents/CODE/LST1ANALYSIS/DL3") label="Crab_20230117" dl3_path = base_dir / label target_position = SkyCoord.from_name("Crab Nebula")
- SED & LC: post_DL3_analysis.ipynb
Change/check:
base_dir = Path("/Users/monicava/Documents/CODE/LST1ANALYSIS/DL3") label="Crab_20230117" dl3_path = base_dir / label filename = dl3_path / "dl3_LST-1.Run11699.fits" target_position = SkyCoord.from_name("Crab", frame='icrs')
DL3 cuts
The fill_X_cut is used by the main pyirf function pyirf.cuts.calculate_percentile_cut to give a "fill value" for bins, where the number of events is lower than min_event_p_en_bin (100 by default). Here X can be for gammaness, theta or alpha. In principle, yes, one should change both max_X_cut and fill_X_cut appropriately. However, in lstchain, we have also added a "rolling" feature on the cut values for such bins, update_fill_cuts, which takes the values in the nearest filled bin