Difference between revisions of "LST1Analysis MVA2"
(→DL1 to DL2) |
(→SLURM COMMANDS) |
||
| (13 intermediate revisions by the same user not shown) | |||
| Line 7: | Line 7: | ||
School scripts/notebooks: '''/fefs/aswg/workspace/analysis-school-2024/cta-lstchain''' | School scripts/notebooks: '''/fefs/aswg/workspace/analysis-school-2024/cta-lstchain''' | ||
| + | |||
| + | <font style="color: red"> NOTE: the home directory has limited quota. Should use /fefs/aswg/workspace/monica.vazquez for the output instead!</font> | ||
=== Open remote connection === | === Open remote connection === | ||
| Line 108: | Line 110: | ||
<pre style="color: blue"> | <pre style="color: blue"> | ||
cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts | cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts | ||
| − | cp /fefs/aswg/workspace/analysis-school-2024/cta-lstchain/docs/examples/ | + | cp /fefs/aswg/workspace/analysis-school-2024/cta-lstchain/docs/examples/irf_dl3_tool_config.json . |
</pre> | </pre> | ||
The default json has energy dependent gammaness and theta containment cuts of 70% | The default json has energy dependent gammaness and theta containment cuts of 70% | ||
| Line 120: | Line 122: | ||
mkdir IRF | mkdir IRF | ||
DL2MC=/fefs/aswg/data/mc/DL2/AllSky/20230927_v0.10.4_crab_tuned/ | DL2MC=/fefs/aswg/data/mc/DL2/AllSky/20230927_v0.10.4_crab_tuned/ | ||
| − | ./launch_irf_creation.sh $DL2MC/TestingDataset/dec_2276 ./ | + | ./launch_irf_creation.sh $DL2MC/TestingDataset/dec_2276 ./irf_dl3_tool_config.json gh_eff_0.7_th_cont_0.7 |
</pre> | </pre> | ||
| Line 129: | Line 131: | ||
cp /fefs/aswg/workspace/analysis-school-2024/cta-lstchain/docs/examples/irf_dl3_tool_config.json . | cp /fefs/aswg/workspace/analysis-school-2024/cta-lstchain/docs/examples/irf_dl3_tool_config.json . | ||
IRFDIR=IRF/20230927_v0.10.4_crab_tuned/dec_2276/gh_eff_0.7_th_cont_0.7 | IRFDIR=IRF/20230927_v0.10.4_crab_tuned/dec_2276/gh_eff_0.7_th_cont_0.7 | ||
| − | ./launch_dl2_dl3.sh irf_dl3_tool_config.json ./DL2 $IRFDIR Crab\ Nebula | + | ./launch_dl2_dl3.sh ./irf_dl3_tool_config.json ./DL2 $IRFDIR Crab\ Nebula |
</pre> | </pre> | ||
(if source name has a blank space you have to escape it, using a backslash) | (if source name has a blank space you have to escape it, using a backslash) | ||
| Line 187: | Line 189: | ||
[[File:SED_Crab_20230117.png | 400px ]] [[File:LC_Crab_20230117.png | 400px ]] | [[File:SED_Crab_20230117.png | 400px ]] [[File:LC_Crab_20230117.png | 400px ]] | ||
| + | |||
| + | == DL3 cuts == | ||
| + | |||
| + | The '''fill_X_cut''' is used by the main pyirf function [https://pyirf.readthedocs.io/en/latest/api/pyirf.cuts.calculate_percentile_cut.html#pyirf.cuts.calculate_percentile_cut pyirf.cuts.calculate_percentile_cut] to give a "fill value" for bins, where the number of events is lower than min_event_p_en_bin (100 by default). Here X can be for gammaness, theta or alpha. | ||
| + | In principle, yes, one should change both max_X_cut and fill_X_cut appropriately. However, in lstchain, we have also added a "rolling" feature on the cut values for such bins, update_fill_cuts, which takes the values in the nearest filled bin | ||
| + | The default consideration of 100 events, was tested for the energy axes with 5 energy bins per decade. So far, it seems reasonable for any analyses | ||
| + | |||
| + | By number of events you mean MC events with which to compute the cut for the requested efficiency, right? | ||
| + | Yes, for each energy bin | ||
| + | Indeed for the bins with too short MC stats I think adopting the closest (in E) bin with good stats is reasonable | ||
| + | |||
| + | == SLURM COMMANDS == | ||
| + | |||
| + | <pre> | ||
| + | squeue -u monica.vazquez | ||
| + | squeue -t RUNNING | ||
| + | scontrol show jobid XXX | ||
| + | scancel -u monica.vazquez | ||
| + | </pre> | ||
Latest revision as of 11:04, 15 March 2024
LST1 Analysis after Software School 2024
Contents
MVA Analysis pages[edit]
Creating DL3 in the La Palma IT cluster[edit]
School scripts/notebooks: /fefs/aswg/workspace/analysis-school-2024/cta-lstchain
NOTE: the home directory has limited quota. Should use /fefs/aswg/workspace/monica.vazquez for the output instead!
Open remote connection[edit]
ssh cp03 source /fefs/aswg/software/conda/etc/profile.d/conda.sh conda activate lstchain-v0.10.7 jupyter notebook --no-browser Link for browser in printout: http://localhost:88XX/?token=YYYYYYYYYYYY In another terminal: ssh -NfL localhost:88XX:localhost:88XX monica.vazquez@cp03
Select good quality data[edit]
Run notebook: /home/monica.vazquez/LST1SCHOOL2024/notebooks/data_quality.ipynb
data_quality notebook settings to select crab data taken in the night 2023/01/17:
files = glob.glob("/fefs/aswg/data/real/OSA/DL1DataCheck_LongTerm/v0.10/2023????/DL1_datacheck_2023*.h5")
source_coordinates = SkyCoord.from_name("Crab Nebula")
max_zenith = 35 * u.deg
first_date = 20230117
last_date = 20230117
To check the most recent version of all the data use this location:
/fefs/aswg/data/real/OSA/DL1DataCheck_LongTerm/night_wise/all/
Copy the output good run list:
/fefs/aswg/data/real/DL1/20230117/v0.10/tailcut84/dl1_LST-1.Run11699.h5 /fefs/aswg/data/real/DL1/20230117/v0.10/tailcut84/dl1_LST-1.Run11700.h5 /fefs/aswg/data/real/DL1/20230117/v0.10/tailcut84/dl1_LST-1.Run11701.h5 /fefs/aswg/data/real/DL1/20230117/v0.10/tailcut84/dl1_LST-1.Run11702.h5 /fefs/aswg/data/real/DL1/20230117/v0.10/tailcut84/dl1_LST-1.Run11703.h5
to /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts/file_list.txt
DL1 to DL2[edit]
cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts ./launch_dl1_dl2.sh /fefs/aswg/data/models/AllSky/20230927_v0.10.4_crab_tuned/dec_2276 /fefs/aswg/data/models/AllSky/20230927_v0.10.4_crab_tuned/dec_2276/lstchain_config_2023-09-27.json # check batch job status squeue –u monica.vazquez # check they completed well sacct -u monica.vazquez
Processing v0.9 DL1 data[edit]
Process v0.9 DL1 files as if they were v0.10. Use lstchain v0.10 for the analysis, and v0.10 versions of the MC products (RF models and DL2).
Recommended MC trainings[edit]
Model files are located in: /fefs/aswg/data/models/AllSky/
| Recommended trainings | ||
|---|---|---|
|
Source |
Model |
json |
| Crab Nebula tuned | 20230927_v0.10.4_crab_tuned | lstchain_config_2023-09-27.json |
| Standard Dark complete production | 20230901_v0.10.4_allsky_base_prod | lstchain_config_2023-09-01.json |
| Standard Dark recent/incomplete! | 20240131_allsky_v0.10.5_all_dec_base | lstchain_config_2024-01-31.json |
Exploration of DL2 data[edit]
Run notebook: /home/monica.vazquez/LST1SCHOOL2024/notebooks/explore_DL2.ipynb
IRF creation[edit]
Use the test MC of the same declination line used in the DL2 step
cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts cp /fefs/aswg/workspace/analysis-school-2024/cta-lstchain/docs/examples/irf_dl3_tool_config.json .
The default json has energy dependent gammaness and theta containment cuts of 70%
"gh_efficiency": 0.7 "theta_containment": 0.7
cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts mkdir IRF DL2MC=/fefs/aswg/data/mc/DL2/AllSky/20230927_v0.10.4_crab_tuned/ ./launch_irf_creation.sh $DL2MC/TestingDataset/dec_2276 ./irf_dl3_tool_config.json gh_eff_0.7_th_cont_0.7
DL2 to DL3[edit]
cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts cp /fefs/aswg/workspace/analysis-school-2024/cta-lstchain/docs/examples/irf_dl3_tool_config.json . IRFDIR=IRF/20230927_v0.10.4_crab_tuned/dec_2276/gh_eff_0.7_th_cont_0.7 ./launch_dl2_dl3.sh ./irf_dl3_tool_config.json ./DL2 $IRFDIR Crab\ Nebula
(if source name has a blank space you have to escape it, using a backslash)
Create DL3 index files[edit]
cd /home/monica.vazquez/LST1SCHOOL2024/helpful_scripts/DL3 lstchain_create_dl3_index_files -d ./DL3 -p dl3\*.fits.gz
In case the DL3 are unzipped, redo index file creation command:
gzip -d ./DL3/dl3\*.fits.gz lstchain_create_dl3_index_files --overwrite -d ./DL3 -p dl3\*.fits
Output files to be used together with DL3 in gammapy:
hdu-index.fits.gz and obs-index.fits.gz
Running DL3 gammapy analysis locally[edit]
#mamba create -c conda-forge -n lstchain-v0.10.7 python=3.11 lstchain=0.10.7 conda activate lstchain-v0.10.7 #transfer the DL3 files to laptop scp cp03:/home/monica.vazquez/LST1SCHOOL2024/helpful_scripts/DL3/*fits* /Users/monicava/Documents/CODE/LST1ANALYSIS/DL3/Crab_20230117
Run locally the notebooks:
- Theta2: plot_theta2_from_dl3.ipynb
Change/check:
base_dir = Path("/Users/monicava/Documents/CODE/LST1ANALYSIS/DL3")
label="Crab_20230117"
dl3_path = base_dir / label
target_position = SkyCoord.from_name("Crab Nebula")
- SED & LC: post_DL3_analysis.ipynb
Change/check:
base_dir = Path("/Users/monicava/Documents/CODE/LST1ANALYSIS/DL3")
label="Crab_20230117"
dl3_path = base_dir / label
filename = dl3_path / "dl3_LST-1.Run11699.fits"
target_position = SkyCoord.from_name("Crab", frame='icrs')
DL3 cuts[edit]
The fill_X_cut is used by the main pyirf function pyirf.cuts.calculate_percentile_cut to give a "fill value" for bins, where the number of events is lower than min_event_p_en_bin (100 by default). Here X can be for gammaness, theta or alpha. In principle, yes, one should change both max_X_cut and fill_X_cut appropriately. However, in lstchain, we have also added a "rolling" feature on the cut values for such bins, update_fill_cuts, which takes the values in the nearest filled bin The default consideration of 100 events, was tested for the energy axes with 5 energy bins per decade. So far, it seems reasonable for any analyses
By number of events you mean MC events with which to compute the cut for the requested efficiency, right? Yes, for each energy bin Indeed for the bins with too short MC stats I think adopting the closest (in E) bin with good stats is reasonable
SLURM COMMANDS[edit]
squeue -u monica.vazquez squeue -t RUNNING scontrol show jobid XXX scancel -u monica.vazquez
