Difference between revisions of "Onsite analysis (LSTOSA)"

From my_wiki
Jump to: navigation, search
(Created page with "To be completed")
 
Line 1: Line 1:
−
To be completed
+
 
 +
Code: [https://github.com/gae-ucm/lstosa https://github.com/gae-ucm/lstosa]
 +
 
 +
Docs: [https://lstosa.readthedocs.io https://lstosa.readthedocs.io]
 +
 
 +
== How to use lstosa (onsite analysis pipeline for the LST-1) ==
 +
=== Access to the LST IT container ===
 +
* ''' With your personal account (meant for developing and testing)'''
 +
If you have an LDAP account (firstname.lastname@cta-consortium.org), follow these steps:
 +
:- Log in to an allowed machine. Only a few machines (IPs) are allowed to log in to the LST IT center, so it is necessary to jump through them to the IT center.
 +
:- Log in with the general user CtAlAPaLmA to a general login server:
 +
$ ssh -l CtAlAPaLmA 161.72.87.1
 +
:- From this login server, login with your LDAP account either to cp01 or cp02:
 +
$ ssh -l firstname.lastname cp01
 +
* '''Alternatively, with the official lstanalyzer account (meant for data processing)'''
 +
Login to tcs06 machine as lstanalyzer:
 +
$ ssh lstanalyzer@tcs06
 +
 
 +
=== Install miniconda (python package manager) ===
 +
Follow the installation instructions
 +
[https://conda.io/projects/conda/en/latest/user-guide/install/linux.html here]
 +
or alternatively, follow the instructions [[Working at the LST-IT center|here]] to work at the LST-IT container.
 +
 
 +
=== Install lstosa in your directory ===
 +
For the installation of lstosa follow the instructions in:
 +
https://github.com/gae-ucm/lstosa
 +
 
 +
You will end up with a new conda environment with all the python packages needed to run lstosa and lstchain, plus some other for developing, building the documentation, and testing the code. By default the environment is named 'osa-env' if no other name is specified with the -n flag while creating the environment.
 +
 
 +
Make sure that you do the last part after activating the environment: 'pip install -e .' in the lstosa base directory.
 +
 
 +
=== Activate osa environment ===
 +
$ conda activate <name of the environment>
 +
'''Tip''': you can list the environments in your conda installation with 'conda env list'.
 +
 
 +
=== Configuration file for running lstosa ===
 +
Set the proper paths in the configuration file you are using to run osa scripts: sequencer, closer, etc. See default file in cfg/sequencer.cfg
 +
*[ENV]:
 +
:- USER: firstname.lastname (set the name of the user for the usage of SLURM. If you have entered with your personal LDAP account)
 +
*[LSTOSA]:
 +
:- HOMEDIR: home directory (e.g. /fefs/aswg)
 +
:- PYTHONDIR: e.g. /home/firstname.lastname/lstosa
 +
*[LST1] (paths to different data level files):
 +
:- DIR set your work base directory where files are going to be produced (e.g. /fefs/aswg/workspace/firstname.lastname/data/real)
 +
:- PROD-ID = v0.7.5 (main production id for the running_analysis directory)                                                                                   
 +
:- CALIB-PROD-ID = v0.7.5                                                                                         
 +
:- DL1-PROD-ID = tailcut84_dynamic_cleaning (sub production id for the DL1AB subdirectory)                                                                                                               
 +
:- DL2-PROD-ID = tailcut84_dynamic_cleaning (sub production id for the DL2 subdirectory)
 +
The DL1 and DL2 can be different or leave them empty so automatically just lstchain version will be used.
 +
 
 +
With these parameters the running_analysis directory where all analysis products are first produced is:
 +
/fefs/aswg/workspace/firstname.lastname/data/real/running_analysis/YYYYMMDD/v0.7.5
 +
 
 +
DL1/DL2 final directories (after running closer script) will be:
 +
/fefs/aswg/workspace/firstname.lastname/data/real/{DL1,DL2}/YYYYMMDD/v0.7.5/tailcut84_dynamic_cleaning
 +
 
 +
=== Check ELOG ===
 +
Check the ELOG (https://www.lst1.iac.es/elog/LST+commissioning/) sent the morning after data taking finishes. Look for sky-data runs that are going to be processed by lstosa. In addition, look for information about the problems during data taking and weather conditions. For bookkeeping, annotate the runs to be processed by lstosa in the spreadsheet [https://docs.google.com/spreadsheets/d/1ECZldivXPP0RFbVZWrvFsJmBAOkj8dW0dpD1J1kKlW8/edit#gid=176249465 LST-analysis-logbook].
 +
 
 +
=== RunSummary files ===
 +
The official RunSummary files are produced every day by a cron job in the lstanalyzer account. The files of each night can be found in the following path: “/fefs/aswg/data/real/monitoring/RunSummary”.
 +
 
 +
The cron job is:
 +
<nowiki>#</nowiki> [RUN SUMMARY] Produce run summary file the morning after data taking at 6:45 UTC
 +
45 06 * * * obsdate=`date +\%Y\%m\%d -d yesterday`; export obsdate; /fefs/aswg/lstosa/run_summary.sh $obsdate >/dev/null 2>&1
 +
 
 +
'''Note''': one can always produce its own run summary files with [https://github.com/cta-observatory/cta-lstchain/blob/master/lstchain/scripts/lstchain_create_run_summary.py this lstchain script].
 +
 
 +
=== Launch sequencer ===
 +
This is indeed when OSA processing starts.
 +
 
 +
First simulate the sequencer by using the -s option:
 +
$ sequencer -v -s -c cfg/sequencer.cfg  -d 2020_12_07 LST1
 +
 
 +
Then, after checking that everything is in the sequencer table, launch the sequencer script script without the -s option:
 +
$ sequencer -v -c cfg/sequencer.cfg  -d 2020_12_07 LST1
 +
 
 +
The “sequencer” script creates and executes the calibration sequence and prepares a SLURM job array which launches the data sequences for every subrun. The sequence scripts will be launched using sbatch. 
 +
 
 +
A first calibration sequence creates the DRS4 pedestal, charge and time calibration files. Once the calibration sequence has finished, the rest of the sequences, corresponding to sky-data runs that make use of the previously produced calibration files, start executing. DL1 and DL2 files are generated. It takes around 1 hour to execute the calibration sequence and also around 1 hour in total to execute the data sequences (go from R0 to DL2).
 +
 
 +
What does sequencer do?
 +
:- Set up the ‘running_analysis’ directory according to the path and production ID set up in the cfg file.
 +
:- Extract run information from RunSummary.
 +
:- Build the sequences 
(each sequence corresponds to a run).
 +
:- Create jobs for each sequence.
 +
:- Submit jobs to queue.
 +
 
 +
=== Monitor the processing status ===
 +
At any moment you can launch again the same sequencer command used to trigger the processing but this time with the -s option for simulating all the steps. It will display the sequencer table with the processing status for each run.
 +
This is done automatically by sequencer_webmaker script that produces the table display in https://www.lst1.iac.es/datacheck/lstosa/sequencer.xhtml. This page is updated every 5 minutes. So there is no need to login into the LST-IT to check the processing status.
 +
 
 +
'''Note''': this will be done automatically by the autocloser script.
 +
 
 +
=== Launch closer/autocloser ===
 +
'''WARNING''': closer will be called by autocloser in the next future.
 +
Launch the “closer” script:
 +
$ closer -c cfg/sequencer.cfg -d 2020_12_07 LST1
 +
 
 +
The -s option can be used first to simulate the closer launching:
 +
$ closer -c cfg/sequencer.cfg -s -d 2020_12_07 LST1
 +
 
 +
In case of finding errors or non-completed sequences, launching the closer script will not close the day. In that case, the closer script can be launched using the -y option which forces it to close:
 +
$ closer -c cfg/sequencer.cfg -y -s -d 2020_12_07 LST1
 +
 
 +
What does closer do?
 +
:- Check that all jobs have finished properly by checking at the history files
 +
:- Check that all DL1, DL2, DL1 datacheck, Muons files (subrun wise) were produced by looking at the sequencer table.
 +
:- If the above condition is fulfilled, merge DL1 datacheck, DL2 subrun wise files on a run basis.
 +
:- Move the files to their final directory.
 +
:- Produce provenance.
 +
 
 +
'''Note''': the provenance information, merging of different analysis products (DL2, DL1datacheck) are handled by closer script. In previous versions of OSA this was done separately by hand.
 +
 
 +
Provenance files run-wise are produced in these directories:
 +
/fefs/aswg/workspace/firstname.lastname/data/real/{DL1,DL2}/20201207/v0.7.5/tailcut84_dynamic_cleaning/log
 +
 
 +
DL1 files are in
 +
/fefs/aswg/workspace/firstname.lastname/data/real/DL1/20201207/v0.7.5/tailcut84_dynamic_cleaning
 +
 
 +
DL2 files are in
 +
/fefs/aswg/workspace/firstname.lastname/data/real/DL2/20201207/v0.7.5/tailcut84_dynamic_cleaning
 +
 
 +
Muons file are in
 +
/fefs/aswg/workspace/firstname.lastname/data/real/DL1/20201207/v0.7.5
 +
 
 +
DL1 datacheck files are in
 +
/fefs/aswg/workspace/firstname.lastname/data/real/DL1/20201207/v0.7.5/tailcut84_dynamic_cleaning
 +
 
 +
Merged DL1 datacheck h5 and PDF files are produced on a run basis.
 +
 
 +
=== Long-term DL1 monitoring script ===
 +
These are not python scripts but bash scripts in lstosa/utils/
 +
:- Daily (long-term script restricted to one day to facilitate the daily datacheck)
 +
:- All-time monitoring (run over all the runs taken so far after July 2020)
 +
 
 +
=== Copy datacheck files to web (copy_datacheck script) ===
 +
Copy pdf run-wise files to [https://lst1.iac.es/datacheck/dl1/ lst1.iac.es/datacheck/dl1/] and copy the DRS4 and calibration datacheck pdf files to [https://www.lst1.iac.es/datacheck/drs4/ lst1.iac.es/datacheck/drs4/] and [https://www.lst1.iac.es/datacheck/enf_calibration/ lst1.iac.es/datacheck/enf_calibration/], respectively, by launching the copy_datacheck.py script.
 +
 
 +
 
 +
== Backup ==
 +
=== Cron jobs ===
 +
=== Git, python, documentation tips ===
 +
=== [Developers] How to use git/GitHub ===
 +
=== Update lstosa in the IT container ===
 +
=== Look for problematic files during the processing ===

Revision as of 19:36, 3 November 2021

Code: https://github.com/gae-ucm/lstosa

Docs: https://lstosa.readthedocs.io

How to use lstosa (onsite analysis pipeline for the LST-1)

Access to the LST IT container

  • With your personal account (meant for developing and testing)

If you have an LDAP account (firstname.lastname@cta-consortium.org), follow these steps:

- Log in to an allowed machine. Only a few machines (IPs) are allowed to log in to the LST IT center, so it is necessary to jump through them to the IT center.
- Log in with the general user CtAlAPaLmA to a general login server:
$ ssh -l CtAlAPaLmA 161.72.87.1
- From this login server, login with your LDAP account either to cp01 or cp02:
$ ssh -l firstname.lastname cp01
  • Alternatively, with the official lstanalyzer account (meant for data processing)

Login to tcs06 machine as lstanalyzer: 

$ ssh lstanalyzer@tcs06

Install miniconda (python package manager)

Follow the installation instructions here or alternatively, follow the instructions here to work at the LST-IT container.

Install lstosa in your directory

For the installation of lstosa follow the instructions in: https://github.com/gae-ucm/lstosa

You will end up with a new conda environment with all the python packages needed to run lstosa and lstchain, plus some other for developing, building the documentation, and testing the code. By default the environment is named 'osa-env' if no other name is specified with the -n flag while creating the environment.

Make sure that you do the last part after activating the environment: 'pip install -e .' in the lstosa base directory.

Activate osa environment

$ conda activate <name of the environment>

Tip: you can list the environments in your conda installation with 'conda env list'.

Configuration file for running lstosa

Set the proper paths in the configuration file you are using to run osa scripts: sequencer, closer, etc. See default file in cfg/sequencer.cfg

  • [ENV]:
- USER: firstname.lastname (set the name of the user for the usage of SLURM. If you have entered with your personal LDAP account)
  • [LSTOSA]:
- HOMEDIR: home directory (e.g. /fefs/aswg)
- PYTHONDIR: e.g. /home/firstname.lastname/lstosa
  • [LST1] (paths to different data level files):
- DIR set your work base directory where files are going to be produced (e.g. /fefs/aswg/workspace/firstname.lastname/data/real)
- PROD-ID = v0.7.5 (main production id for the running_analysis directory)
- CALIB-PROD-ID = v0.7.5
- DL1-PROD-ID = tailcut84_dynamic_cleaning (sub production id for the DL1AB subdirectory)
- DL2-PROD-ID = tailcut84_dynamic_cleaning (sub production id for the DL2 subdirectory)

The DL1 and DL2 can be different or leave them empty so automatically just lstchain version will be used.

With these parameters the running_analysis directory where all analysis products are first produced is: /fefs/aswg/workspace/firstname.lastname/data/real/running_analysis/YYYYMMDD/v0.7.5

DL1/DL2 final directories (after running closer script) will be: /fefs/aswg/workspace/firstname.lastname/data/real/{DL1,DL2}/YYYYMMDD/v0.7.5/tailcut84_dynamic_cleaning

Check ELOG

Check the ELOG (https://www.lst1.iac.es/elog/LST+commissioning/) sent the morning after data taking finishes. Look for sky-data runs that are going to be processed by lstosa. In addition, look for information about the problems during data taking and weather conditions. For bookkeeping, annotate the runs to be processed by lstosa in the spreadsheet LST-analysis-logbook.

RunSummary files

The official RunSummary files are produced every day by a cron job in the lstanalyzer account. The files of each night can be found in the following path: “/fefs/aswg/data/real/monitoring/RunSummary”.

The cron job is: # [RUN SUMMARY] Produce run summary file the morning after data taking at 6:45 UTC 45 06 * * * obsdate=`date +\%Y\%m\%d -d yesterday`; export obsdate; /fefs/aswg/lstosa/run_summary.sh $obsdate >/dev/null 2>&1

Note: one can always produce its own run summary files with this lstchain script.

Launch sequencer

This is indeed when OSA processing starts.

First simulate the sequencer by using the -s option: 

$ sequencer -v -s -c cfg/sequencer.cfg  -d 2020_12_07 LST1

Then, after checking that everything is in the sequencer table, launch the sequencer script script without the -s option: 

$ sequencer -v -c cfg/sequencer.cfg  -d 2020_12_07 LST1

The “sequencer” script creates and executes the calibration sequence and prepares a SLURM job array which launches the data sequences for every subrun. The sequence scripts will be launched using sbatch.

A first calibration sequence creates the DRS4 pedestal, charge and time calibration files. Once the calibration sequence has finished, the rest of the sequences, corresponding to sky-data runs that make use of the previously produced calibration files, start executing. DL1 and DL2 files are generated. It takes around 1 hour to execute the calibration sequence and also around 1 hour in total to execute the data sequences (go from R0 to DL2).

What does sequencer do?

- Set up the ‘running_analysis’ directory according to the path and production ID set up in the cfg file.
- Extract run information from RunSummary.
- Build the sequences 
(each sequence corresponds to a run).
- Create jobs for each sequence.
- Submit jobs to queue.

Monitor the processing status

At any moment you can launch again the same sequencer command used to trigger the processing but this time with the -s option for simulating all the steps. It will display the sequencer table with the processing status for each run. This is done automatically by sequencer_webmaker script that produces the table display in https://www.lst1.iac.es/datacheck/lstosa/sequencer.xhtml. This page is updated every 5 minutes. So there is no need to login into the LST-IT to check the processing status.

Note: this will be done automatically by the autocloser script.

Launch closer/autocloser

WARNING: closer will be called by autocloser in the next future. Launch the “closer” script: 

$ closer -c cfg/sequencer.cfg -d 2020_12_07 LST1

The -s option can be used first to simulate the closer launching: 

$ closer -c cfg/sequencer.cfg -s -d 2020_12_07 LST1

In case of finding errors or non-completed sequences, launching the closer script will not close the day. In that case, the closer script can be launched using the -y option which forces it to close: 

$ closer -c cfg/sequencer.cfg -y -s -d 2020_12_07 LST1

What does closer do?

- Check that all jobs have finished properly by checking at the history files
- Check that all DL1, DL2, DL1 datacheck, Muons files (subrun wise) were produced by looking at the sequencer table.
- If the above condition is fulfilled, merge DL1 datacheck, DL2 subrun wise files on a run basis.
- Move the files to their final directory.
- Produce provenance.

Note: the provenance information, merging of different analysis products (DL2, DL1datacheck) are handled by closer script. In previous versions of OSA this was done separately by hand.

Provenance files run-wise are produced in these directories: 

/fefs/aswg/workspace/firstname.lastname/data/real/{DL1,DL2}/20201207/v0.7.5/tailcut84_dynamic_cleaning/log

DL1 files are in 

/fefs/aswg/workspace/firstname.lastname/data/real/DL1/20201207/v0.7.5/tailcut84_dynamic_cleaning

DL2 files are in 

/fefs/aswg/workspace/firstname.lastname/data/real/DL2/20201207/v0.7.5/tailcut84_dynamic_cleaning

Muons file are in 

/fefs/aswg/workspace/firstname.lastname/data/real/DL1/20201207/v0.7.5

DL1 datacheck files are in 

/fefs/aswg/workspace/firstname.lastname/data/real/DL1/20201207/v0.7.5/tailcut84_dynamic_cleaning

Merged DL1 datacheck h5 and PDF files are produced on a run basis.

Long-term DL1 monitoring script

These are not python scripts but bash scripts in lstosa/utils/

- Daily (long-term script restricted to one day to facilitate the daily datacheck)
- All-time monitoring (run over all the runs taken so far after July 2020)

Copy datacheck files to web (copy_datacheck script)

Copy pdf run-wise files to lst1.iac.es/datacheck/dl1/ and copy the DRS4 and calibration datacheck pdf files to lst1.iac.es/datacheck/drs4/ and lst1.iac.es/datacheck/enf_calibration/, respectively, by launching the copy_datacheck.py script.


Backup

Cron jobs

Git, python, documentation tips

[Developers] How to use git/GitHub

Update lstosa in the IT container

Look for problematic files during the processing

Retrieved from "https://www.lst1.iac.es/wiki/index.php?title=Onsite_analysis_(LSTOSA)&oldid=3415"