Image cleaning and parametrization
Contents
- 1 Real Data DL1 production
- 1.1 General comments
- 1.2 2020 DL1 Analysis logbook
- 1.3 2019 DL1 Analysis logbook
- 1.4 Standard DL1 v0.5.1 production of 2019 and 2020 data (edited: June 2020)
- 1.5 Standard DL1 v0.4.4 production for 2019 and 2020 data (edited: March 6, 2020)
- 1.6 Standard DL1 v0.4.3 production for 2019 and 2020 data (edited: January 2020)
- 2 MC production
Real Data DL1 production
Real data will be stored in the IT Container following the structure discussed in the Redmine issue https://forge.in2p3.fr/issues/37938, which includes directories whose name depends on the date. The convention UTC+12, by which days change at noon of the next day, is used at DAQ. For instance, for the data taken in the night 2019_11_23 to 2019_11_24 the selected date would be 2019_11_23 and main directories:
/fefs/aswg/data/real/DL1/20191123/v00 for the default analysis
v00 can take the form of XXX_NN where XXX is the initials of a name and NN a version number (e,g, JLC_01 for the first version of Jose Luis Contreras) when it is an alternative version run by an analysis shifter)
--Jose-Luis (talk) 18:09, 22 November 2019 (UTC)
General comments
- Unless otherwise specified, the standard configuration file is used to produce DL1 and DL2 analysis products. You can find it here:
/fefs/aswg/software/virtual_env/ctasoft/cta-lstchain/lstchain/data/lstchain_standard_config.json
In particular, the tailcut cleaning levels are the followings:
"tailcut": { "picture_thresh":6, "boundary_thresh":3, "keep_isolated_pixels":false, "min_number_picture_neighbors":1 }
The only setting we change in LSTOSA is: "timestamps_pointing":"dragon"
to interpolate the pointing position using Dragon-based timestamps since we use valid absolute timestamps as reference.
- Drive information:
/fefs/onsite/monitoring/driveLST1/DrivePositioning/
- NightSummary files:
/fefs/aswg/lstosa/NightSummary
- Note: OFF runs 1927, 1930 and 1933 from 2020_02_01 to be shared with the CTA real-time analysis team for purposes of software development.
2020 DL1 Analysis logbook
Date | DRS4 | CALIB | Runs ON | Runs OFF | Source | Conditions | Comments | Incidences |
---|---|---|---|---|---|---|---|---|
2020_01_14 | 1785 | 1786 | 1787 | 1788 | Crab | Dark | First day of datataking mainly for test purposes | |
2020_01_15 | 1789 | 1791 | 1795, 1796, 1797, 1799, 1800 (moon) | 1798, 1801 (moon) | Crab | Dark + Moon | (1800, 1801) Piggy-back L3 MAGIC moon. Good weather | Discarded run 1795 due to wrong DTs. While taking 1800, shifters report one cluster off in the event display. It shows no data in any event even though it HV is on. This is in addition to the cluster that is not seen only in calibration events. Looking at the pixels where the HV is switched off due to stars, this view and the event view seem not to agree. Not able to fit good muon rings in runs 1799, 1800, 1801 do not fit good muon rings (due to moon?) |
2020_01_17 | 1830 | 1831 | 1813, 1814, 1817, 1818, 1819, 1820, 1821, 1824, 1825, 1826, 1827 | 1815, 1816, 1822, 1823 | Crab | Dark | Piggy-back MAGIC | 1813, 1827 (1 subrun), 1814 (no UCTS). Problems with calib files. Runs 1826 and 1827 had problems with TIB-CameraServer connection |
2020_01_18 | 1830 | 1831 | 1832, 1833, 1834, 1835, 1836, 1843, 1844 | 1837, 1840, 1841, 1842 | Crab | Dark | UCTS+GPS start working. Piggy-back MAGIC. Good weather | Subruns 1836(.0080 and 81) are wrong created (missing files, Event ID always 0, ...) |
2020_01_27 | 1872 | 1873 | 1874, 1875, 1876, 1878, 1879, 1880 | 1877, 1881 | Crab | Dark | Some moon giving high rates | |
2020_01_28 | 1882 | 1883 | 1887, 1888, 1891, 1892 | 1893 | Crab | Dark | 1887, 1888 (no UCTS) | |
2020_01_31 | 1915 | 1917 | 1918, 1919,1921 | 1920 | Mrk421 | Dark | dust~76 µg/m3 (from TNG) | During most of the 1920, 1921 data-taking the L1-LOCAL rate is mostly dominated by a single module in the bottom left corner. At the end od 1921 rates have skyrocketed and thus stop the run. Not able to fit good muon rings in run 1921. |
2020_02_01 | 1922 | 1923 | 1925, 1926, 1928, 1929, 1931, 1932 | 1927, 1930, 1933 | Crab | Moon | Good weather | During data-taking, the L1-LOCAL rate is mostly dominated by a single module in the bottom left corner. Not able to fit good muon rings in runs 1925, 1926, 1927, 1928, 1929, 1930, 1931. |
2020_02_02 | 1944 | 1947 | 1948, 1951, 1952, 1954, 1955 | 1949, 1953 | Crab | Moon | Piggy-back MAGIC | Windy. The rate is dominated by one cluster in the bottom left corner. Not able to fit good muon rings in run 1954 |
2020_02_13 | 1967 | 1968 | 1970 | 1971 | Crab | Dark | Not very good weather | Discarded run 1969 (taken without CDTS) |
2020_02_15 | 1985 | 1986 | 1987, 1988, 1991 | 1990, 1992 | Crab | Dark | Not very good atm transmission. Simultaneous observations with MAGIC | Discarded run 1989 (HV not properly set). Strange pattern in L1 rates in first runs due to MAGIC LIDAR. |
2020_02_17 | 1994 | 1995 | 1996, 1997, 1999, 2000, 2002, 2003 | 1998, 2001, 2004 | Crab | Dark | Simultaneous observations with MAGIC | Cloudy weather starting from run 1999 |
2020_02_18 | 2005 | 2006 | 2007, 2008, 2010, 2011, 2013, 2014 | 2009, 2012 | Crab | Dark | Good weather. Simultaneous observations with MAGIC (SumT until 23:00 UTC) | |
2020_02_27 | 2028 | 2030 | 2031, 2032 | 2033 | Crab | Dark | Good weather | Dragon time not OK after 4e6 events. Not able to fit good muon rings in run 1954 runs 2031, 2032 |
2020_02_28 | 2034 | 2035 | 2036, 2037 | 2038 | Crab | Dark | No interleaved pedestal events (?) |
2019 DL1 Analysis logbook
Date | DRS4 | CALIB | Runs ON | Runs OFF | Source | Conditions | Comments | Incidences |
---|---|---|---|---|---|---|---|---|
2019_11_23 | 1611 | 1614 | 1616, 1618, 1620, 1622 | 1615, 1617, 1619, 1621 | Crab | Dark | Data taken without EvB online corrections (except for first DRS4 run) | Good weather (T9km~0.9, cloudiness~25) |
2019_11_24 | 1623 | 1625 | 1627, 1629, 1631, 1632, 1633 | 1626, 1628, 1630 | Crab | Dark | Data taken with EvB online corrections (except for DRS4 run 1624) | Low transmission9km (<0.5) up to run 1629 |
2019_11_26 | 1645 | 1646 | 1648, 1649, 1650, 1652, 1653 | 1647, 1651 | Crab | Dark | Data taken with EvB online corrections | Good weather (T9km~0.9, cloudiness~30) |
2019_11_29 | 1696 | 1697 | 1701, 1702 | 1700 | Crab | Dark | Data taken with EvB online correction | Not very good atm transmission . Car flash at the end of 1701 |
Standard DL1 v0.5.1 production of 2019 and 2020 data (edited: June 2020)
- Files are in
/fefs/aswg/data/real/DL1/<observation_date>/v0.5.1_v03
, produced withlstchain v0.5.1
- Calibration files used:
/fefs/aswg/data/real/calibration/<date>/v03
- Find all the details about which runs where analyzed in the previous tables 2019 DL1 Analysis logbook and 2020 DL1 Analysis logbook
- As of production v0.5.1 there are also DL1 datacheck plots:
/fefs/aswg/data/real/DL1/<date>/v0.5.1_v03/datacheck_dl1_LST-1.Run*.pdf
- Provenance information:
/fefs/aswg/data/real/DL1/<date>/v0.5.1_v03/log
- DRS4 & CALIB runs from 2020_01_17 are not suitable for producing DRS4 pedestal and calibration files so DRS4 & CALIB runs from the next day 2020_01_18 were used to process the data runs.
- Time calibration file time_calibration.Run1625.0000.hdf5 from 2019_11_24 (v03) was used to process all the DL1 files.
Standard DL1 v0.4.4 production for 2019 and 2020 data (edited: March 6, 2020)
Files are in /fefs/aswg/data/real/DL1/<observation_date>/v0.4.4_v00
, produced with lstchain v0.4.4
Note: these v0.4.4 data are for the first time processed with the intended LocalPeakWindowSum, same as MC.
Standard lstchain configuration file was used: cta-lstchain/lstchain/data/lstchain_standard_config.json
. Hence these cleaning settings were applied:
"tailcut": { "picture_thresh":6, "boundary_thresh":3, "keep_isolated_pixels":false, "min_number_picture_neighbors":1 }
Find all the details about which runs where analyzed in the tables below (DL1 Analysis logbook).
Additional comments
- DRS4 & CALIB runs from 2020_01_17 and 2020_01_25 are not suitable for producing DRS4 pedestal and calibration files so DRS4 & CALIB runs from other days (2020_01_18 and 2020_01_27 respectively) were used to process the data runs from those days.
- Time calibration file time_calibration.Run1625.0000.hdf5 from 2019_11_24 was used to process all the DL1 files.
- By default (from January 2020 on), ucts time-stamps
"timestamps_pointing":"ucts"
are used to interpolate pointing coordinates.
- However for Nov 2019 data, dragon-based time-stamps were used for the interpolation:
"timestamps_pointing":"dragon"
- Corrected new drive log files were used for Nov 2019:
/fefs/aswg/scripts-osa/corrected_drive_logs_Nov19/
--Daniel.morcuende (talk) 11:31, 6 March 2020 (UTC)
Standard DL1 v0.4.3 production for 2019 and 2020 data (edited: January 2020)
Files are in /fefs/aswg/data/real/DL1/<observation_date>/v0.4.3_v00
, produced with lstchain v0.4.3
Standard lstchain configuration file was used. You can find it here:
/fefs/aswg/software/virtual_env/ctasoft/cta-lstchain/lstchain/data/lstchain_standard_config.json
Cleaning levels: 6-3
"tailcut": { "picture_thresh":6, "boundary_thresh":3, "keep_isolated_pixels":false, "min_number_picture_neighbors":1 }
MC production
Have a look at MC_analysis_and_IRF_production
--Thomas Vuillaume (talk) 15:02, 21 November 2019 (UTC)