Image cleaning and parametrization

Real Data DL1 production

Real data will be stored in the IT Container following the structure discussed in the Redmine issue https://forge.in2p3.fr/issues/37938, which includes directories whose name depends on the date. The convention UTC+12, by which days change at noon of the next day, is used at DAQ. For instance, for the data taken in the night 2019_11_23 to 2019_11_24 the selected date would be 2019_11_23 and main directories:

/fefs/aswg/data/real/DL1/20191123/v00 for the default analysis

v00 can take the form of XXX_NN where XXX is the initials of a name and NN a version number (e,g, JLC_01 for the first version of Jose Luis Contreras) when it is an alternative version run by an analysis shifter)

--Jose-Luis (talk) 18:09, 22 November 2019 (UTC)

General comments

• Unless otherwise specified, the standard configuration file is used to produce DL1 and DL2 analysis products. You can find it here:

/fefs/aswg/software/virtual_env/ctasoft/cta-lstchain/lstchain/data/lstchain_standard_config.json

In particular, the tailcut cleaning levels are the followings:

 "tailcut": {
   "picture_thresh":6,
   "boundary_thresh":3,
   "keep_isolated_pixels":false,
   "min_number_picture_neighbors":1
 }

The only setting we change in LSTOSA is: "timestamps_pointing":"dragon" to interpolate the pointing position using Dragon-based timestamps since we use valid absolute timestamps as reference.

• Drive information: /fefs/onsite/monitoring/driveLST1/DrivePositioning/
• NightSummary files: /fefs/aswg/lstosa/NightSummary

• Note: OFF runs 1927, 1930 and 1933 from 2020_02_01 to be shared with the CTA real-time analysis team for purposes of software development.

2020 DL1 Analysis logbook

Date	DRS4	CALIB	Runs ON	Runs OFF	Source	Conditions	Comments	Incidences
2020_01_14	1785	1786	1787	1788	Crab	Dark	First day of datataking mainly for test purposes
2020_01_15	1789	1791	1795, 1796, 1797, 1799, 1800 (moon)	1798, 1801 (moon)	Crab	Dark + Moon	(1800, 1801) Piggy-back L3 MAGIC moon. Good weather	Discarded run 1795 due to wrong DTs. While taking 1800, shifters report one cluster off in the event display. It shows no data in any event even though it HV is on. This is in addition to the cluster that is not seen only in calibration events. Looking at the pixels where the HV is switched off due to stars, this view and the event view seem not to agree. Not able to fit good muon rings in runs 1799, 1800, 1801 do not fit good muon rings (due to moon?)
2020_01_17	1830	1831	1813, 1814, 1817, 1818, 1819, 1820, 1821, 1824, 1825, 1826, 1827	1815, 1816, 1822, 1823	Crab	Dark	Piggy-back MAGIC	1813, 1827 (1 subrun), 1814 (no UCTS). Problems with calib files. Runs 1826 and 1827 had problems with TIB-CameraServer connection
2020_01_18	1830	1831	1832, 1833, 1834, 1835, 1836, 1843, 1844	1837, 1840, 1841, 1842	Crab	Dark	UCTS+GPS start working. Piggy-back MAGIC. Good weather	Subruns 1836(.0080 and 81) are wrong created (missing files, Event ID always 0, ...)
2020_01_27	1872	1873	1874, 1875, 1876, 1878, 1879, 1880	1877, 1881	Crab	Dark		Some moon giving high rates
2020_01_28	1882	1883	1887, 1888, 1891, 1892	1893	Crab	Dark		1887, 1888 (no UCTS)
2020_01_31	1915	1917	1918, 1919,1921	1920	Mrk421	Dark	dust~76 µg/m3 (from TNG)	During most of the 1920, 1921 data-taking the L1-LOCAL rate is mostly dominated by a single module in the bottom left corner. At the end od 1921 rates have skyrocketed and thus stop the run. Not able to fit good muon rings in run 1921.
2020_02_01	1922	1923	1925, 1926, 1928, 1929, 1931, 1932	1927, 1930, 1933	Crab	Moon	Good weather	During data-taking, the L1-LOCAL rate is mostly dominated by a single module in the bottom left corner. Not able to fit good muon rings in runs 1925, 1926, 1927, 1928, 1929, 1930, 1931.
2020_02_02	1944	1947	1948, 1951, 1952, 1954, 1955	1949, 1953	Crab	Moon	Piggy-back MAGIC	Windy. The rate is dominated by one cluster in the bottom left corner. Not able to fit good muon rings in run 1954
2020_02_13	1967	1968	1970	1971	Crab	Dark	Not very good weather	Discarded run 1969 (taken without CDTS)
2020_02_15	1985	1986	1987, 1988, 1991	1990, 1992	Crab	Dark	Not very good atm transmission. Simultaneous observations with MAGIC	Discarded run 1989 (HV not properly set). Strange pattern in L1 rates in first runs due to MAGIC LIDAR.
2020_02_17	1994	1995	1996, 1997, 1999, 2000, 2002, 2003	1998, 2001, 2004	Crab	Dark	Simultaneous observations with MAGIC	Cloudy weather starting from run 1999
2020_02_18	2005	2006	2007, 2008, 2010, 2011, 2013, 2014	2009, 2012	Crab	Dark	Good weather. Simultaneous observations with MAGIC (SumT until 23:00 UTC)
2020_02_27	2028	2030	2031, 2032	2033	Crab	Dark	Good weather	Dragon time not OK after 4e6 events. Not able to fit good muon rings in run 1954 runs 2031, 2032
2020_02_28	2034	2035	2036, 2037	2038	Crab	Dark	No interleaved pedestal events (?)

2019 DL1 Analysis logbook

Date	DRS4	CALIB	Runs ON	Runs OFF	Source	Conditions	Comments	Incidences
2019_11_23	1611	1614	1616, 1618, 1620, 1622	1615, 1617, 1619, 1621	Crab	Dark	Data taken without EvB online corrections (except for first DRS4 run)	Good weather (T9km~0.9, cloudiness~25)
2019_11_24	1623	1625	1627, 1629, 1631, 1632, 1633	1626, 1628, 1630	Crab	Dark	Data taken with EvB online corrections (except for DRS4 run 1624)	Low transmission9km (<0.5) up to run 1629
2019_11_26	1645	1646	1648, 1649, 1650, 1652, 1653	1647, 1651	Crab	Dark	Data taken with EvB online corrections	Good weather (T9km~0.9, cloudiness~30)
2019_11_29	1696	1697	1701, 1702	1700	Crab	Dark	Data taken with EvB online correction	Not very good atm transmission . Car flash at the end of 1701

Standard DL1 v0.5.1 production of 2019 and 2020 data (edited: June 2020)

• Files are in /fefs/aswg/data/real/DL1/<observation_date>/v0.5.1_v03 , produced with lstchain v0.5.1

• Calibration files used: /fefs/aswg/data/real/calibration/<date>/v03

• Find all the details about which runs where analyzed in the previous tables 2019 DL1 Analysis logbook and 2020 DL1 Analysis logbook

• As of production v0.5.1 there are also DL1 datacheck plots: /fefs/aswg/data/real/DL1/<date>/v0.5.1_v03/datacheck_dl1_LST-1.Run*.pdf

• Provenance information: /fefs/aswg/data/real/DL1/<date>/v0.5.1_v03/log

• DRS4 & CALIB runs from 2020_01_17 are not suitable for producing DRS4 pedestal and calibration files so DRS4 & CALIB runs from the next day 2020_01_18 were used to process the data runs.

• Time calibration file time_calibration.Run1625.0000.hdf5 from 2019_11_24 (v03) was used to process all the DL1 files.

Standard DL1 v0.4.4 production for 2019 and 2020 data (edited: March 6, 2020)

Files are in /fefs/aswg/data/real/DL1/<observation_date>/v0.4.4_v00 , produced with lstchain v0.4.4

Note: these v0.4.4 data are for the first time processed with the intended LocalPeakWindowSum, same as MC.

Standard lstchain configuration file was used: cta-lstchain/lstchain/data/lstchain_standard_config.json. Hence these cleaning settings were applied:

 "tailcut": {
   "picture_thresh":6,
   "boundary_thresh":3,
   "keep_isolated_pixels":false,
   "min_number_picture_neighbors":1
 }

Find all the details about which runs where analyzed in the tables below (DL1 Analysis logbook).

Additional comments

• DRS4 & CALIB runs from 2020_01_17 and 2020_01_25 are not suitable for producing DRS4 pedestal and calibration files so DRS4 & CALIB runs from other days (2020_01_18 and 2020_01_27 respectively) were used to process the data runs from those days.

• Time calibration file time_calibration.Run1625.0000.hdf5 from 2019_11_24 was used to process all the DL1 files.

• By default (from January 2020 on), ucts time-stamps "timestamps_pointing":"ucts" are used to interpolate pointing coordinates.

• However for Nov 2019 data, dragon-based time-stamps were used for the interpolation: "timestamps_pointing":"dragon"

• Corrected new drive log files were used for Nov 2019: /fefs/aswg/scripts-osa/corrected_drive_logs_Nov19/

--Daniel.morcuende (talk) 11:31, 6 March 2020 (UTC)

Standard DL1 v0.4.3 production for 2019 and 2020 data (edited: January 2020)

Files are in /fefs/aswg/data/real/DL1/<observation_date>/v0.4.3_v00 , produced with lstchain v0.4.3

Standard lstchain configuration file was used. You can find it here:

/fefs/aswg/software/virtual_env/ctasoft/cta-lstchain/lstchain/data/lstchain_standard_config.json

Cleaning levels: 6-3

 "tailcut": {
   "picture_thresh":6,
   "boundary_thresh":3,
   "keep_isolated_pixels":false,
   "min_number_picture_neighbors":1
 }

MC production

Have a look at MC_analysis_and_IRF_production

--Thomas Vuillaume (talk) 15:02, 21 November 2019 (UTC)