Difference between revisions of "Image cleaning and parametrization"

From my_wiki
Jump to: navigation, search
(→‎MC DL1 production)
m (→‎Real Data DL1 production)
 
(11 intermediate revisions by 3 users not shown)
Line 2: Line 2:
 
==  Real Data DL1 production ==
 
==  Real Data DL1 production ==
  
−
Real data will be stored in the IT Container following the structure discussed in the Redmine issue https://forge.in2p3.fr/issues/37938, which includes directories whose name depends on the date.  We will use the convention UTC+12 as used in DAQ, by which days change at noon. For instance, for the data take in the night 2019_11_22 to 2019_11_23 the selected date would be 2019_11_23 and main directories:
+
Real data will be stored in the IT Container following the structure discussed in the Redmine issue https://forge.in2p3.fr/issues/37938, which includes directories whose name depends on the date.  The convention UTC+12, by which days change at noon of the next day, is used at DAQ. For instance, for the data taken in the night 2019_11_23 to 2019_11_24 the selected date would be 2019_11_23 and main directories:
  
  
−
'' /fefs/aswg/data/real/dl1/20191122/v00 '' for the default analysis  
+
'' /fefs/aswg/data/real/DL1/20191123/v00 '' for the default analysis  
 
    
 
    
 
v00 can take the form of XXX_NN where XXX is the initials of a name and NN a version number (e,g, JLC_01 for the first version of Jose Luis Contreras) when it is an alternative version run by an analysis shifter)
 
v00 can take the form of XXX_NN where XXX is the initials of a name and NN a version number (e,g, JLC_01 for the first version of Jose Luis Contreras) when it is an alternative version run by an analysis shifter)
Line 12: Line 12:
 
--[[User:Jose-Luis|Jose-Luis]] ([[User talk:Jose-Luis|talk]]) 18:09, 22 November 2019 (UTC)
 
--[[User:Jose-Luis|Jose-Luis]] ([[User talk:Jose-Luis|talk]]) 18:09, 22 November 2019 (UTC)
  
−
==  MC DL1 production ==
+
__INDEX__
  
−
Latest DL1 MC production to use can be found there:
+
=== General comments ===
  
−
* <code> /fefs/aswg/data-tmp/mc/DL1/20190415/proton/south_pointing/20191109_v0.1_v00/    </code>
+
* Unless otherwise specified, the standard configuration file is used to produce DL1 and DL2 analysis products. You can find it here:
−
* <code> /fefs/aswg/data-tmp/mc/DL1/20190415/gamma/south_pointing/20191109_v0.1_v00/    </code>
 
−
* <code>/fefs/aswg/data-tmp/mc/DL1/20190415/gamma-diffuse/south_pointing/20191109_v0.1_v00/  </code>
 
−
* <code> /fefs/aswg/data-tmp/mc/DL1/20190415/electron/south_pointing/20191109_v0.1_v00/ </code>
 
  
 +
<code>/fefs/aswg/software/virtual_env/ctasoft/cta-lstchain/lstchain/data/lstchain_standard_config.json</code>
  
−
The DL1 have been reprocessed to get DL1b with different image cleaning (they can be found in the same directory with extension `_tailcut1` and `_tailcut2`):
+
In particular, the tailcut cleaning levels are the followings:
  
−
Tailcut 1 :
+
  "tailcut": {
−
<code>
+
    "picture_thresh":6,
−
picture_thresh:10, boundary_thresh:5, keep_isolated_pixels:false, min_number_picture_neighbors:1
+
    "boundary_thresh":3,
−
</code>
+
    "keep_isolated_pixels":false,
 +
    "min_number_picture_neighbors":1
 +
  }
  
−
Tailcut 2:
+
The only setting we change in LSTOSA is: <code>"timestamps_pointing":"dragon"</code> to interpolate the pointing position using Dragon-based timestamps since we use valid absolute timestamps as reference.
−
<code>
 
−
picture_thresh:8, boundary_thresh:4, keep_isolated_pixels:false, min_number_picture_neighbors:1
 
−
</code>
 
  
 +
* Drive information: <code>/fefs/onsite/monitoring/driveLST1/DrivePositioning/</code>
 +
* NightSummary files: <code>/fefs/aswg/lstosa/NightSummary</code>
 +
 +
* Note: OFF runs 1927, 1930 and 1933 from 2020_02_01 to be shared with the CTA real-time analysis team for purposes of software development.
 +
 +
=== 2020 DL1 Analysis logbook ===
 +
{| class="wikitable"
 +
!Date  !!  DRS4  !!  CALIB  !!  Runs ON  !!  Runs OFF  !!  Source  !!  Conditions  !!  Comments  !!  Incidences
 +
|-
 +
| 2020_01_14 || 1785 || 1786 || 1787  ||  1788  ||  Crab  ||  Dark || First day of datataking mainly for test purposes ||
 +
|-
 +
| 2020_01_15 || 1789 || 1791 || 1795, 1796, 1797, 1799, 1800 (moon) || 1798, 1801 (moon) || Crab || Dark + Moon || (1800, 1801) Piggy-back L3 MAGIC moon. Good weather || Discarded run 1795 due to wrong DTs. While taking 1800, shifters report one cluster off in the event display. It shows no data in any event even though it HV is on. This is in addition to the cluster that is not seen only in calibration events. Looking at the pixels where the HV is switched off due to stars, this view and the event view seem not to agree. Not able to fit good muon rings in runs 1799, 1800, 1801 do not fit good muon rings (due to moon?)
 +
|-
 +
| 2020_01_17 || 1830 || 1831 || 1813, 1814, 1817, 1818, 1819, 1820, 1821, 1824, 1825, 1826, 1827 || 1815, 1816, 1822, 1823 || Crab || Dark || Piggy-back MAGIC || 1813, 1827 (1 subrun), 1814 (no UCTS). Problems with calib files. Runs 1826 and 1827 had problems with TIB-CameraServer connection
 +
|-
 +
| 2020_01_18 || 1830 || 1831 || 1832, 1833, 1834, 1835, 1836, 1843, 1844 || 1837, 1840, 1841, 1842 || Crab || Dark || UCTS+GPS start working. Piggy-back MAGIC. Good weather || Subruns 1836(.0080 and 81) are wrong created (missing files, Event ID always 0, ...)
 +
|-
 +
| 2020_01_27 || 1872 || 1873 || 1874, 1875, 1876, 1878, 1879, 1880 || 1877, 1881 || Crab || Dark  ||  || Some moon giving high rates
 +
|-
 +
| 2020_01_28 || 1882 || 1883 || 1887, 1888, 1891, 1892 || 1893 || Crab || Dark ||  || 1887, 1888 (no UCTS)
 +
|-
 +
| 2020_01_31 || 1915 || 1917 || 1918, 1919,1921 || 1920 || Mrk421 || Dark || dust~76 µg/m3 (from TNG) || During most of the 1920, 1921 data-taking the L1-LOCAL rate is mostly dominated by a single module in the bottom left corner.  At the end od 1921 rates have skyrocketed and thus stop the run. Not able to fit good muon rings in run 1921.
 +
|-
 +
| 2020_02_01 || 1922 || 1923 || 1925, 1926, 1928, 1929, 1931, 1932 || 1927, 1930, 1933 || Crab || Moon || Good weather || During data-taking, the L1-LOCAL rate is mostly dominated by a single module in the bottom left corner. Not able to fit good muon rings in runs 1925, 1926, 1927, 1928, 1929, 1930, 1931.
 +
|-
 +
| 2020_02_02 || 1944 || 1947 || 1948, 1951, 1952, 1954, 1955 || 1949, 1953 || Crab  || Moon || Piggy-back MAGIC || Windy. The rate is dominated by one cluster in the bottom left corner. Not able to fit good muon rings in run 1954
 +
|-
 +
| 2020_02_13 || 1967|| 1968 || 1970 || 1971 || Crab || Dark || Not very good weather || Discarded run 1969 (taken without CDTS)
 +
|-
 +
| 2020_02_15 || 1985|| 1986 || 1987, 1988, 1991 || 1990, 1992 || Crab || Dark || Not very good atm transmission. Simultaneous observations with MAGIC || Discarded run 1989 (HV not properly set). Strange pattern in L1 rates in first runs due to MAGIC LIDAR.
 +
|-
 +
| 2020_02_17 || 1994 || 1995 || 1996, 1997, 1999, 2000, 2002, 2003 || 1998, 2001, 2004 || Crab || Dark || Simultaneous observations with MAGIC || Cloudy weather starting from run 1999
 +
|-
 +
| 2020_02_18 || 2005 || 2006 || 2007, 2008, 2010, 2011, 2013, 2014 || 2009, 2012 || Crab || Dark || Good weather. Simultaneous observations with MAGIC (SumT until 23:00 UTC)
 +
|-
 +
| 2020_02_27 || 2028 || 2030 || 2031, 2032 || 2033 || Crab || Dark || Good weather  || Dragon time not OK after 4e6 events. Not able to fit good muon rings in run 1954 runs 2031, 2032
 +
|-
 +
| 2020_02_28 || 2034 || 2035 || 2036, 2037 || 2038 || Crab || Dark || No interleaved pedestal events (?)  ||
 +
|}
 +
 +
 +
=== 2019 DL1 Analysis logbook ===
 +
{| class="wikitable"
 +
!Date  !!  DRS4  !!  CALIB  !!  Runs ON  !!  Runs OFF  !!  Source  !!  Conditions  !!  Comments  !!  Incidences
 +
|-
 +
| 2019_11_23 || 1611 || 1614 || 1616, 1618, 1620, 1622 || 1615, 1617, 1619, 1621 || Crab || Dark  || Data taken without EvB online corrections (except for first DRS4 run) || Good weather (T9km~0.9, cloudiness~25)
 +
|-
 +
| 2019_11_24 || 1623 || 1625 || 1627, 1629, 1631, 1632, 1633 || 1626, 1628, 1630 || Crab || Dark  || Data taken with EvB online corrections (except for DRS4 run 1624) || Low transmission9km (<0.5) up to run 1629
 +
|-
 +
| 2019_11_26 || 1645 || 1646 || 1648, 1649, 1650, 1652, 1653 || 1647, 1651 || Crab || Dark  || Data taken with EvB online corrections || Good weather (T9km~0.9, cloudiness~30)
 +
|-
 +
| 2019_11_29 || 1696 || 1697 || 1701, 1702 || 1700 || Crab || Dark  || Data taken with EvB online correction || Not very good atm transmission . Car flash at the end of 1701
 +
|}
 +
 +
 +
 +
=== Standard DL1 v0.5.1 production of 2019 and 2020 data (edited: June 2020) ===
 +
 +
* Files are in <code> /fefs/aswg/data/real/DL1/<observation_date>/v0.5.1_v03 </code>, produced with <code>lstchain v0.5.1</code>
 +
 +
* Calibration files used: <code>/fefs/aswg/data/real/calibration/<date>/v03 </code>
 +
 +
* Find all the details about which runs where analyzed in the previous tables 2019 DL1 Analysis logbook and 2020 DL1 Analysis logbook
 +
 +
* As of production v0.5.1 there are also DL1 datacheck plots: <code>/fefs/aswg/data/real/DL1/<date>/v0.5.1_v03/datacheck_dl1_LST-1.Run*.pdf </code>
 +
 +
* Provenance information: <code> /fefs/aswg/data/real/DL1/<date>/v0.5.1_v03/log </code>
 +
 +
* DRS4 & CALIB runs from 2020_01_17 are not suitable for producing DRS4 pedestal and calibration files so DRS4 & CALIB runs from the next day 2020_01_18 were used to process the data runs.
 +
 +
* Time calibration file ''time_calibration.Run1625.0000.hdf5'' from 2019_11_24 (v03) was used to process all the DL1 files.
 +
 +
 +
=== Standard DL1 v0.4.4 production for 2019 and 2020 data (edited: March 6, 2020) ===
 +
 +
Files are in <code> /fefs/aswg/data/real/DL1/<observation_date>/v0.4.4_v00 </code>, produced with <code>lstchain v0.4.4</code>
 +
 +
'''Note: these v0.4.4 data are for the first time processed with the intended LocalPeakWindowSum, same as MC.'''
 +
 +
Standard lstchain configuration file was used: <code>cta-lstchain/lstchain/data/lstchain_standard_config.json</code>. Hence these cleaning settings were applied:
 +
 +
  "tailcut": {
 +
    "picture_thresh":6,
 +
    "boundary_thresh":3,
 +
    "keep_isolated_pixels":false,
 +
    "min_number_picture_neighbors":1
 +
  }
 +
 +
Find all the details about which runs where analyzed in the tables below (DL1 Analysis logbook).
 +
 +
 +
=====Additional comments=====
 +
 +
* DRS4 & CALIB runs from 2020_01_17 and 2020_01_25 are not suitable for producing DRS4 pedestal and calibration files so DRS4 & CALIB runs from other days (2020_01_18 and 2020_01_27 respectively) were used to process the data runs from those days.
 +
 +
* Time calibration file ''time_calibration.Run1625.0000.hdf5'' from 2019_11_24 was used to process all the DL1 files.
 +
 +
* By default (from January 2020 on), ucts time-stamps <code> "timestamps_pointing":"ucts" </code> are used to interpolate pointing coordinates.
 +
 +
* However for Nov 2019 data, dragon-based time-stamps were used for the interpolation: <code> "timestamps_pointing":"dragon" </code>
 +
 +
* Corrected new drive log files were used for Nov 2019: <code> /fefs/aswg/scripts-osa/corrected_drive_logs_Nov19/ </code>
 +
 +
 +
--[[User:Daniel.morcuende|Daniel.morcuende]] ([[User talk:Daniel.morcuende|talk]]) 11:31, 6 March 2020 (UTC)
 +
 +
 +
=== Standard DL1 v0.4.3 production for 2019 and 2020 data (edited: January 2020) ===
 +
 +
Files are in <code> /fefs/aswg/data/real/DL1/<observation_date>/v0.4.3_v00 </code>, produced with <code>lstchain v0.4.3</code>
 +
 +
Standard lstchain configuration file was used. You can find it here:
 +
 +
<code>/fefs/aswg/software/virtual_env/ctasoft/cta-lstchain/lstchain/data/lstchain_standard_config.json</code>
 +
 +
Cleaning levels: 6-3
 +
 +
  "tailcut": {
 +
    "picture_thresh":6,
 +
    "boundary_thresh":3,
 +
    "keep_isolated_pixels":false,
 +
    "min_number_picture_neighbors":1
 +
  }
 +
 +
==  MC production ==
 +
 +
Have a look at [[MC_analysis_and_IRF_production]]
  
 
--[[User:Thomas.vuillaume|Thomas Vuillaume]] ([[User talk:Thomas.vuillaume|talk]]) 15:02, 21 November 2019 (UTC)
 
--[[User:Thomas.vuillaume|Thomas Vuillaume]] ([[User talk:Thomas.vuillaume|talk]]) 15:02, 21 November 2019 (UTC)

Latest revision as of 19:22, 9 June 2020

Real Data DL1 production[edit]

Real data will be stored in the IT Container following the structure discussed in the Redmine issue https://forge.in2p3.fr/issues/37938, which includes directories whose name depends on the date. The convention UTC+12, by which days change at noon of the next day, is used at DAQ. For instance, for the data taken in the night 2019_11_23 to 2019_11_24 the selected date would be 2019_11_23 and main directories:


/fefs/aswg/data/real/DL1/20191123/v00 for the default analysis

v00 can take the form of XXX_NN where XXX is the initials of a name and NN a version number (e,g, JLC_01 for the first version of Jose Luis Contreras) when it is an alternative version run by an analysis shifter)


--Jose-Luis (talk) 18:09, 22 November 2019 (UTC)


General comments[edit]

  • Unless otherwise specified, the standard configuration file is used to produce DL1 and DL2 analysis products. You can find it here:

/fefs/aswg/software/virtual_env/ctasoft/cta-lstchain/lstchain/data/lstchain_standard_config.json

In particular, the tailcut cleaning levels are the followings: 

 "tailcut": {
   "picture_thresh":6,
   "boundary_thresh":3,
   "keep_isolated_pixels":false,
   "min_number_picture_neighbors":1
 }

The only setting we change in LSTOSA is: "timestamps_pointing":"dragon" to interpolate the pointing position using Dragon-based timestamps since we use valid absolute timestamps as reference.

  • Drive information: /fefs/onsite/monitoring/driveLST1/DrivePositioning/
  • NightSummary files: /fefs/aswg/lstosa/NightSummary
  • Note: OFF runs 1927, 1930 and 1933 from 2020_02_01 to be shared with the CTA real-time analysis team for purposes of software development.

2020 DL1 Analysis logbook[edit]

Date DRS4 CALIB Runs ON Runs OFF Source Conditions Comments Incidences
2020_01_14 1785 1786 1787 1788 Crab Dark First day of datataking mainly for test purposes
2020_01_15 1789 1791 1795, 1796, 1797, 1799, 1800 (moon) 1798, 1801 (moon) Crab Dark + Moon (1800, 1801) Piggy-back L3 MAGIC moon. Good weather Discarded run 1795 due to wrong DTs. While taking 1800, shifters report one cluster off in the event display. It shows no data in any event even though it HV is on. This is in addition to the cluster that is not seen only in calibration events. Looking at the pixels where the HV is switched off due to stars, this view and the event view seem not to agree. Not able to fit good muon rings in runs 1799, 1800, 1801 do not fit good muon rings (due to moon?)
2020_01_17 1830 1831 1813, 1814, 1817, 1818, 1819, 1820, 1821, 1824, 1825, 1826, 1827 1815, 1816, 1822, 1823 Crab Dark Piggy-back MAGIC 1813, 1827 (1 subrun), 1814 (no UCTS). Problems with calib files. Runs 1826 and 1827 had problems with TIB-CameraServer connection
2020_01_18 1830 1831 1832, 1833, 1834, 1835, 1836, 1843, 1844 1837, 1840, 1841, 1842 Crab Dark UCTS+GPS start working. Piggy-back MAGIC. Good weather Subruns 1836(.0080 and 81) are wrong created (missing files, Event ID always 0, ...)
2020_01_27 1872 1873 1874, 1875, 1876, 1878, 1879, 1880 1877, 1881 Crab Dark Some moon giving high rates
2020_01_28 1882 1883 1887, 1888, 1891, 1892 1893 Crab Dark 1887, 1888 (no UCTS)
2020_01_31 1915 1917 1918, 1919,1921 1920 Mrk421 Dark dust~76 µg/m3 (from TNG) During most of the 1920, 1921 data-taking the L1-LOCAL rate is mostly dominated by a single module in the bottom left corner. At the end od 1921 rates have skyrocketed and thus stop the run. Not able to fit good muon rings in run 1921.
2020_02_01 1922 1923 1925, 1926, 1928, 1929, 1931, 1932 1927, 1930, 1933 Crab Moon Good weather During data-taking, the L1-LOCAL rate is mostly dominated by a single module in the bottom left corner. Not able to fit good muon rings in runs 1925, 1926, 1927, 1928, 1929, 1930, 1931.
2020_02_02 1944 1947 1948, 1951, 1952, 1954, 1955 1949, 1953 Crab Moon Piggy-back MAGIC Windy. The rate is dominated by one cluster in the bottom left corner. Not able to fit good muon rings in run 1954
2020_02_13 1967 1968 1970 1971 Crab Dark Not very good weather Discarded run 1969 (taken without CDTS)
2020_02_15 1985 1986 1987, 1988, 1991 1990, 1992 Crab Dark Not very good atm transmission. Simultaneous observations with MAGIC Discarded run 1989 (HV not properly set). Strange pattern in L1 rates in first runs due to MAGIC LIDAR.
2020_02_17 1994 1995 1996, 1997, 1999, 2000, 2002, 2003 1998, 2001, 2004 Crab Dark Simultaneous observations with MAGIC Cloudy weather starting from run 1999
2020_02_18 2005 2006 2007, 2008, 2010, 2011, 2013, 2014 2009, 2012 Crab Dark Good weather. Simultaneous observations with MAGIC (SumT until 23:00 UTC)
2020_02_27 2028 2030 2031, 2032 2033 Crab Dark Good weather Dragon time not OK after 4e6 events. Not able to fit good muon rings in run 1954 runs 2031, 2032
2020_02_28 2034 2035 2036, 2037 2038 Crab Dark No interleaved pedestal events (?)


2019 DL1 Analysis logbook[edit]

Date DRS4 CALIB Runs ON Runs OFF Source Conditions Comments Incidences
2019_11_23 1611 1614 1616, 1618, 1620, 1622 1615, 1617, 1619, 1621 Crab Dark Data taken without EvB online corrections (except for first DRS4 run) Good weather (T9km~0.9, cloudiness~25)
2019_11_24 1623 1625 1627, 1629, 1631, 1632, 1633 1626, 1628, 1630 Crab Dark Data taken with EvB online corrections (except for DRS4 run 1624) Low transmission9km (<0.5) up to run 1629
2019_11_26 1645 1646 1648, 1649, 1650, 1652, 1653 1647, 1651 Crab Dark Data taken with EvB online corrections Good weather (T9km~0.9, cloudiness~30)
2019_11_29 1696 1697 1701, 1702 1700 Crab Dark Data taken with EvB online correction Not very good atm transmission . Car flash at the end of 1701


Standard DL1 v0.5.1 production of 2019 and 2020 data (edited: June 2020)[edit]

  • Files are in /fefs/aswg/data/real/DL1/<observation_date>/v0.5.1_v03 , produced with lstchain v0.5.1
  • Calibration files used: /fefs/aswg/data/real/calibration/<date>/v03
  • Find all the details about which runs where analyzed in the previous tables 2019 DL1 Analysis logbook and 2020 DL1 Analysis logbook
  • As of production v0.5.1 there are also DL1 datacheck plots: /fefs/aswg/data/real/DL1/<date>/v0.5.1_v03/datacheck_dl1_LST-1.Run*.pdf
  • Provenance information: /fefs/aswg/data/real/DL1/<date>/v0.5.1_v03/log
  • DRS4 & CALIB runs from 2020_01_17 are not suitable for producing DRS4 pedestal and calibration files so DRS4 & CALIB runs from the next day 2020_01_18 were used to process the data runs.
  • Time calibration file time_calibration.Run1625.0000.hdf5 from 2019_11_24 (v03) was used to process all the DL1 files.


Standard DL1 v0.4.4 production for 2019 and 2020 data (edited: March 6, 2020)[edit]

Files are in /fefs/aswg/data/real/DL1/<observation_date>/v0.4.4_v00 , produced with lstchain v0.4.4

Note: these v0.4.4 data are for the first time processed with the intended LocalPeakWindowSum, same as MC.

Standard lstchain configuration file was used: cta-lstchain/lstchain/data/lstchain_standard_config.json. Hence these cleaning settings were applied: 

 "tailcut": {
   "picture_thresh":6,
   "boundary_thresh":3,
   "keep_isolated_pixels":false,
   "min_number_picture_neighbors":1
 }

Find all the details about which runs where analyzed in the tables below (DL1 Analysis logbook).


Additional comments[edit]
  • DRS4 & CALIB runs from 2020_01_17 and 2020_01_25 are not suitable for producing DRS4 pedestal and calibration files so DRS4 & CALIB runs from other days (2020_01_18 and 2020_01_27 respectively) were used to process the data runs from those days.
  • Time calibration file time_calibration.Run1625.0000.hdf5 from 2019_11_24 was used to process all the DL1 files.
  • By default (from January 2020 on), ucts time-stamps "timestamps_pointing":"ucts" are used to interpolate pointing coordinates.
  • However for Nov 2019 data, dragon-based time-stamps were used for the interpolation: "timestamps_pointing":"dragon"
  • Corrected new drive log files were used for Nov 2019: /fefs/aswg/scripts-osa/corrected_drive_logs_Nov19/


--Daniel.morcuende (talk) 11:31, 6 March 2020 (UTC)


Standard DL1 v0.4.3 production for 2019 and 2020 data (edited: January 2020)[edit]

Files are in /fefs/aswg/data/real/DL1/<observation_date>/v0.4.3_v00 , produced with lstchain v0.4.3

Standard lstchain configuration file was used. You can find it here:

/fefs/aswg/software/virtual_env/ctasoft/cta-lstchain/lstchain/data/lstchain_standard_config.json

Cleaning levels: 6-3 

 "tailcut": {
   "picture_thresh":6,
   "boundary_thresh":3,
   "keep_isolated_pixels":false,
   "min_number_picture_neighbors":1
 }

MC production[edit]

Have a look at MC_analysis_and_IRF_production

--Thomas Vuillaume (talk) 15:02, 21 November 2019 (UTC)

Retrieved from "https://www.lst1.iac.es/wiki/index.php?title=Image_cleaning_and_parametrization&oldid=2686"