DL1 data check
Contents
Where to find the DL1 data check products
The DL1 data check products can be found inside this Browsable directory Always use the most recent version vx.y of the check products which is available for the night of interest. As of March 2024, the most recent version is v0.10.
Inside each of the night-wise YYYYMMDD directories, e.g. v0.10/20231118, you can find three files: DL1_datacheck_20231118.[html, log, h5], which contain information relative to the whole night of observations (for the sky runs that were successfully processed by lstosa). There is also a pdf subdirectory which contains one .pdf file (with further useful information) per run. More details on these products are provided below.
Reporting issues
If problems are found in the data, experts should be contacted as soon as possible. This can be done via Slack (in the appropriate channel, depending on the problem) and / or submitting a reply to the ELOG entry of the corresponding night. Whenever a problem is found in the data check it is convenient to check the ELOG to verify if the shift crew reported anything that may be related to the problem (e.g. a car flash).
The html DL1_datacheck file
This is the most relevant one for checking the good behaviour of the telescope during the night. It is an interactive webpage showing various quantities (both pixel-wise and camera-wise) averaged over the whole run. Many of the plots have a slider which allows you to select the run number.
Here is an example: DL1_datacheck_20231214.html. Go to datacheck/dl1/v0.10 to find the latest available nights.
General comments:
- Some of the quantities are displayed together with limits (min/max, indicated by dashed orange and red lines) within which good data are usually found. The limits may change through the night (from run to run), because the limits are sometimes pointing-dependent (e.g. rates of showers - "cosmics" - depend significantly on the zenith angle). Note that the current limits were set in 2021 and some of them may require regular updates. The values are as of now hard-coded in the lstchain script lstchain_longterm_dl1_check.py, which is the one which produces the night-wise DL1 check outputs.
- The limits, and even the computation of some of the data check quantities, like e.g. everything related to muon rings, will not be valid for data taken under very high NSB conditions (which can be identified by the large values of the mean and std deviation of charges recorded in interleaved pedestal events). If limits are exceeded for such data, there is no reason to conclude that the telescope is malfunctioning.
- The quantities displayed in the html file are all run averages, there is no information about the evolution within a run. Inside the "pdf" directory under each of the YYYYMMDD folders you will find a pdf file for each of the runs of the night. They display the evolution subrun-by-subrun (i.e., every few seconds) of some quantities, and may be helpful to understand some of the possible issues spotted when checking the run-wise averages in the html file.
- By moving the mouse over the plots you can obtain useful information, like run number, pixel / cluster number, numerical value of the displayed quantity, etc.
- Note that sometimes data points may be beyond the ranges shown by default. You can use the "Bokeh plot tools" on the top right of each tab to zoom out (and in) any of the plots.
What to check in each tab of the DL1 datacheck html file?
- Event rates
- This tab shows the average rate (vs. run number) of different types of events: interleaved pedestals; interleaved flatfield (FF); "cosmics" (i.e. physics triggers, which are mostly showers when everything works well); and "contained muon rings" (those cosmics recognized as muons, and not truncated by the camera edges).
- The limits for cosmics and muon rings are for data taken in good conditions (and with not too high NSB). Data taken in moon conditions (with reduced HV in the PMTs) will be beyond limits, even if the telescope is perfectly fine.
- The rate of interleaved pedestals and interleaved FF is smaller than 100 Hz because of dead time. If they fall below ~90 Hz it may be due to increased dead time (e.g. because of occasional DAQ issues). Sometimes they may be completely missing - this may indicate either a problem with the event tagging, or perhaps the interleaved events were not even generated. This is a serious problem and should be reported immediately.
- Only sky runs are displayed in the html file. If you see gaps in the time axis of the plots, it may be simply due to a pause in observations. If the run numbers before and after the gap are not consecutive (you can know the run number placing the mouse over a point) it may mean that non-sky runs (for example, "pedcal" calibration runs) were taken. You can check this in the ELOG of the corresponding night (use the "Find" utility if it is not the latest night). If the missing runs are not calibrations, but sky runs, then the LST on-site analysis team should be informed.
- Trigger tags (both tabs)
- these tabs are obsolete since 20231206, when we started version 6 of the Event builder! Ignore them for newer data!
- TBD! (note for developers): keep the EvB5-specific info (like UCTS jumps) for older data in a dedicated tab. Replace the rest by a tab showing checks that the recorded images for the different event types look like they should (fraction of events surviving cleaning may be a good observable).
- Pointing
- Just to see where the telescope was pointing. May be useful to understand if a change of behaviour observed in some other quantity is perhaps related to a change of pointing (e.g. a drop in rates might be due to a worse atmosphere in the new pointing direction).
- Interleaved pedestals
- This allows one to check the level of noise on the camera. The top row of plots shows the mean pedestal charge for each pixel (use the sliders on the left to check different runs), while the bottom row shows its standard deviation. The mean is not around zero because we use a biased waveform integrator (which looks for the highest peak). The mean and the standard deviation are very correlated, both depend on the level of night sky background falling on a given pixel. Hence we only have set a limit (a maximum) on the standard deviation.
- Having a pixel or a group of pixels exceeding the limit is normally not an issue, and simply due to stars in the field of view. It should not be reported as a problem unless a pixel or group of pixels is always too noisy, for runs at different pointings. This can be checked better in the "Pixel problems" tab.
- Interleaved flat field, charge
- This tabs shows the mean and standard deviation of the pixel charges recorded during the interleaved flatfield events of every run. Note: the limits on the mean charge should perhaps be revised a bit downwards.
- The charge may be occasionally too low for some pixels. This can happen when their high voltage is reduced as a result of a star increasing the noise in the pixel. The conversion factor (to p.e.) is then no longer valid, and the amount of light is underestimated. Eventually we will re-calibrate pixels (in the so-called Cat-B calibration), so this is not a real issue.
- Like in the case of interleaved pedestals, one should only report an it as a problem if a pixel or group of pixels shows systematically too low (or too high) flat-field charge, regardless of the pointing.
- If the whole camera has too low or too high FF charge values, one possibility is that the calibox had a non-standard filter combination. In case it is too low, it can also be due to wrong event tagging... Perhaps the events tagged as FF include some which are not FF, but either cosmics or pedestals. We recommend to check the pdf file of the corresponding run. In the first page there is a plot of "Fraction of events surviving cleaning" vs. "subrun index". All genuine FF events should survive the image cleaning. If the plotted fraction is <1.0 at any time during the run, it means that non-FF events are being tagged as FF. This is a serious issue and should be immediately reported to daq / camera experts.
- Interleaved flat field, time
- This tab is useful to check the proper time-flatfielding of the camera, and the precision of the timing. The values are relative to the camera-averaged time in each flatfield event. This is so because what matters physics-wise is the relative timing between pixels. The absolute position of pulses within the readout window may jitter by ~nanoseconds, but consistently for all pixels (it is a "trigger jitter") and hence has no effect on the reconstruction.
- As usual, occasional deviations of single pixels from the expected range may be related to stars (causing pixel HV reduction) and are not a worry, unless a pixel is found to misbehave consistently in all runs.
- Cosmics (first tab)
- Here we check whether the rate of pulses (of >10 p.e. and >30 p.e.) in cosmics is the one expected from showers in good conditions. If the rates are too low (only the >30 p.e. plot has limits) this may just be the result of poor atmospheric conditions (e.g. calima or haze), we recommend to check the ELOG to verify this possibility.
- Too high rates in some runs are most often due to car flashes or other external sources of light which can trigger LST, like the MAGIC LIDAR. We recommend to check the page titled "COSMICS, relative frequency of pixel charges, camera averages" in the pdf file (see above) of the corresponding run. Check also the ELOG, perhaps the shift crew noticed during observations.
- While both too low and too high values of the rates in this tab indicate a sub-optimal quality of the data, there is not much we can do about it, so in general this is not an issue that needs to be reported to experts (unless you notice it becomes more frequent).
- Cosmics (second tab)
- Shows the distribution on the camera of the centers of gravity (CoG) of the shower images.
- Note: the rate axes of the plots here (z-axis in camera displays, y axis in the middle column, x-axis on the right column) have a range determined by the full span of the rates during the whole night. So in case there was some strong flash during the night (like car flashes) you will have to use the "z_range" slider of camera displays to be able to see details of the distribution.
- Accumulation of CoGs in certain points may be due to car flashes, MAGIC-LIDAR-induced events, or some misbehaving group of pixels (in the latter case these should also show up as noisy in the Interleaved pedestals tab).
- Localized regions of lower density of CoGs are most often due to the presence of stars. The image cleaning uses higher thresholds (determined from the noise in the pixel) around stars, to avoid having a large rate of spurious signals which survive the cleaning. These higher thresholds create a deficit of CoGs in those regions. As long as these regions are not persistent for different telescope pointings, there is no reason to worry.
- Pixel problems
- This tab contains summary plots for how often each pixels is outside the limits of different quantities. Click on the "z-range" slider, leaving it at its default position 0.50. Now the pixels which are beyond limits (on the "issue" that can be selected with the left-side slider) for more than half of the runs of the night will appear as red.
- Muons
- Interleaved pedestals, averages
- Interleaved FF, averages
- Cosmics, averages
List of known problems
Below is a (very incomplete!) list of some runs with known problems based on the data checks, which should not be used for higher data analysis (unless some special treatment is applied to solve their issues). You can use this to add the problems you spot in the data.
- 20201123/v0.9/tailcut84/dl1_LST-1.Run03038 "CDTS not connected"
- 20210513/v0.9/tailcut84/dl1_LST-1.Run04738 No pedestals (No TIB no CDTS)
- 20220302/v0.9/tailcut84/dl1_LST-1.Run07217 No pedestals
- 20220501/v0.9/tailcut84/dl1_LST-1.Run08053 No pedestals “TIB and CDTS are both 0”
- 20220503/v0.9/tailcut84/dl1_LST-1.Run08091 “Broken Camera Configuration, Data not usable most likely”
- 20220504/v0.9/tailcut84/dl1_LST-1.Run08121 CDTS and TIB are 0
- 20220916/v0.9/tailcut84/dl1_LST-1.Run09228,29,30,31,32 No pedestals according to data check, UCTS and TIB seem k.o. despite no Log message about it
- 20220929/v0.9/tailcut84/dl1_LST-1.Run09419 No pedestals, UCTS and TIB seem k.o. despite no Log message about it
- 20221018/v0.9/tailcut84/dl1_LST-1.Run09706 No pedestals, UCTS and TIB k.o.
- 20221020/v0.9/tailcut84/dl1_LST-1.Run09789 No pedestals, UCTS and TIB k.o.
- 20221024/v0.9/tailcut84/dl1_LST-1.Run09882 No pedestals, UCTS and TIB k.o.
- 20221026/v0.9/tailcut84/dl1_LST-1.Run09925 No pedestals, UCTS and TIB k.o. Not even mentioned in ELOG
- 20221028/v0.9/tailcut84/dl1_LST-1.Run09978 No pedestals, UCTS and TIB k.o.
- 20221101/v0.9/tailcut84/dl1_LST-1.Run10060 No pedestals, UCTS and TIB k.o.
- 20221102/v0.9/tailcut84/dl1_LST-1.Run10106 No pedestals, UCTS and TIB k.o. Runs to test DAQ problems!
- 20221102/v0.9/tailcut84/dl1_LST-1.Run10147 No pedestals, UCTS and TIB k.o. Runs to test DAQ problems!
- 20221102/v0.9/tailcut84/dl1_LST-1.Run10188 No pedestals, UCTS and TIB k.o. Runs to test DAQ problems!
- 20221104/v0.9/tailcut84/dl1_LST-1.Run10304 No pedestals, UCTS and TIB k.o. (DAQ stability tests)
- 20221105/v0.9/tailcut84/dl1_LST-1.Run10353 No pedestals, UCTS and TIB k.o. No message in Log, but anyway this is a high moon run, rate ~1k, Extremely low FF charges (for the whole night actually), and probably very low HV.
