User:Cdelgado

Management

List of open problems and commissioning plan activities [1]

Camera commissioning

Useful links

1. Data taking procedure [2]
2. VLANS topology [3]
3. ACADA LST camera FSM mapping [4]

Operation

Top 10 of "CAM startup failures" since June 21st

1. ECC/CAM power cycle (error state) 1
2. Module 6.28 down (power cycle relay 1) 14
3. Module 5.16 down (power cycle relay 2) 1
4. SIS "undefined" 1
5. network interface DOWN p1p1+p3p1 (osaka) 1
6. network interface DOWN p2p2 (osaka) 3
7. network interface DOWN p3p2 (osaka) 2
8. EVB(2) crashed in GOTOOBSERVING 1
9. TIB error state 255 3
10. EVB(2) only connecting to one group/switch (probably IF DOWN) 1

• From this table, out main problem is the Module 6.28 power up problem.
• I would say the second one is network interface going down
• Third one is TIB error state

Camera commissioning plan status

This page provides a summary of the status of the deliverables commissioning plan of the camera, with actions , for each point. What remains to be clarified is if providing this set of deliverables really guarantees the system is commissioned. My impression is that not necessarily.

1. Mechanics

Deliverables description

• Mechanics. Opening and closing front door (commissioning of the hydraulic system). Opening and closing backdoor. Moving the Star Imaging Screen (SIS). Movement and visibility from dish center of the SIS.

Comments

• It is not clear what has to be delivered and what is really needed to consider the system commissioned. All the declared deliverables have been performed and there are documents proving it pictures and movies in some cases).

Deliverables status

Actions

• 03/07/2019 Ask experts to decide what is required

2. Thermalization

Deliverables description

• a. Tests without Drive: Plot Temperature vs time of various T sensors, auxiliary and internal module sensors. T must stabilize within 2 hours and T of the modules must be within the operating range of the electronics with at least 10 degrees margin. Minimum run duration is 10 hours without any interruption.
• b. Tests with Drive: Plot of the water pressure vs zenith/azimuth angle.

Comments

Deliverables status

• For Deliverable a, plots exist when possible (there are no module sensor, but air sensor for modules channels), although limited to up to 8 hours to C. Delgado knowledge, and shows dependence due to direct sun light reaching to one side of the camera.

• Deliverable b does not make a lot of sense, as the important quantity are the temperature as a function of the zenith/azimuth angle. Information seems available.

Actions

• 03/07/2019 Ask Daniel to modify deliverable b to temperature vs t vs zenith
• 03/07/2019 Prepare a long run with telescope movement to provide all plots

3. Connectivity: DONE!

Deliverables description

• a. All modules (means Dragon and Backplane) must respond to their foreseen IP and must be able to start up (boot) at the start up sequence. Plot: T for backplane and Humidity in camera coordinates for every Dragon and corresponding backplane.
• b. Backplanes must see each other (L1, Clock, PPS distribution) and L0 distribution. Plot:

reconstructed network topology

Comments

• This is done, but deliverable b does not prove all the signals, but only L1. For L0 there are dedicated tests, that failed for few modules.

Deliverables status

• All done from monitoring and dedicated test before changing module numbering to the current one.

Actions

• 03/07/2019 Improve plots. Add dates.

4. State Machine

Deliverables description

• Software state machine must be verified: Safe -> StandBy -> Ready -> StandBy -> Safe. Plot: State vs Time to measure transition times between states.

Comments

• Still pretty unstable. From ELOG entry 304:
  • Not always all modules power up in the first trial (especially 6.28)
  • BP trigger UP delay is not checked yet. And PPS synchronization fails in module 114 sometimes.
  • L1 thresholds are not optimized. We should either implement Module rate control or prepare Threshold Table for various target total rate.
  • TIB stops sending triggers after some observations. The reason is unclear.
  • CaCo is not fully ready EVBv3 is not fully ready.
  • lst-chain seems not to be compatible with EVBv3 data. Quick analysis is impossible.

Deliverables status

• Not ready to do it

Actions

• 03/07/2019 Follow up.

5. Trigger Timing with Test Pulse. DONE! but quality of plots could be improved or uniformized

Deliverables description

• Using test pulses, the delays must be set such that a trigger issued by the Trigger Interface Board or Local Module Trigger the pulses are well within the Readout Window of 40 cells. Plot: 2D plot of camera coordinates and pulse position as z-axis. Plot: 1D overlay of all pulses in Region Of Interest.

Data available

• ​Run 00115 2019-02-19 21:14 for all modules triggering. Declared as showing pulses in all modules.
• Run 00232 2019-03-08 10:32 trigger only in central
• Run 00234 2019-03-08 11:10 trigger only in central
• Run 00416 ​2019-05-22 12:07 all modules triggering
• Run 00418 ​2019-05-22 12:24 trigger only in central
• Run 00535 ?????????? ????? ???????????????????????
• Runs 00632 to 00636

Comments

• Something like that already exist, but dependes of BP calibration, so should be posterior to it.
• Synchronization of all pulses should reached after BP calibration.

Deliverables status

• Plots can be improved, but we have them.

RUN 115 -- All modules triggering

RUN 418 -- trigger only in central

Run 535 -- conditions unknown

Actions

• 03/07/2019 Investigate and solve BP instabilities
• 03/07/2019 Maybe move this step further in the list.

6. L0/L1 rate scans with HV=0. FIRST THING TO BE DONE from scratch!

Deliverables description

• L0 rate scans with noise and with test pulses of a known amplitude. THIS SHOULD SAY L0 and L1 rate scans instead
  • a. Plot: Overlay of L0 rate scans with electronic noise only, with limits for the acceptable noise. Plot: Overlay of L0 rate scans with test pulse of a known amplitude, with limits for the acceptable discriminator level. Plot L0 threshold at 50% for each pulse amplitude.
  • b. L1 rate scans for every module for every adder individually with electronic noise only. Plot: Overlay of L1 rate scans with electronic noise only, with limits for the acceptable noise. L1 rate scans with a test pulse of a known amplitude. Plot: Overlay of L1 rate scans with test pulse of a known amplitude, with limits for the acceptable discriminator level. Plot L1 threshold of 50% rate for every pulse amplitude.
  • c. Repeat L1 scans and plots for different trigger modes (itself, 2 neighbours, 3 neighbours).

Data available

• I have plenty of files from ELOG and mal threads. However it is difficult to do anything with them because the conditions seems to be not fully documented.
• In addition, most of the files I have are only for L1. For L0 I almost have nothing.
• Here is the result of the rate scan we perform today with only noise and TP of 5 p.e. for mode 1 and mode 3. For each, we repeat three times the opreration.[...] With only noise, seem fine. [...] With TP5p.e, for mode1, the only thing strange for us is to see that for very high DT, we still see 300 Hz.[…] With TP5p.e. and mode 3: for dac 1, contrary to the one with HV, it seems ok, only module 48 has a strange behavior in the three trials and it had exactly the same in the rate scan dac1 of Tuesday night with HV. For dac2, it seems fine.
  • Link with data for L1 scans at [5]
  • New data tking from WLOG entry 316: mod 3
    • Newest L0 rate scan at [6]
    • Newest L1 rate scan at [7]

Comments

• Probably something has been done, but plots and data are scattered around, so it is better to do it systematically.
• I do not know up to which point we know the pulse amplitude.
• What about settings of L0? These affect the L1 scan, so could require a calibration before making sense of L1 rate scans.
• What does it mean acceptable discriminator levels? It is just in terms of the rate itself or the meaning of the discriminator level in terms of p.e.?

Deliverables status

• Missing, but quickly achievable (one or two days, during daytime,in parking position) once we answer few questions above.

Actions

• 03/07/2019 Ask for discussion with expert.

Data taking plan

7. Data taking with HV=0.

Deliverables description

• Data taking with a pedestal trigger of a fixed frequency and writing data to the disk. Vary input frequency, change from fixed frequency to Poissonian regime. Plot: Data writing rate vs. Input frequency ( a) fixed or b) Poisson).

Comments

• Some plots are there. Still stability is poor enough as to prevent stating the step is fulfilled. Instead we can say that can be performed.

Deliverables status

• With EVB2: Run 381 with Poissoniasn TP random trigger, whereas 382 with constant frequency.

• With EBV3,from ELOG entry 305. Still transitioning from Osaka to Okinawa. Using Poissonian random trigger. Anyway one plot is missing.

• With EBV3,from ELOG entry 323, running in Osaka with gain selection.Using Poissonian random trigger. Anyway one plot is missing.

Actions

• 03/07/2019 Ask for remaining plots and run conditions.

8. Pedestal with HV.

Deliverables description

• Pedestal runs in different configurations: HV off shutter closed, HV on shutter closed, HV on shutter open. Plot Ped RMS each pixel in a histogram. Plot Ped RMS per pixel vs. time. Plot: Mean Ped Camera vs Time.

Comments

• Again I think all this has been already done, but information is scattered around, so it is difficult to retrieve. Therefore we should have dedicated time to do it in a single run, involving analyzers to get the result.
• I think the pedestals are still unstable, but I should confirm.
• Some runs already taken (see ELOG entry 314)

Deliverables status

Actions

• 03/07/2019 Confirm stability of pedestals

9. Calibration Box, charge calibration.

Deliverables description

• a. Use calibration pulses to make HV flatfielding of the camera, i.e. equalize extracted charge in the readout for all pixels by adjusting the HV of the pixels. Plot: Extracted charge histogram before and after the flatfielding procedure. RMS after the flatfielding should be below few %. Repeat for different intensities of the calibration pulses.
• b. Use calibration pulses to make after HV flatfielding to convert ADC counts to phe. Use two methods: F-factor method and known gain factor method for every PMT. Plot: compare the two methods, the difference must be below 10%.

Comments

• Plenty of runs available and some nice analysis

Deliverables status

• From franca. Not clear which calibration method has been used

• Missing plot comparing the methods.

Actions

• 03/07/2019

10. Calibration Box, timing calibration.

Deliverables description

• Use calibration pulses to calibrate delays of the trigger signal propagation from the modules to the central backplane and back. Plot: Arrival time (backplane and in the Domino Readout) of pulses before the calibration and after the calibration. After the calibration, the RMS must be below 1ns.

Comments

• The RMS must be calculated in a event by event basis, because there is quite some jitter in the camera trigger.
• I think a big part of the information is somewhere, but it is not finished

Deliverables status

• From Taka: spread of the trigger down (triggering only in the central BP with Flash). It shows the pulse position in the ROI. It miss compensation of the FPGAs skews, and solving the instability. I do not know if DRS timing is corrected. Without outliers it is close to the goal.

• PLot of one of the runs, showing that, removing the outliers, it is possible to get an RMS smaller than 1
• Same plot, but substracting the pulse position from other run, taken as an offset due to what is missing in the calibration (skew from FPAG to FPGA, for example). It shows the best that can be done (outliers are outside the plot)

Actions

• 03/07/2019

11. Calibration box, rate scans.

Deliverables description

• Use calibration box with flatfielded HV to make rate scans with L0 and L1.
  • a. For L0, calibrate attenuation of the signal to equalize the amplitudes. Plot: L0 scan all pixels with a given calibration pulse amplitude before L0 amplitude attenuation and after. Plot: Histogram of L0 amplitudes after attenuation calibration. RMS must be better than 2%. Repeat for a different amplitude of the calibration pulses.
  • b. L1 rate scans for a give amplitude of the calibration pulses. Plot: Overlay of rate scans for different modules. Repeat for different L1 trigger modes. Calibrate L1 thresholds (defined as L1 DT at which the rate is half of the input pulse rate). Use different intensities of calibration pulses to check L1 thresholds are linear with light intensity. Plot: DT per pixel as function of light intensity, check linearity (must be better than 2%)

Comments

• This requires calibrate the L0
• For the 1L1 it requires the L0 calibration to include the L0 delay lines
• It is not clear to me what does it mean DT per pixel for L1

Deliverables status

Actions

• 03/07/2019

12. NSB rate scans

Deliverables description

• Point the telescope to a dark sky and take pedestal runs and L1 rate scans at dark sky. Define L1 suitable for this NSB level (allow 10% NSB triggers, rest must be showers). Plot trigger rates L1 and Camera vs DT levels.

Available data

• First one I am aware is ELOG entry 145 with nominal HV (whatever that means at the moment March 13th 2019). It is not clear to me where was it pointing to.
  • Data in [8]
• On top of that it seems that we have dedicated runs, but how the DT was selected is unknown to me:
  • With nominal HV, adder mode 3, tracking files are declared in next directories, but I do no know where:
    • /var/log/CTA/Caco/calibrations_results/ScanL1_2019-05-06-01-15-13_DAC1_tracking.result
    • /var/log/CTA/Caco/calibrations_results/ScanL1_2019-05-06-01-15-13_DAC2_tracking.result

Comments

• Which L0 settings to use? Which trigger mode? What is camera trigger vs DT level (the DT is module wise)?
• Detailed study is needed to get the DT level accordingly to the deliverable definition. So the observation should be reproducible.

Deliverables status

• Nothing yet.

Actions

• 03/07/2019

13. Stability and Repeatability

Deliverables description

• Repeat 9, 10, 11 and 12 at different days, different ambient temperatures, different NSB levels (12 only). Plot quantities of the tests from 9, 10 and 11 as a function of time.

Available data

Comments

• Need to create a routine.
• We need to define what is several days, and how to track the ambient temperature in data.
• Nevertheless the system is still quite unstable

Deliverables status

Actions

• 03/07/2019

14. Take pedestal runs.

Deliverables description

• Confirm pedestal RMS is stable within 2% with a constant DC level. Plot Pedestal RMS (mean camera) vs time

Available data

Comments

Deliverables status

Actions

• 03/07/2019

15. Take (Charge) Calibration runs.

Deliverables description

• Confirm calibration constants are stable (variation less than 2%) vs time. Plot calibration constants vs time

Comments

• Need to get to sufficiently stable operation (maybe is it already the case)

Deliverables status

Actions

• 03/07/2019

16. NSB rate scans at different NSB levels

Deliverables description

• Repeat Tests and Plots of Nr. 12 at different levels of NSB. Confirm the telescope is operational at 10 x dark NSB

Comments

• Has quite some overlap with 13. Stability and Repeatability
• The only difference is the requirement of 10 x dark NSB (how do we get to this? Monitoring DC?)
• Plots not define, so we should move either 13 to here or this to 13.

Deliverables status

Actions

• 03/07/2019

17. Take pedestal runs at different Zd/Az.

Deliverables description

• a. Confirm the pedestal RMS does not depend on Zd/Az. Plot pedestal RMS as a function of Zd and Az.
• b. Confirm relation between pedestal RMS and DC levels. Plot Pedestal RMS (mean camera) vs Mean DC. Confirm the camera is operational with 10 x dark NSB

Comments

Deliverables status

Actions

• 03/07/2019

18. Take (Charge) Calibration runs at different Zd/Az angles and NSB levels.

Deliverables description

• a. Confirm calibration constants are stable (variation less than 2%) vs time and independent of Zd/Az angle. Plot calibration constants vs time.
• b. Confirm there is no dependence on the NSB level. Confirm the camera is operational with 10 x dark NSB.

Comments

Deliverables status

Actions

• 03/07/2019

19. Take data runs at different Zd/Az.

Deliverables description

• Take data runs with flatfielded HV and adjusted trigger settings. Confirm data rates are stable as a function of time and rate has a cos^a relation with the zenith angle. Plot: Rate vs time vs Zd angle. Confirm there is no dependence on the azimuth angle. Plot image parameters (Size, Length, Width) as a function of time, confirm there is no dependence on time

Comments

Deliverables status

Actions

• 03/07/2019

20. Take data runs at different NSB levels

Deliverables description

• 'Take data runs with flatfielded HV and adjusted trigger settings at different NSB levels. Confirm data rates are stable as a function of time and rate has a cos^a relation with the zenith angle for a given NSB level. Plot: Rate vs time vs NSB level. Plot image parameters (Size, Length, Width) as a function of time, confirm there is no dependence on time.

Comments

Deliverables status

Actions

• 03/07/2019