User:Cdelgado

Camera open points

Not always all modules power up in the first trial (especially 6.28)

Removing it from EVB seems problematic because data format is not prepared (not uniform with the case with all modules)

BP trigger UP delay is checked with TP, but it is unstable. And PPS synchronization fails in module 114 sometimes.

Trigger up maybe not very important, as far as the error keeps pulses inside the ROI

What is the impact of PPS synchronization?

Trigger Down is unstable: settings seem to fail from reinitialization to reinitialization

L1 thresholds are not optimized. We should either implement Module rate control or prepare Threshold Table for various target total rate.

In addition L1 dependes on L0 settings and L1 trigger mode!

TIB stops sending triggers after some observations. The reason is unclear.

CaCo is not fully ready

Currently we are having problems either in CaCo or ClusCo (ELOG entries 307 - 311)

EVBv3 is not fully ready.

Dragon problem: with random trigger rate higher than 15 kHz, One of the Dragons gets stuck within ~2 min. The buffer in Dragon is filled up (we can check it by slow control) and send busy.

lst-chain seems not to be compatible with EVBv3 data. Quick analysis is impossible.

Operation

Top 10 of "CAM startup failures" since June 21st

1. ECC/CAM power cycle (error state) 1
2. Module 6.28 down (power cycle relay 1) 14
3. Module 5.16 down (power cycle relay 2) 1
4. SIS "undefined" 1
5. network interface DOWN p1p1+p3p1 (osaka) 1
6. network interface DOWN p2p2 (osaka) 3
7. network interface DOWN p3p2 (osaka) 2
8. EVB(2) crashed in GOTOOBSERVING 1
9. TIB error state 255 3
10. EVB(2) only connecting to one group/switch (probably IF DOWN) 1

• From this table, out main problem is the Module 6.28 power up problem.
• I would say the second one is network interface going down
• Third one is TIB error state

Camera commissioning plan status

This page provides a summary of the status of the deliverables commissioning plan of the camera, with actions , for each point. What remains to be clarified is if providing this set of deliverables really guarantees the system is commissioned. My impression is that not necessarily.

1. Mechanics

Deliverables description

• Mechanics. Opening and closing front door (commissioning of the hydraulic system). Opening and closing backdoor. Moving the Star Imaging Screen (SIS). Movement and visibility from dish center of the SIS.

Comments

• It is not clear what has to be delivered and what is really needed to consider the system commissioned. All the declared deliverables have been performed and there are documents proving it pictures and movies in some cases).

Deliverables status

Actions

• 03/07/2019 Ask experts to decide what is required

2. Thermalization

Deliverables description

• a. Tests without Drive: Plot Temperature vs time of various T sensors, auxiliary and internal module sensors. T must stabilize within 2 hours and T of the modules must be within the operating range of the electronics with at least 10 degrees margin. Minimum run duration is 10 hours without any interruption.
• b. Tests with Drive: Plot of the water pressure vs zenith/azimuth angle.

Comments

Deliverables status

• For Deliverable a, plots exist when possible (there are no module sensor, but air sensor for modules channels), although limited to up to 8 hours to C. Delgado knowledge, and shows dependence due to direct sun light reaching to one side of the camera.

• Deliverable b does not make a lot of sense, as the important quantity are the temperature as a function of the zenith/azimuth angle. Information seems available.

Actions

• 03/07/2019 Ask Daniel to modify deliverable b to temperature vs t vs zenith
• 03/07/2019 Prepare a long run with telescope movement to provide all plots

3. Connectivity: DONE!

Deliverables description

• a. All modules (means Dragon and Backplane) must respond to their foreseen IP and must be able to start up (boot) at the start up sequence. Plot: T for backplane and Humidity in camera coordinates for every Dragon and corresponding backplane.
• b. Backplanes must see each other (L1, Clock, PPS distribution) and L0 distribution. Plot:

reconstructed network topology

Comments

• This is done, but deliverable b does not prove all the signals, but only L1. For L0 there are dedicated tests, that failed for few modules.

Deliverables status

• All done from monitoring and dedicated test before changing module numbering to the current one.

Actions

• 03/07/2019 Improve plots. Add dates.

4. State Machine

Deliverables description

• Software state machine must be verified: Safe -> StandBy -> Ready -> StandBy -> Safe. Plot: State vs Time to measure transition times between states.

Comments

• Still pretty unstable. From ELOG entry 304:
  • Not always all modules power up in the first trial (especially 6.28)
  • BP trigger UP delay is not checked yet. And PPS synchronization fails in module 114 sometimes.
  • L1 thresholds are not optimized. We should either implement Module rate control or prepare Threshold Table for various target total rate.
  • TIB stops sending triggers after some observations. The reason is unclear.
  • CaCo is not fully ready EVBv3 is not fully ready.
  • lst-chain seems not to be compatible with EVBv3 data. Quick analysis is impossible.

Deliverables status

• Not ready to do it

Actions

• 03/07/2019 Follow up.

5. Trigger Timing with Test Pulse. DONE! but quality of plots could be improved or uniformized

Deliverables description

• Using test pulses, the delays must be set such that a trigger issued by the Trigger Interface Board or Local Module Trigger the pulses are well within the Readout Window of 40 cells. Plot: 2D plot of camera coordinates and pulse position as z-axis. Plot: 1D overlay of all pulses in Region Of Interest.

Data available

• ​Run 00115 2019-02-19 21:14 for all modules triggering. Declared as showing pulses in all modules.
• Run 00232 2019-03-08 10:32 trigger only in central
• Run 00234 2019-03-08 11:10 trigger only in central
• Run 00416 ​2019-05-22 12:07 all modules triggering
• Run 00418 ​2019-05-22 12:24 trigger only in central
• Run 00535 ?????????? ????? ???????????????????????
• Runs 00632 to 00636

Comments

• Something like that already exist, but dependes of BP calibration, so should be posterior to it.
• Synchronization of all pulses should reached after BP calibration.

Deliverables status

• Plots can be improved, but we have them.

RUN 115 -- All modules triggering

RUN 418 -- trigger only in central

Run 535 -- conditions unknown

Actions

• 03/07/2019 Investigate and solve BP instabilities
• 03/07/2019 Maybe move this step further in the list.

6. L0/L1 rate scans with HV=0. FIRST THING TO BE DONE!

Deliverables description

• L0 rate scans with noise and with test pulses of a known amplitude. THIS SHOULD SAY L0 and L1 rate scans instead
  • a. Plot: Overlay of L0 rate scans with electronic noise only, with limits for the acceptable noise. Plot: Overlay of L0 rate scans with test pulse of a known amplitude, with limits for the acceptable discriminator level. Plot L0 threshold at 50% for each pulse amplitude.
  • b. L1 rate scans for every module for every adder individually with electronic noise only. Plot: Overlay of L1 rate scans with electronic noise only, with limits for the acceptable noise. L1 rate scans with a test pulse of a known amplitude. Plot: Overlay of L1 rate scans with test pulse of a known amplitude, with limits for the acceptable discriminator level. Plot L1 threshold of 50% rate for every pulse amplitude.
  • c. Repeat L1 scans and plots for different trigger modes (itself, 2 neighbours, 3 neighbours).

Data available

• I have plenty of files from ELOG and mal threads. However it is difficult to do anything with them because the conditions seems to be not fully documented.
• In addition, most of the files I have are only for L1. For L0 I almost have nothing.

Comments

• Probably something has been done, but plots and data are scattered around, so it is better to do it systematically.
• I do not know up to which point we know the pulse amplitude.
• What about settings of L0? These affect the L1 scan, so could require a calibration before making sense of L1 rate scans.
• What does it mean acceptable discriminator levels? It is just in terms of the rate itself or the meaning of the discriminator level in terms of p.e.?

Deliverables status

• Missing, but quickly achievable (one or two days, during daytime,in parking position) once we answer few questions above.

Actions

• 03/07/2019 Ask for discussion with experts:

7. Data taking with HV=0.

Deliverables description

• Data taking with a pedestal trigger of a fixed frequency and writing data to the disk. Vary input frequency, change from fixed frequency to Poissonian regime. Plot: Data writing rate vs. Input frequency ( a) fixed or b) Poisson).

Comments

• Some plots are there. Still stability is poor enough as to prevent stating the step is fulfilled. Instead we can say that can be performed.

Deliverables status

• With EVB2: Run 381 with Poissoniasn TP random trigger, whereas 382 with constant frequency.

• With EBV3,from ELOG entry 305. Still transitioning from Osaka to Okinawa. Not clear to me if it is constant frequency or Poissonian. Anyway one plot is missing.

Actions

• 03/07/2019 Ask for remaining plots and run conditions.

8. Pedestal with HV.

Deliverables description

• Pedestal runs in different configurations: HV off shutter closed, HV on shutter closed, HV on shutter open. Plot Ped RMS each pixel in a histogram. Plot Ped RMS per pixel vs. time. Plot: Mean Ped Camera vs Time.

Comments

• Again I think all this has been already done, but information is scattered around, so it is difficult to retrieve. Therefore we should have dedicated time to do it in a single run, involving analyzers to get the result.
• I think the pedestals are still unstable, but I should confirm.
• Some runs already taken (see ELOG entry 314)

Deliverables status

Actions

• 03/07/2019 Confirm stability of pedestals

9. Calibration Box, charge calibration.

Deliverables description

• a. Use calibration pulses to make HV flatfielding of the camera, i.e. equalize extracted charge in the readout for all pixels by adjusting the HV of the pixels. Plot: Extracted charge histogram before and after the flatfielding procedure. RMS after the flatfielding should be below few %. Repeat for different intensities of the calibration pulses.
• b. Use calibration pulses to make after HV flatfielding to convert ADC counts to phe. Use two methods: F-factor method and known gain factor method for every PMT. Plot: compare the two methods, the difference must be below 10%.

Comments

Deliverables status

Actions

• 03/07/2019

10. Calibration Box, timing calibration.

Deliverables description

Comments

Deliverables status

Actions

• 03/07/2019

11. Calibration box, rate scans.

Deliverables description

Comments

Deliverables status

Actions

• 03/07/2019

6.

Deliverables description

Comments

Deliverables status

Actions

• 03/07/2019

6.

Deliverables description

Comments

Deliverables status

Actions

• 03/07/2019