Logbook Camera-Works-Mirca-August-September2018

From my_wiki
Revision as of 22:27, 18 September 2018 by Tysaito (talk | contribs) (→‎Enter comments in reverse time order)
Jump to: navigation, search

Enter comments in reverse time order

2018-09-18 || Taka, Daniel K.|| Powering up test|| Very often one or two modules are not booted if we power up in the standard way with ECC. But if we manually activate the relays, all modules work perfectly. We found that the difference is the timing of the relay activation. If we activate relays 1 min after making ECC state to "Ready", then no modules has a problem. waiting for 30 sec is not enough. It seems this 1 min is related to the startup of Data switches. It takes one minutes to be standby and probably lots of current is consumed during this 1 min? This one min waiting scheme should be implemented in the standard ECC powering up.
Date Who did it? Action Comments Documents
2018-09-18 Taka, Daniel K. Mezzanine test

We made a rate scan. We found that the mezzanine of the same Module (102 in ClusCo) showed the same problem as the one set before. Rate drops earlier in threshold than others. But test pulse amplitude shows no problem in DRS4 data. There may be a problem in Dragon connector, or trigger branch. Also, during the TIB test, we found that one mezzanine creates a crazy L1 rate (too high). It was working with no problem by today. We exchange mezzanines of this module. By now, we have 4 modules with not-perfectly healthy rate scan results in the camera.

2018-09-17 Taka, Daniel K. Mezzanine test

One mezzanine, which worked fine last week, does not work at all today (no L0, no L1). There are 6 more non-perfect mezzanines. We replaced 4 of them + non working one with spares.

2018-09-17 Taka, Daniel K. DAQ test

No problem in DAQ with "hoffmann" account. But there is a problem with "dragon" account. The reason is being investigated by Julien.

2018-09-17 Taka, Daniel K. Powering up test

We always have 1 or 2 modules that are not booted (we miss connection to them). This doesn't happen if we activate the relays manually by direct serial connection. It happens only if relays are activated by ECC.

2018-09-17 Taka, Daniel Camera Network Configuration

In order to complete the camera network, we had to collect information by asking relevant people by email. It seems one or two hardware components are missing for WR connection. Anyway, we give up. No WR connection in Mirca. Expert should make it work in ORM.

2018-09-14 Taka, Seiya Investigation of undershoot problem

Now there is the LED laser used at IFAE in June here. So after we closed the shutter and turned off the light, we applied HV(1000V) at all pixels and injected the LED laser pulse from the a bit small gap of the shutter. LED setting is;

  • bias voltage : 16.5mV
  • LED num : 1

We took data with this setup and L1 local trigger, and it didn't seem that there are any undershoot.

Media:DSC_0195.JPG Media:DSC_0198.JPG Media:Ch0Ev06.gif
2018-09-14 Daniel.K, Taka, Seiya Problem of trigger distribution

We could not get camera trigger at every module by setting TIB bypass at only central BP by now. We did some tests with help of Gustavo and Carlos with 7 modules. Camera trigger was distributed at central one and upper 3 modules by setting TIB bypass at only central BP. These 7 modules are central-BP type,that is they have other two connectors for communication with TIB. We found that the cables for communication with TIB were connected at not only central BP but also lower 3 BPs, at which camera trigger was not distributed. So we unplugged these cables from non-central BPs(lower 3 BPs), then camera trigger was distributed at every module. The cables may be acting as antennas and bringing some noise into the backplanes

Media:TIBbypass_20180914.JPG
2018-09-14 Daniel.K, Taka, Seiya Problem of power on procedure

After power on from ECC(to Ready state), we could not communicate with BP about 1 module(IP10.1.6.12), but we could ping this module and communicate with Dragon. After that we powered off and on remotely, we could not ping one module(IP10.1.5.16) and it didn't recovered by 3 trials of power off and on. This problem happened yesterday and this was the same one. By powering up with the cable manually we could communicate with all modules.

2018-09-13 Daniel.K, Taka, Seiya TIB and UCTS problem

We did some tests following Otger's suggestion and the procedure of tests is below; (10.1.4.64 = default IP of TIB)

  • going to safe and going to ready
  • ping 10.1.4.64: nothing
  • plugging cable from caco (em2) to slow control switch
  • ping 10.1.4.64: nothing
  • removing cable from caco (em2) to slow control switch and plugging cable from caco (em2) to TIB main ethrnet port (the one going to the slow control switch)
  • ping 10.1.4.64: WE SEE IT
  • going back in the previous configuration
  • ping 10.1.4.64: nothing
  • plugging a new clable from TIB to Slow control switch (byassing the actual one)
  • ping 10.1.4.64: WE SEE IT
  • replacing the old cable from the TIB to the slow control switch (so exact same configuration as on the line 3)
  • ping 10.1.4.64: WE SEE IT
  • removing the additionnal cable from caco (em2) to Slow control switch
  • ping 10.1.4.64: WE SEE IT

We got clues more or less, but it is still not understood.

2018-09-12 Daniel.K, Taka, Seiya TIB and UCTS problem

TIB and UCTS doesn't work well now because of network problem and we cannot even ping them(TIB:10.1.4.64(default), UCTS:10.1.8.4) from osaka and CaCo server. TIB and UCTS need dhcp server and should get IP address from it. In the same network there is movistar router(dhcp server) and this setup may cause some "confuses". Now we are investigating this problem with help of Luis and Otger.

2018-09-10 Taka, Seiya Module tests in camera

We investigated the problem of camera trigger distribution with remote help of Carlos.D and Dirk. At first Carlos configured the modules with his own scripts and it worked well. Next we tried to configure them with ClusCo as like yesterday we did, but it didn't work. As a result we didn't know why camera trigger was not distributed, but we found a temporary solution. When we set "TIB bypass" with not only central BP but also all BPs, camera trigger was distributed and we took data with camera trigger and 265 modules. One possible but unlikely explanation is that a BP is configured to act as central in addition to the correct one, and hence this solution helped camera trigger distribution.

2018-09-07 Taka, Yuji, Seiya, Shunsuke, Daniel.K Module tests in camera

We installed one module in the camera and it worked well about DragonQC. So now we have 265 good modules in the camera and 2 spare modules from CIEMAT.

We had a plan to do some tests with EVB, but we didn't do it because of the problem about trigger distribution. The configuration we wanted to prepare was (1) inject test pulse at the central module (2) camera trigger was distributed from central BP to all modules. At first we wanted to configure the TIB, but it didn't work. Next we tried to use TIB bypass at the central backplane. We used almost all the same configuration files as one used at IFAE(init6.uic, pulse_injection_central.uic, ByPassTIB.uic) and L1 local trigger was generated at central one, but all modules didn't seem to receive camera trigger, even not the central Dragon. We also didn't take data with 7 modules setup, but we could take data with single module setup.

2018-09-06 Taka, Yuji, Seiya, Shunsuke, Daniel.K Module tests in camera

We put 9 modules of which the functionality test results were good in the camera. After that we did DragonQC again with all modules in the camera. About only one module DragonQC didn't finish because of communication error with SCB(We could inject test pulse and change pulse height, but could not read fact code and temparature etc..). After we took out this module, we assembled different SCB with the same Dragon and communicated with it, SCB had some problem.

While only this module failed QC, some modules were also seen some error at analysis because some value exceeded the limit we set at some measurement. However this was not a big problem, so we decided to use these ones.

We installed a spare module instead of bad module and tried to do DragonQC in the camera. QC result was bad, the value about EEPROM was strange. So we have to install another module there.

2018-09-05 Taka, Yuji, Seiya, Shunsuke, Daniel.K Module functionality test
  • DragonQC

We did DragonQC with 13 modules outside the camera. 10 modules out of them were good. The results are below,

ID / IP / QC result

1-097 -- 10.1.6.13 -- good

1-051 -- 10.1.5.43 -- bad(the behavior of the 3 DRS4 chip was strange)

1-038 -- 10.1.5.17 -- good

1-053 -- 10.1.5.150 -- good

2-123 -- 10.1.6.7 -- good

1-098 -- 10.1.6.43 -- good

1-001 -- 10.1.6.28 -- good(noise dist. is a bit wider at ch3, but it is not a big problem)

1-136 -- 10.1.5.157 -- bad(bad linearity, but the different Dragon with this PMT head also had the same problem, so this problem was caused by SCB or PACTA)

2-140 -- 10.1.6.26 -- good

2-054 -- 10.1.5.55 -- good

2-039 -- 10.1.6.27 -- bad(The right value was written at EEPROM(register xFFFFFC**), but the register of FPGA about EEPROM(register xFFFFFF**) was strange)

1-055 -- 10.1.5.4 -- good

2-021 -- 10.1.5.56 -- good

  • Laser pulse with HV

After DragonQC we took some data with PMT and HV for functionality check. At first we measured about one photoelectron with 1400V and estimated filter wheel configuration for what we wanted to see with nominal HV. Then we took data with nominal HV and the results were good at all 10 modules.

Media:DSC_1562.JPG Media:DSC_1564.JPG Media:Linearity_0_0.pdf Media:AveragePulse_0_0.pdf
2018-09-04 Taka, Yuji, Seiya, Shunsuke, Daniel.K Prepare for module functionality test

We decided to do functionality test for new modules, Dragon QC and check of pulse shape with HV and flasher in the dark box. We replaced the diode laser with new one and checked if we could see the pulse with HV and it worked well. Moreover we prepared the connection between Raspberry Pi and filter wheel, so we can change filter combination remotely. We have a plan to do these test tomorrow and put them in the camera.

2018-09-04 Taka, Yuji, Seiya, Shunsuke, Daniel.K Replace Dragons and PMTs with new one
  • We replaced bad DragonQC board (5 modules) and the Al flame broken module (1 module) with new Dragon. Spare Dragon is out of stock now, so Seiya will being new spare Dragon on October.
  • We replaced short PMT(not electrical short) with new one. About problematic PMT on which HV was not applied at IFAE, we could apply HV with this PMT here. So we didn't replace it, but we exchanged PMT head(from S.0172 to S.0068) just in case and this PMT head stayed without Dragon.
2018-09-03 Taka, Yuji, Seiya, Shunsuke, Daniel.K Investigation of bad Dragon QC modules We summarized the results of DragonQC. Some problem are not still fixed about 5 modules(ID 1-137(noisy), 2-90(bad linearity), 1-45, 1-63(bad cell linearity), 2-45(data not reached)). We may exchange these boards with new Dragons. Media:20180903_DragonQC.pdf
2018-09-03 Taka, Yuji, Seiya, Shunsuke, Daniel.K Take out and in modules

We took out 3 modules and in 2 modules(See below), so 9 modules are outside the camera now.

  • guide pin missing modules(3modules)

We took 2 modules(ID 2-72, 2-117) in the camera, which are missing guide pin. However guide pin was not fixed about one module, so we put more araldite and this module still stay outside the camera now.

  • bad Dragon QC(5modules)

We took out one more module(ID 1-63) from the camera. Cell linearity was bad about this module. We are investigating these 5 modules with single modules setup.

  • broken PMT modules(2 modules)

One(ID 2-140) cannot apply HV to only one pixel and another module(ID 1-98) has a short PMT. We will exchange this one with new PMT and these 2 modules are still outside the camera now.

  • Broken Al flame(ID 2-134)

This module are still outside the camera.

2018-08-31 Yuji, Seiya, Shunsuke Take out modules We took out 6 modules today, 3 modules for putting a guide pin on an alminium plate and 3 modules for the investigation of bad QC modules). There are 257 modules in the camera and the following 8 modules are out of the camera.
  • ID, IP, position, reason
  • 045_02 -- 10.1.6.13 (2,4) -- bad DragonQC
  • 134_02 -- 10.1.5.17 (8,16) -- Al flame was broken
  • 072_02 -- 10.1.5.158 (6,15) -- missing guide pin
  • 117_02 -- 10.1.6.25 (3,5) -- missing guide pin
  • 123_02 -- 10.1.6.7 (1,6) -- missing guide pin
  • 1-137 -- 10.1.5.150 (6,7) -- noisy
  • 090_02 -- 10.1.5.157 (6,14) -- bad linearity
  • 1-45 -- 10.1.5.43 (10,14) -- bad cell linearity
2018-08-30 Yuji, Seiya, Shunsuke Bad Dragon study About one module(ID 2-45), We didn't take any data in the camera, so we took out this module from camera and studied with single modules setup. We read some registers about FIFO memory,then It seemed data wasn't sent to PC and FIFO buffer went to be full and it became busy state, not socket connection lost. We may need exchange this Dragon with new one.
2018-08-30 Yuji, Seiya, Shunsuke DragonQC We did Dragon QC column by column yesterday concerning about column 6,8,9,10,11, in which there are more bad modules then others. Today we checked these all summary plots.
  • High cross talk problem disappeared. About 2 modules(ID 1-63,1-72) cross talk was a bit high, but it seemed this result was not a big problem.
  • Almost all of modules we used yesterday were shown broad noise distribution.
  • About one modules(ID 1-45) it showed bad cell linearity. This result was shown for the third time with the same module, This Dragon may have some problem.
 Media:noise1_20180830.JPG Media:noise2_20180830.JPG Media:celllin_20180830.JPG
2018-08-30 Yuji, Seiya, Shunsuke Dark box We finished assembling a dark box. So we can now apply HV with modules in the dark box. Media:DSC_1533.JPG Media:DSC_1534.JPG Media:DSC_1535.JPG
2018-08-30 Gustavo, Carlos Counters check We checked the PPS, clock cycle and trigger number in the data stream using the legacy DAQ, in order to check signal distribution from BP to Dragon. No problem found, and all modules are properly synchronized. The only remark that has to be done is that in about 1% of Dragons, 10 MHz clock cycle counter was not updated in a first run while the back plane was counting it, and was solved after resetting all modules.
2018-08-30 Gustavo, Carlos , Seiya, Shunsuke, Yuji Visual inspection of modules We performed a visual inspection of modules failing n the connectivity test. We found one mezzanine with a problem in the polarization of L1 input. Another two modules were inspected and are kept out of the camera because one of the does not fully pass the Dragon QC, and the other has a faulty mechanical module assembly, to be fixed. Media:DSC_1527.JPG
2018-08-29 Gustavo, Carlos, Seiya, Shunsuke, Yusuke, Yuji Data taking with HV for investigation of "undershoot" problem After we covered the frontal part of the camera by white board and turn off room light, we applied 1000V with only 1 module(all pixel) and took some data. The light source we used was LED which Gustavo and Carlos bought at the shop yesterday and we used a pulse generated circuit developed by Yusuke. We set L1 local trigger synchronized with internal clock(133 MHz) and adjusted trigger delay in Dragon FPGA, so we can see pulse in ROI. The pulse width was a bit wide(~25ns), but we could not see "undershoot" for quick check. Now analysis is on going. Media:DSC_1517.JPG Media:mod101_HV1000_LED.JPG
2018-08-29 Gustavo, Carlos BP Clock phase calibration Using the test pulse and CIEMAT software, we manage to measure the latency of trigger distribution for all the camera with setting calibrated at IFAE. We find that the latencies are uniform except for some modules due to differences in the distribution routes between IFAE and Mirca camera setups. Making use of this we were able to adjust the clock phases in the whole camera. The issue seen at IFAE with the clock phase calibration is understood and solved. Current setting can be used as default for data taking, keeping in mind that they are wrong for few modules, until we perform a new calibration a ORM with the calibration box. We will produce configuration files with these settings for ClusCo.
2018-08-29 Yusuke, Yuji Assembly of new module We assembled new 3 modules by using some parts which Shunsuke and Seiya brought from Japan.
2018-08-28 Seiya, Shunsuke, Yusuke, Yuji Dragon QC We summarized Dragon QC results. Concerning about some "bad QC" modules, we did DtragonQC again. We could not even take any data with only one module(IP10.1.6.13), it seemed socket connection was lost again and again. Media:DragonQC20180828.pdf
2018-08-28 Seiya, Yusuke, Yuji Changing Bad dragon We replaced the dragon board of bad modules (1-58, 2-71, 2-52). and We re-assembled modules which were arrived from CIEMAT.
2018-08-28 Yusuke, Yuji Building the dark box The wooden plate was painted with a black. Current status: 95% done.
2018-08-28 Gustavo, Carlos Connectivity test Connectivity test is finished. Almost all connections are ok, except for the ones that cannot be tested due to mezzanine issues. Three mezzanines don't produce L1. One module keep the L1 always high with our settings (IP 10.1.5.19). Another module has strange behaviour for one L0 neigbour (to be checked in detail, IP 10.1.5.174 or neigbour)
2018-08-27 Seiya,Shunsuke Data taking for investigating an undershoot problem We took test pulse data with internal clock, test pulse trigger for investigating an undershoot problem. For quick check, we could not see undershoot. We are going to analyze it more deeply tomorrow.
2018-08-27 Seiya,Shunsuke Dragon QC data check We checked the all data of Dragon QC. We found some problems. Some modules did not finish QC procedures, it caused from a threshold in the script. And it seems that cross talk result is strange. We are going to investigate the reason of that later.
2018-08-27 Yusuke,Yuji Building the dark box Current status: 70% done.
2018-08-27 Gustavo, Carlos Connectivity test We run the ip search, neighbours search and l0 connection test. We found all the ips, and localize the position of all modules. In addition we find three of the four l1 faulty mezzanines. We need to understand some features of two modules regarding the connectivity test results, and what is the fault mode of one of the mezzanines with failing l1. The rest of connections tested so far are ok. A document is coming soon.
2018-08-23 Riccardo, Carlos, Miguel, Tony, Daniel K, Shunsuke, Yuji Module Installation We installed 65 modules (from column 6 half to column 1) to the camera. In total 265 modules were installed. We found about 15 mezzanine problems and 12 Light Guide problems. Light Guides were fixed by Yuji, then those 27 modules were installed. We have to pull out modules to fix the problem later. We finished installing modules for the next test. We succeed to power on the full camera at once, and to ping to all modules and read out BP firmware and temperature from all modules. For the optical fiber problem, Osaka can not communicate with all modules. then Dragon QC was not performed yet. https://drive.google.com/open?id=1DdRUGOgVrMz2Pivtg7P4Zre2Va-aYO1C
2018-08-22 Riccardo, Daniel K, Shunsuke, Yuji Module Installation We installed 94 modules (from column 12 to column 6 half) to the camera. In total 200 modules were installed. For 35 modules (from column 9 to column 10), we tried performed sanity and connection test. We were able to ping all modules, then we could read firmware and tempaature for all modules. About Dragon QC, for the fiber problem, we could communicate with only 7 modules. Results were fine. https://drive.google.com/open?id=1DdRUGOgVrMz2Pivtg7P4Zre2Va-aYO1C
2018-08-21 Riccardo, Carlos, Miguel, Tony, Daniel K, Shunsuke, Yusuke, Yuji Module Installation We installed 98 modules (from column 18 to column 12) to the camera. In total 106 modules were installed. For 89 modules (from column 19 to column 13), we tried to perform sanity and connection test. We were able to ping all modules, then we could read firmware and tempaature for all modules. At first look, L0 and L1 scans went fine, but deeper investigation needed. About Dragon QC, results were fine. However only for 2 modules, Dragon QC has stopped in the middle. for column12, we will perform the test tomorrow. We also found 2 data optical fibers through which we could not communicate with the modules, we used the one spare optical fiber we have and we replaced the second one with the optical fiber for the UCTS. https://drive.google.com/open?id=1DdRUGOgVrMz2Pivtg7P4Zre2Va-aYO1C
2018-08-20 Shunsuke Brief check of Mezzanine checked sanity of the Mezzanine for 3 rack = 57 modules and more. I found 9 more mezzanine missing parts. In 9 mezzanines, some strange mezzanine found. It was L0 ASIC not covered by material, missing parts and fixed without parts? And a small parts was soldered with strange angle. See pictures. https://drive.google.com/open?id=1G_3LFysZG-ia6awGCnLEbGedMsC1UVvI
2018-08-20 Carlos, Daniel K, Shunsuke Test of the traveling modules 1-58 and C008.140_02 were transported from IFAE in the camera. So we checked the effect of travel. We did L0 rate scan, L1 rate scan with test pulse and Backplane resistor reading. Daniel and Shunsuke confirmed a consistency of result of the test to one took in IFAE. These were fine. But one of the mezzanine's pin was bend. It means the bend pin does not effect with communication. https://drive.google.com/open?id=1jBe10SAx17wu7F7rRw-qSGyjg6ZwLde7
2018-08-20 Riccardo, Carlos, Miguel, Tony, Daniel, Shunsuke, Yusuke, Yuji Module Installation for Column 19th We started install modules to camera, with confirming procedures one by one. Procedures start from powering on generator, power supply, camera, and modules proceed to preparing of PMT modules. Then we discussed about procedures, we check again tomorrow. Today we finished to install 1st column (col19th), without some tests for checking communications. https://drive.google.com/open?id=1DdRUGOgVrMz2Pivtg7P4Zre2Va-aYO1C
2018-08-20 Yusuke, Yuji LG assembly Assembled 59 modules. In total 268 modules were assembled. last 2 modules will not be installed to camera. So these were not assembled. Basically, this work has done.
2018-08-17 Daniel K, Shunsuke Dragon QC for checking Dragon For checking a sanity of Dragon board, I tried to perform Dragon QC by my PC. But it was failure. So I connected Dragon (2-128) to Osaka directly (through em3), execute QCDaq.sh. It was performed fine. It has not analysed yet. https://drive.google.com/open?id=1qZn6j4hAUSn2enz7DvvBQ3rBdhGzwVOd
2018-08-17 Carlos, Miguel, Tony, Daniel K, Shunsuke Camera opening Carlos, Miguel and Tony were arrived. We have cleaned up around the camera. After that, they removed materials from a camera transport structure. We can access the both side of the camera now. https://drive.google.com/open?id=1jBe10SAx17wu7F7rRw-qSGyjg6ZwLde7
2018-08-17 Toko, Daniel K, Yusuke, Shunsuke, Yuji Assembly LG to PMT module assembled 76 modules. 2 or 3 mezzanines had a problem in each racks, and we found that missing a capacitor and an inductor on Mezzanine. This may be a pretty problem. Those parts should be soldered. In total 209 modules were assembled. And Al plates for transportation were all removed from modules. So we can attach a new LG to the module immediately. https://drive.google.com/open?id=1nQkuRGRz-w8eayIwtCO98NLbOyxmQN4Y
2018-08-16 Toko, Daniel K, Yusuke, Shunsuke, Yuji Assembly LG to PMT module assembled 76 modules. This corresponds to four module racks. 2 or 3 mezzanines had a problem in each racks, these problems have fixed by Daniel. In total 133 modules were assembled.
2018-08-15 Toko, Daniel, Yusuke, Shunsuke, Yuji Assembly LG to PMT module assembled 52 modules. 11 mezzanines has pins problems. After Daniel was arrived in Mirca, these problems have fixed by him. The modules were covered by plastic bag. The pictures are in the following web. https://drive.google.com/open?id=1nNdVpm95B9fIDlFu7MplnQDmV4I2afrc
2018-08-14 Toko, Yusuke, Shunsuke, Yuji Assembly LG to PMT module assembled new LG for 5 modules. during that works, we found 2 problematic things. At first, scree for fixing modules to module molder was fall out. (see the link.) and another, broken connectors of the mezzanine. some pins bend like pictures. we checked about 20 modules, problems were found in 5 modules. https://drive.google.com/open?id=1nNdVpm95B9fIDlFu7MplnQDmV4I2afrc
2018-08-14 Toko, Yusuke, Shunsuke, Yuji Picking up Mr. Yamamoto also went to buy a staff for LG assembly. Alcohol for wiping cathode plane.
2018-08-14 Yusuke, Shunsuke, Yuji Preparation for Light Guide assembly Opened transportation boxes of LG, and moved them in the clean room.
2018-08-14 Yusuke, Shunsuke, Yuji Building a Big Black Box Assembled side panels to flames of the dark box, 50 % panels were assembled until now.
2018-08-13 Yusuke, Shunsuke, Yuji Shopping staffs for Light Guide assembly went to buy the materials for a clean room, It will be used in LG assembly. Staffs are, e.g., table, chairs, and clean suits.
2018-08-13 Yusuke, Shunsuke, Yuji Building a Big Black Box Assembled the black box which used in IAC for module QC. We finished almost whole part of frames, but we need additional screws for it.
2018-08-13 Yusuke, Shunsuke, Yuji Unpacking of transportation boxes for PMT modules PMT modules have delivered to Mirca by wooden boxes, and it was fixed by screws. We finished dismounting these screws from boxes. we can use all modules without another preparation.
2018-08-08 Patricia Marquez Camera, chiller and the modules arrived in Mirca
Retrieved from "https://www.lst1.iac.es/wiki/index.php?title=Logbook_Camera-Works-Mirca-August-September2018&oldid=1087"