Logbook CameraCommissioning ORM Oct18

From my_wiki
Revision as of 15:34, 16 October 2018 by DirkHoffmann (talk | contribs) (→‎Enter comments in reverse time order)
Jump to: navigation, search

Enter comments in reverse time order

Glossary at the end!

Date Actor/Author Action summary Comments Documents
2018-10-15 Dirk UCTSd dead. 
● uctsd.service - Execute the UCTS OPC-UA server
   Loaded: loaded (/etc/systemd/system/uctsd.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Di 2018-10-16 13:22:28 WEST; 2h 59min ago
  Process: 152844 ExecStart=/home/dragon/ucm_temp/ucts_opcua_server.sh (code=exited, status=134)
 Main PID: 152844 (code=exited, status=134)

Okt 16 13:22:28 osaka ucts_opcua_server.sh[152844]: (MOS) : Info : 2018-10-13.12:30:47 : Connected to Server : opc.tcp://osaka:48010
Okt 16 13:22:28 osaka ucts_opcua_server.sh[152844]: 
Okt 16 13:22:28 osaka ucts_opcua_server.sh[152844]: (MOS) : Info : 2018-10-13.12:30:47 : Verification of MOS version with lappweb
Okt 16 13:22:28 osaka ucts_opcua_server.sh[152844]: 
Okt 16 13:22:28 osaka ucts_opcua_server.sh[152844]: ********************
Okt 16 13:22:28 osaka ucts_opcua_server.sh[152844]:  Press CTRL-C to shutdown server
Okt 16 13:22:28 osaka ucts_opcua_server.sh[152844]: /home/dragon/ucm_temp/ucts_opcua_server.sh: line 9: 152847 Aborted (core dumped) ./MOS_Device -d /MOS/plugins/Plugin_UCTS/UCTS.xml
Okt 16 13:22:28 osaka systemd[1]: uctsd.service: main process exited, code=exited, status=134/n/a
Okt 16 13:22:28 osaka systemd[1]: Unit uctsd.service entered failed state.
Okt 16 13:22:28 osaka systemd[1]: uctsd.service failed.

Restarted.

2018-10-15 Taka MOXA Switch connected SLOW control connection intact. Drive network can be used from remote tomorrow.
2018-10-16 Léa, Dirk, Julien, taka, Saiya, Mitsunari TIB issues

- TIB goes from state 0 to 4 but then when we enable trigger go to state 255 as the alarms vector

2018-10-16 Léa, Dirk, Julien, taka, Saiya, Mitsunari Small run summary

- 7 GotoSafe and GoToReady for the ECC due to too high temperatures so switch ON/Off of the Modules/BP:

1) All module ON, 2 BPs OFF associated to module 10.1.6.12 and 10.1.6.27

2) All module ON, 2 BPs OFF associated to module 10.1.6.24 and 10.1.6.27

3) All module ON, 1 BP OFF associated to module 10.1.7.171

4) All modules and BPs ON

5) All modules and BPs ON

6) All modules and BPs ON

7) All modules and BPs ON

8) All modules and BPs ON

9) All modules and BPs ON

10) All modules and BPs ON

11) All module ON, 2 BPs OFF associated to module 10.1.7.147 and 10.1.7.149

2018-10-15 Dirk Charging Walkie-Talkies with our private mini-USB adapters, while waiting for the real charger to reappear Alternative: $8.99 on Amazon


2018-10-15 Léa, Dirk, Julien, taka, Saiya, Mitsunari Slow control and uaexpert deconnection 
- Slow control connection lost in ready mode so then no more current in the pulse bar. GotOsafe GOtOready still no curent with the negative value in the pulse bar. We had to switch off and on the 233 and 400 V

- We lost uaexpert that was completely stuck so to have the monitoring back again the DataLogger are now written in /home/cacooperator/CoolingSystem/20181015_005 and 20181015_006

2018-10-15 Léa, Dirk, Julien, taka, Saiya, Mitsunari Small run summary

- 7 GotoSafe and GoToReady for the ECC due to too high temperatures so switch ON/Off of the Modules/BP:

- Cut busy propagation from BP, dragon on local clock

1) 1 module OFF: 10.1.6.28, 2 BPs OFF associated to module 10.1.6.24 and 10.1.6.27

2) 1 module OFF: 10.1.6.28, 3 BPs OFF associated to module 10.1.6.24, 10.1.6.27 and 10.1.7.147

3) 1 module OFF: 10.1.6.28, 1 BP OFF associated to module 10.1.7.147

4) All modules ON, 2 BPs OFF associated to module 10.1.7.147 and 10.1.7.149

5) All modules ON and All BPs ON

- pb of internal/external trigger clock for the Dragon fixed: For DRS4, referential clock is now 10 MHz external clock. - Configuration of UCTS and TIB, two last runs taken with the TIB so with external clock, external trigger and busy propagation. 6) 1 module OFF: 10.1.5.16, 3 BPs OFF associated to module 10.1.6.27 and 10.1.7.146 and 10.1.7.149

7) 1 module OFF: 10.1.6.28, 1 BP OFF associated to module 10.1.6.28

8) ALL modules On


2018-10-15 Léa, Taka, Dirk, Daniel SLOW Control lost

While Camera was on, the SLOW control connection was interrupted in the Drive container to prepare connection of Drive/AMC network.

Consequently the EMC went to SAFE. But also the UaExpert interface was stuck (which is the current base for Camera monitoring). The setup was then restored as well as we could, including DataLogger function.

2018-10-15 Léa, Taka, Julien, Seiya, Dirk writing speed limitation in data taking

1 ZFW: validate speed 300 MB writing speed 8 ZFW: validate 8* 300 MB/s writing speed 16 ZFW: writing speed: 16*150 MB writing speed. Maybe problem due to the disk. To investigate

2018-10-15 Léa, Dirk, Julien Slow control deconnection+ disconnect from OPC-UA
2018-10-14 Léa, Dirk, Julien Small run summary

Runs0016-0019

- No TIB/UCTS

- Cut busy propagation from BP, dragon on local clock

- 7 GotoSafe and GoToReady for the ECC due to too high temperatures so 7 switch ON/Off of the Modules/BP

1) All modules ON, 2 BPs OFF associated to module 10.1.6.24 and 10.1.6.27

2) All modules ON, didn't check the BPs

3) All modules ON, didn't check the BPs

4) All modules ON, 1 BPs OFF associated to module 10.1.7.147

5) All modules ON, 2 BPs OFF associated to module 10.1.7.147 and 10.1.7.171

6) All modules ON, 1 BPs OFF associated to 10.1.7.147

7) All modules ON, 1 BP OFF associated to module 10.1.7.147


2018-10-14 Eric, Dirk All (DATA) fibres straight now!
  • There are straight and crossed fibre patch cords (AB->AB and AB->BA)! They are obviously used indifferently and mixed on our site. :-(
  • The fibres in the IC-PP that are connected to the couplers are all yellow! (No colour code to trace them.)
  • We have chosen the same convention as on the transceivers for input/output of the LC connectors
  • Problem/drawback: All fibres at the IC-PP are now reversed. Need to think/investigate that (last?) point.
  • All PP boxes now closed and secure. Should not be touched any more without agreement by INFRA experts!
 Cisco-Transceiver 13927.jpg
2018-10-14 Dirk Direct measurement of TX lasers

INFO: Direct measurements can be done without danger for Photom-211 

測定範囲 -70 ～ +5dBm

according to datasheet. That is 3.16 mW to 0.1 µW.

2018-10-14 Léa, Dirk, Julien First full-cam data run up to 15kHz! That is what we would have liked to see last week.

Now it's Champagne time. :-)

2018-10-14 Julien, Eric Fibres DATAsp1-6 tested Optically, between DC and Cam. Data1-6spare measurements 20181014.jpg
2018-10-13 Léa, Dirk, Julien Run0015 Still no UCTS(/TIB); fibre broken between DC and Cam. Running with half-cam and two additional missing modules (BP problem): 6.24, 6.27.

- r0015 all events (at runstart), 300Hz and 10kHz, but ZFW problems (testing with 16 instances).

ALL Door knob! Falling apart from the CC door. Urgent action needed. (Bigger screw?)
Taka, Seiya, Mitsunari Fiber Check We checked optical connection between DC and Camera because some labels were lost due to UV damage. We checked Data2, Data 6, SlowControl and UCTS. Only UCTS had a problem (no splicing at Drive PP). The rest where OK.
Taka, Seiya, Mitsunari Labeling fibers We labeled optical fibers of the data (DATA 1 - 6) at patch panels in Drive contaniner and in IT container. The spare cables have not been done yet because a ribbon ran out.
Seiya, Mitsunari Connection validation We validated 12 optical fiber connection (No. 1-6, 13-18) from Drive coontainer to IT container. Strength is -35 to -38 dBm.
2018-10-12 Léa, Taka, Julien, Seiya, Dirk Runs0012 sqq. No UCTS/TIB today.

These runs have 3 modules missing (as identified in the preparatory phase: 6.21, 6.25, 6.28). According to a quick check, all EventNb=TriggerNb otherwise for all runs today. See RunCatalog for details.

Dirk Creation of logbook
Dirk, Taka, Julien, Seiya, Léa Data acquisition

- Pb with the ClusCo on tcs01. The root propagation for the BPs for the trigger doesn't work. Using exactly the same script it works on CacoOperator.

- We validate for 3 fibers the new connections to the dataswitch fiber. Eric is fixing the one missing or broken. So for now only the right part of the camera is used for data acquisition

- No TIB/ UCTS

- Few runs were taken with no external trigger from TIB. 3 Modules didn't appear busy but didn't sent any data. In those test the busy from the CBP was cut. Those missing modules have to be investigated in more details but due to a lot of slow control deconnection problem and high temperature in the camera it was not possible. Script used in CLusCo: init7_noextTrigger_Test.uic. To not cut the busy, name is: init7_noextTrigger_noBusyCut_Test.uic

- One try with no external trigger and clock but with the CBP delivering the clock and pps and using 10 MHz clock as default clock for the dragons. L1 local Trigger didn't generated. No we come back to a configuration of dragon on their local clock but this issue has to be investigated. Script used in CLusCo: init7_noextTriggerClock_Test.uic

Dirk, Taka, Julien, Seiya, Léa Too high temperatures in the Camera

- Limit fix to 27 for the aire temperature inside and 35 for the BP temperature. Pression also get some alarms

- During the day, due to high temperatures we have to gotosafe to wait for the camera to cool almost 11 times but never the BP max temperature went more than 34 degree. The air inside reach at the maximum 26.5.

Dirk, Taka, Julien, Seiya, Léa ECC lost connection (2 times!)

- In the afternoon, 3 lost of ECC slow control communication due to the interruption between IT-Container/Driver-Container. Miscomunnication with AMC people... First time, temperature was already high in the camera, we had to switch OFF the 233 and 400V for security reasons. Two other times, we get the ECC connection back quite fast and ECC was in the same state that when the connection was lost meaning state 2 ready. Just no more current in the pulse bar so we have to gotosafe and gotoready both times. After that the current was -247 in the pulse bar... Not understood for the moment - The second interruption happened, when the Moxa switch was reconnected, probably not correctly configured. It was disconnected again. Presently this impacts AMC and drive operation, until the Moxa can be reconnected.

Léa, Taka, Julien, Seiya, Dirk Discovered SLOW control fiber lost, fibers changed connection recover Interruption. Using UCTS section for replacement.
2018-10-11 Eric, Armand Cable splicing UCTS fibres ready and checked.
Léa, Taka, Julien, Seiya, Dirk DATA5-upstream broken Located between DC-PP. and IC-PP. Eric is going to have a look on Friday, when working on the other (spare) fibres.
Léa, Taka, Julien, Seiya, Dirk Found correct order of DATA1-DATA6 We eventually found that the fibres DATA1-6 were connected in (exactly) wrong order to the camera, which lead to a mismatch of switches/modules with respect to interfaces/addresses in osaka.

This is an item for our "learned lessons": The indoor fibres had been labelled (switch-interface), but stayed in Mirca. The new fibres had been confectioned at ORM, and labels had to be "guessed" in one way or the other.

Glossary

  • CC = Commissioning Container (present LST1 Control Room)
  • DC = Drive Container
  • IC = IT-Container
Retrieved from "https://www.lst1.iac.es/wiki/index.php?title=Logbook_CameraCommissioning_ORM_Oct18&oldid=1162"