Last week we put in sevice the new ALS completly modified:

_ Main System with new reducer.

_ New Speed parameters for Main System (motor is at full speed but due to reducer the spped is slow). Decceleration sensors not installed since the speed is very slow.

_ All the sensors were replaced by new ones (2 logic embedded in each sensor (NC + NO) ) = only one type of sensor in the ALS. Sensor with connector to ease maintenance.

_ All sensor cabling were changed and new ducts were installed to ease maintenance.

_ The damaged cable of the loadcell was welded again and protected (Heat shrink tube + tape).

_ Add mechanical part to protect loadcell cable when opening over the opening position. It's now impossible to damage Loadcell cable.

_ Increase speed of plier. We add a deceleration sensor in each plier.

_ All the sensors support are new to avoid dammage if there is over position due to manual operation.

The ALS is now functionnal with human interface. The system are so slow 180s !!!, They were not integrated into automatic cycles.

Shifters must manage the Opening/Closing of the ALS at the Start/End od telescope operations.

A new error code is ride up if a door is opened during motion (3002).

In the late afternoon shifters noticed the semi-expert drive GUI was not updating, several restarts did not help. Checking into the issue with the drive team connecting from remote it was found that the MOS server is not running and cannot be restarted. This issue could not be solved and will be revisited tomorrow, Jan 17, during the day.

Actions applied for solving this issue: (T. Le Flour)

The status is the following

• The MOS OPCUA Server cannot be relauched since the port used by server was not properly released after the last stop.
• We decided to reboot the VM running on TCS02 but after this operation , the concerned port was still in use.
• By using the commend lsof , we relased the port and relaunched successfully the server.
• Actions : releasing the port has been included in the script used to launch the server to prevent such an issue.

On Dec 6, Dario, Frederic and Yuji showed to Koji Noda and Daniel Mazin how to operate the new ALS. The idea was to do an unlock and lock again, as the weather was still bad. The automatic part went more or less OK ending in error state (excpected). We then went to EAST ALS to open the main actuator manually. After moving the actuator manually, the status of the ALS was either locked not unlocked. We realized that secondary sensor for South part was reacting incorrectly (close to metal OFF, far from metal ON), and we thought it is the reason. We removed the endswictch screw but the status did not change. After pressing again Unlock we realized that the secondary South plier squeezed the sensor cable and destroyed it (see Attachment 1). We consulted Ino and closed the ALS EAST manually (all three actuators) because the eather was getting worse.

Next day, Dec 7, we (Daniel and Koji) examined the sensor cable and confirmed it is completely distroyed. We introduced a patch cable and few adapters as a temporary repair (See attachements 2,3,4 and 5). We confirmed the reading of the pressure of the South and North sensors of teh secondary pliers are good again (attachment 6). See pictures.

Few first images at 5s with LED @1

The aim was to observe several stars with different magnitude and check the sensitivity of the CDM. We wanted to use a semi-automatic procedure with command sent to the drive via a script to the Acs

After we realized that the telescope was not moving after a StarTracking command we try to use the drive system through the Acs server. StartTracking command not sent to the drive but the other commands (such as StopTracking) worked. 

Instead we used the Semi-expert Gui to make our test. All fine. list of stars observed:

AlphaAqr mag 3.11

10 images with LEDs and OARL

100 images + 100 dark

Homam mag 3.39

10 images with LEDs and OARL

100 images + 100 dark

48-peg mag 3.66

10 images with LEDs and OARL

100 images + 100 dark

24-peg mag 3.87

10 images with LEDs and OARL

100 images + 100 dark

47-peg mag 4.13

10 images with LEDs and OARL

120 images + 120 dark

17 And mag 4.26

10 images with LEDs and OARL

120 images + 120 dark

84Peg mag 4.69

10 images with LEDs and OARL

120 images + 120 dark

56Peg mag 4.91

10 images with LEDs and OARL

180 images + 180 dark

81Peg mag 5.13

10 images with LEDs and OARL

300 images + 300 dark

63 Peg mag 5.70

10 images with LEDs and OARL

400 images + 400 dark

85 Peg mag 5.87

10 images with LEDs and OARL

400 images + 400 dark

- CDM images at park out position.
Exposure 100ms.
OARL on.
Leds 255-200-150-100-50-25-10-1

- CDM image  Az=0 Zd=75-70-65-60-55-50-45-40-35-30-25-20-15-10-5-0
 exposure 100ms
OARL on
Leds On 255

Then we track serie of stars, all affected by clouds unfortunately

- Gamma Peg
- Markab
- Vega
- Altair
- Deneb
For each we took 1 image 100ms led/oarl on + 60 1s images led/oarl off
Then we moved by 2 deg on dark region at took 60 1s images
 

We found out a bug in drive repointing that was affecting the image quality in the first seconds after repositionning. It has been correct in drive SW.

We tracked stars at various zenith angles and changed the Az and Zd offsets, taking images with the PSF camera for each position. After Armand's changes during the day the offset is the same for all Zenith angles.

Star Zd Az
dAz, dZd image

Polaris 61.5 0.7
0, 0 1568923765.fits
0.5, 0 1568923985.fits
0, 0.5 1568924088.fits
0.5, -0.5 1568924324.fits

Eltanin 29.2 327.9
0, 0 1568924558.fits
0.5, 0 1568924683.fits
0, 0.5 1568924747.fits

Benetnasch 66.6 314.7
Vega 15.6 313.8
0, 0 1568925023.fits
0.5, 0 1568925090.fits
0, 0.5

Kochab 57.3 342.0
Sulafat 8.7 299.5
0, 0 1568925265.fits
0.5, 0 1568925420.fits
0, 0.5 1568925485.fits

Alphecca 52.6 283.0
Arcturus 72.2 282.5
Oph Alpha-55 32.2 246.0
Albireo 1.2 229.1
0, 0 1568925734.fits
0.5, 0 1568925842.fits
0, 0.5 1568925917.fits

Nunki 55.8 190.6
Altair 20.2 168.3
Enif 35.7 115.2
Alpheratz 59.0 71.4
Caph 54.8 35.6
Deneb 21.2 33.9
Polaris 61.5 0.7

Thierry&Ino on site

Ievgen, Vitalli and Jean-Luc remotely.

TCU tests to perform the parkin-in and park-out procedure from the Drive.

Results :

• Actions can be triggered from the TCU and correctly executed thru the ACS Drive Bridge-MOS-Drive PLC.
• Monitoring information is not correctly pushed from the bridge to the TCU (positions , ...)
  • Investigation brings many options. For the time being, a simple change in the ACS xml description will fix the problem. Anyway, this solution is not satifactory and a new solution will be implement asap.

Plans : Monday September 16th (2pm localtime) : new TCU tests to validate the intermediate solution for park-in, park-out and probably for some others action like Tracking,  GotoCircularPosition, ...

Tuesday September 110th :

• Tests park-in and park-out to manage the different type of failures 
• Tests failure during park-in phase with a telescope in a position different from the park-out position
• Tests (from the Drive semi-expert GUI) to send the telecope to the parking area from any positions. Useful to close the camera shutter without parking-in the telescope.
• Night : tracking tests for fixing the shifhting issues : Problems with the PSF camera  

Wenesday September 11th :

• Night : Still tracking tests for fixing the shifhting issues. PSF camera OK. Many pictures taken. Small shifting not yet understood

Drive GUI stacked due to a wrong shutdown procedure (Jul. 27th 01-02UTC). This was simply a human error.

First, we pushed ParkIn button as usual in the end of the night. We confirmed the telescope was in Locking Position in GUI, but most probably overlooked to check the statuses of access tower and CSS.

Next, we went to the drive container, turn the key into Service mode and pull it out. (Before get into the drive container, we saw the telescope in parking position, but apparently this inspection was not enough.)

Then, we visually found the camera shutter/SIS was not closed/put back (this was also our misoperation). BUT, at this time, we couldn't notice that access tower was open. (Since it was complete dark time, it was not clear to see even using a fully charged flash light.)

After closing/putting back the camera shutter/SIS (visually checked), we inserted the elevation bolt and locked the azimuth properly. Then, inserted the key back and went back to CC.

Finally, we found that the status of access tower/CSS was opened/unlocked. At first we though this was a bug of the GUI, so closed and opened the GUI, but (of course) it didn't change.

Then, we tried to push the parkin button after turning the key into Automatic mode, but the GUI didn't response. So, we contacted to the drive expert at this point (01:35UTC).

Ino asked us to visullay check the access tower and CSS (01:52UTC). After turning the key into Service mode, we confirmed that the CSS seemed to be locked but the tower was indeed open (02:00UTC). We turned the key into Automatic mode and got out the telescope area. Ino successfully locked and closed remotely (02:21UTC). We turned the key into Service mode and closed the shop.

Regarding the problem on the Flywheel PLC (see here), we entered UC1 under the supervision of Daniel M. and Koji (11:16 local time).

First we checked the LEDs of the black Ethernet cable. LINK in green was constantly ON and ACT in orange was blinking.

Then, we disconnected the black cable and reconnected again after 20 sec (11:30). But, the LEDs of the PLC in the right side were OFF.

Finally, we removed the two red power lines connected to the PLC with a screw driver and reconnected again (11:52). The LEDs of the PLC turned ON. The UC1 web page was back (12:08).

After finishing the normal start-up procedure, we tried to park out by GUI, but got an error (22:30 local time): "MSG_WARNINGPark_Out_TimeOut"

The Drive, FSM and Command Statuses were "Parked", "in parking Area" and "No command in progress," respectively.

We closed&reopend the GUI and "Park Out" three times, but everytime got the same error.

We then send a message to the WhatsApp group of Drive experts (22:32).

Thierry asked us to restart the server on 10.1.8.3 and we killed the corresponding process (ps -ef | grep MOS), but the statuses didn't change (still FSM "in parking Area").

Thierry asked us to press the "General Abort" button, but didn't improve the situation.

After Ino's intervention (probably, we don't know detail), Thierry asked us to restart the GUI and we did so. The statuses got improved (Drive&FSM: Parked, Command: Started). So, we pushed the park out button, but the same error happend.

Ino suspected if the telescope wasn't unlocked by ALS, but we could visually confirm the ALSs unlocked the telescope properly.

Ino found the problem that "the Flywheel interlock says no motion allowed so PLC refuse to do Park out". He wanted to see the screen shot of Flywheel page. But, we could not connect to the server 10.1.200.1 which means the Flywheel PLC is probably down.

Daniel M. asked us to go to UC1 and we went there (23:45). After some discussion between Daniel and Ino, they asked us to try "stop" -> "run" -> "reset" in the PLC (lower white one, in the cabinet opposite side of the Flywheels). But, still there was no connection to 10.1.200.1 (also no two LEDs in the right side of the PLC turned on). See attachment 1, the LEDs were ON when it was working fine. Attachment 2 was taken tonight (the LEDs were OFF).

Daniel M. suggested us to close the shop because we need Koji's help to fix this problem (00:01).

Tonight, we tracked Vega to test (a correction of) the pointing offset of the Drive.

Around 22:30UTC, we tried different Az offsets of the pointing in the Semi-expert GUI, followed by taking an image with the PSF camera to see the effect of the offset choice.

We tried delta_az = 0.3°, 0.1°, 0.26°, 0.6°. Strangely, we saw some unexpected (much too large) shifting of the spot for delta_az = 0.3° and delta_az = 0.6° (but looked reasonable for delta_az = 0.26°), which appeared quite puzzling to us (see specific Camera Elog here).

After having tracked with delta_az = 0.6°, we stopped again, and selected delta_az = 0.4°. We submitted the command ("Start"). Some seconds after submission, we became unsure whether we used the right field in the GUI where we entered the delta_az values. We hit "abort current action" (to check our previous command), when probably the slewing to the new offset position wasn't finished yet (about 5 seconds after the previous "Start" command). Then, the GUI became stuck displaying "abort current action", and not displaying the field to enter new positions/targets. According to the GUI, the telescope had in fact stopped tracking, but without any possibility to receive further commands.

We restarted the GUI, but the state remained the same, and checked the telescope outside. Then we called the experts, who could solve the problems after some non-trivial intervention.

At parking in, the telescope got stuck at 94.254°. We checked in the container that fuse Q21 went off. We did not see any errors both in the Drive cointainer and the GUI. We texted the experts. After 15 minutes, we pressed again "ParkIn" (without having touched the fuse). After nothing happend in the GUI, we called the experts. They authorized us to switch back the fuse. After that, still heavy intervention by the experts was needed, who finally made the telescope successfully park (fuse was on during all the trials). Thanks to the experts solving the issue at late night!

When parking in the telescope, accidently hit "Shutdown" button instead of "Parkin" in Drive Semi-epert GUI. Nothing seem to happen for minutes despite telling command in progress "Shutdown"

Texted Drive experts, they told to just hit again the correct "Parkin" button.

Near the end of parkin in, procedure got stuck. Drive expert identified fuse Q21 having gone off. Switched back the fuse, acknowledged error, retried park in. Then, tower close. After expert intervention, did close and everything was fine.

When checking the telescope visually outside, while it was in the "park-out" position, a car happened to pass by from the upper road (coming from the other telescopes). Its lights flashed directly into the camera. It seems that the road is in direct prolongation of our current park-out position.

• We saw the CAM HV going down several times in the park-out position (where the shutter is typically opened) during the last weeks. Can we change this standard position slightly (higher or lower)?
• Another question is of course, why cars drive with lights on in that place. The "no luz larga" sign is below the MAGIC site, if I remember correctly.

On park-in command, telescope went to the right position 95/-1, but did not recognise that it was there.

• Pushed Park-In again
• Acknowledged one error
• Then everything fine.

When parking in yesterday, the telescope drove to the nominal position (approx. 95° zenith angle and -1° azimuth), but then it took a long time and never showed the "park out" button again.

In the driver container, we saw two alerts to acknowledge, then the GUI indicated "telescope parked". We confirmed visually that the position was correct and we could lock azimuth and elevation. Informed experts on WhatsApp and got confirmation that everything was done right.

This morning we were asked by Ino to rearm fuse Q21 (24V), which was the reason for the wrong statement from the UI.

(For Thierry with love <3 :-)

The semi-expert drive control panel uses various terms for the zenith angle: "Altitude (zenith angle)", "Elevation", "Angle". "Zenith Angle". 
It would be less confusing, if only one and the same term were used over all places in the GUI.

After we issued the tracking command, giving RA/Dec, the GUI displayed

• "Tracking in process" when the telescope was still moving, and the tracking point was not reached.
• "Action in progress.Please wait ...", when the telescope was actually tracking.

We believe these two should be exchanged. Maybe even a colour code (yellow when still going to the tracking point, indicating that the telescop is not in nominal operation, and green when tracking is ongoing) could help the shiftcrew to understand in one glimpse where we are.

The Current&Power Monitoring was not updated and hidden by two emtpy modal dialogue windows.

Restart of display (via icon on desktop) not successfull. ("MODBUS #2 communication failed.")

Called expert (ThLF), who advised how to call the old instance, then restart worked.

Ino explained: MOS was totally desynchronised with PLC. (It was assuming that the telescope was parked.) He parked the telescope manually. Thanks!
Experts will investigate on Monday. But the shift crews can continue to use the  Drive Control and tracking.

After tracking Mk421, we abort current action (acknowledged) and try to go to a new source: RA=308.1, Dec=40.96. Drive Control refuses these parameters ("check your parameters").

General Abort without effect for 2 minutes. Called expert (Ino). He will look into it.

During this weeks we solve this problems:

_ Park In: Sometimes the final locking position was too limit and the "In Position" sensor was lost during the locking witch put the sytems in bad state vs safety constaint (we can't unlock if the sensor In Position is not active). So now the final poition is now adjusted dynamically with the difference betwwen motors and encoders to be sure to stop on torque limit. No more problems since this modification.

_ Problem of communication between the 2 main PLC of the Drive. Due to this error randomly the PLC receive a 2ndTracking command after a Cancel Command. An other random error is to receive the Start Tracking before the Goto Position witch gives errors in the PLC sequence. Communication was rewritten between the 2 PLC. No more problems since this modification.

_ Some change were done in Cancel and Emergency Stop sequence.

_ Now the Power and the Intensity of Main Electrical Cabinet can be available in the OPCUA of the Drive.

_ The High speed of west ALS was decreased since sometimes the variator gives overspeed error. The speed are now identical between the two ALS

+ other minors improvement.

During this 2 weeks we solve this problems:

_ Park In: Sometimes the final locking position was too limit and the "In Position" sensor was lost

We sent a tracking with with RA211,08 and DEC29,52

1st time the goto position of azimuth was sent at +42° and the trajectory requierts  -52° !!! (speed limit of the tracking 0,001 rad/s)

After a cancel the azimuth was sent at -54° !!!

Yesterday we had the same problem with the same RAD/DEC coordinates. -31° and after a cancel +270° !!!

Is it a problem of trajectory with the 1st point positionning or with this coordinates does we need a higher speed in tracking ?

- During shutter tests, we were requested to move the telescope many times between two positions at different zenith. Sometimes, once the position is reached and in the log I got the message 'GoToTelescopePosition Done received', the command panel stay stack with the "action in progress" bar without possibility to perform other atctions. I need to click to 'general abort' to stop the action and  get the panel comand back. It happened 3 times

- During tracking, the message on the screen switch from "tracking in progress" to "action in progress, please wait..." The latter is quite misleading for non-familiar user of the gui.

The 2 swicthes are too far from the target at 56.48 elevation.  The sensor have been tuned in parkin position and the distance was correct (2mm) and at 56.48 is about 10mm. (impossible to tune correctly the sensor and the atrger has to be tuned)

• Different ParkOut and Parkin done with different values for testing the torque limitation. 
  • By decreasing the torque and the position , the parkin failed because we are not in the CSS locking sensor (in locking position)
  • By setting back the original values, th parkin is ok but the bolt cannot be properly inserted.

After pointing Polaris yesterday, it seems that the zero position of the telescope was incorrect but It's never had been calibrated (zero was equals to parking position towards center of tower).

So I shift the zero position of telescope by 0.869°.

As the consequence the new Parking Position = -0.869° don't be surprised.

• Tracking at position :
  • RA = 101.288541 RA Offset = -0.6
  • DEC = -16.71343 DEC Offset = 0.7

• Cabling the azimuth end switches.

• 1 degree has been added to the Elevation accordingly to the last measurement given by Olger

• On Wenesday morning (27/03/19) :
  • With Holger and Robert (MPI) , many telescope "park in/park out" operations have been performed in order to refine the positionning in the acces tower and at the elevation locking bolt
    Holger and Robert applied many mechanical modifications on the acces tower to make the telescope park operations very smooth
    ==> Conclusive operations. 

• Regulation of the motors (PID) for slow and fast tracking and motor oscillations.
• Implementation and Debugging of the Semi-Expert control software for the drive.
  • GUIs + MOS plugin (OPCUA Server)
•  Results are quite encouraging in good wheather condiftions.
  • additionnal tests in different (wind) conditions are envisaged to validate completly the process

new session of high speed drive tests after issues with Power Supply.

• Tests of emergency stop (with a speed at 25% and 43 % GRB speed) with a modification of the jerk value from 0.08 to 0.2 ==> OK
• Tests of azimuth movements : (-90 to 270 degres)
  • at 60% of GRB speed. ==> Ok for 6 movements
  • at 80% of GRB speed
    • Ok for 4 movements
    • Failures due to resistance  (sur-tension). Breaking modules in failure then reset
    • Ok for 2 next movements
  • at 100% of GRB speed :
    • 3 movements from -90 to 90 ==> OK
    • Temperature of breaking resistance are getting very high (at least the top one) 

• Tracking procedure has been debugged from the GUI-MOS (OPCUA Server with tracking plugin)-PLC
• Still some improvments to apply but globally the procedure is running well.

• On Thrusday :
  • new movements at 45% from -90 to 270 have been done.
  • No possible movement at 60% because the FW and UPS issues are not solved yet
    • Ino will back on site when HITACHI team will be on site for FW maintenance
      • Tests at 60% will be done to validate (or not) the correction applied by Hitachi team.

• Debugging of the MOS plugin corresponding to the tracking 

• Test at 60% maximal speed from -90 to 0
  •  Check of the FW discharge : No error (FW 2 : 95% FW1 : 100%)
• Test at 60% maximal speed from -90 to 270
  • Check of the FW discharge ==> Error (FW 2 : 90 % FW1 : 100%)
  • Drive stops at 250 degree.
  • Power consumption = 60kw
•  

During a park-in action , the acces tower did not close correctly.

• The FW issue is not solved . No movement can be done
• The FW is going to be restarted ==> It takes several hours to put it down and up again
• Drive movement are stucked 

. Fisrt day of the high speed drive tests with IFAE team.

• Tests for tuning and generating motor plots at 10% (0.0234 Rad/s) of the maximal speed (GRB alert speed)
• Tuning of the emergency stop from 0.05 Rad/s2 up to 0.2 Rad/s2
• 6 movements at 25% speed. (from -90 to 270)
• repositionnning to -90 at 40% 
• movements from -90 to 270 and reverse at 40%
• 3 movements from -90 to 270 and reverse at 60%  : the FW generates an alarm and the drive stops the movement.
  • The FW level was ok but the the status (GRB Fast Rotation and UPS system) got red.
  • Koji has been contacted to look at this issue.

Same operations as yesterday ==> Polaris position

Debugging of the tracking sequence :

• PLC sequence : trajectory tracking data correctly managed
• PLC tuning : regulation OK ==> need to refine.
• Tracking algorithm :
  • Changes in Drive.ccp to allow azimuth negative values.
  • Tracking duration : even by changing the duration of he tracking (from 1'to 5') ==> not correctly managed (still 1' tracking) ==> to be checked
•  

Same operations as yesterday ==> pointing to Polaris with the same coordinates

Pointing to polaris ==>

Azimuth = 0.808

Elevation = 59.941 + 1 degree (correction)

Start of the tracking debug ==> PLC and MOS 

Tests and debugging of the tracking procedure. Many components are involved : PLC MOS GUIs.

Each component needs to be well debbuged to achieve a safe movement of the telescope

==> IN PROGRESS

• Debugging of the "Parkout" sequence with errors ==> 
  • MOS received the error and set the FSM in error state
  • User from semi-expert GUI should acknowledge the error ==> MOS FSM acknowledge methosd is call (just authorized from the error state) to acknowledge the error to the PLC.
  • PLC set up all the needed variables to inform the MOS and GUI of the result of the acknowlegment 
• Today, an PLC error sends the MOS FSM in the ERROR state which can the only go "Parked" state via the "Acknowlegde" method.

Many differents succesfull  "goto position" operations ... MOS configuration file (XML) modified to aplly to correct Azimth and Elevation ranges for secure movement 

• GotoPosition azimuth range : [-126..354]
• GotoPosition elevation range : [0..75]
• FastGotoPosition azimuth range : [-96..324]
• FastGotoPosition elevation range : [0..50]

Many parkin operations done. To confirm the different movements , a Parkin test with a azimuth position 20 and Elevation 70.

This is to validate the complete "Parkin" scenario with "random" az and el positions

The brake of the elevation issue has been fixed. Operations on drive can be resumed.

After issues during the parkin procedures ( 100% elveation torque ) :

• a visual inpsection has been done :  access is not involved , 2 possible causes :
  • rolls on elevation structure because they are rolling (4 of them yes , 4 of them not)
  • motor break ==> under investigation

Emergency stop applied because of a azimuth value(encoder) not in the pre-defined range +-0.015

• first parkin ==> Azimuth 0.013 ==> Emergency because offset even if in the correct range
• 2nd parkin ==> Azimuth 0.004 ==> Emenrgency stop because offset not in the correct range (eye control not drive control)

• To define precisely the correction to apply to Azimuth positionning, a set of movements have been done to calculte the correction coefficient.
• A scan from 0 deg. to 270 deg. with a step of 10 deg. and a reverse scan from 270 deg. to 0 deg. with a step of 50.deg realized to have a better approximation

1. Many parkin and parkout operations with different torque  (with the clamping system of the access tower (from 18% to 8% = 26N/m). With 8%, the locking position cannot be reached
2. Some minors have been corrected

Installation of the new PLC code and re-initilalisation of the system.

Here is the first entry in the DRIVE ELOG logbook