========================================== 96Jun26 00:14 Operator On Duty: T. Briscoe ------------------------------------------------------------ 96Jun26 00:15 Job Start #5728 96Jun26 00:41 Job End #5728 ------------------------------------------------------------ 96Jun26 01:20 Job Start #5729 96Jun26 01:47 Job End #5729 ------------------------------------------------------------ 96Jun26 01:49 Job Start #5730 96Jun26 02:20 Job End #5730 ------------------------------------------------------------ 96Jun26 02:26 Job Start #5731 At 5h52m the job emitted a several (~20) msgs of that said "writing visibilities X ticks behind" where X decreased by one each iteration. 96Jun26 02:54 Job End #5731 ------------------------------------------------------------ 96Jun26 02:56 Job Start #5732 96Jun26 03:24 Job End #5732 ------------------------------------------------------------ 96Jun26 03:24 Job Start #5733 More of the "writing visibilities X ticks behind" msgs 96Jun26 03:50 Job End #5733 ------------------------------------------------------------ 96Jun26 03:55 Job Start #5734 96Jun26 04:23 Job End #5734 ------------------------------------------------------------ 96Jun26 04:23 Job Start #5735 96Jun26 04:48 Job End #5735 ------------------------------------------------------------ 96Jun26 04:49 Job Start #5736 KP failed to prime and was slow to get into sync. 96Jun26 05:17 Job End #5736 ------------------------------------------------------------ 96Jun26 05:18 Job Start #5737 96Jun26 05:49 Job End #5737 ------------------------------------------------------------ 96Jun26 05:51 Job Start #5738 96Jun26 06:22 Job End #5738 ------------------------------------------------------------ 96Jun26 06:22 Job Start #5739 More "writing visibilities X ticks behind" msgs. 96Jun26 06:49 Job End #5739 ========================================== 96Jun26 06:49 Operator On Duty: switz 96Jun26 06:49 Comment: systest *96Jun26 Down-time category:OHD 07:00-08:00 *96Jun26 Down-time category:MNT 08:00-11:30 ------------------------------------------------------------ 96Jun26 11:33 Job Start #1722 96Jun26 11:42 Job End #1722 *96Jun26 Down-time category:MNT 11:44 ========================================== 96Jun26 12:00 Operator On Duty: Betty Ragan *96Jun26 Down-time category:TST 11:30-14:25 *96Jun26 Down-time category:OHD 14:25-14:56 ------------------------------------------------------------ 96Jun26 14:56 Job Start #6701 96Jun26 15:00 Comment: GO STALLed until jt 16:43, then goes into SEARCH. RO in SEARCH thru the entire job. 96Jun26 15:06 Job End #6701 ------------------------------------------------------------ 96Jun26 15:48 Job Start #6600 96Jun26 15:48 Job #6600 Killed by system ------------------------------------------------------------ 96Jun26 15:54 Job Start #6600 96Jun26 15:58 Comment: "CALC terminated due to internal errors in subroutine: UT1G" errors. 96Jun26 16:01 Comment: TI43 in SEARCH, finding times a few seconds behind. 96Jun26 16:05 Comment: TD goes into SEARCH, jt 11:39. 96Jun26 16:06 Job End #6600 ------------------------------------------------------------ 96Jun26 16:08 Job Start #6600 96Jun26 16:08 Comment: same errors 96Jun26 16:17 Comment: TI43 still in SEARCH, consistently about 20 s behind. TD agian in SEARCH jt 11:39. 96Jun26 16:21 Job End #6600 ------------------------------------------------------------ 96Jun26 16:29 Job Start #6600 96Jun26 16:32 Comment: same, but now UD in SEARCH. 96Jun26 16:33 Job #6600 Killed by ------------------------------------------------------------ 96Jun26 16:33 Job Start #6600 96Jun26 16:39 Comment: TI43, UD in SEARCH. TD slewing. No errors this time, though. 96Jun26 16:40 Comment: UD head not moving properly. Command reads -2817, position is -1706. 96Jun26 16:41 Job End #6600 ------------------------------------------------------------ 96Jun26 17:03 Job Start #6600 96Jun26 17:08 Comment: UD (now on pbd 10) syncs this time. 96Jun26 17:10 Job End #6600 *96Jun26 Down-time category:TST 14:56-17:10 (C. Lewis) 96Jun26 17:11 Comment: running systest 96Jun26 17:12 Comment: getting errors: S-record file open error can't open defctrl.mcs in hcbLoadXil() and HCB timeouts. 96Jun26 17:14 Correlator rebooted. Area: code Reason: errors 96Jun26 17:16 Comment: getting missed tick errors. Window into v47c has hung. 96Jun26 17:17 Comment: Going to try rebooting one more time before I call Chuck. 96Jun26 17:18 Correlator rebooted. Area: code Reason: error 96Jun26 17:20 Comment: Rebooted OK, though I did get a "Timed out on 132 ms tic" error. Running systest. 96Jun26 17:33 Comment: Systest ran fine & came out clean. 96Jun26 17:37 Correlator rebooted. Area: jbenson Reason: test jobs ------------------------------------------------------------ 96Jun26 17:39 Job Start #172 96Jun26 17:41 Comment: LA in SEARCH. NL has weak track 33. PT has low weights, high parities on most tracks. OV has high parities on multiple tracks, min. weight about .65. 96Jun26 17:45 Comment: Just had a power failure. 96Jun26 18:15 Comment: I've reset the MG and things are back up again. *96Jun26 Down-time category:PWR 17:39-18:15 96Jun26 18:15 Correlator rebooted. Area: jbenson Reason: power failure ------------------------------------------------------------ 96Jun26 18:17 Job Start #172 96Jun26 18:22 Comment: Runs just as it did before it was rudely interrupted. 96Jun26 18:28 Comment: NL track 33 much improved by jt 20:44. 96Jun26 18:31 Comment: LA now in slew, jt 20:49 ========================================== 96Jun26 18:45 Operator On Duty: T. Briscoe 96Jun26 18:45 Job End #172 ------------------------------------------------------------ 96Jun26 18:47 Job Start #148 96Jun26 19:01 Job End #148 96Jun26 19:09 Correlator rebooted. Area: CODE Reason: Production *96Jun26 Down-time category: 19:09 21:20 ------------------------------------------------------------ 96Jun26 19:45 Job Start #2642 96Jun26 20:17 Job End #2642 ------------------------------------------------------------ 96Jun26 20:19 Job Start #2643 96Jun26 20:44 Job End #2643 ------------------------------------------------------------ 96Jun26 21:12 Job Start #5740 96Jun26 21:45 Job End #5740 ------------------------------------------------------------ 96Jun26 21:46 Job Start #5741 96Jun26 22:10 Job End #5741 ------------------------------------------------------------ 96Jun26 22:11 Job Start #5742 96Jun26 22:36 Job End #5742 96Jun26 22:38 Comment: Switching to archive disk 3 ------------------------------------------------------------ 96Jun26 22:38 Job Start #5743 Hmm.. this is very strange.. no significant errors, but the job only wrote 8 MB (151 MB expected). Also, I noticed that all but two of the stations were disabled (below the horizon?).. min wt of 2.00 96Jun26 23:01 Job End #5743 ------------------------------------------------------------ 96Jun26 23:02 Job Start #5744 96Jun26 23:28 Job End #5744 96Jun27 00:03 Comment: The job ended with the following: recs writn, flagd, lost = 1692 38262 20 ------------------------------------------------------------ 96Jun26 23:37 Job Start #5745 96Jun27 00:02 Job End #5745