========================================== 96May17 00:06 Operator On Duty: switz ------------------------------------------------------------ 96May17 00:06 Job Start #1391 96May17 00:31 Job End #1391 ------------------------------------------------------------ 96May17 00:31 Job Start #1392 96May17 00:55 Job End #1392 ------------------------------------------------------------ 96May17 00:55 Job Start #1393 96May17 01:19 Job End #1393 ------------------------------------------------------------ 96May17 01:19 Job Start #1394 96May17 01:45 Job End #1394 *96May17 Down-time category:OHD 01:45-02:08 ------------------------------------------------------------ 96May17 02:08 Job Start #3922 96May17 02:58 Job End #3922 ------------------------------------------------------------ 96May17 02:58 Job Start #3923 96May17 03:44 Job End #3923 ------------------------------------------------------------ 96May17 03:44 Job Start #3924 96May17 04:31 Job End #3924 ------------------------------------------------------------ 96May17 04:31 Job Start #3925 96May17 05:19 Job End #3925 ------------------------------------------------------------ 96May17 05:19 Job Start #3926 96May17 05:56 Job End #3926 96May17 05:57 Comment: systest 96May17 06:18 Correlator rebooted. Area: code Reason: post systest ------------------------------------------------------------ 96May17 06:26 Job Start #4020 96May17 06:45 Comment: KP, SC and MK all doing poorly. KP showing high numbers of bad headers on tracks 24, 25, 29 & 33. ========================================== 96May17 06:48 Operator On Duty: Betty Ragan 96May17 06:50 Job End #4020 $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $ 96May17 06:51 Playback Tape Move Report $ Proj. Code: BT022 $ Job Number: 4020/4021 $ Station: SC $ VSN: VLBA1127 $ Recorder: 1 $ Cause for move: poor playback $ Move From PBD: 7 $ To PBD: 11 $ Result of move: no help $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ *96May17 Down-time category:TAP 06:50-07:03 ------------------------------------------------------------ 96May17 07:03 Job Start #4021 96May17 07:15 Comment: KP, MK, SC still having problems. SC has high parities on tracks 7-9, 15-22. I will write up the site recorder for SC. KP has already been written up. 96May17 07:37 Job End #4021 96May17 07:38 Comment: running systest 96May17 07:54 Correlator rebooted. Area: code Reason: post-systest 96May17 07:55 Comment: Oops. Hadn't realized Stephan had already run the systest. ------------------------------------------------------------ 96May17 07:56 Job Start #4022 96May17 07:58 Comment: KP, MK, SC still poor. Will move MK for next job. 96May17 08:20 Job End #4022 $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $ 96May17 08:21 Playback Tape Move Report $ Proj. Code: BT022 $ Job Number: 4022/4023 $ Station: MK $ VSN: VLBA0525 $ Recorder: 1 $ Cause for move: poor playback $ Move From PBD: 10 $ To PBD: 12 $ Result of move: no better $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ *96May17 Down-time category:TAP 08:20-08:34 ------------------------------------------------------------ 96May17 08:34 Job Start #4023 96May17 08:47 Comment: MK is worse. Mow it's even having trouble staying in sync, at least at the beginning of the job. 96May17 09:07 Job End #4023 ------------------------------------------------------------ 96May17 09:08 Job Start #4024 96May17 09:08 Job #4024 Killed by op. vacuum loss on pbd 12 ------------------------------------------------------------ 96May17 09:13 Job Start #4024 96May17 09:18 Comment: Playback problems continue. 96May17 09:36 Job End #4024 ------------------------------------------------------------ 96May17 09:36 Job Start #4025 96May17 10:00 Job End #4025 ------------------------------------------------------------ 96May17 10:00 Job Start #4026 96May17 10:23 Job End #4026 ------------------------------------------------------------ 96May17 10:23 Job Start #4027 96May17 10:47 Job End #4027 ------------------------------------------------------------ 96May17 10:47 Job Start #4028 96May17 11:11 Job End #4028 ------------------------------------------------------------ 96May17 11:11 Job Start #4029 96May17 11:35 Job End #4029 ------------------------------------------------------------ 96May17 11:35 Job Start #4030 96May17 11:59 Job End #4030 Bus Error Program Counter: 0x00016348 Status Register: 0x3018 Access Address : 0xeeeeeeee Special Status : 0x0525 Task: 0x131bbd0 "stnTask_05c" tt ec30 _vxTaskEntry +10 : _stnTask (1737f74, 1393fdc, 2, 0, 0, 0, 0, 0, 0, 0) 1e034dc _stnTask +1332: _ctskTaskDelet ([14c7d00, 0, 0, 0, 0]) 1edd0e0 _ctskTaskDelet +40 : _ctskNameUniq (1f08240, 0, 4, 131b5d8) 1edbc8a _ctskNameUniq +1d6: _taskNameToId (131b5d8) 326c6 _taskNameToId +1e : _qFifoEach (&_activeQHead, 326ea, 131b5d8) 2eb34 _qFifoEach +1e : 326ea (14d44c4, 131b5d8) 32704 _taskNameToId +5c : _strcmp ([eeeeeeee, 131b5d8, 131b5d8, 131b55c, 2eb36]) 96May17 12:00 Comment: Got hung waiting error. *96May17 Down-time category:TST 12:00-13:25 (B. Rowan) 96May17 13:26 Correlator rebooted. Area: code Reason: production ------------------------------------------------------------ 96May17 13:29 Job Start #4031 96May17 13:53 Job End #4031 ------------------------------------------------------------ 96May17 13:53 Job Start #4032 96May17 13:57 Comment: Same playback problems continue, but now FD is also having problems: popping in and out of sync and showing weights of zero. 96May17 14:16 Job End #4032 ------------------------------------------------------------ 96May17 14:16 Job Start #4033 96May17 14:18 Comment: FD fine this job. 96May17 14:40 Job End #4033 *96May17 Down-time category:TST 14:41-15:20 (J. Greenberg) *96May17 Down-time category:OHD 15:20-15:25 96May17 15:25 Correlator rebooted. Area: code Reason: production ------------------------------------------------------------ 96May17 15:27 Job Start #4120 96May17 15:29 Comment: FD has low weights; I think this is a known problem. 96May17 15:33 Job End #4120 ------------------------------------------------------------ 96May17 15:34 Job Start #4121 96May17 15:51 Job End #4121 ------------------------------------------------------------ 96May17 15:53 Job Start #4122 96May17 15:54 Comment: PT in SEARCH until jt 7:31, though it has bouncy weights for a while once it syncs. 96May17 16:19 Job End #4122 ------------------------------------------------------------ 96May17 16:20 Job Start #4123 96May17 16:21 Comment: PT OK now. 96May17 16:42 Job End #4123 96May17 16:43 Comment: Joe ran the systest earlier, so I'm not going to repeat it now. 96May17 16:44 Correlator rebooted. Area: code Reason: thought I needed to change areas here ------------------------------------------------------------ 96May17 16:46 Job Start #2120 96May17 16:47 Comment: MC has dead tracks 8 & 22. 96May17 16:54 Job End #2120 *96May17 Down-time category:OHD 16:54-17:02 ------------------------------------------------------------ 96May17 17:02 Job Start #2125 96May17 17:22 Comment: J.t. 22:54, MC & LA have low weights, high parities. 96May17 17:30 Comment: MC better by 23:10. 96May17 17:43 Comment: LA better by 23:37, has low weights again on the last pass. 96May17 17:55 Job End #2125 96May17 17:57 Correlator rebooted. Area: browen Reason: test jobs ------------------------------------------------------------ 96May17 18:00 Job Start #1701 96May17 18:00 Job #1701 Killed by system. :no tape table row has vsn" Only FD and KP show up on the job monitor screen. The other two stations have question marks in place of station IDs. ------------------------------------------------------------ 96May17 18:01 Job Start #1702 96May17 18:01 Job #1702 Killed by system. Same deal. Skipping these tests. *96May17 Down-time category:OHD 18:02-18:14 96May17 18:09 Correlator rebooted. Area: code Reason: prod. ------------------------------------------------------------ 96May17 18:14 Job Start #3820 96May17 18:18 Comment: BR, SC, HN have low weights. SC already has a MAINT form on it. 96May17 18:20 Comment: Just got a lot of messages of the form "writing visibilities" xx "ticks behind." The number was in the double digits, but was down to 8 before the errors stopped. 96May17 18:30 Job End #3820 $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $ 96May17 18:30 Playback Tape Move Report $ Proj. Code: BD031 $ Job Number: 3820/3821 $ Station: BR $ VSN: VLBA0253 $ Recorder: 1 $ Cause for move: low min. weights $ Move From PBD: 4 $ To PBD: 11 $ Result of move: $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $ 96May17 18:30 Playback Tape Move Report $ Proj. Code: BD031 $ Job Number: 3820/3821 $ Station: HN $ VSN: SVLB0172 $ Recorder: 1 $ Cause for move: low min. weights $ Move From PBD: 7 $ To PBD: 12 $ Result of move: $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ *96May17 Down-time category:TAP 18:30-18:38 ------------------------------------------------------------ 96May17 18:38 Job Start #3821 ========================================== 96May17 18:44 Operator On Duty: switz 96May17 18:45 Comment: HN showing marginal parity and bad headers, all tracks. BR showing the same and high parity on track 2. 96May17 18:55 Comment: HN and BR both written up. Maint form numbers 16795 & 16796. 96May17 19:05 Job End #3821 ------------------------------------------------------------ 96May17 19:08 Job Start #3822 96May17 19:26 Job End #3822 ------------------------------------------------------------ 96May17 19:26 Job Start #3823 96May17 19:42 Comment: BR and HN look good this forward pass. 96May17 19:47 Job End #3823 ------------------------------------------------------------ 96May17 19:47 Job Start #3824 96May17 19:55 Comment: a bunch of error messages just got dumped to checker, of the form: writing visibilities N ticks behind. Where N started as 60 and decreased by about 1 per error message. 96May17 20:05 Job End #3824 ------------------------------------------------------------ 96May17 20:05 Job Start #3825 96May17 20:23 Job End #3825 ------------------------------------------------------------ 96May17 20:23 Job Start #3826 96May17 20:37 Comment: another swarm of N ticks behind messages. 96May17 20:40 Job End #3826 *96May17 Down-time category:OHD 20:40-21:30 96May17 21:27 Comment: systest ------------------------------------------------------------ 96May17 21:40 Job Start #3630 96May17 22:01 Comment: FD is off in search with filtered time troubles. 96May17 22:07 Job End #3630 ------------------------------------------------------------ 96May17 22:11 Job Start #3631 96May17 22:36 Comment: FD is in stall. 96May17 23:39 Job End #3631 ------------------------------------------------------------ 96May17 23:39 Job Start #3632 96May18 00:45 Job End #3632