========================================== 05Oct14 15:16 Operator On Duty: Alan Kerr 05Oct14 15:41 Comment: RDV53 pilot ------------------------------------------------------------ 05Oct14 15:42 Job Start #1524 05Oct14 15:52 Comment: WF, OSOD-109, is in GETVSN. looks like some sort of "O" vs "0" problem. killing the job to unload and load PBD16. 05Oct14 15:53 Job # Killed by operator ------------------------------------------------------------ 05Oct14 15:54 Job Start #1524 (2nd) 05Oct14 16:09 Comment: jt 2301. KK1 on PBD13 lost vacuum at BOT when it "unloaded" rather than stopping! killing job 05Oct14 16:10 Job # Killed by operator ------------------------------------------------------------ 05Oct14 16:39 Job Start #1524 (3rd) 05Oct14 16:45 Comment: jt 2243. KK1 solid, was in Search for a while. R@7k HP79 All. 05Oct14 16:56 Comment: jt 2305. all four non-vlba disk sites (WF,WZ,GG,ON) will not stay in the job. on each "scan row" they all sync up for about 30-60 seconds and then drop to START. letting it run so that analysts can look at it on monday. 05Oct14 17:26 Job End #1524 ============================================================ 05Oct18 20:06 Job Start #1520 05Oct18 20:10 Comment: jt 1804. GC1 all tracks, 40-60% INVAL. F@3k HP-350 All. (YES! HP is -350 and *not* -319 like the first Fwd pass should be). 05Oct18 20:18 Comment: jt 1820. GC1 solid. F@9k. 05Oct18 20:55 Job End #1520 ------------------------------------------------------------ 05Oct18 20:55 Job Start #1521 05Oct18 20:58 Comment: KK1 failed to prime: expected from script. 05Oct18 21:12 Comment: jt 2000. boht WZ5 and GG5 are in WAIT and have been since the start of the job. both are "new disks" for this job. killing job to restart job. 05Oct18 21:15 Job # Killed by operator ------------------------------------------------------------ 05Oct18 21:25 Job Start #1521 (2nd) 05Oct18 21:26 Comment: jt 1934. again, WZ5 and GG5 in WAIT. the System Media window has the "motion" as "A:" for both... they SHOULD be "PLA" or "ARM". killing job again. 05Oct18 21:28 Job # Killed by operator 05Oct18 21:29 Comment: I have "keyed-off" and then "keyed-on" both the WZ5 and GG5 disks. ------------------------------------------------------------ 05Oct18 21:30 Job Start #1521 (3rd) 05Oct18 21:35 Comment: jt 1942. GG5, WZ5 as before. GG5 should be in the job already. WZ5 should join at 1959. but both are still in WAIT and not in START. ...letting it run. 05Oct18 21:46 Comment: jt 2005. I've used the ListModule command and both diskpacks have data at the times that they should. only thing left that I can think of is that both PBD22 and PBD23 need to be rebooted. calling Walter first. killing job. 05Oct18 21:48 Job # Killed by operator 05Oct18 22:00 Comment: have contacted Walter... He's looking into the problem. ------------------------------------------------------------ 05Oct18 22:19 Job Start #1521 (4th) 05Oct18 22:20 Comment: GG5, WZ5 still in WAIT. 05Oct18 22:21 Job # Killed by operator 05Oct18 22:21 Comment: going to next job to see if the disks will work in that job. ------------------------------------------------------------ 05Oct18 22:22 Job Start #1522 05Oct18 22:40 Comment: jt 2110. KP6 in Wait. Array log log says that KP changed packs at jt 2104. this job should be 2 jobs. 05Oct18 22:45 Job End #1522 05Oct18 22:50 Comment: OK... job 1521, two packs will not join the job. job 1522, KP changes packs in mid-job and needs to be split into two jobs... I give up on this project. going to take tapes down and run BR100AB and let Walter and analysts look at things in the morning. ------------------------------------------------------------ 05Oct19 13:49 Job Start #1521 05Oct19 13:50 Comment: Well WZ and GG are in wait again. I will keep an eye on them. 05Oct19 14:36 Job End #1521 ------------------------------------------------------------ 05Oct19 14:37 Job Start #1522 05Oct19 14:37 Comment: Well WZ and GG are now running, and they look good. 05Oct19 14:40 Comment: ON and WZ just dropped out. 05Oct19 14:41 Job End #1522 ------------------------------------------------------------ 05Oct19 14:46 Job Start #1523 05Oct19 14:47 Comment: KP is in wait. I will watch it. 05Oct19 14:53 Comment: KP is now solid and just fine. 05Oct19 15:35 Job End #1523 05Oct19 15:35 Comment: Job #1524 was run as the pilot. Going on to 1525. ------------------------------------------------------------ 05Oct19 15:39 Job Start #1525 ========================================== 05Oct19 15:59 Operator On Duty: Alan Kerr 05Oct19 15:59 Job End #1525 ------------------------------------------------------------ 05Oct19 16:08 Job Start #1526 05Oct19 16:46 Job End #1526 ------------------------------------------------------------ 05Oct19 16:46 Job Start #1527 05Oct19 17:37 Job End #1527 ------------------------------------------------------------ 05Oct19 17:37 Job Start #1528 05Oct19 17:42 Job End #1528 05Oct19 17:42 Comment: OK this is weird... The project wants GC1, HST00182, AK18 next. I have the tape up. the "small label" says HST00182 but the VSN sticker says HST00474! using tapeassvsn to "trick" the system. ------------------------------------------------------------ 05Oct19 17:48 Job Start #1529 05Oct19 17:51 Comment: GC1 failed to prime. expected from a note on the tape itself. 05Oct19 17:59 Comment: jt 0336. GC1 in Search. miniD finding nothing. F@2k HP-350 (yes, -350) All. 05Oct19 18:18 Job End #1529 05Oct19 18:19 Correlator rebooted. Area:code Reason: systest 05Oct19 18:23 Comment: running systest 05Oct19 18:48 Comment: systest NOT CLEAN, rerunning. error is in Channel 6 between FFT03 and FFT10. 05Oct19 18:48 Correlator rebooted. Area:code Reason: systest 05Oct19 18:53 Comment: re-running systest 05Oct19 19:07 Comment: systest still NOT CLEAN. same error. FFT03 shut off. Work Order is 117066. 05Oct19 19:07 Correlator rebooted. Area:code Reason: queue ------------------------------------------------------------ 05Oct19 19:18 Job Start #1530 05Oct19 19:21 Comment: GC1 failed to prime. 05Oct19 19:23 Comment: jt 0403. GC1 in Search. miniD finding nothing. F@13k. 05Oct19 19:27 Comment: jt 0411. ON5 dropped out. 05Oct19 19:29 Job End #1530 ------------------------------------------------------------ 05Oct19 19:30 Job Start #1531 05Oct19 19:31 Comment: GC1 failed to prime 05Oct19 19:35 Comment: jt 0423. GC1 in Search. R@17k HP0 (yes "0"). 05Oct19 19:36 Job End #1531 ------------------------------------------------------------ 05Oct19 19:36 Job Start #1532 05Oct19 19:39 Comment: GC1 failed to prime 05Oct19 19:41 Comment: jt 0426. GC1 in Search. R@15k. 05Oct19 20:02 Comment: jt 0506. bizarre... GC1 had stopped at BOT and is sitting in WAIT. there is a fwd pass in a few seconds... 05Oct19 20:05 Comment: jt 0513. GC1 in Search F@2k HP-295. 05Oct19 20:06 Comment: jt 0514. TC1 dropped out. 05Oct19 20:10 Job End #1532 ------------------------------------------------------------ 05Oct19 20:10 Job Start #1533 05Oct19 20:14 Comment: GC1 failed to prime. 05Oct19 20:21 Comment: cross bar cannot be set. moving TC1 from PBD05 to PBD09 05Oct19 20:21 Job # Killed by operator ------------------------------------------------------------ 05Oct19 20:24 Job Start #1533 (2nd) 05Oct19 20:29 Comment: jt 0523. GC1 in Search. 05Oct19 21:13 Job End #1533 ------------------------------------------------------------ 05Oct19 21:13 Job Start #1534 05Oct19 21:19 Comment: GC1: no prime: in Search. 05Oct19 21:21 Job End #1534 ------------------------------------------------------------ 05Oct19 21:21 Job Start #1535 05Oct19 21:29 Comment: GC1... 05Oct19 22:05 Job End #1535 ------------------------------------------------------------ 05Oct19 22:05 Job Start #1536 05Oct19 22:09 Comment: GC1... 05Oct19 22:43 Comment: OBJ_UNAVAILABLE!!! rebooting and rerunning. 05Oct19 22:44 Correlator rebooted. Area:code Reason: clear error ------------------------------------------------------------ 05Oct19 22:49 Job Start #1536 (2nd) 05Oct19 22:53 Comment: GC1 continues. I've never seen the miniD find ANY time. Power has always been about 0.2. This tape seems to be blank. 05Oct19 23:29 Job End #1536 ------------------------------------------------------------ 05Oct19 23:29 Job Start #1537 05Oct19 23:32 Comment: GC1... ========================================== 05Oct19 23:57 Operator On Duty: Betty Ragan 05Oct20 00:12 Job End #1537 ========================================== 05Oct20 00:13 Operator On Duty: Betty Ragan 05Oct20 00:14 Correlator rebooted. Area: code Reason: systest 05Oct20 00:16 Comment: I got half of "opsSystest" typed, and the v47c window hung on me. I'll push the red button and reboot. 05Oct20 00:21 Comment: OK, apparently the problem was with Reber. It got these "Timer expired" messages, and everything froze for a while. Seems to be OK now. 05Oct20 00:21 Correlator rebooted. Area: code Reason: systest 05Oct20 00:25 Comment: Running systest 05Oct20 00:37 Comment: Same problem as before. FFT 3 is still shut off. 05Oct20 00:37 Correlator rebooted. Area: code Reason: post-systest ------------------------------------------------------------ 05Oct20 00:42 Job Start #1538 05Oct20 00:47 Comment: GC1 continues in SEARCH. 05Oct20 00:56 Job End #1538 ------------------------------------------------------------ 05Oct20 00:56 Job Start #1539 05Oct20 00:58 Comment: GC1 same. 05Oct20 01:48 Job End #1539 ------------------------------------------------------------ 05Oct20 01:48 Job Start #1540 05Oct20 01:49 Comment: GC1 same. 05Oct20 01:55 Comment: jt 12:29, ~9k ft in rev, OV1 wts. bouncing, slightly poor on all tracks. 05Oct20 01:57 Job #1540 Killed by op. $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $ 05Oct20 01:57 Playback Tape Move Report $ Job# / Project: 1540; RDV53 $ VSN / Shelf : HST00449; AD107 $ Cause for move: bouncy wts $ From Site/PBD: OV1; 2 $ To Site/PBD: 11 $ Maint Form #: $ Result of move: better $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ ------------------------------------------------------------ 05Oct20 02:03 Job Start #1540 05Oct20 02:24 Job End #1540 ------------------------------------------------------------ 05Oct20 02:24 Job Start #1541 05Oct20 02:32 Comment: GC1 dead on track 33 at start of fwd pass, jt 12:57. KK1 wts bouncing intermittently. 05Oct20 02:33 Comment: GC1 better, jt 13:00. 05Oct20 02:38 Job End #1541 $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $ 05Oct20 02:38 Playback Tape Move Report $ Job# / Project: 1541/1542; RDV53 $ VSN / Shelf : HST00354; AK11 $ Cause for move: intermittent bouncy wts $ From Site/PBD: KK1; 10 $ To Site/PBD: 12 $ Maint Form #: $ Result of move: better $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ ------------------------------------------------------------ 05Oct20 02:41 Job Start #1542 05Oct20 02:50 Job End #1542 ------------------------------------------------------------ 05Oct20 02:50 Job Start #1543 05Oct20 03:21 Job End #1543 ------------------------------------------------------------ 05Oct20 03:22 Job Start #1544 05Oct20 03:30 Job End #1544 ------------------------------------------------------------ 05Oct20 03:30 Job Start #1545 05Oct20 03:36 Comment: NL2 slow to sync. SC is in SEARCH with TapeTask error: filtered time error at targeted track. 05Oct20 03:38 Job #1545 Killed by op. $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $ 05Oct20 03:38 Playback Tape Move Report $ Job# / Project: 1545; RDV53 $ VSN / Shelf : EVNT0908; AD118 $ Cause for move: SEARCHing $ From Site/PBD: SC2; 2 $ To Site/PBD: 5 $ Maint Form #: $ Result of move: still SEARCHing $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ ------------------------------------------------------------ 05Oct20 03:42 Job Start #1545 05Oct20 03:43 Comment: SC2 popping in and out of SEARCH. 05Oct20 03:48 Comment: SC2 in sync now, jt 14:46, ~5k ft in fwd. 05Oct20 03:51 Job End #1545 ------------------------------------------------------------ 05Oct20 03:51 Job Start #1546 05Oct20 04:24 Job End #1546 ------------------------------------------------------------ 05Oct20 04:25 Job Start #1547 05Oct20 04:32 Comment: SC2 popping in and out of SEARCH again at start of fwd pass, jt 16:09. Better by ~5k ft. 05Oct20 04:46 Comment: jt 16:35, ~13k ft in fwd, SC2 now has 0 min wts, poor on odd tracks. BEtter in rev, jt 16:49. 05Oct20 05:09 Job End #1547 ------------------------------------------------------------ 05Oct20 05:09 Job Start #1548 05Oct20 05:17 Comment: SC2 still having SEARCHing problems at start of fwd pass. 05Oct20 05:20 Job End #1548 ------------------------------------------------------------ 05Oct20 05:20 Job Start #1549 05Oct20 05:22 Comment: SC2 in SEARCH. Seems to be finding time from an hour and a half behind where it should be. 05Oct20 05:31 Job End #1549 ------------------------------------------------------------ ------------------------------------------------------------ 05Oct23 17:06 Job Start #1529 05Oct23 17:11 Job # Killed by operator 05Oct23 17:11 Comment: system reports "Can't find FFT for PBD=11". (TC1). moving tapes *05Oct23 Down-time category:OHD 17:12-17:22 ------------------------------------------------------------ 05Oct23 17:22 Job Start #1529 (2nd) 05Oct23 17:23 Comment: great!! now PBD15 can't find an FFT in the Xbar... killing job. *05Oct23 Down-time category:OHD 17:24-17:27 05Oct23 17:24 Comment: actually, I'm going to swap PBD10 and 14. I *think* that will allow a legal Xbar. ------------------------------------------------------------ 05Oct23 17:27 Job Start #1529 (3rd) 05Oct23 17:40 Comment: jt 0348. NL2 track 33, 35-40% INVAL. was not seen on the first correlation. 4 of 8 channels at 0.90. R@11k HP319 All. 05Oct23 17:45 Comment: jt 0358. NL2 continues. killing job to swap... R@7k. 05Oct23 17:46 Job # Killed by operator $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $ 05Oct23 17:50 Playback Tape Move Report $ Job# / Project: 1529/RDV53 $ VSN / Shelf : VLBA1155/AD102 $ Cause for move: weak T33 $ From Site/PBD: NL2/03 $ To Site/PBD: OV2/06 $ Maint Form #: C117088 (PBD03) $ Result of move: OV2 now has the weak T33 $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ *05Oct23 Down-time category:TAP 17:46-18:18 ------------------------------------------------------------ 05Oct23 18:18 Job Start #1529 (4th) 05Oct23 18:20 Comment: jt OV2, track 33, 35-40% INVAL. R@14k HP319 All. 05Oct23 18:24 Comment: jt 0334. OV2 continues. killing job to move OV2 off of PBD03. 05Oct23 18:25 Job # Killed by operator $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $ 05Oct23 18:25 Playback Tape Move Report $ Job# / Project: 1529/RDV53 $ VSN / Shelf : SVLB0038/AD106 $ Cause for move: weak track 33 $ From Site/PBD: OV2/03 $ To Site/PBD: 12 $ Maint Form #: C117088 (PBD03) $ Result of move: solid $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ *05Oct23 Down-time category:PBD 18:25-18:43 ------------------------------------------------------------ 05Oct23 18:43 Job Start #1529 (5th) 05Oct23 19:03 Job End #1529 ------------------------------------------------------------ 05Oct23 19:04 Job Start #1530 05Oct23 19:11 Job End #1530 ------------------------------------------------------------ 05Oct23 19:11 Job Start #1531 05Oct23 19:14 Comment: NL1 failed to prime. 05Oct23 19:19 Job End #1531 ------------------------------------------------------------ 05Oct23 19:19 Job Start #1532 05Oct23 19:50 Job End #1532 05Oct23 19:50 Comment: starting shutdown. 05Oct23 20:38 Operator Off Duty: Alan Kerr ========================================== 05Oct24 07:51 Operator On Duty: Alan Kerr 05Oct24 07:55 Comment: the correlator is being worked on. 05Oct24 08:02 Comment: correlator is fixed. systest is clean. 05Oct24 08:02 Correlator rebooted. Area:code Reason: queue 05Oct24 08:25 Comment: RDV53 partial recorrelation continues. ------------------------------------------------------------ 05Oct24 08:25 Job Start #1533 05Oct24 09:14 Job End #1533 ------------------------------------------------------------ 05Oct24 09:14 Job Start #1534 05Oct24 09:18 Job End #1534 ------------------------------------------------------------ 05Oct24 09:18 Job Start #1535 05Oct24 09:26 Comment: system is locked up. no response from Checker, Correlator windows or v47c window. RED BUTTON reboot! 05Oct24 09:27 Correlator rebooted. Area:code Reason: clear HANG ------------------------------------------------------------ 05Oct24 09:38 Job Start #1535 (2nd) 05Oct24 10:21 Job End #1535 05Oct24 10:21 Comment: a compressor on the Contempos has failed and temp is rising. shutting down correlator. 05Oct24 15:37 Correlator rebooted. Area:code Reason: see if things work ------------------------------------------------------------ 05Oct24 15:45 Job Start #1536 05Oct24 15:46 Job # Killed by operator 05Oct24 15:46 Comment: still tripping the alarm. ========================================== 05Oct24 16:00 Operator On Duty: Betty Ragan 05Oct24 16:36 Comment: Well, they cleaned some connections and jiggled some wires in the aircon. It's dubious whether that'll have fixed anything, but we're going to start up a job and see how the temps go. 05Oct24 16:36 Correlator rebooted. Area: code Reason: testing ------------------------------------------------------------ 05Oct24 16:38 Job Start #1536 05Oct24 16:47 Comment: Temperatures are rising... 05Oct24 16:47 Job #1536 Killed by op. 05Oct24 16:49 Comment: Correlator has been shut down again. 05Oct24 16:59 Comment: Looks like nothing is going to be fixed until morning. 05Oct24 16:59 Operator Off Duty: Betty Ragan ========================================== 05Oct24 23:47 Operator On Duty: A Taylor 05Oct24 23:59 Comment: End Log 05Oct25 15:36 Comment: RDV53 continues ------------------------------------------------------------ 05Oct25 15:36 Job Start #1536 05Oct25 15:40 Comment: jt 0818. FD5 in PBD18A, has had Wts of 0.00 for the job so far. FD5 Dir62 05Oct25 15:43 Comment: killing job, will key-off and key-on the diskpack. 05Oct25 15:43 Job # Killed by operator ------------------------------------------------------------ 05Oct25 15:47 Job Start #1536 (2nd) ========================================== 05Oct25 15:50 Operator On Duty: Betty Ragan 05Oct25 15:50 Comment: FD5 is same. Interestingly, it only has zero wts. on one out of eight channels. 05Oct25 15:50 Job #1536 Killed by op. ------------------------------------------------------------ 05Oct25 15:53 Job Start #1536 (3rd) 05Oct25 15:56 Comment: Walter is investigating. 05Oct25 16:08 Job #1536 Killed by op. ------------------------------------------------------------ 05Oct25 17:05 Job Start #1536 05Oct25 17:06 Comment: Looks good! 05Oct25 17:26 Comment: TC1 wts bouncing, jt 8:53, start of fwd pass. Mostly better by ~4k ft, though still dropping very briefly once in a while. 05Oct25 17:37 Job End #1536 ------------------------------------------------------------ 05Oct25 17:38 Job Start #1537 05Oct25 18:18 Job End #1537 ------------------------------------------------------------ 05Oct25 18:19 Job Start #1538 05Oct25 18:27 Job End #1538 ------------------------------------------------------------ 05Oct25 18:27 Job Start #1539 05Oct25 18:45 Comment: TC1 bouncing briefly from time to time again. 05Oct25 19:17 Job End #1539 ------------------------------------------------------------ 05Oct25 19:18 Job Start #1540 05Oct25 19:30 Job End #1540 ------------------------------------------------------------