From analysts@aoc.nrao.edu Wed Jan 4 10:15:08 2006 Date: Wed, 04 Jan 2006 17:15:05 +0000 From: Data Analysts To: vlbiobs@aoc.nrao.edu Subject: [Fwd: Re: [IVS-ops] MV-3 GGAO Stop Message] [ Part 2: "Included Message" ] Date: Thu, 22 Dec 2005 10:35:10 -0500 From: Ed Himwich To: Dan Smythe Cc: mpoirier@haystack.mit.edu, Craig Walker , Data Analysts Subject: Re: [IVS-ops] MV-3 GGAO Stop Message Hi, Mike is apparently out this week. We can try to resolve this after the end of the year festivities. As a warning to the correlator: It is not clear what happened, but apparently only one disk pack was sent from Gg. So there is probably no data after 2005.349.14:37:10.18. However, the naive reading of the log file is that the disk pack was not changed and something was written in the middle of the pack some time after 2005.349.14:37:10.18. It seems very unlikely that this actually happen, but the situation was apparently complicated and some operational assumptions that the FS makes (like there is a second pack installed and the Mark5 is run with formal parsing) were not meet. I would like to find out what really happened so that the FS can be fixed to properly record what is happening in cases like this. It is easier to learn from mistakes if you can see a record of them. :-) Merry Xmas, Ed Dan Smythe wrote: > Ed, > Mike P. was on the 'phone with Skip during this activity. > Maybe he can help you figure out what happened. > Try calling him at 781-981-5554. > ... Dan > > At 01:55 PM 12/21/2005, you wrote: > > > GGAO VLBI Operator wrote: > > > > > Missed scans from 18:06:49 to 21:00:23 due to antenna controller > > > failure. > > > Missed scans from 15:14:26 to 16:19:45 due to filled disk pack. > > > Replaced disk pack, but port not open. Rebooted disk pack at > > > 16:39:40. > > > Code 4 error message on FS, but all other indicators show disk pack > > > recording. > > > Skip Gordon MV-3 GGAO > > > > > > I haven't quite been able to figure out what happened there from the > > log. It seems like disk pack filled and (maybe?) there was no pack in > > the second bank for it to switch to. Then (maybe?) a new pack was > > inserted. Then (maybe?) the Mark5 was rebooted and restarted with > > informal parsing (old out-of-date configuration) instead of formal > > parsing. There is some indication that it eventually started recording > > again at the end of some already partially recorded disk pack, but there > > is no indication in the log what that disk pack was. In fact the literal > > information seems to imply that it kept recording on the original pack > > starting somewhere in the middle, but that is a wild guess. Do you know > > what happened? The scan_checks don't come out right at the end, but > > maybe that is because of informal parsing. > > > > > From analysts@aoc.nrao.edu Wed Jan 4 10:15:22 2006 Date: Wed, 04 Jan 2006 17:15:19 +0000 From: Data Analysts To: vlbiobs@aoc.nrao.edu Subject: [Fwd: Re: [IVS-ops] MV-3 GGAO Stop Message RDV54] [ Part 2: "Included Message" ] Date: Thu, 22 Dec 2005 11:38:47 -0500 From: Ed Himwich To: Craig Walker , Data Analysts Cc: Mike Poirier , Dan Smythe , Jay R. Redmond , Sessions , Pete Bolis Subject: Re: [IVS-ops] MV-3 GGAO Stop Message RDV54 Hi, Mike called me on the phone. We talked about it a bit. I guess the best we can say is that it is a mystery. I will be interested to hear what the correlator finds on this disk. It may have been that the old disk was withdrawn while the mark5 was trying to write to it. With one disk full and the second slot empty, I *think* the correct process would have been to insert a new pack in the empty bank and turned the key switch to "locked" when the system was not recording and will be idle for the next 20 seconds. At the next scan set-up, the FS should then switch to the new pack. The old disk could then have been "unlocked" and removed at the next convenient opportunity when the system was not recording. This process might be clearer if the FS did not issue the change pack instructions until it finds a disk successfully in the new pack to record to. However, if the pack is full there needs to be an indication that some action is needed, so instead maybe a second message saying to insert a new pack in the other bank before removing the old pack is needed. (?) After thinking about the situation a little more I am a loss to explain why another VSN does not appear in the RDV54 log if another pack was in fact inserted. The logic in the FS is such that logging the VSN is inhibited *only* if the Mark5 responds that a VSN that is the same as the previous one reported (in fact we had a bug for a while where it reported already recorded VSNs on each scan). So I am reduced to thinking that through some complicated sequence of problems that the original disk was partially overwritten is some way. I am hoping not. Mike did mention that Seth reported that he was getting a low voltage warning for the Mark5 power supply. Perhaps this contributed to the strange behavior at this time. Mile is in the process of asking Pete to send a replacement. Ed Mike Poirier wrote: > Ed, > I am in!! Been sick!! > > GGAO problem, I was on the phone with Skip when he informed me that the > there was a mk5 error occurring on the field system. I asked him if he > could > check the position on the disk pack and he could not. He told me that the > record lights were not going out between scans. He also informed me that > the > previous disk pack indicated that it was full. He changed that disk pack > and > this problems has been occurring. At that point we hard booted the mk5 and > it seemed to record normally. > > > Mike > > -----Original Message----- > From: Ed Himwich [mailto:weh@ivscc.gsfc.nasa.gov] Sent: Thursday, December > 22, 2005 10:35 AM > To: Dan Smythe > Cc: mpoirier@haystack.mit.edu; Craig Walker; Data Analysts > Subject: Re: [IVS-ops] MV-3 GGAO Stop Message > > Hi, > > Mike is apparently out this week. We can try to resolve this after the end > of the year festivities. > > As a warning to the correlator: It is not clear what happened, but > apparently only one disk pack was sent from Gg. So there is probably no > data after 2005.349.14:37:10.18. > > However, the naive reading of the log file is that the disk pack was not > changed and something was written in the middle of the pack some time > after 2005.349.14:37:10.18. It seems very unlikely that this actually > happen, but the situation was apparently complicated and some operational > assumptions that the FS makes (like there is a second pack installed and > the Mark5 is run with formal parsing) were not meet. I would like to find > out what really happened so that the FS can be fixed to properly record > what is happening in cases like this. It is easier to learn from mistakes > if you can see a record of them. :-) > > Merry Xmas, Ed > > Dan Smythe wrote: > > > Ed, > > Mike P. was on the 'phone with Skip during this activity. > > Maybe he can help you figure out what happened. > > Try calling him at 781-981-5554. > > ... Dan > > > > At 01:55 PM 12/21/2005, you wrote: > > > > > > > GGAO VLBI Operator wrote: > > > > > > > > > > Missed scans from 18:06:49 to 21:00:23 due to antenna controller > > > > failure. > > > > Missed scans from 15:14:26 to 16:19:45 due to filled disk pack. > > > > Replaced disk pack, but port not open. Rebooted disk pack at > > > > 16:39:40. > > > > Code 4 error message on FS, but all other indicators show disk pack > > > > recording. > > > > Skip Gordon MV-3 GGAO > > > > > > > > > I haven't quite been able to figure out what happened there from the > > > log. It seems like disk pack filled and (maybe?) there was no pack in > > > the second bank for it to switch to. Then (maybe?) a new pack was > > > inserted. Then (maybe?) the Mark5 was rebooted and restarted with > > > informal parsing (old out-of-date configuration) instead of formal > > > parsing. There is some indication that it eventually started recording > > > again at the end of some already partially recorded disk pack, but > > > there is no indication in the log what that disk pack was. In fact the > > > literal information seems to imply that it kept recording on the > > > original pack starting somewhere in the middle, but that is a wild > > > guess. Do you know what happened? The scan_checks don't come out right > > > at the end, but maybe that is because of informal parsing. > > > > > > > > > > > > >