By Craig ------------------------- Sept. 4, 2007 ----------------------------- I am trying to read the Mark5B logs with fs2db. With the original log, I use these inputs: *inp Inputs for fs2db ================ (DATADESCRIP)========= antenna. . . . . WF proposal . . . . RD0606 session. . . . . uttimerange. . . 2006AUG25 at 16:20:00 to 2006AUG26 at 18:05:00 (DATAADDED)=========== polarization . . * lo_offset. . . . 0 (DATALOCATION)======== logdirectory . . /home/vlbiobs/astronomy/aug06/rd0606 logfile. . . . . rd06065b.log database . . . . ORACLE/VLBA verbose. . . . . NO And get these messages: *go fs2db EXECUTION ===== ========= Reading /home/vlbiobs/astronomy/aug06/rd0606/rd06065b.log Reading Field System log file /home/vlbiobs/astronomy/aug06/rd0606/rd06065b.log Data being stored for station WF Field system version 9.7.7 started 2006.235.16:26:19.17 2006.235.16:26:19.17;LOCATION,MK5-665 ,76.83,39.02,24.5 Line does not begin with valid day/time DRIVERVERSION 801 ; Disk HOB-0002 begins at 23-AUG-2006 16:26:22 Truncated line: ' ' Illegal bitrate at 2006.235.17:59:50.00: 0 You have mail in /var/mail/analysts shire.analysts$ pwd I think the mail message is unrelated. There is a blank line in the input file. Remove it and try again. *go fs2db EXECUTION ===== ========= Reading /home/vlbiobs/astronomy/aug06/rd0606/rd06065b.log Reading Field System log file /home/vlbiobs/astronomy/aug06/rd0606/rd06065b.log Data being stored for station WF Field system version 9.7.7 started 2006.235.16:26:19.17 2006.235.16:26:19.17;LOCATION,MK5-665 ,76.83,39.02,24.5 Line does not begin with valid day/time DRIVERVERSION 801 ; Disk HOB-0002 begins at 23-AUG-2006 16:26:22 Illegal bitrate at 2006.235.17:59:50.00: 0 The line without a valid day/time must come before the specification of the disk. There is a DRIVERVERSION 801 ; in the file - at the end of a very long line. Is that line just too long and is wrapped? Try splitting the line in two. *go fs2db EXECUTION ===== ========= Reading /home/vlbiobs/astronomy/aug06/rd0606/rd06065b.log Reading Field System log file /home/vlbiobs/astronomy/aug06/rd0606/rd06065b.log Data being stored for station WF Field system version 9.7.7 started 2006.235.16:26:19.17 2006.235.16:26:19.17;LOCATION,MK5-665 ,76.83,39.02,24.5 Disk HOB-0002 begins at 23-AUG-2006 16:26:22 Illegal bitrate at 2006.235.17:59:50.00: 0 You have mail in /var/mail/analysts That time is when the scan turns on. I think the format may not be getting set. Add the second of the two lines below whereever the first appears. 2006.235.18:00:43.69&setupsx/"form=m,16.000,1:2 2006.235.18:00:43.69&setupsx/form=m,16.000,1:2 Now it ran all the way through the schedule. It looks like it might have worked, but it says at the end that: 0 scans processed The log file has many entries of the sort: WARNING - :ST with no preceeding ET at 2006.235.18:05:06.00 Null observation ignored at 2006.235.18:05:56.00 LO1=8.080 GHz USB R LO2=2.020 GHz USB R LO3=8.580 GHz USB R The time is a scan stop time (disk_record=off and data_valid=off). In fs2db, that line is triggered if numChan==0. So I guess it's not picking up the channels. I have to go for the day. Get back to this and see if there is something else in the setupsx or something that starts with " that shouldn't. There are a number of possibilities like "tracks. --------------------- Sept. 6, 2007 -------------------------- Copied the tracks lines and unquoted. 2006.235.16:26:19.18&setupsx/"tracks=v0,v1,v2,v3 2006.235.16:26:19.18&setupsx/tracks=v0,v1,v2,v3 That didn't help. It seems that fs2db needs the association of tracks and bbc's. Add trkfsx specification by uncommenting trkfsx and adding the definition of trkfsx 2006.235.16:26:19.18&setupsx/"trkfsx 2006.235.16:26:19.18&setupsx/trkfsx later - copied from the wf file, add (right after &pcalon): 2006.235.17:32:29.89&trkfsx/trackform= 2006.235.17:32:29.89&trkfsx/trackform=2,1us,6,1ls,10,2us,14,3us,18,4us,22,5us,26,6us,30,7us 2006.235.17:32:29.89&trkfsx/trackform=3,8us,7,8ls,11,9us,15,10us,19,11us,23,12us,27,13us 2006.235.17:32:29.89&trkfsx/trackform=31,14us Now fs2db runs and generates scans. Try running cjobgen - that will be new for me. --------------------------- Sept. 12, 2007 -------------------- I'll try cjobgen. First need to run sub2db. Inputs for sub2db ================= (JOB_PARAMETERS)====== proposal . . . . RD0606 uttimerange. . . 2006AUG25 at 16:20:00 to 2006AUG26 at 18:05:00 database . . . . ORACLE/VLBA tempspace. . . . /tmp (SUBFIND_PARAMETERS)== timedelta. . . . 5 angledelta . . . -1 freqdelta. . . . 15 refant . . . . . * maxchan. . . . . 32 bandwidthmix . . NO * ERROR - No data found To proceed against the advice of software, type "Y " (Anything else to return to inputs): harumph I thought I had made it work. Ran fs2db for SV Ended with 473 scans processed. ERROR - No data found To proceed against the advice of software, type "Y " (Anything else to return to inputs): Y Proposal RD0606 Starttime 25-AUG-2006 16:20:00 Stoptime 26-AUG-2006 18:05:00 No data found for proposal RD0606 *Inputs for sub2db I think I've run both stations ok. But sub2db is not finding the data. I talked to Barry about getting a summary of what's in the data base. He doesn't have anything except typing SQL commands. The analysts might have something, but they are gone for the day. I think I'll fold for the day. ------------------------- Sept. 14, 2007 --------------------- Lets try again. There is blank after the station name in the location line. Try deleting it. That went rather further before hitting the same problem. There was a second location line. Still getting: FS2DB: ORA-01480: trailing null missing from STR bind value Try editobs: (DATASELECTION)======= proposal . . . . RD0606 antenna. . . . . SV uttimerange. . . 2006AUG25 at 16:20:00 to 2006AUG26 at 18:05:00 database . . . . ORACLE/VLBA (CHANGE_ITEMS)======== barrel_roll. . . * telescope. . . . * lofreq . . . . . 0.000 copyfreq . . . . * shadow . . . . . * zap. . . . . . . YES pulsar . . . . . * segment. . . . . * filename . . . . * blackbelt. . . . NO *utt 2006AUG23 at 16:20:00 to 2006AUG24 at 18:05:00 *go editobs EXECUTION ======= ========= For station SV, for proposal RD0606, from 23-AUG-2006 16:20:00 to 24-AUG-2006 18:05:00 EDITOBS: ORA-01405: fetched column value is NULL So not sure what happened. I went and asked Barry. It seems that the problem is that the proposal code has 6 characters where the VLBA is limited to 5. Changed proposal to RD060, and it ran. 473 scans processed * Now try sub2db Inputs for sub2db ================= (JOB_PARAMETERS)====== proposal . . . . RD060 uttimerange. . . 2006AUG23 at 16:20:00 to 2006AUG24 at 18:05:00 database . . . . ORACLE/VLBA tempspace. . . . /tmp (SUBFIND_PARAMETERS)== timedelta. . . . 5 angledelta . . . -1 freqdelta. . . . 15 refant . . . . . * maxchan. . . . . 32 bandwidthmix . . NO sub2db EXECUTION ====== ========= Proposal RD060 Starttime 23-AUG-2006 16:20:00 Stoptime 24-AUG-2006 18:05:00 SUB2DB: ORA-01405: fetched column value is NULL last message is not informative. I don't know if this ran or not. Put Barry on it again. This time, he found that there are no source names in the WF log. It turns out that the source lines in the log file have been commented out. So I will do a global replace of '"source' with 'source' and try reloading wf. Ran editobs to try to remove WF. *inp Inputs for editobs ================== (DATASELECTION)======= proposal . . . . RD0606 antenna. . . . . WF uttimerange. . . 2006AUG23 at 16:20:00 to 2006AUG24 at 18:05:00 database . . . . ORACLE/VLBA (CHANGE_ITEMS)======== barrel_roll. . . * telescope. . . . * lofreq . . . . . 0.000 copyfreq . . . . * shadow . . . . . * zap. . . . . . . YES pulsar . . . . . * segment. . . . . * filename . . . . * blackbelt. . . . NO *go Then ran fs2db Inputs for fs2db ================ (DATADESCRIP)========= antenna. . . . . WF proposal . . . . RD060 session. . . . . uttimerange. . . 2006AUG23 at 16:20:00 to 2006AUG24 at 18:05:00 (DATAADDED)=========== polarization . . * lo_offset. . . . 0 (DATALOCATION)======== logdirectory . . /home/vlbiobs/astronomy/aug06/rd0606 logfile. . . . . rd06065b.log database . . . . ORACLE/VLBA verbose. . . . . YES Last few lines of output: Disk HOB-0002 begins at 23-AUG-2006 16:26:22 Scan started at 2006.235.17:59:50.00 VC 2 FR 172.990 BW 8.0000 VC 6 FR 272.890 BW 8.0000 VC 11 FR 236.990 BW 8.0000 FS2DB: ORA-00001: unique constraint (VLBA.OBSSTA_PK) violated So now we have another problem. Maybe there is still old WF data. Try editobs again with wider time range. Also change the project code. Actually that is probably all I need. (DATASELECTION)======= proposal . . . . RD060 antenna. . . . . WF uttimerange. . . 2006AUG23 at 16:20:00 to 2006AUG24 at 18:05:00 database . . . . ORACLE/VLBA (CHANGE_ITEMS)======== barrel_roll. . . * telescope. . . . * lofreq . . . . . 0.000 copyfreq . . . . * shadow . . . . . * zap. . . . . . . YES pulsar . . . . . * segment. . . . . * filename . . . . * blackbelt. . . . NO *go editobs EXECUTION ======= ========= For station WF, for proposal RD060, from 23-AUG-2006 16:20:00 to 24-AUG-2006 18:05:00 Deleting data from 23-AUG-2006 16:20:00 to 24-AUG-2006 18:05:00 (Observing occurrs from 23-AUG-2006 17:59:50 to 24-AUG-2006 18:01:01 Deleting following tape records: Tape H0B-0002 at 23-AUG-2006 16:26:22 Deleting 0 rows from meteorology table Try fs2db again, same parameters 412 scans processed Now sub2db. Inputs for sub2db ================= (JOB_PARAMETERS)====== proposal . . . . RD060 uttimerange. . . 2006AUG23 at 16:20:00 to 2006AUG24 at 18:05:00 database . . . . ORACLE/VLBA tempspace. . . . /tmp (SUBFIND_PARAMETERS)== timedelta. . . . 5 angledelta . . . -1 freqdelta. . . . 15 refant . . . . . * maxchan. . . . . 32 bandwidthmix . . NO Lots of outputs, ending in: 3C371 24-AUG-2006 16:54:35 SV 0458-020 24-AUG-2006 17:05:12 WF 1749+096 24-AUG-2006 17:06:49 SV DA426 24-AUG-2006 17:09:29 SV 1101+384 24-AUG-2006 17:21:03 SV 1308+326 24-AUG-2006 17:31:27 SV Commiting subarray Handling channel remaps (if any). Commiting channel remaps Reconciling database entries.... Program RD060 did not appear in schedule. Inserting program RD060, segment ' ', with status SCHEDULED. SUB2DB: ORA-00947: not enough values Talked to Barry again. Some time ago, they added an autorelease field to the data base. It is not unlikely that we have not had any projects that don't have VLBA antennas since. With VLBA antennas, the project is added to the schedule when the VLBA monitor data comes in. So sub2db may not have had to add a project since that field was added. Barry will look into it. But we are probably both going home tonight before that gets done. ------------------------- Sept. 17, 2007 --------------- Never got to it. ------------------------- Sept. 24, 2007 --------------- Another try, but I've only got 20 minutes to the next meeting. Now, where was I? Barry says he fixed the data base for the problem I had with not enough values. So try it. Run sub2db Inputs for sub2db ================= (JOB_PARAMETERS)====== proposal . . . . RD060 uttimerange. . . 2006AUG23 at 16:20:00 to 2006AUG24 at 18:05:00 database . . . . ORACLE/VLBA tempspace. . . . /tmp (SUBFIND_PARAMETERS)== timedelta. . . . 5 angledelta . . . -1 freqdelta. . . . 15 refant . . . . . * maxchan. . . . . 32 bandwidthmix . . NO This time showed lots of scans and ended with: 0458-020 24-AUG-2006 17:05:12 WF 1749+096 24-AUG-2006 17:06:49 SV DA426 24-AUG-2006 17:09:29 SV 1101+384 24-AUG-2006 17:21:03 SV 1308+326 24-AUG-2006 17:31:27 SV Commiting subarray Handling channel remaps (if any). Commiting channel remaps Reconciling database entries.... Subarraying of RD060 complete There are no new files, but maybe that is expected. Try cjobgen. Inputs for cjobgen ================== (DATA_SELECTION)====== proposal . . . . RD060 uttimerange. . . 2006AUG23 at 16:20:00 to 2006AUG24 at 18:05:00 correlation. . . * pass . . . . . . * antennas . . . . * channels . . . . * database . . . . ORACLE/VLBA tempspace. . . . /tmp (JOB_PARAMETERS)====== jobid. . . . . . * singletape . . . YES joblength. . . . 1.5 outputsize . . . 660 gapsize. . . . . 7 prime. . . . . . 20 meteorology. . . NO tables . . . . . ALL *go cjobgen EXECUTION ======= ========= Verifying inputs No correlation parameters for the given proposal and time. Set correlation parameters or correct input parameters. Jobs will be made for 2 stations. Processing 2 subarray(s). ERRORS /home/packages/oms/bin/oms Paul says that have to set a correlation row in OMS. Probably Peggy will have to put the project in. Need speedup. Paul Dyer added something and now there is info. There were complaints about no clocks. And I had to say to continue despite errors. But despite all that, it created jobs - 130 of them! The start of one job has the following: !*** An ERROR was encounted during job preparation *! !* ERROR - Polarization processing requested, but no recorded crosspolarization *! Fair enough. The job has lots of scans, only a few minutes each. So redo with pol off. First delete the output. Oops, the polarization setting is in OMS and I'm not sure I can access it. Paul Dyer did it for me earlier. When I tried OMS from my login, it didn't work. Try from the shire login. Didn't work there either - failed to establish some connection. Try from vlbaops on Jansky. Taking a long time. Again failed to establish a connection to the database. So I don't understand the problem. Fix it tomorrow. --------------------------- Sept. 25, 2007 -------------------------- Talked to Ken Hartley. He will switch rd060 to RCP only. Trying cjobgen again Inputs for cjobgen ================== (DATA_SELECTION)====== proposal . . . . RD060 uttimerange. . . 2006AUG23 at 16:20:00 to 2006AUG24 at 18:05:00 correlation. . . * pass . . . . . . * antennas . . . . * channels . . . . * database . . . . ORACLE/VLBA tempspace. . . . /tmp (JOB_PARAMETERS)====== jobid. . . . . . * singletape . . . YES joblength. . . . 1.5 outputsize . . . 660 gapsize. . . . . 7 prime. . . . . . 20 meteorology. . . NO tables . . . . . NONE Got a lot of: WARNING - patched 38.0 second hole in obs for station WF WARNING - patched 48.0 second hole in obs for station WF WARNING - patched 189.0 second hole in obs for station WF WARNING - patched 86.0 second hole in obs for station WF WARNING - patched 88.0 second hole in obs for station WF WARNING - patched 507.0 second hole in obs for station WF WARNING - patched 39.0 second hole in obs for station WF WARNING - patched 117.0 second hole in obs for station WF I'm not sure what those are about. That produced no cal tables and 33 job scripts. It started at 5454, which I think was the highest number job from my last attempt - so it was trying not to write over what it had already made. So I think we have what Walter needs. The condensed version of what was done: Changes to the Mark5B log file: Remove blank line. Remove the " mark before "form" in the following lines. 2006.235.18:00:43.69&setupsx/"form=m,16.000,1:2 to make 2006.235.18:00:43.69&setupsx/form=m,16.000,1:2 Otherwise the format is not getting set. This might or might not match what was really used. Do the same for the tracks lines: 2006.235.16:26:19.18&setupsx/"tracks=v0,v1,v2,v3 Not clear if this helped, but it was done. It seems that fs2db needs the association of tracks and bbc's. Add trkfsx specification by uncommenting trkfsx and adding the definition of trkfsx Change 2006.235.16:26:19.18&setupsx/"trkfsx to 2006.235.16:26:19.18&setupsx/trkfsx later - copied from the wf file (Mark5A), add (right after &pcalon): 2006.235.17:32:29.89&trkfsx/trackform= 2006.235.17:32:29.89&trkfsx/trackform=2,1us,6,1ls,10,2us,14,3us,18,4us,22,5us,26,6us,30,7us 2006.235.17:32:29.89&trkfsx/trackform=3,8us,7,8ls,11,9us,15,10us,19,11us,23,12us,27,13us 2006.235.17:32:29.89&trkfsx/trackform=31,14us Of course, for Mark5B, this is probably all lies, but at least fs2db would read it. The job scripts are likely to need modification. Also deleted a blank after the station name in the location line. Change to a 5 character proposal code. sub2db couldn't handle rd0606. All of the source lines in the log had been commented out. Undo that with a global replacement of "source with source. Get the dates right (we didn't translate from DOY to MMDDYY correctly!). Then Barry had to fix the data base. It seems that a new parameter was added not too long ago. Things are fine if the first logs loaded are from the VLBA, but if they come from fs2db and there are not VLBA observations, that new parameter is not made. Then sub2db croaks. Someone has to put the observation into OMS (and get a doable polarization spec). That's it so far - not counting many dead ends.