Different search results depending on whether <EXEC> is used to start the search.

barry.marcus
Posts: 288
Joined: Thu Nov 16, 2006 1:05 pm

Different search results depending on whether <EXEC> is used to start the search.

Post by barry.marcus »

We have a Vortex application in which the user has the option of either waiting for Texis to finish searching the database for query results before the application proceeds to display those results on the screen (i.e., "foreground processing") or to have the application start a separate process to run the query (i.e, "background processing" using <exec> with the BKGND option), in which case the app returns immediately without results. The background process stores the results when it completes. We are noticing that with certain queries we get different results depending on whether the code is run in the "foreground" or the "background." For those queries for which the foreground vs. background processing return different results, the results are consistent.

Is there a reason why this would be? The settings are the same and the query is the same. The only difference, as far we can tell, is that in one case the app waits for the results and in the other it does not.
User avatar
John
Site Admin
Posts: 2597
Joined: Mon Apr 24, 2000 3:18 pm
Location: Cleveland, OH
Contact:

Different search results depending on whether <EXEC> is used to start the search.

Post by John »

No, the results would not be different. It sounds as if there is a different path through the code that is causing a difference in settings, either because the code is different, or certain variables that are set in the CGI environment aren't present when exec'd to avoid mistakenly running as a CGI.
John Turnbull
Thunderstone Software
barry.marcus
Posts: 288
Joined: Thu Nov 16, 2006 1:05 pm

Different search results depending on whether <EXEC> is used to start the search.

Post by barry.marcus »

Where should we look for those CGI settings that are resulting in different result sets? We clearly need the results to be the same regardless of how the search is started. I have a small program that demonstrates (on our server and with our production database, at least) the difference. In this demo program the only settings that are *explicitly* made are the same for both the "foreground"/CGI instance and the "background"/non-CGI instance. They are:

<set indexwithin = 7>
<set maxlinearrows = 0>
<apicp alpostproc 1>
<apicp allinear 1>
<apicp alintersects 1>
<apicp intersects -1>
<apicp alwithin 1>
<apicp minwordlen 5>
<apicp qmaxsetwords 10000>
<apicp qmaxwords 10000>
<apicp withinmode "word span">
<apicp keepnoise 1>
barry.marcus
Posts: 288
Joined: Thu Nov 16, 2006 1:05 pm

Different search results depending on whether <EXEC> is used to start the search.

Post by barry.marcus »

Forgot a few more...

<apicp eqprefix "D:\Crosshairs\equivs\Prg008">
<apicp alequivs 1>
<apicp keepeqvs 1>
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

Different search results depending on whether <EXEC> is used to start the search.

Post by mark »

I think John was suggesting possible form inputs or other CGI variables being used by the script or in the query may not be passed to the background process.

Are both foreground and background processes able to access the equiv file? Try removing that setting and see if the results match up.

Also check the vortex.log and/or HTML result output to see if there are any error/warning messages for either.
barry.marcus
Posts: 288
Joined: Thu Nov 16, 2006 1:05 pm

Different search results depending on whether <EXEC> is used to start the search.

Post by barry.marcus »

If anything other than what I listed above is being passed to the background script, I don't know where it would be coming from. As I described, what is running in both cases is pretty self-contained and concise, with the same calls in both scripts. Although they are kind of long for this forum, here are the two scripts. The first, RunQuery, runs the foreground query and also is used to "spawn" the background search. The second, RunQueryBackground, runs the background search:

<script language=vortex>
<!-- RunQuery -->
<a name=main PUBLIC>
<vxcp timeout -1>
<db="D:\Crosshairs\database1">

<form action="$url" method="post">
Enter SQL SELECT query with a single returned field aliased to "rtnVal":<br>
<textarea name="theQuery" rows="10" cols="140" wrap="soft">$theQuery</textarea><br>
<input type="submit" value="Run Query in Foreground" name="doWhat">
<input type="submit" value="Run Query in Background" name="doWhat">
</form>

<sandr ">>= +" "" $theQuery>
<$theQuery=$ret>

<if ($doWhat eq "Run Query in Background") and ($theQuery ne "")>
<$queryFile="D:\Crosshairs\bgndQueryResult.txt">

<!-- Store the query to a file to be read by the spawned script -->
<write append $queryFile>
$theQuery
</write>

<!-- Spawn the script -->
<SpawnBackgroundQuery queryFile=$queryFile>
<br>Results will be found in $queryFile (Allow time for query to finish.)<br>
</if>

<if ($doWhat eq "Run Query in Foreground") and ($theQuery ne "")>
<SetControlParameters>
<sql row $theQuery>
<if $savnum eq "">
<if $loop eq 0>
The query results are:<br>
</if>
$rtnVal<br>
</if>
</sql>

<if $loop eq 1>
<br>$loop result<br>
<else>
<br>$loop results<br>
</if>
</if>
</a>

<a name=SetControlParameters PRIVATE>
<apicp eqprefix "D:\Crosshairs\appData1\equivfiles\Prg008">
<apicp alequivs 1>
<apicp keepeqvs 1>
<set indexwithin = 7>
<set maxlinearrows = 0>
<apicp alpostproc 1>
<apicp allinear 1>
<apicp alintersects 1>
<apicp intersects -1>
<apicp alwithin 1>
<apicp minwordlen 5>
<apicp qmaxsetwords 10000>
<apicp qmaxwords 10000>
<apicp withinmode "word span">
<apicp keepnoise 1>
</a>

<a name=SpawnBackgroundQuery PRIVATE queryFile>
<local theCommand texisC>

<strfmt "D:\\Crosshairs\\script\\utility\\thunderstone\\RunQueryBackground">
<$theCommand=$ret>
<$texisC="C:\MORPH3\texis.exe">
<exec bkgnd -- $texisC $theCommand></exec>
</a>
</script>

<script language=vortex>
<!-- RunQueryBackground -->
<a name=main>
<vxcp timeout -1>
<db="D:\Crosshairs\database1">
<$queryFile="D:\Crosshairs\bgndQueryResult.txt">

<!-- Read the query from the file -->
<readln rev row $queryFile>
<$theQuery=$ret>
<break>
</readln>

<SetControlParameters>
<sql row $theQuery>
<if $loop eq 0>
<write append $queryFile>
The query results are:
</write>
</if>
<write append $queryFile>
$rtnVal
</write>
</sql>

<if $loop eq 1>
<write append $queryFile>
$loop result
</write>
<else>
<write append $queryFile>
$loop results
</write>
</if>

<write append $queryFile>
<fmt "\n\n">
</write>
</a>

<a name=SetControlParameters PRIVATE>
<apicp eqprefix "D:\Crosshairs\appData1\equivfiles\Prg008">
<apicp alequivs 1>
<apicp keepeqvs 1>
<set indexwithin = 7>
<set maxlinearrows = 0>
<apicp alpostproc 1>
<apicp allinear 1>
<apicp alintersects 1>
<apicp intersects -1>
<apicp alwithin 1>
<apicp minwordlen 5>
<apicp qmaxsetwords 10000>
<apicp qmaxwords 10000>
<apicp withinmode "word span">
<apicp keepnoise 1>
</a>
</script>


Yes, both scripts can access the equiv file. The query is very complex, involving many terms found ONLY in the equiv file. (We do this often, i.e., defining terms in our equiv files that are guaranteed NEVER to appear in the actual data. We use these as "root" terms for many equivalent synonyms. Using the root terms rather than the individual synonyms allows us to simplify our queries somewhat.) The queries are so dependent on the equiv file, that if I remove the equiv file neither query returns any results at all.

There is no error in the vortex log for either script, and the HTML source is as clean as a whistle.

It's a mystery!
barry.marcus
Posts: 288
Joined: Thu Nov 16, 2006 1:05 pm

Different search results depending on whether <EXEC> is used to start the search.

Post by barry.marcus »

If you need the equiv file and/or the query itself let me know.
User avatar
John
Site Admin
Posts: 2597
Joined: Mon Apr 24, 2000 3:18 pm
Location: Cleveland, OH
Contact:

Different search results depending on whether <EXEC> is used to start the search.

Post by John »

Does it make a difference if the exec is background or not? I'm not sure why you have two separate scripts, but I'd create a single script which does both the exec and the direct SQL to the same function in the same script to compare the results.
John Turnbull
Thunderstone Software
barry.marcus
Posts: 288
Joined: Thu Nov 16, 2006 1:05 pm

Different search results depending on whether <EXEC> is used to start the search.

Post by barry.marcus »

There was no good reason why I split it into two scripts other than... Well, it just didn't occur to me at the time. But I changed it (see script below) with identical results as before. That is, the results are still *different* when run background vs. foreground.

Now, though, with the script structured the way it is, I am getting an error when I remove the BKGND option from the <EXEC>. It has to do with not being able to overwrite the file that's currently running. How should I structure the script to avoid this? The error is:

006 2012-01-03 11:17:15 D:\Crosshairs\script\utility\thunderstone\RunQuery: Cannot rename D:\Crosshairs\script\utility\thunderstone\RunQuery.vtc to .vtx: File exists
118 2012-01-03 11:17:15 /utility/thunderstone/runquery:116: <EXEC> command C:\MORPH3\texis.exe returned exit code 52: Cannot rename file?

<script language=vortex>
<!-- RunQuery -->

<a name=main PUBLIC>
<form action="$url" method="post">
Enter SQL SELECT query with a single returned field aliased to "rtnVal":<br>
<textarea name="theQuery" rows="10" cols="140" wrap="soft">$theQuery</textarea><br>
<input type="submit" value="Run Query in Foreground" name="doWhat">
<input type="submit" value="Run Query in Background" name="doWhat">
</form>

<sandr ">>= +" "" $theQuery>
<$theQuery=$ret>

<if ($doWhat eq "Run Query in Background") and ($theQuery ne "")>
<$queryFile="D:\Crosshairs\bgndQueryResult.txt">

<!-- Store the query to a file to be read by the spawned script -->
<write append $queryFile>
$theQuery
</write>

<!-- Spawn the script -->
<SpawnBackgroundQuery queryFile=$queryFile>
<br>Results will be found in $queryFile (Allow time for query to finish.)<br>
</if>

<if ($doWhat eq "Run Query in Foreground") and ($theQuery ne "")>
<DoQuery background="0">
</if>
</a>

<a name=DoQuery PUBLIC background="1">
<vxcp timeout -1>
<db="D:\Crosshairs\database1">

<if $background eq "1">
<!-- Read the query from the file -->
<$queryFile="D:\Crosshairs\bgndQueryResult.txt">
<readln rev row $queryFile>
<$theQuery=$ret>
<break>
</readln>
</if>

<SetControlParameters>

<sql row $theQuery>
<if $background eq "1">
<if $loop eq 0>
<write append $queryFile>
The query results are:
</write>
</if>
<write append $queryFile>
$rtnVal
</write>
</if>

<if $background eq "0">
<if $loop eq 0>
The query results are:<br>
</if>
$rtnVal<br>
</if>
</sql>

<if $background eq "1">
<if $loop eq 1>
<write append $queryFile>
$loop result
</write>
<else>
<write append $queryFile>
$loop results
</write>
</if>
<write append $queryFile>
<fmt "\n\n">
</write>
</if>

<if $background eq "0">
<if $loop eq 1>
<br>$loop result<br>
<else>
<br>$loop results<br>
</if>
</if>
</a>

<a name=SetControlParameters PRIVATE>
<apicp eqprefix "D:\Crosshairs\appData1\equivfiles\Prg008">
<apicp alequivs 1>
<apicp keepeqvs 1>
<set indexwithin = 7>
<set maxlinearrows = 0>
<apicp alpostproc 1>
<apicp allinear 1>
<apicp alintersects 1>
<apicp intersects -1>
<apicp alwithin 1>
<apicp minwordlen 5>
<apicp qmaxsetwords 10000>
<apicp qmaxwords 10000>
<apicp withinmode "word span">
<apicp keepnoise 1>
</a>

<a name=SpawnBackgroundQuery PRIVATE queryFile>
<local theCommand texisC>

<strfmt "D:\\Crosshairs\\script\\utility\\thunderstone\\RunQuery\\DoQuery.txt">
<$theCommand=$ret>
<$texisC="C:\MORPH3\texis.exe">
<exec bkgnd -- $texisC $theCommand></exec>
</a>
</script>
User avatar
John
Site Admin
Posts: 2597
Joined: Mon Apr 24, 2000 3:18 pm
Location: Cleveland, OH
Contact:

Different search results depending on whether <EXEC> is used to start the search.

Post by John »

It shouldn't need to recompile the script unless you have a clock time stamp issue. The timestamp on the .vtx should be newer than the source file once it's compiled by the foreground process, so the background process won't need to recompile. If you are transferring files over the network you might have out of sync clocks.

What is the difference in results? Can you identify which produces the "right" results, and how the other results are different?
John Turnbull
Thunderstone Software
Post Reply