This page describes the log_compare function:
= Description =
log_compare is called from a test script's test_processing() function to
scan the test's logfile and indicate whether the test passed or failed.

It can be used several times in a test_processing() function, to check
different categories of test results.  This includes checking for both
the number of positive results (sub-tests passed) or negative results
(sub-tests failed).

Arguments [edit section]

= Arguments = 
Position arguments:
 * $1 = reflog_prefix = prefix of reference parsed log filename
 * $2 = match_count = number of matching results
 * $3 = results_expression 
 * $4 = results_category (either 'p', 'n', or something else)

The first argument is a string used in the filename of the reference parsed log.  Internally this is used as part of a filename, but externally, all callers pass in $TESTDIR.  In the case of Functional.bzip2, the string
passed in is "Functional.bzip2", so it appears to match the test name.
However, in Functional.posixtestsuite, the prefix passed in is:
"posixtestsuite-1.5.2", which matches the first part of the name of the
tarball containing the source for the test suite.  It appears this
can be used to distinguish between multiple versions of the same
test.  The reference logs appear to be placed in the test area, alongside
the test script.

Hence, the posixtestsuite reference parsed logs are at:
 * /home/jenkins/fuego/engine/tests/Functional.posixtestsuite/posixtestsuite-1.5.2_p.log
 * /home/jenkins/fuego/engine/tests/Functional.posixtestsuite/posixtestsuite-1.5.2_n.log

The second argument is the match_count, specifying the number of lines
in the log file that should match the results_expression (described next).
In some tests, the match_count is specified via an environment variable
such as: $LTP_OPEN_POSIX_SUBTEST_COUNT_POS.  For extremely complex tests that have board-dependent or platform-dependent results it is useful
to allow the match_count to be specified in the board file.

The third argument is a regular expressionx to
search for in the log file.  Internally this is referred to as the "Criteria",
but I prefer to call it the results_expression.  This is passed as an
expression to 'grep -E', so the syntax used by that external command applies
here.  See [[http://www.gnu.org/software/grep/manual/grep.html#Regular-Expressions|the grep man page]] for details.

The results_expression is searched for in the logfile for this run.
The number of results is the number of lines in the logfile that match
that regular expression.

A "parsed" logfile is created, wich consists of the original test logfile
filtered through 'grep -E ${results_expression}'. 

This parsed logfile is tested against a reference log for this test,
to see if it matches.  If it does not match, then an error is logged.

Two checks are made:
 * the number of lines in the original logfile that match the results_expression
 * whether the lines in the parsed logfile file exactly match the lines in a reference parsed logfile, for a specific category

The final parameter to the function is the "result category".  This is usually
either 'p' or 'n', for positive results and negative results.  But it can be
any string.  The posixtestsuite uses 'unr' for items reported as unresolved
the test results log.

The result_category is used as part of the name of the reference parsed filename.  Indeed, the name used for comparisons is:
${WORKSPACE}/../ref_logs/${JOB_NAME}/${reflog_prefix}_${results_category}.log

After determining the status of the results, the function:
 * check_create_functional_logrun
is called, to record the status of the test.

Examples [edit section]

= Examples =
From the bzip2.sh test script:
 * log_compare "$TESTDIR" "11" "^TEST.*OK" "p"
 * log_compare "$TESTDIR" "0" "^TEST.*FAILED" "n"

This basically says that 11 lines in the log should start with TEST and have
OK in them, and 0 lines in the log should start with TEST and have FAILED in them.

From the posixtestsuite.sh
{{{
        PASSED="PASSED:  922"
        FAILED="FAILED:  69"
        UNRESOLVED="UNRESOLVED:  7"
        UNSUPPORTED="UNSUPPORTED:  93"
        UNTESTED="UNTESTED:  66"

        log_compare "$TESTDIR" "1" "${PASSED}" "p"
        log_compare "$TESTDIR" "1" "${FAILED}" "f"
        log_compare "$TESTDIR" "1" "${UNRESOLVED}" "unr"
        log_compare "$TESTDIR" "1" "${UNSUPPORTED}" "uns"
        log_compare "$TESTDIR" "1" "${UNTESTED}" "unt"
}}}

Note that this test sums the number of results in different categories,
so the match_counts are always 1, and the results_expression for each
category includes the number of results expected.

Return value [edit section]

= Return value =
A return value of 'true' indicates that the test succeeded.

A return value of 'false' indicates that the test failed.

If multiple log_compares are called in sequence, only the return value
of the last one is used as the return value for the calling
function (test_processing).

Notes [edit section]

= Notes =
 * this call should be made inside the "test_processing" functions of Functional tests.
 * how is this function called?
   * see the examples
 * what does check_create_functional_logrun do?
   * called with one of: "test error", "passed", "failed"
   * check_create_functional_logrun only works if BATCH_TESTPLAN is set
 * how is success or failure communicated to Jenkins for their icon?
   * it appears to be entirely based on the return code from the last call to log_compare
     * which becomes the return code for test_processing(), and "source functional.sh" and the last value returned by the test script itself.
 * where is the final result stored in Jenkins?
   * I don't know

Location [edit section]

== Location ==
This function is located in /home/jenkins/scripts/functions.sh.
It is called by about 2/3rds of the functional tests, at the time of this writing.

dead functionality [edit section]

== dead functionality ==
There appears to be an intended feature that checks the parsed logfile
against a reference parsed logfile (either positive or negative).  If the
result of the test does not match exactly the reference parsed log, then
an error is emitted with the difference, with the string:
 * "Fuego error reason: Unexpected test log output:\n$TMP\n"

The path for the reference parsed logfile used in the current system
doesn't match any files.  However
reference_logs are defined in each test directory. There's currently a
bug in the shell fragment that does this comparison, so that it always
yields a result of 0 (or success), so the test is completely meaningless.

This is the only place in the routine that uses the first positional parameter
($1), and it's part of the reference parsed log filename.  So I highly doubt
that the meaning for parameter $1 of "tarball template" is correct.

creating reference parsed log files [edit section]

== creating reference parsed log files ==
FIXTHIS - should document how to create parsed reference log files in the system, or make a tool to do so. 

Maybe you just copy the parsed log file, creating the
filename with "${reflog_prefix}_p.log" or "{reflog_prefix}_n.log".
That's easy to do manually.

fragility of the script [edit section]

== fragility of the script ==
The call to source functional.sh should be the last thing in a
test script.  This seems fragile.

call tree [edit section]

== call tree ==
{{{
 <test_name>/config.xml
    <test_name>/test_script.sh
       source functional.sh - call to test_processing
          <test_name>/<test_script>.sh:function test_processing
             scripts/functions.sh:log_compare
                scripts/reports.sh:check_create_functional_logrun
}}}    

FIXTHIS - the log_compare and function_log_compare pages should be merged