log compare

This page describes the log_compare function:

Description [edit section]

log_compare is called from a test script's test_processing() function to scan the test's logfile and indicate whether the test passed or failed.

It can be used several times in a test_processing() function, to check different categories of test results. This includes checking for both the number of positive results (sub-tests passed) or negative results (sub-tests failed).

Arguments [edit section]

Position arguments:

The first argument is a string used in the filename of the reference parsed log. Internally this is used as part of a filename, but externally, all callers pass in $TESTDIR. In the case of Functional.bzip2, the string passed in is "Functional.bzip2", so it appears to match the test name. However, in Functional.posixtestsuite, the prefix passed in is: "posixtestsuite-1.5.2", which matches the first part of the name of the tarball containing the source for the test suite. It appears this can be used to distinguish between multiple versions of the same test. The reference logs appear to be placed in the test area, alongside the test script.

Hence, the posixtestsuite reference parsed logs are at:

FIXTHIS - are the pn logs still used by Fuego (in 1.2?)

The second argument is the match_count, specifying the number of lines in the log file that should match the results_expression (described next). In some tests, the match_count is specified via an environment variable such as: $LTP_OPEN_POSIX_SUBTEST_COUNT_POS. For extremely complex tests that have board-dependent or platform-dependent results it is useful to allow the match_count to be specified in the board file.

The third argument is a regular expressionx to search for in the log file. Internally this is referred to as the "Criteria", but I prefer to call it the results_expression. This is passed as an expression to 'grep -E', so the syntax used by that external command applies here. See the grep man page for details.

The results_expression is searched for in the logfile for this run. The number of results is the number of lines in the logfile that match that regular expression.

A "parsed" logfile is created, wich consists of the original test logfile filtered through 'grep -E ${results_expression}'.

This parsed logfile is tested against a reference log for this test, to see if it matches. If it does not match, then an error is logged.

Two checks are made:

The final parameter to the function is the "result category". This is usually either 'p' or 'n', for positive results and negative results. But it can be any string. The posixtestsuite uses 'unr' for items reported as unresolved the test results log.

The result_category is used as part of the name of the reference parsed filename. Indeed, the name used for comparisons is: ${WORKSPACE}/../ref_logs/${JOB_NAME}/${reflog_prefix}_${results_category}.log

After determining the status of the results, the function:

Examples [edit section]

From the bzip2.sh test script:

This basically says that 11 lines in the log should start with TEST and have OK in them, and 0 lines in the log should start with TEST and have FAILED in them.

From the posixtestsuite.sh 

        PASSED="PASSED:  922"
        FAILED="FAILED:  69"
        UNRESOLVED="UNRESOLVED:  7"
        UNSUPPORTED="UNSUPPORTED:  93"
        UNTESTED="UNTESTED:  66"

        log_compare "$TESTDIR" "1" "${PASSED}" "p"
        log_compare "$TESTDIR" "1" "${FAILED}" "f"
        log_compare "$TESTDIR" "1" "${UNRESOLVED}" "unr"
        log_compare "$TESTDIR" "1" "${UNSUPPORTED}" "uns"
        log_compare "$TESTDIR" "1" "${UNTESTED}" "unt"

Note that this test sums the number of results in different categories, so the match_counts are always 1, and the results_expression for each category includes the number of results expected.

Return value [edit section]

A return value of 'true' indicates that the test succeeded.

A return value of 'false' indicates that the test failed.

If multiple log_compares are called in sequence, only the return value of the last one is used as the return value for the calling function (test_processing).

Notes [edit section]

Location [edit section]

This function is located in /home/jenkins/scripts/functions.sh. It is called by about 2/3rds of the functional tests, at the time of this writing.

dead functionality [edit section]

There appears to be an intended feature that checks the parsed logfile against a reference parsed logfile (either positive or negative). If the result of the test does not match exactly the reference parsed log, then an error is emitted with the difference, with the string:

The path for the reference parsed logfile used in the current system doesn't match any files. However reference_logs are defined in each test directory. There's currently a bug in the shell fragment that does this comparison, so that it always yields a result of 0 (or success), so the test is completely meaningless.

This is the only place in the routine that uses the first positional parameter ($1), and it's part of the reference parsed log filename. So I highly doubt that the meaning for parameter $1 of "tarball template" is correct.

creating reference parsed log files [edit section]

FIXTHIS - should document how to create parsed reference log files in the system, or make a tool to do so.

Maybe you just copy the parsed log file, creating the filename with "${reflog_prefix}_p.log" or "{reflog_prefix}_n.log". That's easy to do manually.

fragility of the script [edit section]

The call to source functional.sh should be the last thing in a test script. This seems fragile.

call tree [edit section] 

 <test_name>/config.xml
    <test_name>/test_script.sh
       source functional.sh - call to test_processing
          <test_name>/<test_script>.sh:function test_processing
             scripts/functions.sh:log_compare
                scripts/reports.sh:check_create_functional_logrun

FIXTHIS - the log_compare and function_log_compare pages should be merged