Summary

Add partial results (by calling post-test) when a test times out

Owner

Tim

Reporter

Tim

Status

open

Priority

high

Description [edit section]

As of Fuego version 1.3, when a job times out, ftc will just kill it immediately. This ends up not gathering the test log from the target board, and misses an opportunity to show diagnostic information which would help determine the cause of the timeout.

Maybe the timeout just needs to be a bit longer. Maybe the board hung. Maybe the test program hung. Without a log, or any attempt to gather additional information (like - is the board still up?, or what does the serial console say?), then it's hard to diagnose the problem.

Modify the code in ftc to send a signal (SIGTERM) to the test, rather than kill it immediately, to allow it to call post_test and possibly provide some partial testcase results, or diagnose the problem.

Notes [edit section]

I tried this in the 1.4 release, but it ended up that the test showed partial results that resulted up in a "SUCCESS" test result. This can happen when you have a log_compare($TESTDIR, "not ok", 0, "n") and partial results are all successful (but incomplete) (e.g. 5 out of 6 testcase results are in the log, all successful, but testcase 6 was not reported due to a timeout).

Here is the code I tried:

$ git diff
diff --git a/engine/scripts/ftc b/engine/scripts/ftc
index 92e1d3f..a029a10 100755
--- a/engine/scripts/ftc
+++ b/engine/scripts/ftc
@@ -3161,12 +3161,24 @@ def ftc_exec_command(command, timeout):
             time.sleep(.1)

     except FTC_INTERRUPT:
-        print "Job interrupted!"
+        print("Job interrupted! Timeout of '%s' was exceeded." % timeout)
+        # try graceful termination of test - this may allow post_test to run
+        p.send_signal(signal.SIGTERM)
+        for i in range(3):
+            if p.poll() == None:
+                time.sleep(10)
+                p.send_signal(signal.SIGTERM)
+
         # abort with prejudice...
-        p.kill()
+        if p.poll() == None:
+            p.kill()
+
+
     finally:
+        # cancel the timeout alarm
         signal.alarm(0)

+    dprint("test return code is %d" % p.returncode)
     return p.returncode

Probably, we need to modify post_test operations in functions.sh, so that when a job is interrupted it is not possible to return success from main.sh (even if the parser or log_compare show no testcase fail results).

backlink

Fuego Issues List