[buildbots] False pass results for segmentation fault in test

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[buildbots] False pass results for segmentation fault in test

mmuetzel
It looks like a segmentation fault in the last step of running the test suite on the build bots is still marked as "success" if there were no failed test up to that point. E.g.:
http://buildbot.octave.org:8010/#/builders/12/builds/1525/steps/6/logs/stdio

Maybe any output to stderr in that step should result in it to be marked as "failure"?

Markus

Reply | Threaded
Open this post in threaded view
|

Re: [buildbots] False pass results for segmentation fault in test

mmuetzel
Am 13. Januar 2020 um 16:21 Uhr schrieb "Markus M├╝tzel":
> It looks like a segmentation fault in the last step of running the test suite on the build bots is still marked as "success" if there were no failed test up to that point. E.g.:
> http://buildbot.octave.org:8010/#/builders/12/builds/1525/steps/6/logs/stdio
>
> Maybe any output to stderr in that step should result in it to be marked as "failure"?

I only have a very vague idea of how configuring the buildbots works. And I have no way of testing this. But maybe something similar to this change would work:

diff -r e02079ee69d1 master.cfg
--- a/master.cfg Fri Feb 08 15:58:06 2019 +0000
+++ b/master.cfg Mon Jan 13 19:44:13 2020 +0100
@@ -324,6 +324,7 @@
     self.passed = 0
     self.failed = 0
     self.warnings = 0
+    self.errors = 0
     self.summary_found = False
     self.complete = False
 
@@ -352,6 +353,11 @@
           self.rc = FAILURE
           self.failed += num
 
+  def errLineRecived (self, line)
+    self.total += 1
+    self.errors += 1
+    self.rc = FAILURE
+
 
 class octave_test (Test):
 
@@ -365,7 +371,7 @@
       passed = self.observer.total - self.observer.failed
 
       self.setTestResults (total = self.observer.total,
-                           failed = self.observer.failed,
+                           failed = self.observer.failed + self.observer.errors,
                            passed = self.observer.passed,
                            warnings = self.observer.warnings)