[buildbots] False pass results for segmentation fault in test

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[buildbots] False pass results for segmentation fault in test

mmuetzel
It looks like a segmentation fault in the last step of running the test suite on the build bots is still marked as "success" if there were no failed test up to that point. E.g.:
http://buildbot.octave.org:8010/#/builders/12/builds/1525/steps/6/logs/stdio

Maybe any output to stderr in that step should result in it to be marked as "failure"?

Markus

Reply | Threaded
Open this post in threaded view
|

Re: [buildbots] False pass results for segmentation fault in test

mmuetzel
Am 13. Januar 2020 um 16:21 Uhr schrieb "Markus Mützel":
> It looks like a segmentation fault in the last step of running the test suite on the build bots is still marked as "success" if there were no failed test up to that point. E.g.:
> http://buildbot.octave.org:8010/#/builders/12/builds/1525/steps/6/logs/stdio
>
> Maybe any output to stderr in that step should result in it to be marked as "failure"?

I only have a very vague idea of how configuring the buildbots works. And I have no way of testing this. But maybe something similar to this change would work:

diff -r e02079ee69d1 master.cfg
--- a/master.cfg Fri Feb 08 15:58:06 2019 +0000
+++ b/master.cfg Mon Jan 13 19:44:13 2020 +0100
@@ -324,6 +324,7 @@
     self.passed = 0
     self.failed = 0
     self.warnings = 0
+    self.errors = 0
     self.summary_found = False
     self.complete = False
 
@@ -352,6 +353,11 @@
           self.rc = FAILURE
           self.failed += num
 
+  def errLineRecived (self, line)
+    self.total += 1
+    self.errors += 1
+    self.rc = FAILURE
+
 
 class octave_test (Test):
 
@@ -365,7 +371,7 @@
       passed = self.observer.total - self.observer.failed
 
       self.setTestResults (total = self.observer.total,
-                           failed = self.observer.failed,
+                           failed = self.observer.failed + self.observer.errors,
                            passed = self.observer.passed,
                            warnings = self.observer.warnings)
 


Reply | Threaded
Open this post in threaded view
|

Re: [buildbots] False pass results for segmentation fault in test

mmuetzel
Am 13. Januar 2020 um 19:46 Uhr schrieb "Markus Mützel":

> Am 13. Januar 2020 um 16:21 Uhr schrieb "Markus Mützel":
> > It looks like a segmentation fault in the last step of running the test suite on the build bots is still marked as "success" if there were no failed test up to that point. E.g.:
> > http://buildbot.octave.org:8010/#/builders/12/builds/1525/steps/6/logs/stdio
> >
> > Maybe any output to stderr in that step should result in it to be marked as "failure"?
>
> I only have a very vague idea of how configuring the buildbots works. And I have no way of testing this. But maybe something similar to this change would work:
>
> diff -r e02079ee69d1 master.cfg
> --- a/master.cfg Fri Feb 08 15:58:06 2019 +0000
> +++ b/master.cfg Mon Jan 13 19:44:13 2020 +0100
> @@ -324,6 +324,7 @@
>      self.passed = 0
>      self.failed = 0
>      self.warnings = 0
> +    self.errors = 0
>      self.summary_found = False
>      self.complete = False
>  
> @@ -352,6 +353,11 @@
>            self.rc = FAILURE
>            self.failed += num
>  
> +  def errLineReceived (self, line)
> +    self.total += 1
> +    self.errors += 1
> +    self.rc = FAILURE
> +
>  
>  class octave_test (Test):
>  
> @@ -365,7 +371,7 @@
>        passed = self.observer.total - self.observer.failed
>  
>        self.setTestResults (total = self.observer.total,
> -                           failed = self.observer.failed,
> +                           failed = self.observer.failed + self.observer.errors,
>                             passed = self.observer.passed,
>                             warnings = self.observer.warnings)


These errors still occur occasionally. E.g. here: http://buildbot.octave.org:8010/#/builders/12/builds/1599/steps/6/logs/stdio

Since the overall test appears as "green", they are quite hard to spot.

Could be please apply a change similar to the one outlined above to the buildbot master?

Should I better open a bug report?

Markus