BuildBots no longer run make check?

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

BuildBots no longer run make check?

mmuetzel
Hi,

Since the buildbots were moved to a new version, I no longer see the results
of "make check":
http://buildbot.octave.org:8011/#/waterfall

Do they still run the test suite?

Markus



--
Sent from: http://octave.1599824.n4.nabble.com/Octave-Maintainers-f1638794.html

Reply | Threaded
Open this post in threaded view
|

Re: BuildBots no longer run make check?

John W. Eaton
Administrator
On 06/01/2018 02:19 PM, mmuetzel wrote:
> Hi,
>
> Since the buildbots were moved to a new version, I no longer see the results
> of "make check":
> http://buildbot.octave.org:8011/#/waterfall
>
> Do they still run the test suite?

I disabled the test build step when I began the transition to buildbot
1.x because I didn't know how to update the rules on the master.  I
looked at it again today and now we should be running tests again.  But
something is not quite the same and the filter that looks at the output
of the build steps is displaying the test step as failed when it should
be a success.  So far, I just forced one build for testing:

   http://buildbot.octave.org:8011/#/builders/7

The summary of the test step is

   PASS                            14463
   FAIL                                0
   XFAIL (reported bug)               32
   SKIP (missing feature)            616
   SKIP (run-time condition)          12

I'm not yet sure what in the configuration is wrong so that this is
flagged as a failure.

The other builds should also run tests the next time they run.

jwe

Reply | Threaded
Open this post in threaded view
|

Aw: Re: BuildBots no longer run make check?

mmuetzel
> I disabled the test build step when I began the transition to buildbot
> 1.x because I didn't know how to update the rules on the master. I
> looked at it again today and now we should be running tests again.

Thanks.

> But
> something is not quite the same and the filter that looks at the output
> of the build steps is displaying the test step as failed when it should
> be a success. So far, I just forced one build for testing:
>
> http://buildbot.octave.org:8011/#/builders/7[http://buildbot.octave.org:8011/#/builders/7]
>
> The summary of the test step is
>
> PASS 14463
> FAIL 0
> XFAIL (reported bug) 32
> SKIP (missing feature) 616
> SKIP (run-time condition) 12

Is there some way to access this summary from the web interface? That way
it would be possible to manually check quickly whether everything is still
working as expected. Even when the overall test is (erroneously) flagged
as failed.

Markus

Reply | Threaded
Open this post in threaded view
|

Re: Aw: Re: BuildBots no longer run make check?

John W. Eaton
Administrator
On 06/04/2018 02:36 PM, "Markus Mützel" wrote:

>> I disabled the test build step when I began the transition to buildbot
>> 1.x because I didn't know how to update the rules on the master. I
>> looked at it again today and now we should be running tests again.
>
> Thanks.
>
>> But
>> something is not quite the same and the filter that looks at the output
>> of the build steps is displaying the test step as failed when it should
>> be a success. So far, I just forced one build for testing:
>>
>> http://buildbot.octave.org:8011/#/builders/7[http://buildbot.octave.org:8011/#/builders/7]
>>
>> The summary of the test step is
>>
>> PASS 14463
>> FAIL 0
>> XFAIL (reported bug) 32
>> SKIP (missing feature) 616
>> SKIP (run-time condition) 12
>
> Is there some way to access this summary from the web interface? That way
> it would be possible to manually check quickly whether everything is still
> working as expected. Even when the overall test is (erroneously) flagged
> as failed.

Yes.  Following the link

   http://buildbot.octave.org:8011/#/builders/7

select the latest build, look at the test step, then you can download
the log file or select the link to view all 7481 lines in your browser.

I can also put the hg archive of the configuration files on the web to
make it easier for others to help with managing the buildbots, but first
I have to figure out how best to store some private info that is
currently embedded in the master config file.

jwe

Reply | Threaded
Open this post in threaded view
|

Aw: Re: Re: BuildBots no longer run make check?

mmuetzel
jwe wrote:

> >> The summary of the test step is
> >>
> >> PASS 14463
> >> FAIL 0
> >> XFAIL (reported bug) 32
> >> SKIP (missing feature) 616
> >> SKIP (run-time condition) 12
> >
> > Is there some way to access this summary from the web interface? That way
> > it would be possible to manually check quickly whether everything is still
> > working as expected. Even when the overall test is (erroneously) flagged
> > as failed.
>
> Yes. Following the link
>
> http://buildbot.octave.org:8011/#/builders/7[http://buildbot.octave.org:8011/#/builders/7]
>
> select the latest build, look at the test step, then you can download
> the log file or select the link to view all 7481 lines in your browser.

I was referring to the actual 5 lines summary you pasted. I couldn't find
that in the stdio logs.

> I can also put the hg archive of the configuration files on the web to
> make it easier for others to help with managing the buildbots, but first
> I have to figure out how best to store some private info that is
> currently embedded in the master config file.

That might be helpful.

This might be to complicated: But is it possible to make the log files of the
MXE builds accessible? Or at least the latest log from the "log" directory in
case a package failed to build.

Markus

Reply | Threaded
Open this post in threaded view
|

Re: Aw: Re: Re: BuildBots no longer run make check?

mmuetzel
mmuetzel wrote

> jwe wrote:
>> >> The summary of the test step is
>> >>
>> >> PASS 14463
>> >> FAIL 0
>> >> XFAIL (reported bug) 32
>> >> SKIP (missing feature) 616
>> >> SKIP (run-time condition) 12
>> >
>> > Is there some way to access this summary from the web interface? That
>> way
>> > it would be possible to manually check quickly whether everything is
>> still
>> > working as expected. Even when the overall test is (erroneously)
>> flagged
>> > as failed.
>>
>> Yes. Following the link
>>
>> http://buildbot.octave.org:8011/#/builders/7[http://buildbot.octave.org:8011/#/builders/7]
>>
>> select the latest build, look at the test step, then you can download
>> the log file or select the link to view all 7481 lines in your browser.
>
> I was referring to the actual 5 lines summary you pasted. I couldn't find
> that in the stdio logs.

Never mind. Found it... :-]

Markus




--
Sent from: http://octave.1599824.n4.nabble.com/Octave-Maintainers-f1638794.html

Reply | Threaded
Open this post in threaded view
|

Re: Aw: Re: Re: BuildBots no longer run make check?

John W. Eaton
Administrator
In reply to this post by mmuetzel
On 06/04/2018 03:48 PM, "Markus Mützel" wrote:

> I was referring to the actual 5 lines summary you pasted. I couldn't find
> that in the stdio logs.

Look around line 999 here:

http://buildbot.octave.org:8011/#/builders/7/builds/38/steps/6/logs/stdio

>> I can also put the hg archive of the configuration files on the web to
>> make it easier for others to help with managing the buildbots, but first
>> I have to figure out how best to store some private info that is
>> currently embedded in the master config file.
>
> That might be helpful.
>
> This might be to complicated: But is it possible to make the log files of the
> MXE builds accessible? Or at least the latest log from the "log" directory in
> case a package failed to build.

Yes, this is software... anything is possible.  :-)

I'm just not sure how best to do it.  The full logs for a complete
windows build are about 130MB uncompressed.

jwe

Reply | Threaded
Open this post in threaded view
|

Aw: Re: Re: Re: BuildBots no longer run make check?

mmuetzel
jwe wrote:
> > This might be to complicated: But is it possible to make the log files of the
> > MXE builds accessible? Or at least the latest log from the "log" directory in
> > case a package failed to build.
>
> Yes, this is software... anything is possible. :-)
>
> I'm just not sure how best to do it. The full logs for a complete
> windows build are about 130MB uncompressed.

In most cases, the log file with the newest time-stamp would suffice. And that only
if a package failed to build.

Markus

Reply | Threaded
Open this post in threaded view
|

Re: Aw: Re: BuildBots no longer run make check?

John W. Eaton
Administrator
In reply to this post by John W. Eaton
On 06/04/2018 03:24 PM, John W. Eaton wrote:

> I can also put the hg archive of the configuration files on the web to
> make it easier for others to help with managing the buildbots, but first
> I have to figure out how best to store some private info that is
> currently embedded in the master config file.

I fixed the master config file so that the worker password info is
hidden in a separate file that is not in the hg archive and created a
new repo starting with the latest files.  I omitted previous changes
because those files did contain password info.  You can find the repo here:

   http://hg.octave.org/octave-buildbot

Note that I'm not an expert in python or buildbot.  Any suggestions and
improvements are welcome, especially for filtering the log files to
properly report success/failure for the tests with buildbot version 1.x.

jwe

Reply | Threaded
Open this post in threaded view
|

Aw: Re: Re: BuildBots no longer run make check?

mmuetzel
jwe wrote:

> I fixed the master config file so that the worker password info is
> hidden in a separate file that is not in the hg archive and created a
> new repo starting with the latest files. I omitted previous changes
> because those files did contain password info. You can find the repo here:
>
> http://hg.octave.org/octave-buildbot
>
> Note that I'm not an expert in python or buildbot. Any suggestions and
> improvements are welcome, especially for filtering the log files to
> properly report success/failure for the tests with buildbot version 1.x.


I really don't know whether I understand what is going on.
But might this change help?:

diff -r 4ad92f00bca5 master.cfg
--- a/master.cfg Tue Jun 05 12:38:50 2018 +0000
+++ b/master.cfg Tue Jun 05 22:04:40 2018 +0200
@@ -272,7 +272,7 @@
     ## FIXME: should probably search for the "^Summary:$" line just
     ## before the PASS/FAIL totals.
 
-    logtext = "".join(self.getLog('stdio').readlines())
+    logtext = "\n".join(self.getLog('stdio').readlines())
 
     m = re.search (r"^ *FAIL +([0-9]+) *$", logtext,
                    flags = re.MULTILINE)

Reply | Threaded
Open this post in threaded view
|

AW: Re: Re: BuildBots no longer run make check?

mmuetzel

> jwe wrote:
> > Note that I'm not an expert in python or buildbot. Any suggestions and
> > improvements are welcome, especially for filtering the log files to
> > properly report success/failure for the tests with buildbot version 1.x.
>
>
> I really don't know whether I understand what is going on.
> But might this change help?:
>
> diff -r 4ad92f00bca5 master.cfg
> --- a/master.cfg Tue Jun 05 12:38:50 2018 +0000
> +++ b/master.cfg Tue Jun 05 22:04:40 2018 +0200
> @@ -272,7 +272,7 @@
>      ## FIXME: should probably search for the "^Summary:$" line just
>      ## before the PASS/FAIL totals.
>
> -    logtext = "".join(self.getLog('stdio').readlines())
> +    logtext = "\n".join(self.getLog('stdio').readlines())
>
>      m = re.search (r"^ *FAIL +([0-9]+) *$", logtext,
>                     flags = re.MULTILINE)

After digging a little deeper in the buildbot docs, I assume that self.getLog('stdio') returns a buildbot.process.log.Log object.
These objects don't seem to have a readlines method. In fact the documentation states [1]:
"This class handles write-only access to log files from running build steps. It does not provide an interface for reading logs - such access should occur directly through the Data API."

Maybe adding a buildbot.process.logobserver.BufferLogObserver and using its getStdout would do what is needed. [2]
Or we could use a LogObserver and wait for the FAIL or REGRESSION line and set a property that can be used in the evaluateCommand function. Somewhat similar to [3].

Unfortunately, I don't have a buildbot test setup and I'm by far not comfortable enough in Python to write some code without testing. I hope this might be helpful anyways.

Markus

[1]: http://docs.buildbot.net/current/developer/cls-log.html
[2]: http://docs.buildbot.net/current/developer/cls-logobserver.html#buildbot.process.logobserver.BufferLogObserver 
[3]: http://docs.buildbot.net/latest/manual/customization.html#adding-logobservers



Reply | Threaded
Open this post in threaded view
|

Re: AW: Re: Re: BuildBots no longer run make check?

John W. Eaton
Administrator
On 06/06/2018 12:51 PM, [hidden email] wrote:

>
>> jwe wrote:
>>> Note that I'm not an expert in python or buildbot. Any suggestions and
>>> improvements are welcome, especially for filtering the log files to
>>> properly report success/failure for the tests with buildbot version 1.x.
>>
>>
>> I really don't know whether I understand what is going on.
>> But might this change help?:
>>
>> diff -r 4ad92f00bca5 master.cfg
>> --- a/master.cfg Tue Jun 05 12:38:50 2018 +0000
>> +++ b/master.cfg Tue Jun 05 22:04:40 2018 +0200
>> @@ -272,7 +272,7 @@
>>       ## FIXME: should probably search for the "^Summary:$" line just
>>       ## before the PASS/FAIL totals.
>>
>> -    logtext = "".join(self.getLog('stdio').readlines())
>> +    logtext = "\n".join(self.getLog('stdio').readlines())
>>
>>       m = re.search (r"^ *FAIL +([0-9]+) *$", logtext,
>>                      flags = re.MULTILINE)
>
> After digging a little deeper in the buildbot docs, I assume that self.getLog('stdio') returns a buildbot.process.log.Log object.
> These objects don't seem to have a readlines method. In fact the documentation states [1]:
> "This class handles write-only access to log files from running build steps. It does not provide an interface for reading logs - such access should occur directly through the Data API."
>
> Maybe adding a buildbot.process.logobserver.BufferLogObserver and using its getStdout would do what is needed. [2]
> Or we could use a LogObserver and wait for the FAIL or REGRESSION line and set a property that can be used in the evaluateCommand function. Somewhat similar to [3].
>
> Unfortunately, I don't have a buildbot test setup and I'm by far not comfortable enough in Python to write some code without testing. I hope this might be helpful anyways.
>
> Markus
>
> [1]: http://docs.buildbot.net/current/developer/cls-log.html
> [2]: http://docs.buildbot.net/current/developer/cls-logobserver.html#buildbot.process.logobserver.BufferLogObserver
> [3]: http://docs.buildbot.net/latest/manual/customization.html#adding-logobservers

I also don't have a way to easily test changes other than to modify the
Octave buidlbot configuration and force a build to run.

These rules worked for the earlier 0.8.x installation.  I'm pretty sure
I copied and modified them slightly from an example that I found
(unfortunately, not sure where).

My recollection is there was some kind of default log file processing
for the test step but it didn't do exactly the right thing.  Is there no
longer something like that for the test step that we could start with
and modify as needed?

jwe

Reply | Threaded
Open this post in threaded view
|

Re: AW: Re: Re: BuildBots no longer run make check?

John W. Eaton
Administrator
On 06/12/2018 03:40 PM, John W. Eaton wrote:

> My recollection is there was some kind of default log file processing
> for the test step but it didn't do exactly the right thing.  Is there no
> longer something like that for the test step that we could start with
> and modify as needed?

Ah, following the example for Perl(?) from the shell.py file in the
buildbot sources, I came up with something that looks like it might
work.  Here are the changes I made:

   http://hg.octave.org/octave-buildbot/rev/97e7b7963dc9

jwe

Reply | Threaded
Open this post in threaded view
|

Aw: Re: AW: Re: Re: BuildBots no longer run make check?

mmuetzel
jwe wrote:

> > My recollection is there was some kind of default log file processing
> > for the test step but it didn't do exactly the right thing.  Is there no
> > longer something like that for the test step that we could start with
> > and modify as needed?
>
> Ah, following the example for Perl(?) from the shell.py file in the
> buildbot sources, I came up with something that looks like it might
> work. Here are the changes I made:
>
> http://hg.octave.org/octave-buildbot/rev/97e7b7963dc9

Thanks. That was definitely an improvement. But something still seems to be
off. The last run [1] had 3 FAILs but still passed as green:

Summary:
  PASS                            15032
  FAIL                                3
  XFAIL (reported bug)               35
  SKIP (missing feature)             39
  SKIP (run-time condition)          20

[1]: http://buildbot.octave.org:8011/#/builders/19/builds/62/steps/6/logs/stdio

Markus

Reply | Threaded
Open this post in threaded view
|

Re: Aw: Re: AW: Re: Re: BuildBots no longer run make check?

John W. Eaton
Administrator
On 06/13/2018 04:17 AM, "Markus Mützel" wrote:

> Thanks. That was definitely an improvement. But something still seems to be
> off. The last run [1] had 3 FAILs but still passed as green:
>
> Summary:
>    PASS                            15032
>    FAIL                                3
>    XFAIL (reported bug)               35
>    SKIP (missing feature)             39
>    SKIP (run-time condition)          20
>
> [1]: http://buildbot.octave.org:8011/#/builders/19/builds/62/steps/6/logs/stdio

OK, I made a few more changes and it seems to actually notice the number
of failed tests now.

jwe