Re: diagonal matrices specializations

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

Jaroslav Hajek-2
On Tue, Dec 2, 2008 at 8:23 AM, Jaroslav Hajek <[hidden email]> wrote:

> hi all,
>
> this is another proposed improvement in the interpreter - diagonal
> matrix objects.
>
> Rationale:
> Diagonal matrices are common objects in linear algebra. They are
> frequently used to perform scaling and unscaling, and are also
> naturally yielded by the svd and eig factorizations. In Octave and
> Matlab alike, however, direct use of these matrices suffers a
> significant performance penalty, because they're treated as full.
> So, while expressions like `diag(D) *A' are common in manuals and
> examples, they're avoided in practice by using something ugly like `A
> .* repmat (D, 1, columns(A))'. Octave has introduced `dmult' for this
> purpose, however, this function is not much used because it is not
> Matlab compatible and it is not much known even to Octave users. And
> anyway, the whole point of having a language like Octave is to allow
> user to type equations as directly as possible, with the interpreter
> taking care of performance.
>
> Summary of changes:
> There is existing support for rectangular diagonal matrices in
> liboctave. (Yeah, rectangular. Search me why, but after a while I
> started to like the idea, it's not much additional work and it fits
> more uses nicely, SVD for instance). Some modifications were made to
> improve performance (e.g., inline element access) and extend
> functionality.
> In the interpreter code, 4 new octave_value classes are introduced (
> real/complex diagonal matrix + float counterparts), with appropriate
> operators registered and specialized in OPERATORS. Several builtins
> were modified to take advantage of the new functionality (diag, eye,
> inv) while others, due to their fine coding, benefit automatically
> (svd, eig). As a result, some tests need to be modified.
>
> Two issues are left open:
>
> 1. Any assignment into a diagonal matrix converts it to a full matrix,
> even an assigment to a single diagonal element. This is an inherent
> consequence of current Octave's type system, where you can specialize
> assignment based on types, but at the time when indices are resolved,
> the octave_base_value polymorphic instance is already fixed (see
> ops.h, octave_base_value::numeric_assign etc). I don't think this is
> much of an issue, as diagonal matrices are seldom manipulated by
> elements, but still it's a defect.
>
> 2. There was a prior debate how the special diagonal matrices should
> be displayed & saved. Saving to external formats is out of question,
> of course, but displaying and Octave's native saving is debatable.
> Displaying a diagonal matrix by printing out just the diagonal (after
> a notification of diagonality) is no doubt more informative than
> displaying it as full. Similarly, when Octave saves a matrix in its
> native format, it can be useful to preserve the diagonality
> information rather than forgetting it. OTOH, there's compatibility.
> In the patch presented, neither option is exploited, i.e. the matrix
> is always converted to full.
> My personal preference is to alter the printing and leave the saving,
> but I'd like to hear more opinions about this.
>
> In very near future, I intend to contribute a similar improvement
> concerning permutation matrices, another common type of object in
> linear algebra.
>
> The patch (>100K) is available for download here:
>
> http://artax.karlin.mff.cuni.cz/~hajej2am/ulozna/octave/diag-mat.diff
>
> cheers
>
> --
> RNDr. Jaroslav Hajek
> computing expert
> Aeronautical Research and Test Institute (VZLU)
> Prague, Czech Republic
> url: www.highegg.matfyz.cz
>

OK, I've pushed the patches.

Rationale: as in the first mail.

Summary of changes:

Rectangular diagonal matrices are now special objects. They can be
constructed using the "diag" builtin function, as well as returned by
other builtin functions (svd, eig). Row/column scaling & unscaling of
matrices can easily be done using the *, /, \ operators. The division
operators return a minimum-norm solution.
If D is a diagonal matrix, D(:,:) can be used to convert it to a full matrix.

Permutation matrices are also special objects. They can be
conveniently constructed using the syntax
`eye(n)(p,:)', `eye(n)(:,p)', `eye(n)(p,q)' as well as returned by
some builtins (lu, qr). Permuting / unpermuting is easily done using
*, /, \ or '*, *'.
If P is a permutation matrix, P(:,:) can be used to convert it to a full matrix.

The *,/,\ operations with diagonal & permutation matrices have changed
their complexity from O(N^3) to O(N^2), which significantly improves
the speed and allows to use them normally in matrix algebra without
performance penalty. A simplistic benchmark follows (set n to a
suitable value):

n = 3000;
A = rand(n);
D = diag(rand(n,1));
D1 = diag(D);
tic; C1 = dmult (dmult (D1, A), D1); toc
tic; C = D * A * D; toc
norm (C1 - C, 1)

p = randperm (n);
P = eye(n)(p,:);

tic; C1 = A(p,p); toc
tic; C = P * A * P'; toc
norm (C1 - C, 1)


before the changes, I get (Intel Core 2 Duo @ 2.8 GHz)

Elapsed time is 0.328404 seconds.
Elapsed time is 10.1832 seconds.
ans = 0
Elapsed time is 0.06 seconds.
Elapsed time is 1e+01 seconds.
ans = 0

with my changes, I get:

Elapsed time is 0.329819 seconds.
Elapsed time is 0.267672 seconds.
ans = 0
Elapsed time is 0.07 seconds.
Elapsed time is 0.1 seconds.
ans = 0


i.e. a 38x, resp. 100x speed-up (will increase with n).

TODO:
1. special printing/saving of diagonal & permutation matrices
2. make "balance" benefit from the changes
3. I'm sure there's a lot of stuff I forgot.
4. Sparse code is mostly untouched. When dealing with diagonal &
permutation matrices as sparse, the performance hit is much smaller
(not by an order, but a constant factor). As I don't work with sparse
matrices much, I'd appreciate advice about what interactions would be
beneficial to implement (e.g., diagonal-sparse multiplication etc).

I'll attend to 1,2,3 shortly.

cheers

--
RNDr. Jaroslav Hajek
computing expert
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz
Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

John W. Eaton
Administrator
On  3-Dec-2008, Søren Hauberg wrote:

| ons, 03 12 2008 kl. 14:31 +0100, skrev Jaroslav Hajek:
| > >> ans =
| > >> diagonal matrix (5x5)
| > >>    1       ..      0
| > >>        2
| > >>    :       3       :
| > >>                4
| > >>    0       ..      5
| > >>
| > >> the last form seems attractive, but maybe requires a bit more thinking
| > >> about how to place the dots etc.
| > >
| > > This I don't think will work. It would be very hard to read a 20x20
| > > matrix in a terminal with a width of 80 chars. But I agree, that it
| > > looks nice for small matrices.
| > >
| >
| > But anyone trying to work visually with 20x20 matrices on a 80-column
| > terminal is bound to have problems.
|
| Not if you print it like a vector, which I think we should just always
| do.

Since it seems unlikely that we will all agree on the best way to
display diagonal matrices, maybe it would be best to allow different
output formats based on some preference?  Perhaps as additional
options to the format function.  How about something like

  format diag full
  format diag row
  format diag column

I'm not sure about the format you show above.  It looks good for small
matrices with integer elements, but the expectation is that matrices
too wide for a terminal window will be wrapped, and I don't think this
format would look very good when wrapped, but maybe it would work.

jwe

Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

John W. Eaton
Administrator
In reply to this post by Jaroslav Hajek-2
On  4-Dec-2008, Jaroslav Hajek wrote:

| OK, I've pushed the patches.
|
| Rationale: as in the first mail.
|
| Summary of changes:
|
| Rectangular diagonal matrices are now special objects.
|
| [...]
|
| i.e. a 38x, resp. 100x speed-up (will increase with n).
|
| TODO:
| 1. special printing/saving of diagonal & permutation matrices
| 2. make "balance" benefit from the changes
| 3. I'm sure there's a lot of stuff I forgot.
| 4. Sparse code is mostly untouched. When dealing with diagonal &
| permutation matrices as sparse, the performance hit is much smaller
| (not by an order, but a constant factor). As I don't work with sparse
| matrices much, I'd appreciate advice about what interactions would be
| beneficial to implement (e.g., diagonal-sparse multiplication etc).

I think this is a good change and the performance improvements are
impressive.  However, I see the following problem:

  octave:1> x = diag ([1,2,3,4]);
  octave:2> save foo.dat x
  octave:3> load foo.dat
  error: octave_base_value::load_ascii(): wrong type argument `diagonal matrix'
  error: load: trouble reading ascii file `foo.dat'
  error: load: reading file foo.dat

Until this is fixed, I can't make a new snapshot, because this
represents a regression from previous versions and will likely be
noticed immediately by more than a few people who try to use the new
snapshot...

jwe
Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

Jaroslav Hajek-2
On Thu, Dec 4, 2008 at 6:13 PM, John W. Eaton <[hidden email]> wrote:

> On  4-Dec-2008, Jaroslav Hajek wrote:
>
> | OK, I've pushed the patches.
> |
> | Rationale: as in the first mail.
> |
> | Summary of changes:
> |
> | Rectangular diagonal matrices are now special objects.
> |
> | [...]
> |
> | i.e. a 38x, resp. 100x speed-up (will increase with n).
> |
> | TODO:
> | 1. special printing/saving of diagonal & permutation matrices
> | 2. make "balance" benefit from the changes
> | 3. I'm sure there's a lot of stuff I forgot.
> | 4. Sparse code is mostly untouched. When dealing with diagonal &
> | permutation matrices as sparse, the performance hit is much smaller
> | (not by an order, but a constant factor). As I don't work with sparse
> | matrices much, I'd appreciate advice about what interactions would be
> | beneficial to implement (e.g., diagonal-sparse multiplication etc).
>
> I think this is a good change and the performance improvements are
> impressive.  However, I see the following problem:
>
>  octave:1> x = diag ([1,2,3,4]);
>  octave:2> save foo.dat x
>  octave:3> load foo.dat
>  error: octave_base_value::load_ascii(): wrong type argument `diagonal matrix'
>  error: load: trouble reading ascii file `foo.dat'
>  error: load: reading file foo.dat
>
> Until this is fixed, I can't make a new snapshot, because this
> represents a regression from previous versions and will likely be
> noticed immediately by more than a few people who try to use the new
> snapshot...
>
> jwe
>

OK, this is bad.
Currently, octave_base_diag::save_ascii simply densifies the matrix
and calls save_ascii on the dense instance. Apparently the saving
process is more complicated than I thought. Shame on me not to have
tried it - I'll fix it.

--
RNDr. Jaroslav Hajek
computing expert
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz
Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

Søren Hauberg
In reply to this post by John W. Eaton
tor, 04 12 2008 kl. 12:08 -0500, skrev John W. Eaton:
> | Not if you print it like a vector, which I think we should just always
> | do.
>
> Since it seems unlikely that we will all agree on the best way to
> display diagonal matrices, maybe it would be best to allow different
> output formats based on some preference?

Just for the record: while I prefer the vector style printing, I don't
have a strong opinion. So, I'd say, pick one format and stick with that,
instead of implementing them all.

Søren

Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

Jaroslav Hajek-2
In reply to this post by John W. Eaton
On Thu, Dec 4, 2008 at 6:08 PM, John W. Eaton <[hidden email]> wrote:

> On  3-Dec-2008, Søren Hauberg wrote:
>
> | ons, 03 12 2008 kl. 14:31 +0100, skrev Jaroslav Hajek:
> | > >> ans =
> | > >> diagonal matrix (5x5)
> | > >>    1       ..      0
> | > >>        2
> | > >>    :       3       :
> | > >>                4
> | > >>    0       ..      5
> | > >>
> | > >> the last form seems attractive, but maybe requires a bit more thinking
> | > >> about how to place the dots etc.
> | > >
> | > > This I don't think will work. It would be very hard to read a 20x20
> | > > matrix in a terminal with a width of 80 chars. But I agree, that it
> | > > looks nice for small matrices.
> | > >
> | >
> | > But anyone trying to work visually with 20x20 matrices on a 80-column
> | > terminal is bound to have problems.
> |
> | Not if you print it like a vector, which I think we should just always
> | do.
>
> Since it seems unlikely that we will all agree on the best way to
> display diagonal matrices, maybe it would be best to allow different
> output formats based on some preference?  Perhaps as additional
> options to the format function.  How about something like
>
>  format diag full
>  format diag row
>  format diag column
>

This sounds like a good compromise.

> I'm not sure about the format you show above.  It looks good for small
> matrices with integer elements, but the expectation is that matrices
> too wide for a terminal window will be wrapped, and I don't think this
> format would look very good when wrapped, but maybe it would work.
>
> jwe
>



--
RNDr. Jaroslav Hajek
computing expert
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz

Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

Jaroslav Hajek-2
In reply to this post by John W. Eaton
On Thu, Dec 4, 2008 at 6:13 PM, John W. Eaton <[hidden email]> wrote:

> On  4-Dec-2008, Jaroslav Hajek wrote:
>
> | OK, I've pushed the patches.
> |
> | Rationale: as in the first mail.
> |
> | Summary of changes:
> |
> | Rectangular diagonal matrices are now special objects.
> |
> | [...]
> |
> | i.e. a 38x, resp. 100x speed-up (will increase with n).
> |
> | TODO:
> | 1. special printing/saving of diagonal & permutation matrices
> | 2. make "balance" benefit from the changes
> | 3. I'm sure there's a lot of stuff I forgot.
> | 4. Sparse code is mostly untouched. When dealing with diagonal &
> | permutation matrices as sparse, the performance hit is much smaller
> | (not by an order, but a constant factor). As I don't work with sparse
> | matrices much, I'd appreciate advice about what interactions would be
> | beneficial to implement (e.g., diagonal-sparse multiplication etc).
>
> I think this is a good change and the performance improvements are
> impressive.  However, I see the following problem:
>
>  octave:1> x = diag ([1,2,3,4]);
>  octave:2> save foo.dat x
>  octave:3> load foo.dat
>  error: octave_base_value::load_ascii(): wrong type argument `diagonal matrix'
>  error: load: trouble reading ascii file `foo.dat'
>  error: load: reading file foo.dat
>
> Until this is fixed, I can't make a new snapshot, because this
> represents a regression from previous versions and will likely be
> noticed immediately by more than a few people who try to use the new
> snapshot...
>
> jwe
>

Hi John,

saving diagonal & permutation matrices to octave's native ascii and
binary formats is now implemented. The format is like this:

# Created by Octave 3.1.51+, Mon Dec 08 12:45:48 2008 CET <hajek@hajek>
# name: a
# type: diagonal matrix
# rows: 4
# columns: 4
1
2
3
4
# name: p
# type: permutation matrix
# size: 4
# orient: r
1
2
3
4

and similarly for binary data.
Saving to HDF5 doesn't work yet. I think I'd appreciate advice here on
how to do things. It appears that Octave stores typeinfo along with
variables in hdf5 format, just as it does in the native formats, and
expects every class to hdf5-saveable.
But isn't the hdf5 format meant to be an export format? In that case,
maybe the optimized matrices should be converted to full before
saving?

regards


--
RNDr. Jaroslav Hajek
computing expert
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz
Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

John W. Eaton
Administrator
On  8-Dec-2008, Jaroslav Hajek wrote:

| and similarly for binary data.
| Saving to HDF5 doesn't work yet. I think I'd appreciate advice here on
| how to do things. It appears that Octave stores typeinfo along with
| variables in hdf5 format, just as it does in the native formats, and
| expects every class to hdf5-saveable.
| But isn't the hdf5 format meant to be an export format? In that case,
| maybe the optimized matrices should be converted to full before
| saving?

I'm not sure of the purpose of the HDF5 format.  For example, can
Matlab cload HDF files created by Octave?  If so, it would be nice to
preserve that capability, but if they can't be loaded directly now,
then I'm not sure it matters much and you might as well save the new
data types in whatever format makes the most sense.

jwe
Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

dbateman-2
On Tue, Dec 09, 2008 at 12:52:15PM -0500, John W. Eaton wrote:
> I'm not sure of the purpose of the HDF5 format.  For example, can
> Matlab cload HDF files created by Octave?  If so, it would be nice to
> preserve that capability, but if they can't be loaded directly now,
> then I'm not sure it matters much and you might as well save the new
> data types in whatever format makes the most sense.

The Octave HDF5 foormat was a hack before I properly understood HDF5
attributes. No Octqve HDF5 aren't loaded correctly with matlab. The
matlab v7.3 file format is basically an HDF5 with the header of the HDF5
defined to be the matlab MAT-file header. It would be good to implement
this format and then rewrite the Octave format to be similar...

D.


>
> jwe
Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

Jaroslav Hajek-2
On Tue, Dec 9, 2008 at 9:04 PM,  <[hidden email]> wrote:

> On Tue, Dec 09, 2008 at 12:52:15PM -0500, John W. Eaton wrote:
>> I'm not sure of the purpose of the HDF5 format.  For example, can
>> Matlab cload HDF files created by Octave?  If so, it would be nice to
>> preserve that capability, but if they can't be loaded directly now,
>> then I'm not sure it matters much and you might as well save the new
>> data types in whatever format makes the most sense.
>
> The Octave HDF5 foormat was a hack before I properly understood HDF5
> attributes. No Octqve HDF5 aren't loaded correctly with matlab. The
> matlab v7.3 file format is basically an HDF5 with the header of the HDF5
> defined to be the matlab MAT-file header. It would be good to implement
> this format and then rewrite the Octave format to be similar...
>

I see. Is it OK if I just leave it in the current state (no saving of
diagonal matrices) and wait until it gets fixed?

> D.
>
>
>>
>> jwe
>



--
RNDr. Jaroslav Hajek
computing expert
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz
Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

John W. Eaton
Administrator
On 10-Dec-2008, Jaroslav Hajek wrote:

| I see. Is it OK if I just leave it in the current state (no saving of
| diagonal matrices) and wait until it gets fixed?

Would that mean that save would fail with an error if someone
attempted to save a workspace that contained a diagonal matrix?  That
would be surprising and a regression from previous versions of Octave,
so I think we should at least convert them to full matrices that can
be saved in the HDF format.

jwe
Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

Jaroslav Hajek-2
---------- Forwarded message ----------
From: Jaroslav Hajek <[hidden email]>
Date: Wed, Dec 10, 2008 at 7:17 PM
Subject: Re: diagonal matrices specializations
To: "John W. Eaton" <[hidden email]>


On Wed, Dec 10, 2008 at 2:16 PM, John W. Eaton <[hidden email]> wrote:

> On 10-Dec-2008, Jaroslav Hajek wrote:
>
> | I see. Is it OK if I just leave it in the current state (no saving of
> | diagonal matrices) and wait until it gets fixed?
>
> Would that mean that save would fail with an error if someone
> attempted to save a workspace that contained a diagonal matrix?  That
> would be surprising and a regression from previous versions of Octave,
> so I think we should at least convert them to full matrices that can
> be saved in the HDF format.
>

OK, that's right, it will fail. However, the converting is not that
easy, because what the HDF5 saving code does is that it asks an
octave_value "what type are you of?", then stores the type name, and
then calls the value "now save yourself".
Instead, the sequence needs to be changed to:
"are you hdf-5 saveable?"
if yes, then "what type are you of?", write type, "save yourself".
if not, then "convert yourself" (numeric_conversion), and repeat.
No big deal, but a little more work.
Apparently whoever implemented the hdf5 exporting assumed that
Octave's types map exactly to the exported data. If David (or someone
else) wants to rewrite the hdf5 code, then he may want to things in a
completely different manner, so the work will be useless.
Perhaps David can comment on whether the above pictured change is the
way to go, or if he has a different idea.

--
RNDr. Jaroslav Hajek
computing expert
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz



--
RNDr. Jaroslav Hajek
computing expert
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz
Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

Jaroslav Hajek-2
In reply to this post by John W. Eaton
On Wed, Dec 10, 2008 at 7:17 PM, Jaroslav Hajek <[hidden email]> wrote:

> On Wed, Dec 10, 2008 at 2:16 PM, John W. Eaton <[hidden email]> wrote:
>> On 10-Dec-2008, Jaroslav Hajek wrote:
>>
>> | I see. Is it OK if I just leave it in the current state (no saving of
>> | diagonal matrices) and wait until it gets fixed?
>>
>> Would that mean that save would fail with an error if someone
>> attempted to save a workspace that contained a diagonal matrix?  That
>> would be surprising and a regression from previous versions of Octave,
>> so I think we should at least convert them to full matrices that can
>> be saved in the HDF format.
>>
>
> OK, that's right, it will fail. However, the converting is not that
> easy, because what the HDF5 saving code does is that it asks an
> octave_value "what type are you of?", then stores the type name, and
> then calls the value "now save yourself".
> Instead, the sequence needs to be changed to:
> "are you hdf-5 saveable?"
> if yes, then "what type are you of?", write type, "save yourself".
> if not, then "convert yourself" (numeric_conversion), and repeat.
> No big deal, but a little more work.

In particular, it would involve adding an is_hdf5_saveable virtual
method to octave_base_value and
then overriding it in just about every subclass (because making the
default true is a nonsense).
That's why I'm reluctant to do it at once - perhaps it won't be
necessary. Maybe hdf5 format should only recognize a fixed set of
octave's value classes (e.g. array, sparse matrix, scalar...).

> Apparently whoever implemented the hdf5 exporting assumed that
> Octave's types map exactly to the exported data. If David (or someone
> else) wants to rewrite the hdf5 code, then he may want to things in a
> completely different manner, so the work will be useless.
> Perhaps David can comment on whether the above pictured change is the
> way to go, or if he has a different idea.
>
> --
> RNDr. Jaroslav Hajek
> computing expert
> Aeronautical Research and Test Institute (VZLU)
> Prague, Czech Republic
> url: www.highegg.matfyz.cz
>



--
RNDr. Jaroslav Hajek
computing expert
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz
Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

John W. Eaton
Administrator
In reply to this post by Jaroslav Hajek-2
On 11-Dec-2008, Jaroslav Hajek wrote:

| OK, that's right, it will fail. However, the converting is not that
| easy, because what the HDF5 saving code does is that it asks an
| octave_value "what type are you of?", then stores the type name, and
| then calls the value "now save yourself".
| Instead, the sequence needs to be changed to:
| "are you hdf-5 saveable?"
| if yes, then "what type are you of?", write type, "save yourself".
| if not, then "convert yourself" (numeric_conversion), and repeat.

What is preventing us from simply saving the new data types?  Isn't it
just the type name (as usual) plus dimensions (usual) and then the
array that makes up the diagonal (or permutation) matrix?  Why not
just write that code so that saving and loading these data types at
least works, and then users of 3.2 (and snapshots leading up to it)
won't see any surprising regressions from 3.0?

jwe
Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

Jaroslav Hajek-2
On Thu, Dec 11, 2008 at 3:24 PM, John W. Eaton <[hidden email]> wrote:

> On 11-Dec-2008, Jaroslav Hajek wrote:
>
> | OK, that's right, it will fail. However, the converting is not that
> | easy, because what the HDF5 saving code does is that it asks an
> | octave_value "what type are you of?", then stores the type name, and
> | then calls the value "now save yourself".
> | Instead, the sequence needs to be changed to:
> | "are you hdf-5 saveable?"
> | if yes, then "what type are you of?", write type, "save yourself".
> | if not, then "convert yourself" (numeric_conversion), and repeat.
>
> What is preventing us from simply saving the new data types?  Isn't it
> just the type name (as usual) plus dimensions (usual) and then the
> array that makes up the diagonal (or permutation) matrix?  Why not
> just write that code so that saving and loading these data types at
> least works, and then users of 3.2 (and snapshots leading up to it)
> won't see any surprising regressions from 3.0?
>

Yes, that is of course possible, just I wasn't sure whether it's the
right thing. If the
Octave's hdf5 format is only supposed to be read by Octave, then that's OK.
But then I don't understand its purpose as I see no advantages over
Octave's own
binary format.
If, on the contrary, the saved hdf5 files are supposed to be read by
other software
(like R, Matlab, IDL or whatever), then maybe saving a diagonal matrix
into them is
a bad idea, as the other software won't understand it.

If you say that we're in the first situation where Octave's hdf5
format is just for Octave,
I can readily implement the solution just described. But, as David
said, that was not
his intention when implementing hdf5 format. I guess it all comes down
to the question
of whether David is going to do something with the hdf5 format in the
near future
(before 3.2.0 is out). If not, I'll implement hdf5 saving/loading
analogically to what I did
with ASCII/binary format, because crashing is not acceptable.
David, can you comment...?

> jwe
>



--
RNDr. Jaroslav Hajek
computing expert
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz
Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

dbateman-2
On Thu, Dec 11, 2008 at 04:01:20PM +0100, Jaroslav Hajek wrote:

> Yes, that is of course possible, just I wasn't sure whether it's the
> right thing. If the
> Octave's hdf5 format is only supposed to be read by Octave, then that's OK.
> But then I don't understand its purpose as I see no advantages over
> Octave's own
> binary format.
> If, on the contrary, the saved hdf5 files are supposed to be read by
> other software
> (like R, Matlab, IDL or whatever), then maybe saving a diagonal matrix
> into them is
> a bad idea, as the other software won't understand it.

Things that can read general HDF5 files can read the Octave HDF5 file
format and then decide what to do with it. That doesn't mean that Octave
can't have its own structure for the HDF5 files.

My mistake when implementing this has to create a group for each
variable with two sub values. One with a string for the type of the
variable. The right way to do it would have been to save the value
directly and make the type an attribute of the value, much as matlab
does. In this way thing that can load HDF5 files but don't understand a
uint16 can still load it as a double array, and other such things.

As the attribute is added after the value is saved the seting of the
type attribute would be moved to the save_hdf5 method and your
is_hdf5_saveable method becomes redundant.


> If you say that we're in the first situation where Octave's hdf5
> format is just for Octave,
> I can readily implement the solution just described. But, as David
> said, that was not
> his intention when implementing hdf5 format. I guess it all comes down
> to the question
> of whether David is going to do something with the hdf5 format in the
> near future
> (before 3.2.0 is out). If not, I'll implement hdf5 saving/loading
> analogically to what I did
> with ASCII/binary format, because crashing is not acceptable.
> David, can you comment...?

I'd hope the octave HDF5 format was useful elsewhere than Octave, but
suspect its not used much. I'd say what should be done is

* Implement the save/load hdf5 methods for diagonal matrices in an
analogous way to the othe hdf5 load/save methods.
* In the long term (after 3.2), modify the HDF5 code such that the
format is matlab compatiable for the v7.3 file format and dump the
current HDF5 format...
* Write a simple conversion function for the deprecated octave hdf5 file
to the new format.

Frankly, I'm not sure I have the energy to consider this as a personal
project.

Regards
David




>
> > jwe
> >
>
>
>
> --
> RNDr. Jaroslav Hajek
> computing expert
> Aeronautical Research and Test Institute (VZLU)
> Prague, Czech Republic
> url: www.highegg.matfyz.cz
Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

Bill Denney-5
In reply to this post by Jaroslav Hajek-2
Jaroslav Hajek wrote:
> Yes, that is of course possible, just I wasn't sure whether it's the
> right thing. If the
> Octave's hdf5 format is only supposed to be read by Octave, then that's OK.

So, I've not followed all of this in detail, but it seems to me that
loading a variable from disk would take significantly longer than the
determination if it is a diagonal matrix.  How difficult would it be to
save files using the full format (or as a sparse matrix which probably
makes more sense) and upon loading determine if it is diagonal and make
it have the right class then.  That should work for all external formats

I may be way off, but that seems like the best way to deal with diagonal
matrices when loading and with external formats.

Have a good day,

Bill
Reply | Threaded
Open this post in threaded view
|

Re: diagonal matrices specializations

Jaroslav Hajek-2
In reply to this post by dbateman-2
On Thu, Dec 11, 2008 at 9:42 PM,  <[hidden email]> wrote:

> On Thu, Dec 11, 2008 at 04:01:20PM +0100, Jaroslav Hajek wrote:
>> Yes, that is of course possible, just I wasn't sure whether it's the
>> right thing. If the
>> Octave's hdf5 format is only supposed to be read by Octave, then that's OK.
>> But then I don't understand its purpose as I see no advantages over
>> Octave's own
>> binary format.
>> If, on the contrary, the saved hdf5 files are supposed to be read by
>> other software
>> (like R, Matlab, IDL or whatever), then maybe saving a diagonal matrix
>> into them is
>> a bad idea, as the other software won't understand it.
>
> Things that can read general HDF5 files can read the Octave HDF5 file
> format and then decide what to do with it. That doesn't mean that Octave
> can't have its own structure for the HDF5 files.
>
> My mistake when implementing this has to create a group for each
> variable with two sub values. One with a string for the type of the
> variable. The right way to do it would have been to save the value
> directly and make the type an attribute of the value, much as matlab
> does. In this way thing that can load HDF5 files but don't understand a
> uint16 can still load it as a double array, and other such things.
>
> As the attribute is added after the value is saved the seting of the
> type attribute would be moved to the save_hdf5 method and your
> is_hdf5_saveable method becomes redundant.
>
>
>> If you say that we're in the first situation where Octave's hdf5
>> format is just for Octave,
>> I can readily implement the solution just described. But, as David
>> said, that was not
>> his intention when implementing hdf5 format. I guess it all comes down
>> to the question
>> of whether David is going to do something with the hdf5 format in the
>> near future
>> (before 3.2.0 is out). If not, I'll implement hdf5 saving/loading
>> analogically to what I did
>> with ASCII/binary format, because crashing is not acceptable.
>> David, can you comment...?
>
> I'd hope the octave HDF5 format was useful elsewhere than Octave, but
> suspect its not used much. I'd say what should be done is
>
> * Implement the save/load hdf5 methods for diagonal matrices in an
> analogous way to the othe hdf5 load/save methods.
> * In the long term (after 3.2), modify the HDF5 code such that the
> format is matlab compatiable for the v7.3 file format and dump the
> current HDF5 format...
> * Write a simple conversion function for the deprecated octave hdf5 file
> to the new format.
>
> Frankly, I'm not sure I have the energy to consider this as a personal
> project.
>

Unfortunately I don't understand the hdf5 format, so I'm not even sure
how the "analogous way" you refer to in point 1 should look like. So
for the time being I've committed a simplistic FIXME'd hack that
simply detects the special matrices in add_hdf5_data and densifies
them, so that this issue doesn't block the progress for 3.2.
This will be easy to revert if something more proper is to be implemented.

cheers

--
RNDr. Jaroslav Hajek
computing expert
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz