Problem processing data logs

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem processing data logs

Ian McCallion
I will be receiving data logs from several places across the world which I wish to process and present in the appropriate local timezone. Since the event times are in UTC I need a function to convert UTC to time in the appropriate timezone. I have tried:

function lt =  mylocaltime(EventSecondsSinceEpoch, timezonestring)   
   setenv('tz','EST5EDT,M3.2.0/02:00:00,M11.1.0')
   lt = localtime(EventSecondsSinceEpoch)
   setenv('tz','')
endfunction

but (at least on windows 10 with Octave 4.4.1) this does not work. This however does work:

function lt =  mylocaltime(EventSecondsSinceEpoch, timezonestring)     
   setenv('tz','EST5EDT,M3.2.0/02:00:00,M11.1.0')
   pause(1)
   lt = localtime(EventSecondsSinceEpoch)
   setenv('tz','')
endfunction

Unfortunately as well as taking a second for a microsecond's worth of computation it causes unacceptable secondary effects on Octave itself, such as any file open in the Octave editor being reported as out of date.

Is there another way to implement  mylocaltime()  in Octave without recoding from scratch the logic needed to process a timezone string?

Cheers... Ian


Reply | Threaded
Open this post in threaded view
|

Re: Problem processing data logs

Mike Miller-4
On Thu, Jan 03, 2019 at 13:25:23 +0000, Ian McCallion wrote:
> Is there another way to implement  mylocaltime()  in Octave without
> recoding from scratch the logic needed to process a timezone string?

If setting and unsetting the TZ environment variable is a big problem on
Windows, I can't think of another way to do it readily in Octave at the
moment.

However, you might be able to rely on an external program which has a
separate environment space. The following example works for me (on
GNU/Linux).

    function lt = mylocaltime (seconds, tz)    
      fmt = "%S %M %H %d %m %Y";
      cmd = sprintf ("env TZ='%s' date -d@%lu +'%s'", tz, seconds, fmt);
      [status, output] = system (cmd);
      values = str2double (strsplit (output));
      lt.sec = values(1);
      lt.min = values(2);
      lt.hour = values(3);
      lt.mday = values(4);
      lt.mon = values(5);
      lt.year = values(6);
    endfunction

You can tailor this to whichever broken-down time fields you need to
extract.

--
mike



signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Problem processing data logs

apjanke-floss

On 1/3/19 4:44 PM, Mike Miller wrote:

> On Thu, Jan 03, 2019 at 13:25:23 +0000, Ian McCallion wrote:
>> Is there another way to implement  mylocaltime()  in Octave without
>> recoding from scratch the logic needed to process a timezone string?
>
> If setting and unsetting the TZ environment variable is a big problem on
> Windows, I can't think of another way to do it readily in Octave at the
> moment.
>
> However, you might be able to rely on an external program which has a
> separate environment space. The following example works for me (on
> GNU/Linux).
>
>      function lt = mylocaltime (seconds, tz)
>        fmt = "%S %M %H %d %m %Y";
>        cmd = sprintf ("env TZ='%s' date -d@%lu +'%s'", tz, seconds, fmt);
>        [status, output] = system (cmd);
>        values = str2double (strsplit (output));
>        lt.sec = values(1);
>        lt.min = values(2);
>        lt.hour = values(3);
>        lt.mday = values(4);
>        lt.mon = values(5);
>        lt.year = values(6);
>      endfunction
>
> You can tailor this to whichever broken-down time fields you need to
> extract.
>

If your Octave is compiled with Java support, you can do it using Java
objects. The new java.time API has good support for time zone
calculations. That would be faster than shelling out to an external
program and parsing results for each date, and portable too.


function out = utc_to_local_time(time, tz)
   % Convert a date in UTC to an arbitrary time zone
   % tz is a string holding the time zone name in "Olson database" format

   % In production code, you should cache all these parser values in the
fields
   % of an Octave object
   format = 'yyyy-MM-dd kk:mm:ss[.SSS]';
   javaTzUtc = javaMethod('of', 'java.time.ZoneId', 'UTC');
   javaTz = javaMethod('of', 'java.time.ZoneId', tz);
   parser = javaMethod('ofPattern',
'java.time.format.DateTimeFormatter', format);
   javaDate = javaMethod('parse', 'java.time.LocalDateTime', time, parser);
   javaDateUtc = javaDate.atZone(javaTzUtc);
   javaDateLocal = javaDateUtc.withZoneSameInstant(javaTz);

   % This conversion is kinda inefficient. You could speed it up by
grabbing the Unix
   % timestamp value from javaDate and converting that directly to a
datenum. Better
   % yet, vectorize it. I'm just too lazy to work out the math to do the
epoch
   % conversion right now.
   jd = javaDateLocal;
   dnum = datenum(double(jd.getYear()), double(jd.getMonthValue()),
double(jd.getDayOfMonth()), ...
     double(jd.getHour()), double(jd.getMinute()), double(jd.getSecond()));
   nanos = double(jd.getNano());
   nanosPerDay = (10^9) * 60 * 60 * 24;
   dnum = dnum + (nanos / nanosPerDay);
   out = dnum;
end


Tested on Octave 4.4.1 with Java 1.8.

 >> d = utc_to_local_time('2011-12-14 12:34:56', 'America/New_York')
d =  734851.31593
 >> datestr(d)
ans = 14-Dec-2011 07:34:56

Cheers,
Andrew


Reply | Threaded
Open this post in threaded view
|

Re: Problem processing data logs

Ian McCallion
Thank you Mike and Andrew for the swift responses.  Mik'e neat trick of setting the TZ environment variable in an external program works well for my needs (and on windows even!), but Andrew's suggestion to use java did not work because my Octave does not find java, even though it is installed. I have Changed Mike's function slightly (see below) to make it equivalent to the localtime() function, i.e. it produces a time structure compatible with localtime apart from exceptions noted in the help text.

Cheers... Ian

%% -*- texinfo -*-
%% @deftypefn {} {} localtime2 (@var{Seconds}, @var{TimeZone})
%% Given a time value specified in Seconds Since Epoch (1/1/1900), and a timezone specified
%% return a time structure corresponding to the local time zone of the specified timezone.
%% When called with an empty string for the timezone produces the same results as localtime.
%% Example:
%%
%% @example
%% @group
%% localtime2(lstat('filename').mtime, 'CST6CDT,M3.2.0/02:00:00,M11.1.0') ;
%%    => {
%%       usec = 0
%%       sec =  32
%%       min =  29
%%       hour =  6
%%       mday =  4
%%       mon = 0
%%       year =  119
%%       wday =  5
%%       yday =  3
%%       isdst = 0
%%       gmtoff = 0
%%       zone = EST
%%       }
%% @end group
%% @end example
%% @noindent
%% will return a time structure corresponding to the time
%% (in the specified local time zone) of the time the file was modified.
%%
%% NOTE. Does not currently implement the gmtoff or isdst fields of the time structure.
%%       Does not support fractional seconds.
%% @seealso{localtime}
%% @end deftypefn

%% Author: IMM
   function lt = localtime2 (seconds, tz)     
      cmd = sprintf ("env TZ='%s' date -d@%lu +'%s'", tz, seconds, "%N %S %M %H %d %m %Y %w %j %Z");
      [status, output] = system (cmd);
      values = str2double (strsplit (output));
      lt.usec = values(1)*1000;
      lt.sec  = values(2);
      lt.min  = values(3);
      lt.hour = values(4);
      lt.mday = values(5);
      lt.mon  = values(6)-1;
      lt.year = values(7)-1900;
      lt.wday = values(8);
      lt.yday = values(9)-1;
      lt.isdst = NaN; 
      lt.gmtoff = NaN;
      lt.zone = strsplit(output){10};
    endfunction


On Thu, 3 Jan 2019 at 22:37, Andrew Janke <[hidden email]> wrote:

On 1/3/19 4:44 PM, Mike Miller wrote:
> On Thu, Jan 03, 2019 at 13:25:23 +0000, Ian McCallion wrote:
>> Is there another way to implement  mylocaltime()  in Octave without
>> recoding from scratch the logic needed to process a timezone string?
>
> If setting and unsetting the TZ environment variable is a big problem on
> Windows, I can't think of another way to do it readily in Octave at the
> moment.
>
> However, you might be able to rely on an external program which has a
> separate environment space. The following example works for me (on
> GNU/Linux).
>
>      function lt = mylocaltime (seconds, tz)
>        fmt = "%S %M %H %d %m %Y";
>        cmd = sprintf ("env TZ='%s' date -d@%lu +'%s'", tz, seconds, fmt);
>        [status, output] = system (cmd);
>        values = str2double (strsplit (output));
>        lt.sec = values(1);
>        lt.min = values(2);
>        lt.hour = values(3);
>        lt.mday = values(4);
>        lt.mon = values(5);
>        lt.year = values(6);
>      endfunction
>
> You can tailor this to whichever broken-down time fields you need to
> extract.
>

If your Octave is compiled with Java support, you can do it using Java
objects. The new java.time API has good support for time zone
calculations. That would be faster than shelling out to an external
program and parsing results for each date, and portable too.


function out = utc_to_local_time(time, tz)
   % Convert a date in UTC to an arbitrary time zone
   % tz is a string holding the time zone name in "Olson database" format

   % In production code, you should cache all these parser values in the
fields
   % of an Octave object
   format = 'yyyy-MM-dd kk:mm:ss[.SSS]';
   javaTzUtc = javaMethod('of', 'java.time.ZoneId', 'UTC');
   javaTz = javaMethod('of', 'java.time.ZoneId', tz);
   parser = javaMethod('ofPattern',
'java.time.format.DateTimeFormatter', format);
   javaDate = javaMethod('parse', 'java.time.LocalDateTime', time, parser);
   javaDateUtc = javaDate.atZone(javaTzUtc);
   javaDateLocal = javaDateUtc.withZoneSameInstant(javaTz);

   % This conversion is kinda inefficient. You could speed it up by
grabbing the Unix
   % timestamp value from javaDate and converting that directly to a
datenum. Better
   % yet, vectorize it. I'm just too lazy to work out the math to do the
epoch
   % conversion right now.
   jd = javaDateLocal;
   dnum = datenum(double(jd.getYear()), double(jd.getMonthValue()),
double(jd.getDayOfMonth()), ...
     double(jd.getHour()), double(jd.getMinute()), double(jd.getSecond()));
   nanos = double(jd.getNano());
   nanosPerDay = (10^9) * 60 * 60 * 24;
   dnum = dnum + (nanos / nanosPerDay);
   out = dnum;
end


Tested on Octave 4.4.1 with Java 1.8.

 >> d = utc_to_local_time('2011-12-14 12:34:56', 'America/New_York')
d =  734851.31593
 >> datestr(d)
ans = 14-Dec-2011 07:34:56

Cheers,
Andrew