When messages are %RESCANed hpt doesn't use the __ftsc_date field from the Squish message base, but the 2-second precision DOS date. Which means some messages (~50%) are exported with the time one second off. Some tosser's dupe checkers might fail ...

---
* Origin: . (2:280/464.47)

Hello Oli!

30 Jan 21, Oli wrote to All:

When messages are %RESCANed hpt doesn't use the __ftsc_date field from
the Squish message base, but the 2-second precision DOS date. Which
means some messages (~50%) are exported with the time one second off.

I've never seen uneven timestamps in my editor. Hell, at the moment of writing i see 12:43:49 in the header on the from/to lines. But after saving the mail it switched to 12:43:48, lucky me. ;-)

As far as i remember squish does have a 2 seconds resolution. But i don't know why. If i do not see further than my nose i would say 2 second resolution is the standard and if you have uneven timestamps in your msgbase than that is the bug.

Some tosser's dupe checkers might fail ...

Is this speculation/theory or could you name a real issue?

Regards

Kai

--- GoldED+/LNX 1.1.4.7
* Origin: Monobox (2:240/77)

Kai wrote (2021-02-01):

When messages are %RESCANed hpt doesn't use the __ftsc_date field
from the Squish message base, but the 2-second precision DOS date.
Which means some messages (~50%) are exported with the time one
second off.

I've never seen uneven timestamps in my editor. Hell, at the moment of writing i see 12:43:49 in the header on the from/to lines. But after saving the mail it switched to 12:43:48, lucky me. ;-)

Your messages are dupe save ;).

As far as i remember squish does have a 2 seconds resolution. But i don't know why. If i do not see further than my nose i would say 2 second resolution is the standard and if you have uneven timestamps in your msgbase than that is the bug.

The DOS time format uses a 2 seconds resolution. The creation and received date is stored has DOS time in the Squish message base. Usually you don't see any uneven times in your message editor, but in Golded you can hit I and look at the hexdump of the message header. There you should see the original date in ASCII (only in a Squish base, JAM is different). It's stored in a field called __ftsc_date.

Some tosser's dupe checkers might fail ...

Is this speculation/theory or could you name a real issue?

This is what other nodes have reported and I have no reason to doubt it. Don't you think there are tossers out there which believe that messages with different time stamps are not the same? But it really doesn't matter, because per Squish specification the __ftsc_date should be used on export (rescan). What would be the benefit of not using the exact time when it's available? (hpt does store the original time in __ftsc_date, it just ignores it on export).

---
* Origin: . (2:280/464.47)

Hello Oli!

01 Feb 21, Oli wrote to Kai Richter:

say 2 second resolution is the standard and if you have uneven
timestamps in your msgbase than that is the bug.

The DOS time format uses a 2 seconds resolution. The creation and
received date is stored has DOS time in the Squish message base.

After export the pkt message does no longer have creation date? As for FTS-0001 it's the last edited date?

Some tosser's dupe checkers might fail ...

Isn't the MSGID the solution for that problem?

Is this speculation/theory or could you name a real issue?

This is what other nodes have reported and I have no reason to doubt
it. Don't you think there are tossers out there which believe that
messages with different time stamps are not the same?

I don't know (out there). I think because of ASCII DateTime would not effect DOS 16-bit file format date stamps it should be ok to see different time stamps as different mails.

I don't doubt that the problem exist but for fault confirmation and trouble shooting it is easier to know the software in use.

doesn't matter, because per Squish specification the __ftsc_date

I will forward you my messy research for the format. In short: For FTS-0001 DateTime is the creation date of the message. A packed message use the last edited date. For type 2 pkt format it's the pkt creation date.

After reading those specs i would recomment msgid for dupe checking. ;-)

should be used on export (rescan). What would be the benefit of not
using the exact time when it's available?

Same like if using: Nothing? Even with a fix available - every hpt system have to use that fix to get rid of the problem. And i think we both know that that is most unlikely to happen.

Regards

Kai

--- GoldED+/LNX 1.1.4.7
* Origin: Monobox (2:240/77)

Kai wrote (2021-02-02):

Hello Oli!

01 Feb 21, Oli wrote to Kai Richter:

say 2 second resolution is the standard and if you have uneven
timestamps in your msgbase than that is the bug.

The DOS time format uses a 2 seconds resolution. The creation and
received date is stored has DOS time in the Squish message base.

After export the pkt message does no longer have creation date? As for FTS-0001 it's the last edited date?

Let's consult the Squish Developers Kit. This stuff is stored in the message header in the .sqd file.

The XMSG structure has the following format:
[...]
date_written SCOMBO 164 Date that the message was written.

date_arrived SCOMBO 168 Date that the message was placed in
[...]
__ftsc_date char[20] 218 FTS-0001 compatible date. Squish
applications should not access this
field directly. This field is used
exclusively by tossers and scanners
for preserving the original ASCII
message date. Squish applications
should use the binary dates in
date_written and date_arrived to
retrieve the message date.

If a message is exported from the Squish message base and has an non-empty __ftsc_date field, the tosser should use the contents of that field for the date. If the message has been created locally (no __ftsc_date), the tosser uses the date_written field. (SCOMBO is DOS datetime)

Some tosser's dupe checkers might fail ...

Isn't the MSGID the solution for that problem?

Yes, MSGIDs is very effective in sort out dupes, but not all messages have a MSGID (see your argument below: old software).

Is this speculation/theory or could you name a real issue?

This is what other nodes have reported and I have no reason to doubt
it. Don't you think there are tossers out there which believe that
messages with different time stamps are not the same?

I don't know (out there). I think because of ASCII DateTime would not effect DOS 16-bit file format date stamps it should be ok to see
different time stamps as different mails.

I don't doubt that the problem exist but for fault confirmation and trouble shooting it is easier to know the software in use.

hpt does it wrong. I tested it. What else is there to trouble shoot?

doesn't matter, because per Squish specification the __ftsc_date

I will forward you my messy research for the format. In short: For FTS-0001 DateTime is the creation date of the message. A packed message use the last edited date. For type 2 pkt format it's the pkt creation date.

After reading those specs i would recomment msgid for dupe checking. ;-)

not sure this has anything to do with __ftsc_date in the Squish message base.

should be used on export (rescan). What would be the benefit of not
using the exact time when it's available?

Same like if using: Nothing? Even with a fix available - every hpt system have to use that fix to get rid of the problem. And i think we both know that that is most unlikely to happen.

It only affects systems that use a Squish database and have downlinks that do a %RESCAN. If it's not fixed it will be there forever. If it were fixed, affected systems could be upgraded to a newer hpt version. Just because some sysops are to lazy to use up-to-date software doesn't mean we shouldn't care about bugs and bug fixing.

It's just a simple bug ...

---
* Origin: . (2:280/464.47)

Hello Oli!

03 Feb 21, Oli wrote to Kai Richter:

The XMSG structure has the following format:

Thanks, you found the XMSG definition i was looking for.

I don't doubt that the problem exist but for fault confirmation
and trouble shooting it is easier to know the software in use.

hpt does it wrong. I tested it. What else is there to trouble shoot?

The other side of the medal. As said i'm no coder and my search for hpt found this one:

https://github.com/huskyproject/hpt/search?q=__ftsc_date&type=

src/scan.c
96 msg->destAddr.point = xmsg.dest.point;
97 strcpy((char *)msg->datetime, (char *)xmsg.__ftsc_date);
98 xstrcat(&(msg->subjectLine), (char *)xmsg.subj);

This looks like the xmsg.* is the squish base message.
It looks like the ftsc date is overwritten by some copy -> wrong, i'm no coder.

https://www.cplusplus.com/reference/cstring/strcpy/

char * strcpy ( char * destination, const char * source );

Unlike usual copy the C++ strcpy works reversed, the scan.c of hpt get the original squish base ftsc data and uses it for the datetime field.

It's just a simple bug ...

If it is it would be fixed already.
You reported this one a year ago, don't you?

My knowledge does end here. It was interesting but i can't help you further.

It only affects systems that use a Squish database and have downlinks
that do a %RESCAN.

And it does apply to mails without msgid only. Good luck.

Regards

Kai

--- GoldED+/LNX 1.1.4.7
* Origin: Monobox (2:240/77)

Kai wrote (2021-02-03):

It's just a simple bug ...

If it is it would be fixed already.
You reported this one a year ago, don't you?

I don't know why it doesn't get fixed. Maybe nobody wants to touch that part of the code anymore. A project where critical bugs doesn't get fixed anymore is kind of abandoned.

---
* Origin: . (2:280/464.47)

10 Feb 21 09:56, you wrote to Kai Richter:

It's just a simple bug ...

If it is it would be fixed already.
You reported this one a year ago, don't you?

I don't know why it doesn't get fixed. Maybe nobody wants to touch that part of the code anymore. A project where critical bugs doesn't get
fixed anymore is kind of abandoned.

I suspect the patch below fixes this, but I've not tested it.

It's also necessary to verify it doesn't break when rescanning *.MSG and JAM bases.

diff --git a/src/scanarea.c b/src/scanarea.c
index 60ec71bf..8926aa30 100644
-+- a/src/scanarea.c
+++ b/src/scanarea.c
@@ -88,7 +88,7 @@ void makeMsg(HMSG hmsg, XMSG xmsg, s_message * msg, s_area * echo, int action)
msg->destAddr.point = xmsg.dest.point;
msg->attributes = xmsg.attr & ~MSGLOCAL; /* msg should not have MSGLOCAL bit set */

- sc_time((union stamp_combo *)&(xmsg.date_written), (char *)msg->datetime); + strcpy((char *)msg->datetime, (char *) xmsg.__ftsc_date);
xstrcat(&msg->toUserName, (char *)xmsg.to);
xstrcat(&msg->fromUserName, (char *)xmsg.from);
xstrcat(&msg->subjectLine, (char *)xmsg.subj);

--- GoldED+/BSD 1.1.5-b20180707
* Origin: Blizzard of Ozz, Melbourne, Victoria, Australia (3:633/267)

andrew wrote (2021-02-10):

I suspect the patch below fixes this, but I've not tested it.

It's also necessary to verify it doesn't break when rescanning *.MSG and JAM bases.

- sc_time((union stamp_combo *)&(xmsg.date_written), (char *)msg->datetime);
+ strcpy((char *)msg->datetime, (char *) xmsg.__ftsc_date);

What happens if xmsg.__ftsc_date is empty? Local messages don't have an __ftsc_date and it could also be missing for other reasons (messages converted from another message base format, copied/moved in a message editor or tossed by a tosser that doesn't copy the __ftsc_date).

---
* Origin: . (2:280/464.47)

Hello Oli,

Wednesday February 10 2021, Oli wrote to Kai Richter:

It's just a simple bug ...

If it is it would be fixed already.
You reported this one a year ago, don't you?

I don't know why it doesn't get fixed. Maybe nobody wants to touch
that part of the code anymore. A project where critical bugs doesn't
get fixed anymore is kind of abandoned.

So, where is a pull request from you? I don't see it.

As to me, I have a different view on what is critical and what is not so critical. I have a long list of what should be done in Husky and I pick the items from the list according to my idea of which item is more important to me. I may also pick a somewhat less important item if it requires a little time. And I have my life besides Fidonet, particularly I still have a job despite the fact that I am 73 y/o. So I cannot work for Fidonet full time, but only when I have some time for it.

Michael

... node (at) f1042 (dot) ru
--- GoldED+/LNX 1.1.5-b20180707
* Origin: Moscow, Russia (2:5020/1042)

10 Feb 21 11:04, you wrote to me:

I suspect the patch below fixes this, but I've not tested it.

It's also necessary to verify it doesn't break when rescanning
*.MSG and JAM bases.

- sc_time((union stamp_combo *)&(xmsg.date_written), (char
*)msg->datetime); + strcpy((char *)msg->datetime, (char *)
xmsg.__ftsc_date);

What happens if xmsg.__ftsc_date is empty? Local messages don't have an __ftsc_date and it could also be missing for other reasons (messages converted from another message base format, copied/moved in a message editor or tossed by a tosser that doesn't copy the __ftsc_date).

Good point. But it's easy to test for that condition.

if (*xmsg.__ftsc_date == '\0')
{
/* __ftsc_date is empty, so use xmsg.date_written */
sc_time( ...
}
else
{
/* use xmsg.__ftsc_date */
strcpy( ...
}

--- GoldED+/BSD 1.1.5-b20180707
* Origin: Blizzard of Ozz, Melbourne, Victoria, Australia (3:633/267)

andrew wrote (2021-02-10):

10 Feb 21 11:04, you wrote to me:

What happens if xmsg.__ftsc_date is empty? Local messages don't
have an __ftsc_date and it could also be missing for other reasons
(messages converted from another message base format, copied/moved
in a message editor or tossed by a tosser that doesn't copy the
__ftsc_date).

Good point. But it's easy to test for that condition.

if (*xmsg.__ftsc_date == '\0')
{
/* __ftsc_date is empty, so use xmsg.date_written */
sc_time( ...
}
else
{
/* use xmsg.__ftsc_date */
strcpy( ...
}

That seems to work:

$ pktview outbound/23e73b00.pkt | grep 'Date'

Date and time : 09 Feb 21 13:14:00
Date and time : 09 Feb 21 06:31:33
Date and time : 09 Feb 21 05:08:19
Date and time : 07 Feb 21 18:49:35
Date and time : 10 Feb 21 03:55:02
Date and time : 10 Feb 21 14:58:42

Last one is a message with an empty __ftsc_date field.
(I had to use an hex editor, because Golded does write the __ftsc_date for messages created in the editor, even the Squish developer documentation strongly advises against it).

I haven't tested JAM and .MSG message bases.

---
* Origin: . (2:280/464.47)

Hello Oli!

10 Feb 21, Oli wrote to andrew clarke:

Date and time : 10 Feb 21 14:58:42

Last one is a message with an empty __ftsc_date field.
(I had to use an hex editor, because Golded does write the __ftsc_date
for messages created in the editor, even the Squish developer
documentation strongly advises against it).

__ftsc_date char[20] 218 FTS-0001 compatible date. Squish
applications should not access this
field directly. This field is used
exclusively by tossers and scanners
for preserving the original ASCII
message date.

If the first tosser/scanner touches the message, where does the original ASCII date comes from?

Regards

Kai

--- GoldED+/LNX 1.1.4.7
* Origin: Monobox (2:240/77)

andrew wrote (2021-02-10):

10 Feb 21 09:56, you wrote to Kai Richter:

It's just a simple bug ...

I suspect the patch below fixes this, but I've not tested it.

It's also necessary to verify it doesn't break when rescanning *.MSG and JAM bases.

I tested it with a JAM base and it still works. But it turned out a rescan from a JAM base also modifies time stamps:

Original:
Date and time : 09 Feb 21 06:31:33
Date and time : 09 Feb 21 05:08:19
Date and time : 07 Feb 21 18:49:35
Date and time : 10 Feb 21 03:55:02
Date and time : 10 Feb 21 08:48:36
Date and time : 10 Feb 21 12:23:00
Date and time : 10 Feb 21 13:11:33

After rescan from JAM:
Date and time : 09 Feb 21 06:31:32
Date and time : 09 Feb 21 05:08:18
Date and time : 07 Feb 21 18:49:34
Date and time : 10 Feb 21 03:55:02
Date and time : 10 Feb 21 08:48:36
Date and time : 10 Feb 21 12:23:00
Date and time : 10 Feb 21 13:11:32

Which means Squish can be fixed, but JAM has the same problem (I tested JAM with hpt from github master without any patch).

Maybe a problem/bug within SMAPI (JAM jammed into the Squish API)?

---
* Origin: . (2:280/464.47)

Michael wrote (2021-02-10):

I don't know why it doesn't get fixed. Maybe nobody wants to touch
that part of the code anymore. A project where critical bugs doesn't
get fixed anymore is kind of abandoned.

So, where is a pull request from you? I don't see it.

I'm not a hpt user and I'm not involved with the projects.

As to me, I have a different view on what is critical and what is not so critical. I have a long list of what should be done in Husky and I pick the items from the list according to my idea of which item is more important to me. I may also pick a somewhat less important item if it requires a little time. And I have my life besides Fidonet, particularly
I still have a job despite the fact that I am 73 y/o. So I cannot work
for Fidonet full time, but only when I have some time for it.

I didn't address you directly. Of course you are free to do with hpt / husky whatever you like and label bugs and todos as you wish.

hpt is a software that interacts with the network. When there is a bug that could cause problems in the network, I call that critical. That's my opinion, nothing more. It's an open source project, nobody have to do anything. But if these problems don't get fixed over an extended period of time, I don't think it's wrong to address it and remind people that they are using broken software. If people get annoyed by these reminders, there is an easy solution: fix the bugs that affects the network. Or don't and ignore it. But don't expect that people will not complain.

Btw, hunting down a bug, writing a bug report and testing the fixes is also time consuming.

---
* Origin: . (2:280/464.47)

14 Feb 21 11:28, you wrote to me:

I tested it with a JAM base and it still works. But it turned out a
rescan from a JAM base also modifies time stamps:

Original:
Date and time : 09 Feb 21 06:31:33

After rescan from JAM:
Date and time : 09 Feb 21 06:31:32

Which means Squish can be fixed, but JAM has the same problem (I tested JAM with hpt from github master without any patch).

Maybe a problem/bug within SMAPI (JAM jammed into the Squish API)?

api_jam.c:JamReadMsg() has this:

scombo = (SCOMBO *)(&(msg->date_written));
scombo = TmDate_to_DosDate(s_time, scombo);
/* ftsdate = msg->__ftsc_date; */
ftsdate = (unsigned char *)sc_time(scombo, (char *)(msg->__ftsc_date));

But maybe the correct code should be:

if (*msg->__ftsc_date)
{
ftsdate = msg->__ftsc_date;
}
else
{
scombo = (SCOMBO *)(&(msg->date_written));
scombo = TmDate_to_DosDate(s_time, scombo);
ftsdate = (unsigned char *)sc_time(scombo, (char *)(msg->__ftsc_date));
}

--- GoldED+/BSD 1.1.5-b20180707
* Origin: Blizzard of Ozz, Melbourne, Victoria, Australia (3:633/267)

andrew wrote (2021-02-14):

14 Feb 21 11:28, you wrote to me:

I tested it with a JAM base and it still works. But it turned out a
rescan from a JAM base also modifies time stamps:

Original:
Date and time : 09 Feb 21 06:31:33

After rescan from JAM:
Date and time : 09 Feb 21 06:31:32

Which means Squish can be fixed, but JAM has the same problem (I
tested JAM with hpt from github master without any patch).

Maybe a problem/bug within SMAPI (JAM jammed into the Squish API)?

api_jam.c:JamReadMsg() has this:

scombo = (SCOMBO *)(&(msg->date_written));
scombo = TmDate_to_DosDate(s_time, scombo);
/* ftsdate = msg->__ftsc_date; */
ftsdate = (unsigned char *)sc_time(scombo, (char

*)(msg->> __ftsc_date));

But maybe the correct code should be:

if (*msg->__ftsc_date)
{
ftsdate = msg->__ftsc_date;
}
else
{
scombo = (SCOMBO *)(&(msg->date_written));
scombo = TmDate_to_DosDate(s_time, scombo);
ftsdate = (unsigned char *)sc_time(scombo, (char *)(msg->__ftsc_date)); }

Mhh, a JAM message base doesn't have an __ftsc_date field, I don't understand this piece of the code.

What about the TmDate_to_DosDate conversion?

union stamp_combo *_fast TmDate_to_DosDate(struct tm *tmdate, union stamp_combo *dosdate)
{
if(tmdate && dosdate)
{
dosdate->msg_st.date.da = tmdate->tm_mday;
dosdate->msg_st.date.mo = tmdate->tm_mon + 1;
dosdate->msg_st.date.yr = tmdate->tm_year - 80;

dosdate->msg_st.time.hh = tmdate->tm_hour;
dosdate->msg_st.time.mm = tmdate->tm_min;
dosdate->msg_st.time.ss = tmdate->tm_sec >> 1;
^^^^^^^^^^^^
}

return dosdate;
}

AFAIK the JAM format doesn't use DOS Time at all. It seems to be an unnecessary lossy conversion.

* Origin: . (2:280/464.47)

Hello andrew,

Sunday February 14 2021, andrew clarke wrote to Oli:

api_jam.c:JamReadMsg() has this:

scombo = (SCOMBO *)(&(msg->date_written));
scombo = TmDate_to_DosDate(s_time, scombo);
/* ftsdate = msg->__ftsc_date; */
ftsdate = (unsigned char *)sc_time(scombo, (char *)(msg->__ftsc_date));

But maybe the correct code should be:

if (*msg->__ftsc_date)
{
ftsdate = msg->__ftsc_date;
}
else
{
scombo = (SCOMBO *)(&(msg->date_written));
scombo = TmDate_to_DosDate(s_time, scombo);
ftsdate = (unsigned char *)sc_time(scombo, (char *)(msg->__ftsc_date));
}

"ftsdate" variable is assigned but it is never used, so I've already commented it out but not in master branch yet.

Michael

... node (at) f1042 (dot) ru
--- GoldED+/LNX 1.1.5-b20180707
* Origin: Moscow, Russia (2:5020/1042)

14 Feb 21 16:47, you wrote to me:

But maybe the correct code should be:

if (*msg->__ftsc_date)
{
ftsdate = msg->__ftsc_date;
}
else
{
scombo = (SCOMBO *)(&(msg->date_written));
scombo = TmDate_to_DosDate(s_time, scombo);
ftsdate = (unsigned char *)sc_time(scombo, (char
*)(msg->__ftsc_date)); }

Mhh, a JAM message base doesn't have an __ftsc_date field, I don't understand this piece of the code.

Yeah, disregard that, I wrote it in a hurry.

I've also never used JAM much so don't recall much about the file format, but see below.

dosdate->msg_st.time.mm = tmdate->tm_min;
dosdate->msg_st.time.ss = tmdate->tm_sec >> 1;
^^^^^^^^^^^^
}

return dosdate;
}

AFAIK the JAM format doesn't use DOS Time at all. It seems to be an unnecessary lossy conversion.

Scott Dudley's MSGAPI uses lossy timestamps internally for XMSG->date_written and XMSG->date_arrived and that's folded into the SMAPI fork because nobody's ever thought to change it. See _stamp in huskylib/cvtdate.h.

(The _stamp struct is in Huskylib now, but it was originally self-contained in SMAPI, which predates the Husky project.)

Scott's idea of using his lossy timestamps to recreate XMSG->__ftsc_date is nonsense and as you know only has a 50/50 chance of working accurately.

Now, the Squish format stupidly uses those lossy timestamps but there's no reason the whole API has to. Conceivably that could be fixed with a bit of work so as not to corrupt Squish bases.

In the mean time JAM's msgh->Hdr.DateWritten is a 32-bit unsigned integer. (Fortunately being unsigned it will wrap around in 2106, not in 2038.)

The JAM spec says:

"An ulong representing the number of seconds since midnight, January 1, 1970."

Presumably that's 1970-01-01 00:00 UTC, not local time.

When writing messages, JamWriteMsg() calls ConvertXmsgToJamHdr() (which is not used by JamReadMsg), which does this:

jamhdr->DateWritten = mktime(ptm) + gettz();

One problem here is that ptm is assigned the return value of DosDate_to_TmDate(), which I suspect is lossy. I don't know why the code does that.

That value is then fed into mktime() to return the number of seconds since 1970-01-01, then a timezone offset is added with gettz().

But why is gettz() called every time JamWriteMsg() is called? Wouldn't that mean HPT adds the local timezone to the DateWritten field every time it tosses a message to a JAM base? That's a bug, isn't it? Unless I'm reading the code wrong.

I mean, JamReadMsg() could convert msgh->Hdr.DateWritten to FTSC date format then write that to msg->__ftsc_date, but will the __ftsc_date of rescanned messages have the same timestamp they originated with, or will they be offset by local time?

If the latter, it's not just a matter of reversing things by subtracting the offset of local time, because the local time zone can change due to daylight savings.

Though I just noticed gettz() seems to unexpectedly disable DST when it calculates the time zone offset! So maybe you /could/ subtract the offset and the rescanned timestamps would be accurate... unless you moved into a different time zone.

api_jam.c also hints at supporting the TZUTC kludge, but it does nothing to calculate dates with it. It just stores it as a JAM subfield because the spec says it can.

The ftsdate variable in JamReadMsg() doesn't actually seem to be used. Abandoned code, I guess.

--- GoldED+/BSD 1.1.5-b20180707
* Origin: Blizzard of Ozz, Melbourne, Victoria, Australia (3:633/267)

14 Feb 21 22:50, you wrote to me:

"ftsdate" variable is assigned but it is never used, so I've already commented it out but not in master branch yet.

I see that now, thanks.

Instead of commenting-out code I like to use #if 0, and explain why:

#if 0
/* this code is deliberately broken, never run it */

p = 0;
strcpy(p, "AAA");
#endif

--- GoldED+/BSD 1.1.5-b20180707
* Origin: Blizzard of Ozz, Melbourne, Victoria, Australia (3:633/267)

andrew wrote (2021-02-15):

Scott's idea of using his lossy timestamps to recreate XMSG->__ftsc_date is nonsense and as you know only has a 50/50 chance of working accurately.

I didn't think he expected it to work accurately, more like it is not relevant to preserve the exact time and 2 second resolution is good enough. The dupe checker in Squish also uses a 2 second granularity.

As a human I really don't care about the seconds, but for machine processing I see no benefits in modifying the time.

Now, the Squish format stupidly uses those lossy timestamps but there's
no reason the whole API has to. Conceivably that could be fixed with a
bit of work so as not to corrupt Squish bases.

Not sure if I understand. What kind of potential corruption do you have in mind?

In the mean time JAM's msgh->Hdr.DateWritten is a 32-bit unsigned
integer. (Fortunately being unsigned it will wrap around in 2106, not in 2038.)

The JAM spec says:

"An ulong representing the number of seconds since midnight, January 1, 1970."

Presumably that's 1970-01-01 00:00 UTC, not local time.

I don't think it's meant to be UTC. From the JAM spec:

(1) All timestamps created locally (i.e. those not imported from
other systems) are stored in local time.

The DateWritten field is also used for imported messages. Why should the time ever be converted to UTC? Wouldn't that make everything more complicated? UNIX time in JAM is not a real Unix timestamp, more UNIX-style (UNIXish).

That value is then fed into mktime() to return the number of seconds
since 1970-01-01, then a timezone offset is added with gettz().

which seems to be correct behavior.

But why is gettz() called every time JamWriteMsg() is called? Wouldn't that mean HPT adds the local timezone to the DateWritten field every time it tosses a message to a JAM base? That's a bug, isn't it? Unless I'm reading the code wrong.

[...]

I haven't looked at the code, but wouldn't this already be discovered, if it were a problem?

api_jam.c also hints at supporting the TZUTC kludge, but it does nothing to calculate dates with it. It just stores it as a JAM subfield because the spec says it can.

It'd the JAM way. It stores a few important kludges in special fields (like MSGID, REPLY, TZUTC, Via, PATH, SEENBY).

---
* Origin: . (2:280/464.47)

15 Feb 21 13:31, you wrote to me:

Scott's idea of using his lossy timestamps to recreate
XMSG->__ftsc_date is nonsense and as you know only has a 50/50
chance of working accurately.

I didn't think he expected it to work accurately, more like it is not relevant to preserve the exact time and 2 second resolution is good enough. The dupe checker in Squish also uses a 2 second granularity.

But it's necessary to work accurately if you're doing a rescan. That's the whole point of this bug and is probably why Scott had to store the ftsc_date field.

It's been so long since I used the original SquishMail (ie. squish.exe) that I don't recall if rescans were a supported feature.

("squish.exe rescan 3:633/267 fidosoft.husky" maybe?)

Now, the Squish format stupidly uses those lossy timestamps but
there's no reason the whole API has to. Conceivably that could be
fixed with a bit of work so as not to corrupt Squish bases.

Not sure if I understand. What kind of potential corruption do you have
in mind?

The API just needs to use 2 second resolution for I/O on Squish bases and 1 second resolution everywhere else.

But currently I suspect if you change _stamp in the API to allow 1 second resolution it will break the Squish code.

It will probably also break the ABI if someone is using a DLL version of SMAPI and tries to mix old & new versions, but that situation isn't unique to HPT/SMAPI.

All of that can be ironed out though.

In the mean time JAM's msgh->Hdr.DateWritten is a 32-bit unsigned
integer. (Fortunately being unsigned it will wrap around in 2106,
not in 2038.)

The JAM spec says:

"An ulong representing the number of seconds since midnight,
January 1, 1970."

Presumably that's 1970-01-01 00:00 UTC, not local time.

I don't think it's meant to be UTC. From the JAM spec:

(1) All timestamps created locally (i.e. those not imported from
other systems) are stored in local time.

The DateWritten field is also used for imported messages. Why should the time ever be converted to UTC? Wouldn't that make everything more complicated? UNIX time in JAM is not a real Unix timestamp, more UNIX-style (UNIXish).

You could be right, but at the very least it's ambiguous since AFAIK the convention is for time_t on Linux/BSD be stored as UTC.

But at this point it's really just a matter of throwing mud at the code to see what sticks...

--- GoldED+/BSD 1.1.5-b20180707
* Origin: Blizzard of Ozz, Melbourne, Victoria, Australia (3:633/267)

andrew wrote (2021-02-16):

It's been so long since I used the original SquishMail (ie. squish.exe) that I don't recall if rescans were a supported feature.

("squish.exe rescan 3:633/267 fidosoft.husky" maybe?)

SQUISH RESCAN <area_tag> <node>

The JAM spec says:

"An ulong representing the number of seconds since midnight,
January 1, 1970."

Presumably that's 1970-01-01 00:00 UTC, not local time.

I don't think it's meant to be UTC. From the JAM spec:

[...]

You could be right, but at the very least it's ambiguous since AFAIK the convention is for time_t on Linux/BSD be stored as UTC.

especially if it's called "UNIX date" in the spec. If it's not a real UNIX time stamp (which is always in UTC), what is it then? And what is the exact meaning of "the number of seconds since midnight, January 1, 1970" when UNIX time is "the number of seconds that have elapsed since the Unix epoch, minus leap seconds"? ;-P

---
* Origin: . (2:280/464.47)

Hello andrew!

16 Feb 21, andrew clarke wrote to Oli:

But it's necessary to work accurately if you're doing a rescan.

It's necessary to do the same workaround again. ;-)

That's the whole point of this bug

The bug is that the normal scan and the rescan created different DateTime.

I don't understand what you are doing but it looks like you do some re-invention. Isn't it easier to look how the scan.c does the export and use the same code for the rescan?

Regards

Kai

--- GoldED+/LNX 1.1.4.7
* Origin: Monobox (2:240/77)

15 Feb 21 21:39, you wrote to me:

The bug is that the normal scan and the rescan created different
DateTime.

I don't understand what you are doing but it looks like you do some re-invention. Isn't it easier to look how the scan.c does the export and use the same code for the rescan?

To summarise:

"toss" shouldn't need the message base, instead just memcpy()ing the ftscdate field between .PKTs. This is how PassThru areas work but should also be true for non-PassThru.

"scan" only exports local messages which in the case of a Squish base, only the DateWritten field is supposed to be written to by the mail reader, though Oli recently said that GoldED apparently writes the ftscdate field as well (which then means you can have a one second discrepancy between fields for the same locally written message...).

"rescan" exports both local and non-local messages, which can have a combination of DateWritten field ONLY or DateWritten-and-ftscdate, and HPT incorrectly uses the DateWritten field for non-local messages, which only has a two second resolution in Squish.

I can't say any more about JAM other than what I've already written.

The *.MSG format stores the date solely in ftscdate format, so HPT should really have no choice but to memcpy() the ftscdate field into the outbound PKT on a rescan.

So obviously the moral of the story is for everyone to use *.MSG. ;)

--- GoldED+/BSD 1.1.5-b20180707
* Origin: Blizzard of Ozz, Melbourne, Victoria, Australia (3:633/267)

Hello andrew!

16 Feb 21, andrew clarke wrote to Kai Richter:

Isn't it easier to look how the scan.c does the export and use the
same code for the rescan?

To summarise:

"toss" shouldn't need the message base, instead just memcpy()ing the ftscdate field between .PKTs. This is how PassThru areas work but
should also be true for non-PassThru.

Thanks, i hadn't that in my mind. I thought htp toss would toss into the messagebase only. To be honest i never paid attention at which run the downlink packages were build. Oh wait, are you talking about toss.c the code but not the function hpt toss?

Regards

Kai

--- GoldED+/LNX 1.1.4.7
* Origin: Monobox (2:240/77)

On Tue 2021-02-16 16:29:02, Kai Richter (2:240/77) wrote to andrew clarke:

Isn't it easier to look how the scan.c does the export and use the
same code for the rescan?

To summarise:

"toss" shouldn't need the message base, instead just memcpy()ing
the ftscdate field between .PKTs. This is how PassThru areas work
but should also be true for non-PassThru.

Thanks, i hadn't that in my mind. I thought htp toss would toss into the messagebase only. To be honest i never paid attention at which run the downlink packages were build. Oh wait, are you talking about toss.c the code but not the function hpt toss?

Both I suppose. hpt toss is implemented by toss.c so they're one and the same.

It's possible to have a completely passthru HPT configuration without a messagebase, with the exception of NetmailArea/DupeArea/BadArea.
Though if you wanted to go full Marie Kondo you could point those to /dev/null!

--- GoldED+/BSD 1.1.5-b20180707
* Origin: Blizzard of Ozz, Melbourne, Victoria, Australia (3:633/267)