CTI STIX Subcommittee

I have been going back and forth on the usefulness of the precision field.  Perhaps we could easily get by in a workflow condition to NOT have a precision field as Jason states.   Things I think we can agree on so far: 1) A timestamp format of yyyy-mm-dd-Thh:mm:ss.mmmmmm+-hh:ss MUST be used Examples: 2015-11-23T13:35:12.000000+00:00  (for 1:35:12 in UTC format) 2) All timestamps MUST be in UTC format a UI will change them as needed for an analyst  3) Timestamps will have 6 digits of precision 4) Any values that are not known will be zeroed out, say I only know the date not the time Example: 2015:11:23T00:00:00.000000+00:00 Open Question(s): A) Is it valid to put in a timezone offset from UTC?  Or must the value be actually in UTC. Example 2015-11-23T13:35:12.000000-06:00 B) Do we actually need to manually say what the precision is? Meaning do we need to call out that it is a year , month , day , hour , minute , or second . i) Sean believes we need this ii) Jason does not believe we need this.  I think I am starting to lean towards Jason on this. Lets focus on what we can agree on (the stone in the path) and focus our discussions on the remaining open questions.  This will enable us to drive this seemingly easy win to consensus.   Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg.   On Nov 23, 2015, at 13:02, Jason Keirstead < Jason.Keirstead@ca.ibm.com > wrote: Agree 100% on the nanoseconds - if not useful, they should be dropped. I want to pick up debate here we were having on the Slack channel before it went kapoof. I do not think we should be coming at this from the point of view of this could be theoretically useful for <x> . This is exactly how STIX got so complicated in the first place. We should be coming at this from the point of view of - What is the minimal amount of information to communicate this data point - OK, now, what additional information *beyond the minimum is required to fulfil all identified workflow s. Notice I am using the word workflow , not use case, this is on purpose. All of these decisions should be made from the point of view of an end to end workflow - not only the producer making the data, but also the consumer of the data, and what usefulness it could provide them. So far the requirement for a precision field has assumed that there is a use case on the recpient side for this data - I challenge this. Lets assume we have a mandatory nanosecond-accurate timestamp. What is the workflow by which I would create a timestamp that would not have nanosecond accuracy, send that to a consumer, and then have the consumer improperly process the information or take invalid action based on that? A use case on Slack was presented by @sbarnum that you could use this for high precision temporal analysis - but I assert that said analysis still does not require a precision field, because in the only use cases where you would be doing that action, the data would always have precision (no one is going to take human-generated incident responses and perform millisecond-level temporal analysis on them, that doesn't make any sense) - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown <graycol.gif> Struse, Richard ---11/23/2015 03:42:45 PM---Are there any generally-available tools or technologies that produce timestamps with nanosecond prec From: Struse, Richard < Richard.Struse@HQ.DHS.GOV > To: tony@yaanatech.com < tony@yaanatech.com >, Jordan, Bret < bret.jordan@bluecoat.com >, Trey Darley < trey@SOLTRA.COM > Cc: Jason Keirstead/CanEast/IBM@IBMCA, Jerome Athias < athiasjerome@gmail.com >, cti-stix@lists.oasis-open.org < cti-stix@lists.oasis-open.org >, Wunder, John A. < jwunder@mitre.org >, Patrick Maroney < Pmaroney@Specere.org >, Sean D. Barnum < sbarnum@mitre.org > Date: 11/23/2015 03:42 PM Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000 Sent by: < cti-stix@lists.oasis-open.org > Are there any generally-available tools or technologies that produce timestamps with nanosecond precision today?  If we can't identify any I would suggest that we support 6 digits (microseconds) and be done. This is a trivial but important way that we can communicate to the broader community that we are rooted in real-world practice.

I really don’t agree with #4 here, it’s ambiguous. It means once every (roughly) 1,000,000 documents one will be issued with a precision of “second” rather than “microsecond” because the natural microsecond value is 0. Then every 60,000,000 (granted getting
rare here, but if you talk billions/day) it will accidentally have a precision of “minute” rather than “microsecond”.


This is not to say that we need a precision field, just that if we do it should be explicit rather than implicit.



On Nov 23, 2015, at 3:44 PM, Jordan, Bret < bret.jordan@BLUECOAT.COM > wrote:



I have been going back and forth on the usefulness of the precision field.  Perhaps we could easily get by in a workflow condition to NOT have a precision field as Jason states.  


Things I think we can agree on so far:
1) A timestamp format of yyyy-mm-dd-Thh:mm:ss.mmmmmm+-hh:ss MUST be used
Examples: 2015-11-23T13:35:12.000000+00:00  (for 1:35:12 in UTC format)
2) All timestamps MUST be in UTC format a UI will change them as needed for an analyst 
3) Timestamps will have 6 digits of precision
4) Any values that are not known will be zeroed out, say I only know the date not the time
Example: 2015:11:23T00:00:00.000000+00:00


Open Question(s):
A) Is it valid to put in a timezone offset from UTC?  Or must the value be actually "in" UTC.
Example 2015-11-23T13:35:12.000000-06:00


B) Do we actually need to manually say what the precision is? Meaning do we need to call out that it is a "year", "month", "day", "hour", "minute", or "second".
i) Sean believes we need this
ii) Jason does not believe we need this.  I think I am starting to lean towards Jason on this.


Lets focus on what we can agree on (the stone in the path) and focus our discussions on the remaining open questions.  This will enable us to drive this seemingly easy win to consensus.  












Thanks,


Bret











Bret Jordan CISSP

Director of Security Architecture and Standards Office of the CTO

Blue Coat Systems

PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." 











On Nov 23, 2015, at 13:02, Jason Keirstead < Jason.Keirstead@ca.ibm.com > wrote:



Agree 100% on the nanoseconds - if not useful, they should be dropped.

I want to pick up debate here we were having on the Slack channel before it went kapoof. I do not think we should be coming at this from the point of view of "this could be theoretically useful for <x>". This is exactly how STIX got so complicated in the first
place.

We should be coming at this from the point of view of

- What is the minimal amount of information to communicate this data point

- OK, now, what additional information *beyond the minimum" is required to fulfil all
identified workflow s.

Notice I am using the word "workflow", not use case, this is on purpose. All of these decisions should be made from the point of view of an end to end workflow - not only the producer making the data, but also the consumer of the data, and what usefulness it
could provide them.

So far the requirement for a precision field has assumed that there is a use case on the recpient side for this data - I challenge this. Lets assume we have a mandatory nanosecond-accurate timestamp. What is the workflow by which I would create a timestamp
that would not have nanosecond accuracy, send that to a consumer, and then have the consumer improperly process the information or take invalid action based on that? A use case on Slack was presented by @sbarnum that you could use this for high precision temporal
analysis - but I assert that said analysis still does not require a precision field, because in the only use cases where you would be doing that action, the data would always have precision (no one is going to take human-generated incident responses and perform
millisecond-level temporal analysis on them, that doesn't make any sense)

-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security
www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


<graycol.gif> "Struse, Richard" ---11/23/2015 03:42:45 PM---Are there any generally-available tools or technologies that produce timestamps with nanosecond
prec

From: "Struse, Richard" < Richard.Struse@HQ.DHS.GOV >
To: " tony@yaanatech.com " < tony@yaanatech.com >, "Jordan, Bret" < bret.jordan@bluecoat.com >,
Trey Darley < trey@SOLTRA.COM >
Cc: Jason Keirstead/CanEast/IBM@IBMCA, Jerome Athias < athiasjerome@gmail.com >, " cti-stix@lists.oasis-open.org "
< cti-stix@lists.oasis-open.org >, "Wunder, John A." < jwunder@mitre.org >, Patrick Maroney < Pmaroney@Specere.org >,
"Sean D. Barnum" < sbarnum@mitre.org >
Date: 11/23/2015 03:42 PM
Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000
Sent by: < cti-stix@lists.oasis-open.org >





Are there any generally-available tools or technologies that produce
timestamps with nanosecond precision today?  If we can't identify any I
would suggest that we support 6 digits (microseconds) and be done.

This is a trivial but important way that we can communicate to the broader
community that we are rooted in real-world practice.


Original Message-----
From: cti-stix@lists.oasis-open.org [ mailto:cti-stix@lists.oasis-open.org ]
On Behalf Of Tony Rutkowski
Sent: Monday, November 23, 2015 2:23 PM
To: Jordan, Bret; Trey Darley
Cc: Jason Keirstead; Jerome Athias;
cti-stix@lists.oasis-open.org ; Wunder,
John A.; Patrick Maroney; Sean D. Barnum
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000

It's not inconceivable that fractional microsecond
values matter in virtualization environments.within
the same facility.  On a larger scale, the uncertainties
associated with the timestamp value will make
nanosecond precision moot.

Has anyone articulated what the overhead
differential is of an _expression_ with a precision
of microseconds versus nanoseconds?

-t

On 2015-11-23 01:08 PM, Jordan, Bret wrote:
> I miss typed in my last email, I meant to say micro seconds not
> milliseconds, aka 6 digits of precision not 3 digits of precision.
>  Wireshark and other networking / security tools are able to work with
> and provide 6 digits of precision. That is VERY common. What is not
> really common today is 9 digits of precision.
>

Okay, so lets pull #4 out and put it in the open questions.  Can we agree on 1-3?  Lets focus on what we CAN agree on, and set stones in the path.  Lets get to one stone at a time.   Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg.   On Nov 23, 2015, at 13:51, Wunder, John A. < jwunder@mitre.org > wrote: I really don’t agree with #4 here, it’s ambiguous. It means once every (roughly) 1,000,000 documents one will be issued with a precision of “second” rather than “microsecond” because the natural microsecond value is 0. Then every 60,000,000 (granted getting rare here, but if you talk billions/day) it will accidentally have a precision of “minute” rather than “microsecond”. This is not to say that we need a precision field, just that if we do it should be explicit rather than implicit. On Nov 23, 2015, at 3:44 PM, Jordan, Bret < bret.jordan@BLUECOAT.COM > wrote: I have been going back and forth on the usefulness of the precision field.  Perhaps we could easily get by in a workflow condition to NOT have a precision field as Jason states.   Things I think we can agree on so far: 1) A timestamp format of yyyy-mm-dd-Thh:mm:ss.mmmmmm+-hh:ss MUST be used Examples: 2015-11-23T13:35:12.000000+00:00  (for 1:35:12 in UTC format) 2) All timestamps MUST be in UTC format a UI will change them as needed for an analyst  3) Timestamps will have 6 digits of precision 4) Any values that are not known will be zeroed out, say I only know the date not the time Example: 2015:11:23T00:00:00.000000+00:00 Open Question(s): A) Is it valid to put in a timezone offset from UTC?  Or must the value be actually in UTC. Example 2015-11-23T13:35:12.000000-06:00 B) Do we actually need to manually say what the precision is? Meaning do we need to call out that it is a year , month , day , hour , minute , or second . i) Sean believes we need this ii) Jason does not believe we need this.  I think I am starting to lean towards Jason on this. Lets focus on what we can agree on (the stone in the path) and focus our discussions on the remaining open questions.  This will enable us to drive this seemingly easy win to consensus.   Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg.   On Nov 23, 2015, at 13:02, Jason Keirstead < Jason.Keirstead@ca.ibm.com > wrote: Agree 100% on the nanoseconds - if not useful, they should be dropped. I want to pick up debate here we were having on the Slack channel before it went kapoof. I do not think we should be coming at this from the point of view of this could be theoretically useful for <x> . This is exactly how STIX got so complicated in the first place. We should be coming at this from the point of view of - What is the minimal amount of information to communicate this data point - OK, now, what additional information *beyond the minimum is required to fulfil all identified workflow s. Notice I am using the word workflow , not use case, this is on purpose. All of these decisions should be made from the point of view of an end to end workflow - not only the producer making the data, but also the consumer of the data, and what usefulness it could provide them. So far the requirement for a precision field has assumed that there is a use case on the recpient side for this data - I challenge this. Lets assume we have a mandatory nanosecond-accurate timestamp. What is the workflow by which I would create a timestamp that would not have nanosecond accuracy, send that to a consumer, and then have the consumer improperly process the information or take invalid action based on that? A use case on Slack was presented by @sbarnum that you could use this for high precision temporal analysis - but I assert that said analysis still does not require a precision field, because in the only use cases where you would be doing that action, the data would always have precision (no one is going to take human-generated incident responses and perform millisecond-level temporal analysis on them, that doesn't make any sense) - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown <graycol.gif> Struse, Richard ---11/23/2015 03:42:45 PM---Are there any generally-available tools or technologies that produce timestamps with nanosecond prec From: Struse, Richard < Richard.Struse@HQ.DHS.GOV > To: tony@yaanatech.com < tony@yaanatech.com >, Jordan, Bret < bret.jordan@bluecoat.com >, Trey Darley < trey@SOLTRA.COM > Cc: Jason Keirstead/CanEast/IBM@IBMCA, Jerome Athias < athiasjerome@gmail.com >, cti-stix@lists.oasis-open.org < cti-stix@lists.oasis-open.org >, Wunder, John A. < jwunder@mitre.org >, Patrick Maroney < Pmaroney@Specere.org >, Sean D. Barnum < sbarnum@mitre.org > Date: 11/23/2015 03:42 PM Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000 Sent by: < cti-stix@lists.oasis-open.org > Are there any generally-available tools or technologies that produce timestamps with nanosecond precision today?  If we can't identify any I would suggest that we support 6 digits (microseconds) and be done. This is a trivial but important way that we can communicate to the broader community that we are rooted in real-world practice.
Original Message----- From: cti-stix@lists.oasis-open.org [ mailto:cti-stix@lists.oasis-open.org ] On Behalf Of Tony Rutkowski Sent: Monday, November 23, 2015 2:23 PM To: Jordan, Bret; Trey Darley Cc: Jason Keirstead; Jerome Athias; cti-stix@lists.oasis-open.org ; Wunder, John A.; Patrick Maroney; Sean D. Barnum Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 It's not inconceivable that fractional microsecond values matter in virtualization environments.within the same facility.  On a larger scale, the uncertainties associated with the timestamp value will make nanosecond precision moot. Has anyone articulated what the overhead differential is of an _expression_ with a precision of microseconds versus nanoseconds? -t On 2015-11-23 01:08 PM, Jordan, Bret wrote: > I miss typed in my last email, I meant to say micro seconds not > milliseconds, aka 6 digits of precision not 3 digits of precision. >  Wireshark and other networking / security tools are able to work with > and provide 6 digits of precision. That is VERY common. What is not > really common today is 9 digits of precision. > Attachment: signature.asc Description: Message signed with OpenPGP using GPGMail

Firstly, I apologize for the length of this reply. As always it started with 2 lines and kind of grew from there…..   I can agree with 1-3.   As far as the open questions go:   A)      No. The timestamp should always be in UTC. When performing calculations at scale (e.g. analytics) having all the times in UTC should make things quicker. Also being able to perform text date extractions (say using grep) on STIX doc s would be far easier if we always make sure its UTC. B)       The issue appears to be that mandating a high level of precision in the timestamp format means that we lose the ability to discern how precise the timestamp actually is.   There seems to be three ways around it that have been presented:   i)                     Allow timestamps to vary based on how precise they are (Discounted due to lack of uniformity) ii)                    All timestamps are the same format and we send an additional explicit description of how precise the timestamp is. iii)                  The level of timestamp precision is dependent on the object and the ways that you use the object. All timestamps are the same format, but if a producer doesn’t have the required precision the more detailed values are ‘zeroed’ out.   With option iii) there is the potential for ambiguity. The premise (if I understand it correctly) is that the precision of timestamps required by a consumer for each type of object are different, and that they will naturally match the precision that the producer is able to generate. ·          High precision time measurement appears to have been available in all common processors since Pentium back in 2005. Android, Linux, Windows all support it. ARM and Tegra architectures appear to support it as well. ·          An Observation/Observable Instance generating tool is likely to already use high precision timing, as speed/accuracy matters to that tool. Therefore it will not be onerous to generate high precision STIX timestamps. ·          A STIX threat intel analytics tool will have whatever ability is necessary for processing STIX compliant stuff. If we state that it needs to have microsecond precision, then it will have to support generating that precision. ·          When Threat Analysts are creating content in their Threat Intel analysis tools, they will need to manually enter in the details of things like Threat Actors, and will need to click the ‘save’ button to create the object. At that point, the STIX threat intel tool will be able to write a highly precise timestamp of the exact moment the person pressed the save button. That level of precision may not be that useful for a high-level object such as a Threat Actor, but it will still be possible to do to maintain a consistent timestamp format. Millisecond resolution provides no specific benefit for manually created data . ·          When talking about the estimated start dates of Campaigns or similar, Start_Time, End_Time, are all going to be informed from the detailed Observations that are ultimately associated with the Campaign. Those observations are highly likely to include millisecond level timing.   In the situations that a less accurate timestamp is available (e.g. default Syslog timestamp is accurate to the nearest second) then there are two options available: ·          Configure the log generator to record the milliseconds (e.g. changing rsyslog.conf to use timegenerated rather than timereported) ·          Zero the extra values out. This will be the case for a very small proportion of the population. It also will mean that we are accurate within 1 second minimum.   #4)         IMHO this ties in with B) iii) above.   I think that we have to put this whole discussion in perspective. Most organizations have difficulty in discovering they have a breach within days and weeks, not within a second. So going with B) iii) and having the precision within 1 second realistically is good enough in my opinion. It is far more likely that all the clocks on the network are not synchronized, and all the tools are reporting different and unrelated times and that they are way more than a second out of alignment with each other. The zeroed out millisecond timestamp doesn’t impact us much when we have real-world problems such as that J .   Cheers   Terry MacDonald Senior STIX Subject Matter Expert SOLTRA   An FS-ISAC and DTCC Company +61 (407) 203 206 terry@soltra.com     From: cti-stix@lists.oasis-open.org [mailto:cti-stix@lists.oasis-open.org] On Behalf Of Jordan, Bret Sent: Tuesday, 24 November 2015 8:07 AM To: Wunder, John A. <jwunder@mitre.org> Cc: Jason Keirstead <Jason.Keirstead@ca.ibm.com>; Richard Struse <Richard.Struse@HQ.DHS.GOV>; tony@yaanatech.com; Trey Darley <trey@soltra.com>; Jerome Athias <athiasjerome@gmail.com>; cti-stix@lists.oasis-open.org; Patrick Maroney <Pmaroney@Specere.org>; Sean D. Barnum <sbarnum@mitre.org> Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000   Okay, so lets pull #4 out and put it in the open questions.  Can we agree on 1-3?  Lets focus on what we CAN agree on, and set stones in the path.  Lets get to one stone at a time.     Thanks,   Bret       Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."    On Nov 23, 2015, at 13:51, Wunder, John A. < jwunder@mitre.org > wrote:   I really don’t agree with #4 here, it’s ambiguous. It means once every (roughly) 1,000,000 documents one will be issued with a precision of “second” rather than “microsecond” because the natural microsecond value is 0. Then every 60,000,000 (granted getting rare here, but if you talk billions/day) it will accidentally have a precision of “minute” rather than “microsecond”.   This is not to say that we need a precision field, just that if we do it should be explicit rather than implicit.   On Nov 23, 2015, at 3:44 PM, Jordan, Bret < bret.jordan@BLUECOAT.COM > wrote:   I have been going back and forth on the usefulness of the precision field.  Perhaps we could easily get by in a workflow condition to NOT have a precision field as Jason states.     Things I think we can agree on so far: 1) A timestamp format of yyyy-mm-dd-Thh:mm:ss.mmmmmm+-hh:ss MUST be used Examples: 2015-11-23T13:35:12.000000+00:00  (for 1:35:12 in UTC format) 2) All timestamps MUST be in UTC format a UI will change them as needed for an analyst  3) Timestamps will have 6 digits of precision 4) Any values that are not known will be zeroed out, say I only know the date not the time Example: 2015:11:23T00:00:00.000000+00:00   Open Question(s): A) Is it valid to put in a timezone offset from UTC?  Or must the value be actually "in" UTC. Example 2015-11-23T13:35:12.000000-06:00   B) Do we actually need to manually say what the precision is? Meaning do we need to call out that it is a "year", "month", "day", "hour", "minute", or "second". i) Sean believes we need this ii) Jason does not believe we need this.  I think I am starting to lean towards Jason on this.   Lets focus on what we can agree on (the stone in the path) and focus our discussions on the remaining open questions.  This will enable us to drive this seemingly easy win to consensus.       Thanks,   Bret       Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."    On Nov 23, 2015, at 13:02, Jason Keirstead < Jason.Keirstead@ca.ibm.com > wrote:   Agree 100% on the nanoseconds - if not useful, they should be dropped. I want to pick up debate here we were having on the Slack channel before it went kapoof. I do not think we should be coming at this from the point of view of "this could be theoretically useful for <x>". This is exactly how STIX got so complicated in the first place. We should be coming at this from the point of view of - What is the minimal amount of information to communicate this data point - OK, now, what additional information *beyond the minimum" is required to fulfil all identified workflow s. Notice I am using the word "workflow", not use case, this is on purpose. All of these decisions should be made from the point of view of an end to end workflow - not only the producer making the data, but also the consumer of the data, and what usefulness it could provide them. So far the requirement for a precision field has assumed that there is a use case on the recpient side for this data - I challenge this. Lets assume we have a mandatory nanosecond-accurate timestamp. What is the workflow by which I would create a timestamp that would not have nanosecond accuracy, send that to a consumer, and then have the consumer improperly process the information or take invalid action based on that? A use case on Slack was presented by @sbarnum that you could use this for high precision temporal analysis - but I assert that said analysis still does not require a precision field, because in the only use cases where you would be doing that action, the data would always have precision (no one is going to take human-generated incident responses and perform millisecond-level temporal analysis on them, that doesn't make any sense) - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown <graycol.gif> "Struse, Richard" ---11/23/2015 03:42:45 PM---Are there any generally-available tools or technologies that produce timestamps with nanosecond prec From: "Struse, Richard" < Richard.Struse@HQ.DHS.GOV > To: " tony@yaanatech.com " < tony@yaanatech.com >, "Jordan, Bret" < bret.jordan@bluecoat.com >, Trey Darley < trey@SOLTRA.COM > Cc: Jason Keirstead/CanEast/IBM@IBMCA, Jerome Athias < athiasjerome@gmail.com >, " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >, "Wunder, John A." < jwunder@mitre.org >, Patrick Maroney < Pmaroney@Specere.org >, "Sean D. Barnum" < sbarnum@mitre.org > Date: 11/23/2015 03:42 PM Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000 Sent by: < cti-stix@lists.oasis-open.org > Are there any generally-available tools or technologies that produce timestamps with nanosecond precision today?  If we can't identify any I would suggest that we support 6 digits (microseconds) and be done. This is a trivial but important way that we can communicate to the broader community that we are rooted in real-world practice.
Original Message----- From: cti-stix@lists.oasis-open.org [ mailto:cti-stix@lists.oasis-open.org ] On Behalf Of Tony Rutkowski Sent: Monday, November 23, 2015 2:23 PM To: Jordan, Bret; Trey Darley Cc: Jason Keirstead; Jerome Athias; cti-stix@lists.oasis-open.org ; Wunder, John A.; Patrick Maroney; Sean D. Barnum Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 It's not inconceivable that fractional microsecond values matter in virtualization environments.within the same facility.  On a larger scale, the uncertainties associated with the timestamp value will make nanosecond precision moot. Has anyone articulated what the overhead differential is of an _expression_ with a precision of microseconds versus nanoseconds? -t On 2015-11-23 01:08 PM, Jordan, Bret wrote: > I miss typed in my last email, I meant to say micro seconds not > milliseconds, aka 6 digits of precision not 3 digits of precision. >  Wireshark and other networking / security tools are able to work with > and provide 6 digits of precision. That is VERY common. What is not > really common today is 9 digits of precision. >      

On 24.11.2015 02:36:50, Terry MacDonald wrote: > > I think that we have to put this whole discussion in perspective. > Most organizations have difficulty in discovering they have a > breach within days and weeks, not within a second. So going with > B) iii) and having the precision within 1 second realistically is > good enough in my opinion. It is far more likely that all the > clocks on the network are not synchronized, and all the tools are > reporting different and unrelated times and that they are way > more than a second out of alignment with each other. The zeroed > out millisecond timestamp doesn’t impact us much when we have > real-world problems such as that. > I agree with Terry however... On 23.11.2015 20:51:09, Wunder, John A. wrote: > > This is not to say that we need a precision field, just that if we > do it should be explicit rather than implicit. > ...I also agree with John. On the one hand, I can't count the number of times I've seen investigations get thrown under the bus due to clock-sync issues. On the other hand, recent history has shown that historical assumptions made about datetime can come back to bite us with a vengeance. So I think we should just use the standard explicitly. The producer can use the level of precision they support and intend to communicate, the consumer can easily recognize the level of precision coming from the producer, and handle it appropriately. Doing this in the manner illustrated below imposes less burden on the consumer than handling an optional 'precision' field while accomplishing the same goal. <snip> #!/usr/bin/env python import re nanoseconds = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' seconds = '2015-11-24T09:42:54Z' minutes = '2015-11-24T09:42Z' # You get the idea... timestamps = [nanoseconds, microseconds, milliseconds, seconds, minutes] nano_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{9}Z') micro_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{6}Z') milli_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{3}Z') sec_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}Z') min_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}Z') for ts in timestamps: if nano_re.match(ts): print('Found nanosecond time...') elif micro_re.match(ts): print('Found microsecond time...') elif milli_re.match(ts): print('Found millisecond time...') elif sec_re.match(ts): print('Found second time...') elif min_re.match(ts): print('Found minute time...') </snip There! Entirely explicit, uses RFC 3339 as intended, and there's one clear way of doing things without resorting to optional fields. -- Cheers, Trey -- Trey Darley Senior Security Engineer 4DAA 0A88 34BC 27C9 FD2B A97E D3C6 5C74 0FB7 E430 Soltra An FS-ISAC & DTCC Company www.soltra.com -- "It is always something." --RFC 1925 Attachment: signature.asc Description: PGP signature

I have a slight preference for Trey’s approach but would also be fine with Terry’s. I prefer Trey’s because IMO support for a standard as-is is better than redefining the standard. RFC3339 allows for accuracy to any arbitrary (second) value and therefore the vast majority of date/time libraries will be able to support that. His approach allows us to use the standard as-is. In fact, the requirement for millisecond precision may make things more difficult for those who just want to use RFC3339 processing libraries because they have to get those libraries to produce *our* version of RFC 3339 rather than anything compatible with the standard. I don’t think it will allow accuracy to the day unless we move back to ISO 8601 with all the other openness that it brings. But, I tend to agree with Terry that it will matter very little in almost every circumstance so we shouldn’t worry about it. John > On Nov 24, 2015, at 5:28 AM, Trey Darley <trey@soltra.com> wrote: > > On 24.11.2015 02:36:50, Terry MacDonald wrote: >> >> I think that we have to put this whole discussion in perspective. >> Most organizations have difficulty in discovering they have a >> breach within days and weeks, not within a second. So going with >> B) iii) and having the precision within 1 second realistically is >> good enough in my opinion. It is far more likely that all the >> clocks on the network are not synchronized, and all the tools are >> reporting different and unrelated times and that they are way >> more than a second out of alignment with each other. The zeroed >> out millisecond timestamp doesn’t impact us much when we have >> real-world problems such as that. >> > > I agree with Terry however... > > On 23.11.2015 20:51:09, Wunder, John A. wrote: >> >> This is not to say that we need a precision field, just that if we >> do it should be explicit rather than implicit. >> > > ...I also agree with John. On the one hand, I can't count the number > of times I've seen investigations get thrown under the bus due to > clock-sync issues. On the other hand, recent history has shown that > historical assumptions made about datetime can come back to bite us > with a vengeance. > > So I think we should just use the standard explicitly. The producer > can use the level of precision they support and intend to communicate, > the consumer can easily recognize the level of precision coming from > the producer, and handle it appropriately. Doing this in the manner > illustrated below imposes less burden on the consumer than handling an > optional 'precision' field while accomplishing the same goal. > > <snip> > #!/usr/bin/env python > > import re > > nanoseconds = '2015-11-24T09:42:54.003259294Z' > microseconds = '2015-11-24T09:42:54.003259Z' > milliseconds = '2015-11-24T09:42:54.003Z' > seconds = '2015-11-24T09:42:54Z' > minutes = '2015-11-24T09:42Z' > # You get the idea... > > timestamps = [nanoseconds, microseconds, milliseconds, seconds, minutes] > > nano_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{9}Z') > micro_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{6}Z') > milli_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{3}Z') > sec_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}Z') > min_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}Z') > > for ts in timestamps: > if nano_re.match(ts): > print('Found nanosecond time...') > elif micro_re.match(ts): > print('Found microsecond time...') > elif milli_re.match(ts): > print('Found millisecond time...') > elif sec_re.match(ts): > print('Found second time...') > elif min_re.match(ts): > print('Found minute time...') > </snip > > There! Entirely explicit, uses RFC 3339 as intended, and there's one > clear way of doing things without resorting to optional fields. > > -- > Cheers, > Trey > -- > Trey Darley > Senior Security Engineer > 4DAA 0A88 34BC 27C9 FD2B A97E D3C6 5C74 0FB7 E430 > Soltra An FS-ISAC & DTCC Company > www.soltra.com > -- > "It is always something." --RFC 1925

So I think we are finally getting somewhere.....  Before we claim that we agree, let me paraphrase and summarize just to make sure: 1) A timestamp format of yyyy-mm-dd-Thh:mm:ssZ MUST be used Examples: 2015-11-23T13:35:12Z  (for 1:35:12 in UTC format) 2) All timestamps MUST be in UTC a UI will change them as needed for an analyst  3) A timezone offset will NOT be used, all times will be recorded in UTC 4) Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision Examples: nanoseconds  = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' etc. 5) There will be no extra precision field Open questions A) If the time of day is not known should it be: i) zeroed out Examples: '2015-11-24T11:00:00Z'  (known only to the 11th hour) '2015-11-24T00:00:00Z (known only to the day) ii) not included, just like the sub-second (if we do this, is this a violation of RFC3339) Examples: '2015-11-24T11Z'  (known only to the 11th hour) '2015-11-24TZ (known only to the day) Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg.   On Nov 24, 2015, at 06:24, Wunder, John A. < jwunder@mitre.org > wrote: I have a slight preference for Trey’s approach but would also be fine with Terry’s. I prefer Trey’s because IMO support for a standard as-is is better than redefining the standard. RFC3339 allows for accuracy to any arbitrary (second) value and therefore the vast majority of date/time libraries will be able to support that. His approach allows us to use the standard as-is. In fact, the requirement for millisecond precision may make things more difficult for those who just want to use RFC3339 processing libraries because they have to get those libraries to produce *our* version of RFC 3339 rather than anything compatible with the standard. I don’t think it will allow accuracy to the day unless we move back to ISO 8601 with all the other openness that it brings. But, I tend to agree with Terry that it will matter very little in almost every circumstance so we shouldn’t worry about it. John On Nov 24, 2015, at 5:28 AM, Trey Darley < trey@soltra.com > wrote: On 24.11.2015 02:36:50, Terry MacDonald wrote:  I think that we have to put this whole discussion in perspective.  Most organizations have difficulty in discovering they have a  breach within days and weeks, not within a second. So going with  B) iii) and having the precision within 1 second realistically is  good enough in my opinion. It is far more likely that all the  clocks on the network are not synchronized, and all the tools are  reporting different and unrelated times and that they are way  more than a second out of alignment with each other. The zeroed  out millisecond timestamp doesn’t impact us much when we have  real-world problems such as that. I agree with Terry however... On 23.11.2015 20:51:09, Wunder, John A. wrote: This is not to say that we need a precision field, just that if we do it should be explicit rather than implicit. ...I also agree with John. On the one hand, I can't count the number of times I've seen investigations get thrown under the bus due to clock-sync issues. On the other hand, recent history has shown that historical assumptions made about datetime can come back to bite us with a vengeance. So I think we should just use the standard explicitly. The producer can use the level of precision they support and intend to communicate, the consumer can easily recognize the level of precision coming from the producer, and handle it appropriately. Doing this in the manner illustrated below imposes less burden on the consumer than handling an optional 'precision' field while accomplishing the same goal. <snip> #!/usr/bin/env python import re nanoseconds  = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' seconds      = '2015-11-24T09:42:54Z' minutes      = '2015-11-24T09:42Z' # You get the idea... timestamps = [nanoseconds, microseconds, milliseconds, seconds, minutes] nano_re  = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{9}Z') micro_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{6}Z') milli_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{3}Z') sec_re   = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}Z') min_re   = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}Z') for ts in timestamps:   if nano_re.match(ts):       print('Found nanosecond time...')   elif micro_re.match(ts):       print('Found microsecond time...')   elif milli_re.match(ts):       print('Found millisecond time...')   elif sec_re.match(ts):       print('Found second time...')   elif min_re.match(ts):       print('Found minute time...') </snip There! Entirely explicit, uses RFC 3339 as intended, and there's one clear way of doing things without resorting to optional fields. -- Cheers, Trey -- Trey Darley Senior Security Engineer 4DAA 0A88 34BC 27C9 FD2B  A97E D3C6 5C74 0FB7 E430 Soltra An FS-ISAC & DTCC Company www.soltra.com -- It is always something. --RFC 1925 Attachment: signature.asc Description: Message signed with OpenPGP using GPGMail

I am fine with #1-4
I disagree with #5.
Again, I will assert there IS a need for an explicit precision field. I have tried to explain the context a couple of time but apparently still have not done a good enough job.
Unfortunately, I do not have the cycles today to attempt to explain the scenarios for precision on this topic in any further detail.
Hopefully I will have a chance to do so tomorrow.


I just wanted to make it clear for now that we do not have consensus on #5.


Sean









From: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > on behalf of "Jordan, Bret" < bret.jordan@bluecoat.com >
Date: Tuesday, November 24, 2015 at 1:24 PM
To: John Wunder < jwunder@mitre.org >
Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000





So I think we are finally getting somewhere.....  Before we claim that we agree, let me paraphrase and summarize just to make sure:


1) A timestamp format of yyyy-mm-dd-Thh:mm:ssZ MUST be used
Examples:
2015-11-23T13:35:12Z  (for 1:35:12 in UTC format)
2) All timestamps MUST be in UTC a UI will change them as needed for an analyst 
3) A timezone offset will NOT be used, all times will be recorded in UTC
4) Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision
Examples:
nanoseconds  = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'
etc.
5) There will be no extra precision field




Open questions
A) If the time of day is not known should it be:
i) zeroed out

Examples:
'2015-11-24T11:00:00Z'  (known only to the 11th hour)
'2015-11-24T00:00:00Z (known only to the day)

ii) not included, just like the sub-second (if we do this, is this a violation of RFC3339)









Examples:

'2015-11-24T11Z'  (known only to the 11th hour)
'2015-11-24TZ
(known only to the day)



Thanks,




Bret












Bret Jordan CISSP

Director of Security Architecture and Standards Office of the CTO

Blue Coat Systems

PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." 











On Nov 24, 2015, at 06:24, Wunder, John A. < jwunder@mitre.org > wrote:

I have a slight preference for Trey’s approach but would also be fine with Terry’s.

I prefer Trey’s because IMO support for a standard as-is is better than redefining the standard. RFC3339 allows for accuracy to any arbitrary (second) value and therefore the vast majority of date/time libraries will be able to support that. His approach allows
us to use the standard as-is. In fact, the requirement for millisecond precision may make things more difficult for those who just want to use RFC3339 processing libraries because they have to get those libraries to produce *our* version of RFC 3339 rather
than anything compatible with the standard.

I don’t think it will allow accuracy to the day unless we move back to ISO 8601 with all the other openness that it brings. But, I tend to agree with Terry that it will matter very little in almost every circumstance so we shouldn’t worry about it.

John

On Nov 24, 2015, at 5:28 AM, Trey Darley < trey@soltra.com > wrote:

On 24.11.2015 02:36:50, Terry MacDonald wrote:

 I think that we have to put this whole discussion in perspective.
 Most organizations have difficulty in discovering they have a
 breach within days and weeks, not within a second. So going with
 B) iii) and having the precision within 1 second realistically is
 good enough in my opinion. It is far more likely that all the
 clocks on the network are not synchronized, and all the tools are
 reporting different and unrelated times and that they are way
 more than a second out of alignment with each other. The zeroed
 out millisecond timestamp doesn’t impact us much when we have
 real-world problems such as that.



I agree with Terry however...

On 23.11.2015 20:51:09, Wunder, John A. wrote:

This is not to say that we need a precision field, just that if we
do it should be explicit rather than implicit.



...I also agree with John. On the one hand, I can't count the number
of times I've seen investigations get thrown under the bus due to
clock-sync issues. On the other hand, recent history has shown that
historical assumptions made about datetime can come back to bite us
with a vengeance.

So I think we should just use the standard explicitly. The producer
can use the level of precision they support and intend to communicate,
the consumer can easily recognize the level of precision coming from
the producer, and handle it appropriately. Doing this in the manner
illustrated below imposes less burden on the consumer than handling an
optional 'precision' field while accomplishing the same goal.

<snip>
#!/usr/bin/env python

import re

nanoseconds  = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'
seconds      = '2015-11-24T09:42:54Z'
minutes      = '2015-11-24T09:42Z'
# You get the idea...

timestamps = [nanoseconds, microseconds, milliseconds, seconds, minutes]

nano_re  = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{9}Z')
micro_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{6}Z')
milli_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{3}Z')
sec_re   = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}Z')
min_re   = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}Z')

for ts in timestamps:
  if nano_re.match(ts):
      print('Found nanosecond time...')
  elif micro_re.match(ts):
      print('Found microsecond time...')
  elif milli_re.match(ts):
      print('Found millisecond time...')
  elif sec_re.match(ts):
      print('Found second time...')
  elif min_re.match(ts):
      print('Found minute time...')
</snip

There! Entirely explicit, uses RFC 3339 as intended, and there's one
clear way of doing things without resorting to optional fields.

--
Cheers,
Trey
--
Trey Darley
Senior Security Engineer
4DAA 0A88 34BC 27C9 FD2B  A97E D3C6 5C74 0FB7 E430
Soltra An FS-ISAC & DTCC Company
www.soltra.com
--
"It is always something." --RFC 1925

I am fine with Trey’s proposal too.  Specifically I support all 5 items on Bret’s email. I agree we do not have consensus on #5.
 
A few extra comments though, brought up by a review of the extra restrictions the
Syslog RFC (6.2.3 TIMESTAMP) places on implementations.
 
Is this an acceptable full list for the timestamp? (shamelessly borrowed bits from the syslog rfc).

 
The TIMESTAMP field is a formalized timestamp derived from [RFC3339].
 
Whereas [RFC3339] makes allowances for multiple syntaxes, this
document imposes further restrictions.  The TIMESTAMP value MUST
follow these additional restrictions:
 

1.       
An RFC3339 timestamp format MUST be used.


2.       
All timestamps MUST be in UTC


3.       
A timezone offset MUST NOT be used. All times MUST be recorded in UTC time.

4.       
Usage of the "T" and "Z" characters are REQUIRED.



5.       
The "T" and "Z" characters in this syntax MUST be upper case.

6.       
The producer SHOULD include as much precision as its clock accuracy and performance permit. 


7.       
Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine
the precision
Examples:
nanoseconds  = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'
 
Note: The additional precision field is still up for debate as we currently do not have consensus.
 
What say everybody?
 
Cheers
 

Terry MacDonald
Senior STIX Subject Matter Expert
SOLTRA   An FS-ISAC and DTCC Company
+61 (407) 203 206
terry@soltra.com
 

 


From: cti-stix@lists.oasis-open.org [mailto:cti-stix@lists.oasis-open.org]
On Behalf Of Barnum, Sean D.
Sent: Wednesday, 25 November 2015 5:34 AM
To: Jordan, Bret <bret.jordan@bluecoat.com>; Wunder, John A. <jwunder@mitre.org>
Cc: cti-stix@lists.oasis-open.org
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000


 



I am fine with #1-4


I disagree with #5.


Again, I will assert there IS a need for an explicit precision field. I have tried to explain the context a couple of time but apparently still have not done a
good enough job.


Unfortunately, I do not have the cycles today to attempt to explain the scenarios for precision on this topic in any further detail.


Hopefully I will have a chance to do so tomorrow.


 


I just wanted to make it clear for now that we do not have consensus on #5.


 


Sean




 


From:
" cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > on behalf of "Jordan,
Bret" < bret.jordan@bluecoat.com >
Date: Tuesday, November 24, 2015 at 1:24 PM
To: John Wunder < jwunder@mitre.org >
Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000


 



So I think we are finally getting somewhere.....  Before we claim that we agree, let me paraphrase and summarize just to make sure:


 


1) A timestamp format of yyyy-mm-dd-Thh:mm:ssZ MUST be used


Examples:


2015-11-23T13:35:12Z  (for 1:35:12 in UTC format)
2) All timestamps MUST be in UTC a UI will change them as needed for an analyst 


3) A timezone offset will NOT be used, all times will be recorded in UTC
4) Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision


Examples:


nanoseconds  = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'


etc.
5) There will be no extra precision field


 


 


Open questions


A) If the time of day is not known should it be:


i) zeroed out



Examples:


'2015-11-24T11:00:00Z'  (known only to the 11th hour)


'2015-11-24T00:00:00Z (known only to the day)



ii) not included, just like the sub-second (if we do this, is this a violation of RFC3339)







Examples:


'2015-11-24T11Z'  (known only to the 11th hour)


'2015-11-24TZ (known only to the day)


Thanks,


 


Bret



 


 


 



Bret Jordan CISSP


Director of Security Architecture and Standards Office of the CTO


Blue Coat Systems



PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050


"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." 









 



On Nov 24, 2015, at 06:24, Wunder, John A. < jwunder@mitre.org > wrote:

 

I have a slight preference for Trey’s approach but would also be fine with Terry’s.

I prefer Trey’s because IMO support for a standard as-is is better than redefining the standard. RFC3339 allows for accuracy to any arbitrary (second) value and therefore the vast majority of date/time libraries will be able to support that. His approach allows
us to use the standard as-is. In fact, the requirement for millisecond precision may make things more difficult for those who just want to use RFC3339 processing libraries because they have to get those libraries to produce *our* version of RFC 3339 rather
than anything compatible with the standard.

I don’t think it will allow accuracy to the day unless we move back to ISO 8601 with all the other openness that it brings. But, I tend to agree with Terry that it will matter very little in almost every circumstance so we shouldn’t worry about it.

John




On Nov 24, 2015, at 5:28 AM, Trey Darley < trey@soltra.com > wrote:

On 24.11.2015 02:36:50, Terry MacDonald wrote:




 I think that we have to put this whole discussion in perspective.
 Most organizations have difficulty in discovering they have a
 breach within days and weeks, not within a second. So going with
 B) iii) and having the precision within 1 second realistically is
 good enough in my opinion. It is far more likely that all the
 clocks on the network are not synchronized, and all the tools are
 reporting different and unrelated times and that they are way
 more than a second out of alignment with each other. The zeroed
 out millisecond timestamp doesn’t impact us much when we have
 real-world problems such as that.


I agree with Terry however...

On 23.11.2015 20:51:09, Wunder, John A. wrote:




This is not to say that we need a precision field, just that if we
do it should be explicit rather than implicit.


...I also agree with John. On the one hand, I can't count the number
of times I've seen investigations get thrown under the bus due to
clock-sync issues. On the other hand, recent history has shown that
historical assumptions made about datetime can come back to bite us
with a vengeance.

So I think we should just use the standard explicitly. The producer
can use the level of precision they support and intend to communicate,
the consumer can easily recognize the level of precision coming from
the producer, and handle it appropriately. Doing this in the manner
illustrated below imposes less burden on the consumer than handling an
optional 'precision' field while accomplishing the same goal.

<snip>
#!/usr/bin/env python

import re

nanoseconds  = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'
seconds      = '2015-11-24T09:42:54Z'
minutes      = '2015-11-24T09:42Z'
# You get the idea...

timestamps = [nanoseconds, microseconds, milliseconds, seconds, minutes]

nano_re  = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{9}Z')
micro_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{6}Z')
milli_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{3}Z')
sec_re   = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}Z')
min_re   = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}Z')

for ts in timestamps:
  if nano_re.match(ts):
      print('Found nanosecond time...')
  elif micro_re.match(ts):
      print('Found microsecond time...')
  elif milli_re.match(ts):
      print('Found millisecond time...')
  elif sec_re.match(ts):
      print('Found second time...')
  elif min_re.match(ts):
      print('Found minute time...')
</snip

There! Entirely explicit, uses RFC 3339 as intended, and there's one
clear way of doing things without resorting to optional fields.

--
Cheers,
Trey
--
Trey Darley
Senior Security Engineer
4DAA 0A88 34BC 27C9 FD2B  A97E D3C6 5C74 0FB7 E430
Soltra An FS-ISAC & DTCC Company
www.soltra.com
--
"It is always something." --RFC 1925

 



 

[+1] on use of syslog RFC and for support for Precision (in the literal sense).
[+~] on being able to drive to consensus on this highly critical aspect "our thing" (even though we broke our priority list sequence)


But to Sean's point we do as a practical matter deal with "Uncertainty" (AKA "Precision") in Timestamps.  There are a dozen use cases I can cite, but think I can simply use the "Well, were reporting this incident now as required by Law, but we are only
certain at this point that the activity occurred as far back as yesterday.  


So we can clearly agree that there should be one and only one way of specifying temporal "values" (whether fixed, relative, or intervals).  But his "one way of doing things" means we still need a way to convey "uncertainty" where we need to communicate
same.


Dose that makes sense?






Patrick Maroney
President
Integrated Networking Technologies, Inc.
Office:  (856)983-0001
Cell:      (609)841-5104










From: < cti-stix@lists.oasis-open.org > on behalf of Terry MacDonald < terry@soltra.com >
Date: Tuesday, November 24, 2015 at 2:37 PM
To: Sean Barnum < sbarnum@mitre.org >, Bret Jordan < bret.jordan@bluecoat.com >, John Wunder < jwunder@mitre.org >
Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000








I am fine with Trey’s proposal too.  Specifically I support all 5 items on Bret’s email. I agree we do not have consensus on #5.
 
A few extra comments though, brought up by a review of the extra restrictions the Syslog RFC
(6.2.3 TIMESTAMP) places on implementations.
 
Is this an acceptable full list for the timestamp? (shamelessly borrowed bits from the syslog rfc).

 
The TIMESTAMP field is a formalized timestamp derived from [RFC3339].
 
Whereas [RFC3339] makes allowances for multiple syntaxes, this
document imposes further restrictions.  The TIMESTAMP value MUST
follow these additional restrictions:
 

1.       
An RFC3339 timestamp format MUST be used.


2.       
All timestamps MUST be in UTC


3.       
A timezone offset MUST NOT be used. All times MUST be recorded in UTC time.

4.       
Usage of the "T" and "Z" characters are REQUIRED.

5.       
The "T" and "Z" characters in this syntax MUST be upper case.

6.       
The producer SHOULD include as much precision as its clock accuracy and performance permit. 


7.       
Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision
Examples:
nanoseconds  = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'
 
Note: The additional precision field is still up for debate as we currently do not have consensus.
 
What say everybody?
 
Cheers
 

Terry MacDonald
Senior STIX Subject Matter Expert
SOLTRA   An FS-ISAC and DTCC Company
+61 (407) 203 206
terry@soltra.com
 

 


From:
cti-stix@lists.oasis-open.org [ mailto:cti-stix@lists.oasis-open.org ]
On Behalf Of Barnum, Sean D.
Sent: Wednesday, 25 November 2015 5:34 AM
To: Jordan, Bret < bret.jordan@bluecoat.com >; Wunder, John A. < jwunder@mitre.org >
Cc: cti-stix@lists.oasis-open.org
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000


 



I am fine with #1-4


I disagree with #5.


Again, I will assert there IS a need for an explicit precision field. I have tried to explain the context a couple of time but apparently still have not done
a good enough job.


Unfortunately, I do not have the cycles today to attempt to explain the scenarios for precision on this topic in any further detail.


Hopefully I will have a chance to do so tomorrow.


 


I just wanted to make it clear for now that we do not have consensus on #5.


 


Sean




 


From:
" cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > on behalf of "Jordan,
Bret" < bret.jordan@bluecoat.com >
Date: Tuesday, November 24, 2015 at 1:24 PM
To: John Wunder < jwunder@mitre.org >
Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000


 



So I think we are finally getting somewhere.....  Before we claim that we agree, let me paraphrase and summarize just to make sure:


 


1) A timestamp format of yyyy-mm-dd-Thh:mm:ssZ MUST be used


Examples:


2015-11-23T13:35:12Z  (for 1:35:12 in UTC format)
2) All timestamps MUST be in UTC a UI will change them as needed for an analyst 


3) A timezone offset will NOT be used, all times will be recorded in UTC
4) Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision


Examples:


nanoseconds  = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'


etc.
5) There will be no extra precision field


 


 


Open questions


A) If the time of day is not known should it be:


i) zeroed out



Examples:


'2015-11-24T11:00:00Z'  (known only to the 11th hour)


'2015-11-24T00:00:00Z (known only to the day)



ii) not included, just like the sub-second (if we do this, is this a violation of RFC3339)







Examples:


'2015-11-24T11Z'  (known only to the 11th hour)


'2015-11-24TZ (known only to the day)


Thanks,


 


Bret



 


 


 



Bret Jordan CISSP

Director of Security Architecture and Standards Office of the CTO


Blue Coat Systems



PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050


"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." 









 



On Nov 24, 2015, at 06:24, Wunder, John A. < jwunder@mitre.org > wrote:

 

I have a slight preference for Trey’s approach but would also be fine with Terry’s.

I prefer Trey’s because IMO support for a standard as-is is better than redefining the standard. RFC3339 allows for accuracy to any arbitrary (second) value and therefore the vast majority of date/time libraries will be able to support that. His approach allows
us to use the standard as-is. In fact, the requirement for millisecond precision may make things more difficult for those who just want to use RFC3339 processing libraries because they have to get those libraries to produce *our* version of RFC 3339 rather
than anything compatible with the standard.

I don’t think it will allow accuracy to the day unless we move back to ISO 8601 with all the other openness that it brings. But, I tend to agree with Terry that it will matter very little in almost every circumstance so we shouldn’t worry about it.

John




On Nov 24, 2015, at 5:28 AM, Trey Darley < trey@soltra.com > wrote:

On 24.11.2015 02:36:50, Terry MacDonald wrote:




 I think that we have to put this whole discussion in perspective.
 Most organizations have difficulty in discovering they have a
 breach within days and weeks, not within a second. So going with
 B) iii) and having the precision within 1 second realistically is
 good enough in my opinion. It is far more likely that all the
 clocks on the network are not synchronized, and all the tools are
 reporting different and unrelated times and that they are way
 more than a second out of alignment with each other. The zeroed
 out millisecond timestamp doesn’t impact us much when we have
 real-world problems such as that.


I agree with Terry however...

On 23.11.2015 20:51:09, Wunder, John A. wrote:




This is not to say that we need a precision field, just that if we
do it should be explicit rather than implicit.


...I also agree with John. On the one hand, I can't count the number
of times I've seen investigations get thrown under the bus due to
clock-sync issues. On the other hand, recent history has shown that
historical assumptions made about datetime can come back to bite us
with a vengeance.

So I think we should just use the standard explicitly. The producer
can use the level of precision they support and intend to communicate,
the consumer can easily recognize the level of precision coming from
the producer, and handle it appropriately. Doing this in the manner
illustrated below imposes less burden on the consumer than handling an
optional 'precision' field while accomplishing the same goal.

<snip>
#!/usr/bin/env python

import re

nanoseconds  = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'
seconds      = '2015-11-24T09:42:54Z'
minutes      = '2015-11-24T09:42Z'
# You get the idea...

timestamps = [nanoseconds, microseconds, milliseconds, seconds, minutes]

nano_re  = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{9}Z')
micro_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{6}Z')
milli_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{3}Z')
sec_re   = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}Z')
min_re   = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}Z')

for ts in timestamps:
  if nano_re.match(ts):
      print('Found nanosecond time...')
  elif micro_re.match(ts):
      print('Found microsecond time...')
  elif milli_re.match(ts):
      print('Found millisecond time...')
  elif sec_re.match(ts):
      print('Found second time...')
  elif min_re.match(ts):
      print('Found minute time...')
</snip

There! Entirely explicit, uses RFC 3339 as intended, and there's one
clear way of doing things without resorting to optional fields.

--
Cheers,
Trey
--
Trey Darley
Senior Security Engineer
4DAA 0A88 34BC 27C9 FD2B  A97E D3C6 5C74 0FB7 E430
Soltra An FS-ISAC & DTCC Company
www.soltra.com
--
"It is always something." --RFC 1925

 



 

HI Patrick - please see my earlier email around this topic, specifically the part around end-to-end workflow scenarios... " "Well, were reporting this incident now as required by Law, but we are only certain at this point that the activity occurred as far back as yesterday. " So lets assume for a moment that in the above use case the timestamp said "2015-11-24 00:00:00.000000" and did not indicate any type of "precision" indicating "+/- 24 hours"... why do I as the consumer of that information care.. how does that affect my consumption of that threat report? How can I make more effective use of that information in my system if I have that precision indicator? I am not going to do low-level temporal analysis on this type of indicator, so it doesn't matter from that perspective. What is the workflow where it does matter? - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown Patrick Maroney ---11/24/2015 04:17:59 PM---[+1] on use of syslog RFC and for support for Precision (in the literal sense). [+~] on being able t From: Patrick Maroney <Pmaroney@Specere.org> To: Terry MacDonald <terry@soltra.com>, "Barnum, Sean D." <sbarnum@mitre.org>, "Jordan, Bret" <bret.jordan@bluecoat.com>, "Wunder, John A." <jwunder@mitre.org> Cc: "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org> Date: 11/24/2015 04:17 PM Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 Sent by: <cti-stix@lists.oasis-open.org> [+1] on use of syslog RFC and for support for Precision (in the literal sense). [+~] on being able to drive to consensus on this highly critical aspect "our thing" (even though we broke our priority list sequence) But to Sean's point we do as a practical matter deal with "Uncertainty" (AKA "Precision") in Timestamps. There are a dozen use cases I can cite, but think I can simply use the "Well, were reporting this incident now as required by Law, but we are only certain at this point that the activity occurred as far back as yesterday. So we can clearly agree that there should be one and only one way of specifying temporal "values" (whether fixed, relative, or intervals). But his "one way of doing things" means we still need a way to convey "uncertainty" where we need to communicate same. Dose that makes sense? Patrick Maroney President Integrated Networking Technologies, Inc. Office: (856)983-0001 Cell: (609)841-5104 From: < cti-stix@lists.oasis-open.org > on behalf of Terry MacDonald < terry@soltra.com > Date: Tuesday, November 24, 2015 at 2:37 PM To: Sean Barnum < sbarnum@mitre.org >, Bret Jordan < bret.jordan@bluecoat.com >, John Wunder < jwunder@mitre.org > Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000 I am fine with Trey’s proposal too. Specifically I support all 5 items on Bret’s email. I agree we do not have consensus on #5. A few extra comments though, brought up by a review of the extra restrictions the Syslog RFC (6.2.3 TIMESTAMP) places on implementations. Is this an acceptable full list for the timestamp? (shamelessly borrowed bits from the syslog rfc). The TIMESTAMP field is a formalized timestamp derived from [RFC3339]. Whereas [RFC3339] makes allowances for multiple syntaxes, this document imposes further restrictions. The TIMESTAMP value MUST follow these additional restrictions: 1. An RFC3339 timestamp format MUST be used. 2. All timestamps MUST be in UTC 3. A timezone offset MUST NOT be used. All times MUST be recorded in UTC time. 4. Usage of the "T" and "Z" characters are REQUIRED. 5. The "T" and "Z" characters in this syntax MUST be upper case. 6. The producer SHOULD include as much precision as its clock accuracy and performance permit. 7. Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision Examples: nanoseconds = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' Note: The additional precision field is still up for debate as we currently do not have consensus. What say everybody? Cheers Terry MacDonald Senior STIX Subject Matter Expert SOLTRA An FS-ISAC and DTCC Company +61 (407) 203 206 terry@soltra.com From: cti-stix@lists.oasis-open.org [ mailto:cti-stix@lists.oasis-open.org ] On Behalf Of Barnum, Sean D. Sent: Wednesday, 25 November 2015 5:34 AM To: Jordan, Bret < bret.jordan@bluecoat.com >; Wunder, John A. < jwunder@mitre.org > Cc: cti-stix@lists.oasis-open.org Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 I am fine with #1-4 I disagree with #5. Again, I will assert there IS a need for an explicit precision field. I have tried to explain the context a couple of time but apparently still have not done a good enough job. Unfortunately, I do not have the cycles today to attempt to explain the scenarios for precision on this topic in any further detail. Hopefully I will have a chance to do so tomorrow. I just wanted to make it clear for now that we do not have consensus on #5. Sean From: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > on behalf of "Jordan, Bret" < bret.jordan@bluecoat.com > Date: Tuesday, November 24, 2015 at 1:24 PM To: John Wunder < jwunder@mitre.org > Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 So I think we are finally getting somewhere..... Before we claim that we agree, let me paraphrase and summarize just to make sure: 1) A timestamp format of yyyy-mm-dd-Thh:mm:ssZ MUST be used Examples: 2015-11-23T13:35:12Z (for 1:35:12 in UTC format) 2) All timestamps MUST be in UTC a UI will change them as needed for an analyst 3) A timezone offset will NOT be used, all times will be recorded in UTC 4) Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision Examples: nanoseconds = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' etc. 5) There will be no extra precision field Open questions A) If the time of day is not known should it be: i) zeroed out Examples: '2015-11-24T11:00:00Z' (known only to the 11th hour) '2015-11-24T00:00:00Z (known only to the day) ii) not included, just like the sub-second (if we do this, is this a violation of RFC3339) Examples: '2015-11-24T11Z' (known only to the 11th hour) '2015-11-24TZ (known only to the day) Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." On Nov 24, 2015, at 06:24, Wunder, John A. < jwunder@mitre.org > wrote: I have a slight preference for Trey’s approach but would also be fine with Terry’s. I prefer Trey’s because IMO support for a standard as-is is better than redefining the standard. RFC3339 allows for accuracy to any arbitrary (second) value and therefore the vast majority of date/time libraries will be able to support that. His approach allows us to use the standard as-is. In fact, the requirement for millisecond precision may make things more difficult for those who just want to use RFC3339 processing libraries because they have to get those libraries to produce *our* version of RFC 3339 rather than anything compatible with the standard. I don’t think it will allow accuracy to the day unless we move back to ISO 8601 with all the other openness that it brings. But, I tend to agree with Terry that it will matter very little in almost every circumstance so we shouldn’t worry about it. John On Nov 24, 2015, at 5:28 AM, Trey Darley < trey@soltra.com > wrote: On 24.11.2015 02:36:50, Terry MacDonald wrote: I think that we have to put this whole discussion in perspective. Most organizations have difficulty in discovering they have a breach within days and weeks, not within a second. So going with B) iii) and having the precision within 1 second realistically is good enough in my opinion. It is far more likely that all the clocks on the network are not synchronized, and all the tools are reporting different and unrelated times and that they are way more than a second out of alignment with each other. The zeroed out millisecond timestamp doesn’t impact us much when we have real-world problems such as that. I agree with Terry however... On 23.11.2015 20:51:09, Wunder, John A. wrote: This is not to say that we need a precision field, just that if we do it should be explicit rather than implicit. ...I also agree with John. On the one hand, I can't count the number of times I've seen investigations get thrown under the bus due to clock-sync issues. On the other hand, recent history has shown that historical assumptions made about datetime can come back to bite us with a vengeance. So I think we should just use the standard explicitly. The producer can use the level of precision they support and intend to communicate, the consumer can easily recognize the level of precision coming from the producer, and handle it appropriately. Doing this in the manner illustrated below imposes less burden on the consumer than handling an optional 'precision' field while accomplishing the same goal. <snip> #!/usr/bin/env python import re nanoseconds = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' seconds = '2015-11-24T09:42:54Z' minutes = '2015-11-24T09:42Z' # You get the idea... timestamps = [nanoseconds, microseconds, milliseconds, seconds, minutes] nano_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{9}Z') micro_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{6}Z') milli_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{3}Z') sec_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}Z') min_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}Z') for ts in timestamps: if nano_re.match(ts): print('Found nanosecond time...') elif micro_re.match(ts): print('Found microsecond time...') elif milli_re.match(ts): print('Found millisecond time...') elif sec_re.match(ts): print('Found second time...') elif min_re.match(ts): print('Found minute time...') </snip There! Entirely explicit, uses RFC 3339 as intended, and there's one clear way of doing things without resorting to optional fields. -- Cheers, Trey -- Trey Darley Senior Security Engineer 4DAA 0A88 34BC 27C9 FD2B A97E D3C6 5C74 0FB7 E430 Soltra An FS-ISAC & DTCC Company www.soltra.com -- "It is always something." --RFC 1925

I too want to know this...  As the more I have thought about it, the more I am coming to the idea that the precision field is just theoretical.  It does not have an effect in the actual work flow.  If it does, please give some concrete real world work flow examples of how this is going to help. Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg.   On Nov 24, 2015, at 13:25, Jason Keirstead < Jason.Keirstead@ca.ibm.com > wrote: HI Patrick - please see my earlier email around this topic, specifically the part around end-to-end workflow scenarios... Well, were reporting this incident now as required by Law, but we are only certain at this point that the activity occurred as far back as yesterday. So lets assume for a moment that in the above use case the timestamp said 2015-11-24 00:00:00.000000 and did not indicate any type of precision indicating +/- 24 hours ... why do I as the consumer of that information care.. how does that affect my consumption of that threat report? How can I make more effective use of that information in my system if I have that precision indicator? I am not going to do low-level temporal analysis on this type of indicator, so it doesn't matter from that perspective. What is the workflow where it does matter? - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown <graycol.gif> Patrick Maroney ---11/24/2015 04:17:59 PM---[+1] on use of syslog RFC and for support for Precision (in the literal sense). [+~] on being able t From: Patrick Maroney < Pmaroney@Specere.org > To: Terry MacDonald < terry@soltra.com >, Barnum, Sean D. < sbarnum@mitre.org >, Jordan, Bret < bret.jordan@bluecoat.com >, Wunder, John A. < jwunder@mitre.org > Cc: cti-stix@lists.oasis-open.org < cti-stix@lists.oasis-open.org > Date: 11/24/2015 04:17 PM Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 Sent by: < cti-stix@lists.oasis-open.org > [+1] on use of syslog RFC and for support for Precision (in the literal sense). [+~] on being able to drive to consensus on this highly critical aspect our thing (even though we broke our priority list sequence) But to Sean's point we do as a practical matter deal with Uncertainty (AKA Precision ) in Timestamps. There are a dozen use cases I can cite, but think I can simply use the Well, were reporting this incident now as required by Law, but we are only certain at this point that the activity occurred as far back as yesterday. So we can clearly agree that there should be one and only one way of specifying temporal values (whether fixed, relative, or intervals). But his one way of doing things means we still need a way to convey uncertainty where we need to communicate same. Dose that makes sense? Patrick Maroney President Integrated Networking Technologies, Inc. Office: (856)983-0001 Cell: (609)841-5104 From: < cti-stix@lists.oasis-open.org > on behalf of Terry MacDonald < terry@soltra.com > Date: Tuesday, November 24, 2015 at 2:37 PM To: Sean Barnum < sbarnum@mitre.org >, Bret Jordan < bret.jordan@bluecoat.com >, John Wunder < jwunder@mitre.org > Cc: cti-stix@lists.oasis-open.org < cti-stix@lists.oasis-open.org > Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000 I am fine with Trey’s proposal too. Specifically I support all 5 items on Bret’s email. I agree we do not have consensus on #5. A few extra comments though, brought up by a review of the extra restrictions the Syslog RFC (6.2.3 TIMESTAMP) places on implementations. Is this an acceptable full list for the timestamp? (shamelessly borrowed bits from the syslog rfc). The TIMESTAMP field is a formalized timestamp derived from [RFC3339]. Whereas [RFC3339] makes allowances for multiple syntaxes, this document imposes further restrictions. The TIMESTAMP value MUST follow these additional restrictions: 1. An RFC3339 timestamp format MUST be used. 2. All timestamps MUST be in UTC 3. A timezone offset MUST NOT be used. All times MUST be recorded in UTC time. 4. Usage of the T and Z characters are REQUIRED. 5. The T and Z characters in this syntax MUST be upper case. 6. The producer SHOULD include as much precision as its clock accuracy and performance permit. 7. Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision Examples: nanoseconds = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' Note: The additional precision field is still up for debate as we currently do not have consensus. What say everybody? Cheers Terry MacDonald Senior STIX Subject Matter Expert SOLTRA An FS-ISAC and DTCC Company +61 (407) 203 206 terry@soltra.com From: cti-stix@lists.oasis-open.org [ mailto:cti-stix@lists.oasis-open.org ] On Behalf Of Barnum, Sean D. Sent: Wednesday, 25 November 2015 5:34 AM To: Jordan, Bret < bret.jordan@bluecoat.com >; Wunder, John A. < jwunder@mitre.org > Cc: cti-stix@lists.oasis-open.org Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 I am fine with #1-4 I disagree with #5. Again, I will assert there IS a need for an explicit precision field. I have tried to explain the context a couple of time but apparently still have not done a good enough job. Unfortunately, I do not have the cycles today to attempt to explain the scenarios for precision on this topic in any further detail. Hopefully I will have a chance to do so tomorrow. I just wanted to make it clear for now that we do not have consensus on #5. Sean From: cti-stix@lists.oasis-open.org < cti-stix@lists.oasis-open.org > on behalf of Jordan, Bret < bret.jordan@bluecoat.com > Date: Tuesday, November 24, 2015 at 1:24 PM To: John Wunder < jwunder@mitre.org > Cc: cti-stix@lists.oasis-open.org < cti-stix@lists.oasis-open.org > Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 So I think we are finally getting somewhere..... Before we claim that we agree, let me paraphrase and summarize just to make sure: 1) A timestamp format of yyyy-mm-dd-Thh:mm:ssZ MUST be used Examples: 2015-11-23T13:35:12Z (for 1:35:12 in UTC format) 2) All timestamps MUST be in UTC a UI will change them as needed for an analyst 3) A timezone offset will NOT be used, all times will be recorded in UTC 4) Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision Examples: nanoseconds = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' etc. 5) There will be no extra precision field Open questions A) If the time of day is not known should it be: i) zeroed out Examples: '2015-11-24T11:00:00Z' (known only to the 11th hour) '2015-11-24T00:00:00Z (known only to the day) ii) not included, just like the sub-second (if we do this, is this a violation of RFC3339) Examples: '2015-11-24T11Z' (known only to the 11th hour) '2015-11-24TZ (known only to the day) Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050 Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg. On Nov 24, 2015, at 06:24, Wunder, John A. < jwunder@mitre.org > wrote: I have a slight preference for Trey’s approach but would also be fine with Terry’s. I prefer Trey’s because IMO support for a standard as-is is better than redefining the standard. RFC3339 allows for accuracy to any arbitrary (second) value and therefore the vast majority of date/time libraries will be able to support that. His approach allows us to use the standard as-is. In fact, the requirement for millisecond precision may make things more difficult for those who just want to use RFC3339 processing libraries because they have to get those libraries to produce *our* version of RFC 3339 rather than anything compatible with the standard. I don’t think it will allow accuracy to the day unless we move back to ISO 8601 with all the other openness that it brings. But, I tend to agree with Terry that it will matter very little in almost every circumstance so we shouldn’t worry about it. John On Nov 24, 2015, at 5:28 AM, Trey Darley < trey@soltra.com > wrote: On 24.11.2015 02:36:50, Terry MacDonald wrote: I think that we have to put this whole discussion in perspective. Most organizations have difficulty in discovering they have a breach within days and weeks, not within a second. So going with B) iii) and having the precision within 1 second realistically is good enough in my opinion. It is far more likely that all the clocks on the network are not synchronized, and all the tools are reporting different and unrelated times and that they are way more than a second out of alignment with each other. The zeroed out millisecond timestamp doesn’t impact us much when we have real-world problems such as that. I agree with Terry however... On 23.11.2015 20:51:09, Wunder, John A. wrote: This is not to say that we need a precision field, just that if we do it should be explicit rather than implicit. ...I also agree with John. On the one hand, I can't count the number of times I've seen investigations get thrown under the bus due to clock-sync issues. On the other hand, recent history has shown that historical assumptions made about datetime can come back to bite us with a vengeance. So I think we should just use the standard explicitly. The producer can use the level of precision they support and intend to communicate, the consumer can easily recognize the level of precision coming from the producer, and handle it appropriately. Doing this in the manner illustrated below imposes less burden on the consumer than handling an optional 'precision' field while accomplishing the same goal. <snip> #!/usr/bin/env python import re nanoseconds = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' seconds = '2015-11-24T09:42:54Z' minutes = '2015-11-24T09:42Z' # You get the idea... timestamps = [nanoseconds, microseconds, milliseconds, seconds, minutes] nano_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{9}Z') micro_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{6}Z') milli_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{3}Z') sec_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}Z') min_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}Z') for ts in timestamps: if nano_re.match(ts): print('Found nanosecond time...') elif micro_re.match(ts): print('Found microsecond time...') elif milli_re.match(ts): print('Found millisecond time...') elif sec_re.match(ts): print('Found second time...') elif min_re.match(ts): print('Found minute time...') </snip There! Entirely explicit, uses RFC 3339 as intended, and there's one clear way of doing things without resorting to optional fields. -- Cheers, Trey -- Trey Darley Senior Security Engineer 4DAA 0A88 34BC 27C9 FD2B A97E D3C6 5C74 0FB7 E430 Soltra An FS-ISAC & DTCC Company www.soltra.com -- It is always something. --RFC 1925 Attachment: signature.asc Description: Message signed with OpenPGP using GPGMail

What if a programmatic event occurred at precisely "2015-11-24 00:00:00.000000" how would my logic discern the difference?  My only issue here is with assuming zero values imply range of uncertainty.  I am only arguing that there should be a mechanism
(optional?) to state uncertainty ***explicitly***.  I'm 100% good with everything else here.


There are a number of Work Flow scenarios were this is relevant (User reports:  "Oh yeah I saw an alert from the antivirus thingie sometime last week that warned me "ReallyBadStuff" will happen if I click OK to this, but I see these all the time and just
clicked OK like I always do".  There are also legal implications for organizations in some communities for event reporting and time frames required under law to do so.  In many cases you have to report before you can reasonably establish "when" something happened.
 We need to be able to express "rough" findings rapidly to meet compliance regulation or contractual terms and then refine these dates and time as due diligence investigation occurs.




re: "I too want to know this...  As the more I have thought about it, the more I am coming to the idea that the precision field is just theoretical.  It does not have an effect in the actual work flow. "


It does have an effect on actual workflow for high Capability Maturity organizations that maintain strict device clock synchronization.  Knowing which block of time to pull PCAPs from a Tape Library of PCAPs from a 10GBs channel for re=processing example
can make the difference of hours in processing time (in the case of Netwitness it takes 2 Hours/Terabyte to reprocess raw data packets through a Decoder back into Metadata and File Artifacts).  Every hour of delay in recovering Actionable IOCs and Malware
Artifacts can result in 100's of additional APT compromised systems that need to be mitigated.  So yes, most organizations do not maintain strict infrastructure Time Synchronization, but we should not presume that there is no value/requirement to support those
who have learned these hard lessons and have addressed same in their policies and procedures.


Folks - We are real close on this one.  Is there a way to add Uncertainty as an optional parameter?






Patrick Maroney
President
Integrated Networking Technologies, Inc.
Office:  (856)983-0001
Cell:      (609)841-5104









From: Bret Jordan < bret.jordan@bluecoat.com >
Date: Tuesday, November 24, 2015 at 3:38 PM
To: Jason Keirstead < Jason.Keirstead@ca.ibm.com >
Cc: Patrick Maroney < Pmaroney@Specere.org >, Terry MacDonald < terry@soltra.com >, Sean Barnum < sbarnum@mitre.org >,
John Wunder < jwunder@mitre.org >, " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000





I too want to know this...  As the more I have thought about it, the more I am coming to the idea that the precision field is just theoretical.  It does not have an effect in the actual work flow.  If it does, please give some concrete real world work flow
examples of how this is going to help.











Thanks,


Bret











Bret Jordan CISSP

Director of Security Architecture and Standards Office of the CTO

Blue Coat Systems

PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." 











On Nov 24, 2015, at 13:25, Jason Keirstead < Jason.Keirstead@ca.ibm.com > wrote:



HI Patrick - please see my earlier email around this topic, specifically the part around end-to-end workflow scenarios...

" "Well, were reporting this incident now as required by Law, but we are only certain at this point that the activity occurred as far back as yesterday.
"

So lets assume for a moment that in the above use case the timestamp said "2015-11-24 00:00:00.000000" and
did not indicate any type of "precision" indicating "+/- 24 hours"... why do I as the consumer of that information care.. how does that affect my consumption of that threat report? How can I make more effective use of that information in my
system if I have that precision indicator? I am not going to do low-level temporal analysis on this type of indicator, so it doesn't matter from that perspective. What is the workflow where it does matter?

-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security
www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


<graycol.gif> Patrick Maroney ---11/24/2015 04:17:59 PM---[+1] on use of syslog RFC and for support for Precision (in the literal sense). [+~] on being able t

From: Patrick Maroney < Pmaroney@Specere.org >
To: Terry MacDonald < terry@soltra.com >, "Barnum, Sean D." < sbarnum@mitre.org >, "Jordan, Bret"
< bret.jordan@bluecoat.com >, "Wunder, John A." < jwunder@mitre.org >
Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
Date: 11/24/2015 04:17 PM
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000
Sent by: < cti-stix@lists.oasis-open.org >





[+1] on use of syslog RFC and for support for Precision (in the literal sense).
[+~] on being able to drive to consensus on this highly critical aspect "our thing" (even though we broke our priority list sequence)

But to Sean's point we do as a practical matter deal with "Uncertainty" (AKA "Precision") in Timestamps. There are a dozen use cases I can cite, but think I can simply use the "Well, were reporting this incident now as
required by Law, but we are only certain at this point that the activity occurred as far back as yesterday.


So we can clearly agree that there should be one and only one way of specifying temporal "values" (whether fixed, relative, or intervals). But his "one way of doing things" means we still need a way to convey "uncertainty"
where we need to communicate same.

Dose that makes sense?

Patrick Maroney
President
Integrated Networking Technologies, Inc.
Office: (856)983-0001
Cell: (609)841-5104

From: < cti-stix@lists.oasis-open.org >
on behalf of Terry MacDonald < terry@soltra.com >
Date: Tuesday, November 24, 2015 at 2:37 PM
To: Sean Barnum < sbarnum@mitre.org >,
Bret Jordan < bret.jordan@bluecoat.com >, John Wunder < jwunder@mitre.org >
Cc: " cti-stix@lists.oasis-open.org "
< cti-stix@lists.oasis-open.org >
Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000

I am fine with Trey’s proposal too. Specifically I support all 5 items on Bret’s email. I agree we do not have consensus on #5.

A few extra comments though, brought up by a review of the extra restrictions the Syslog
RFC (6.2.3 TIMESTAMP) places on implementations.


Is this an acceptable full list for the timestamp? (shamelessly borrowed bits from the syslog rfc).




The TIMESTAMP field is a formalized timestamp derived from [RFC3339].

Whereas [RFC3339] makes allowances for multiple syntaxes, this
document imposes further restrictions. The TIMESTAMP value MUST
follow these additional restrictions:


1.
An RFC3339 timestamp format MUST be used.

2.
All timestamps MUST be in UTC

3.
A timezone offset MUST NOT be used. All times MUST be recorded in UTC time.
4.
Usage of the "T" and "Z" characters are REQUIRED.
5.
The "T" and "Z" characters in this syntax MUST be upper case.
6.
The producer SHOULD include as much precision as its clock accuracy and performance permit.

7.
Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision
Examples:
nanoseconds = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'



Note: The additional precision field is still up for debate as we currently do not have consensus.

What say everybody?

Cheers

Terry MacDonald
Senior STIX Subject Matter Expert
SOLTRA An FS-ISAC and DTCC Company
+61 (407) 203 206 terry@soltra.com



From:
cti-stix@lists.oasis-open.org [ mailto:cti-stix@lists.oasis-open.org ]
On Behalf Of Barnum, Sean D.
Sent: Wednesday, 25 November 2015 5:34 AM
To: Jordan, Bret < bret.jordan@bluecoat.com >; Wunder, John
A. < jwunder@mitre.org >
Cc: cti-stix@lists.oasis-open.org
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000

I am fine with #1-4
I disagree with #5.
Again, I will assert there IS a need for an explicit precision field. I have tried to explain the context a couple of time but apparently still have not done a good enough job.
Unfortunately, I do not have the cycles today to attempt to explain the scenarios for precision on this topic in any further detail.
Hopefully I will have a chance to do so tomorrow.

I just wanted to make it clear for now that we do not have consensus on #5.

Sean

From: " cti-stix@lists.oasis-open.org "
< cti-stix@lists.oasis-open.org > on behalf of "Jordan, Bret" < bret.jordan@bluecoat.com >
Date: Tuesday, November 24, 2015 at 1:24 PM
To: John Wunder < jwunder@mitre.org >
Cc: " cti-stix@lists.oasis-open.org "
< cti-stix@lists.oasis-open.org >
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000

So I think we are finally getting somewhere..... Before we claim that we agree, let me paraphrase and summarize just to make sure:


1) A timestamp format of yyyy-mm-dd-Thh:mm:ssZ MUST be used
Examples:
2015-11-23T13:35:12Z (for 1:35:12 in UTC format)
2) All timestamps MUST be in UTC a UI will change them as needed for an analyst

3) A timezone offset will NOT be used, all times will be recorded in UTC
4) Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision
Examples:
nanoseconds = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'
etc.
5) There will be no extra precision field


Open questions
A) If the time of day is not known should it be:
i) zeroed out
Examples:
'2015-11-24T11:00:00Z' (known only to the 11th hour)
'2015-11-24T00:00:00Z (known only to the day)
ii) not included, just like the sub-second (if we do this, is this a violation of RFC3339)
Examples:
'2015-11-24T11Z' (known only to the 11th hour)
'2015-11-24TZ (known only to the day)
Thanks,

Bret



Bret Jordan CISSP
Director of Security Architecture and Standards Office of the CTO
Blue Coat Systems
PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."




On Nov 24, 2015, at 06:24, Wunder, John A. < jwunder@mitre.org > wrote:

I have a slight preference for Trey’s approach but would also be fine with Terry’s.

I prefer Trey’s because IMO support for a standard as-is is better than redefining the standard. RFC3339 allows for accuracy to any arbitrary (second) value and therefore the vast majority of date/time libraries will be able to support that. His approach allows
us to use the standard as-is. In fact, the requirement for millisecond precision may make things more difficult for those who just want to use RFC3339 processing libraries because they have to get those libraries to produce *our* version of RFC 3339 rather
than anything compatible with the standard.

I don’t think it will allow accuracy to the day unless we move back to ISO 8601 with all the other openness that it brings. But, I tend to agree with Terry that it will matter very little in almost every circumstance so we shouldn’t worry about it.

John




On Nov 24, 2015, at 5:28 AM, Trey Darley < trey@soltra.com > wrote:

On 24.11.2015 02:36:50, Terry MacDonald wrote:




I think that we have to put this whole discussion in perspective.
Most organizations have difficulty in discovering they have a
breach within days and weeks, not within a second. So going with
B) iii) and having the precision within 1 second realistically is
good enough in my opinion. It is far more likely that all the
clocks on the network are not synchronized, and all the tools are
reporting different and unrelated times and that they are way
more than a second out of alignment with each other. The zeroed
out millisecond timestamp doesn’t impact us much when we have
real-world problems such as that.


I agree with Terry however...

On 23.11.2015 20:51:09, Wunder, John A. wrote:




This is not to say that we need a precision field, just that if we
do it should be explicit rather than implicit.


...I also agree with John. On the one hand, I can't count the number
of times I've seen investigations get thrown under the bus due to
clock-sync issues. On the other hand, recent history has shown that
historical assumptions made about datetime can come back to bite us
with a vengeance.

So I think we should just use the standard explicitly. The producer
can use the level of precision they support and intend to communicate,
the consumer can easily recognize the level of precision coming from
the producer, and handle it appropriately. Doing this in the manner
illustrated below imposes less burden on the consumer than handling an
optional 'precision' field while accomplishing the same goal.

<snip>
#!/usr/bin/env python

import re

nanoseconds = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'
seconds = '2015-11-24T09:42:54Z'
minutes = '2015-11-24T09:42Z'
# You get the idea...

timestamps = [nanoseconds, microseconds, milliseconds, seconds, minutes]

nano_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{9}Z')
micro_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{6}Z')
milli_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{3}Z')
sec_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}Z')
min_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}Z')

for ts in timestamps:
if nano_re.match(ts):
print('Found nanosecond time...')
elif micro_re.match(ts):
print('Found microsecond time...')
elif milli_re.match(ts):
print('Found millisecond time...')
elif sec_re.match(ts):
print('Found second time...')
elif min_re.match(ts):
print('Found minute time...')
</snip

There! Entirely explicit, uses RFC 3339 as intended, and there's one
clear way of doing things without resorting to optional fields.

--
Cheers,
Trey
--
Trey Darley
Senior Security Engineer
4DAA 0A88 34BC 27C9 FD2B A97E D3C6 5C74 0FB7 E430
Soltra An FS-ISAC & DTCC Company
www.soltra.com
--
"It is always something." --RFC 1925

Question....... How far does the precision requirement really need to go?  I know everyone always wants infinite flexibility....  But.....  Perhaps at the end of the day, the only things that really matters is to know if the precision is to a given day or greater than a day. Let me explain.... Put on my analyst hat..... I am guessing that something happened on my network last week some time.  I think it happened on 2015-11-18..  With that amount of vagueness, that should be sufficient.  And tools and other analyst can only work with that...   If I know that it for sure happened on 2015-11-18 and I am only guessing at the hour or minute, does it really matter if some of the time elements are zeroed???  I am still going to need to look forward and back beyond it.  Yes, I am VERY familiar with full packet capture solution, since I helped build one. So perhaps a compromise is: 1) we allow just a date *2015-11-23 without the T delimiter.  This will make it SUPER easy for code to check for.  This will allow people to give a date.  2) if you know more than just the date, maybe the hour, or maybe the hour and the minute, then things you do not know are zeroed out...   Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg.   On Nov 24, 2015, at 14:08, Patrick Maroney < Pmaroney@Specere.org > wrote: What if a programmatic event occurred at precisely 2015-11-24 00:00:00.000000 how would my logic discern the difference?  My only issue here is with assuming zero values imply range of uncertainty.  I am only arguing that there should be a mechanism (optional?) to state uncertainty ***explicitly***.  I'm 100% good with everything else here. There are a number of Work Flow scenarios were this is relevant (User reports:   Oh yeah I saw an alert from the antivirus thingie sometime last week that warned me ReallyBadStuff will happen if I click OK to this, but I see these all the time and just clicked OK like I always do .  There are also legal implications for organizations in some communities for event reporting and time frames required under law to do so.  In many cases you have to report before you can reasonably establish when something happened.  We need to be able to express rough findings rapidly to meet compliance regulation or contractual terms and then refine these dates and time as due diligence investigation occurs. re: I too want to know this...  As the more I have thought about it, the more I am coming to the idea that the precision field is just theoretical.  It does not have an effect in the actual work flow. It does have an effect on actual workflow for high Capability Maturity organizations that maintain strict device clock synchronization.  Knowing which block of time to pull PCAPs from a Tape Library of PCAPs from a 10GBs channel for re=processing example can make the difference of hours in processing time (in the case of Netwitness it takes 2 Hours/Terabyte to reprocess raw data packets through a Decoder back into Metadata and File Artifacts).  Every hour of delay in recovering Actionable IOCs and Malware Artifacts can result in 100's of additional APT compromised systems that need to be mitigated.  So yes, most organizations do not maintain strict infrastructure Time Synchronization, but we should not presume that there is no value/requirement to support those who have learned these hard lessons and have addressed same in their policies and procedures. Folks - We are real close on this one.  Is there a way to add Uncertainty as an optional parameter? Patrick Maroney President Integrated Networking Technologies, Inc. Office:  (856)983-0001 Cell:      (609)841-5104 From: Bret Jordan < bret.jordan@bluecoat.com > Date: Tuesday, November 24, 2015 at 3:38 PM To: Jason Keirstead < Jason.Keirstead@ca.ibm.com > Cc: Patrick Maroney < Pmaroney@Specere.org >, Terry MacDonald < terry@soltra.com >, Sean Barnum < sbarnum@mitre.org >, John Wunder < jwunder@mitre.org >, cti-stix@lists.oasis-open.org < cti-stix@lists.oasis-open.org > Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 I too want to know this...  As the more I have thought about it, the more I am coming to the idea that the precision field is just theoretical.  It does not have an effect in the actual work flow.  If it does, please give some concrete real world work flow examples of how this is going to help. Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg.   On Nov 24, 2015, at 13:25, Jason Keirstead < Jason.Keirstead@ca.ibm.com > wrote: HI Patrick - please see my earlier email around this topic, specifically the part around end-to-end workflow scenarios... Well, were reporting this incident now as required by Law, but we are only certain at this point that the activity occurred as far back as yesterday. So lets assume for a moment that in the above use case the timestamp said 2015-11-24 00:00:00.000000 and did not indicate any type of precision indicating +/- 24 hours ... why do I as the consumer of that information care.. how does that affect my consumption of that threat report? How can I make more effective use of that information in my system if I have that precision indicator? I am not going to do low-level temporal analysis on this type of indicator, so it doesn't matter from that perspective. What is the workflow where it does matter? - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown <graycol.gif> Patrick Maroney ---11/24/2015 04:17:59 PM---[+1] on use of syslog RFC and for support for Precision (in the literal sense). [+~] on being able t From: Patrick Maroney < Pmaroney@Specere.org > To: Terry MacDonald < terry@soltra.com >, Barnum, Sean D. < sbarnum@mitre.org >, Jordan, Bret < bret.jordan@bluecoat.com >, Wunder, John A. < jwunder@mitre.org > Cc: cti-stix@lists.oasis-open.org < cti-stix@lists.oasis-open.org > Date: 11/24/2015 04:17 PM Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 Sent by: < cti-stix@lists.oasis-open.org > [+1] on use of syslog RFC and for support for Precision (in the literal sense). [+~] on being able to drive to consensus on this highly critical aspect our thing (even though we broke our priority list sequence) But to Sean's point we do as a practical matter deal with Uncertainty (AKA Precision ) in Timestamps. There are a dozen use cases I can cite, but think I can simply use the Well, were reporting this incident now as required by Law, but we are only certain at this point that the activity occurred as far back as yesterday. So we can clearly agree that there should be one and only one way of specifying temporal values (whether fixed, relative, or intervals). But his one way of doing things means we still need a way to convey uncertainty where we need to communicate same. Dose that makes sense? Patrick Maroney President Integrated Networking Technologies, Inc. Office: (856)983-0001 Cell: (609)841-5104 From: < cti-stix@lists.oasis-open.org > on behalf of Terry MacDonald < terry@soltra.com > Date: Tuesday, November 24, 2015 at 2:37 PM To: Sean Barnum < sbarnum@mitre.org >, Bret Jordan < bret.jordan@bluecoat.com >, John Wunder < jwunder@mitre.org > Cc: cti-stix@lists.oasis-open.org < cti-stix@lists.oasis-open.org > Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000 I am fine with Trey’s proposal too. Specifically I support all 5 items on Bret’s email. I agree we do not have consensus on #5. A few extra comments though, brought up by a review of the extra restrictions the Syslog RFC (6.2.3 TIMESTAMP) places on implementations. Is this an acceptable full list for the timestamp? (shamelessly borrowed bits from the syslog rfc). The TIMESTAMP field is a formalized timestamp derived from [RFC3339]. Whereas [RFC3339] makes allowances for multiple syntaxes, this document imposes further restrictions. The TIMESTAMP value MUST follow these additional restrictions: 1. An RFC3339 timestamp format MUST be used. 2. All timestamps MUST be in UTC 3. A timezone offset MUST NOT be used. All times MUST be recorded in UTC time. 4. Usage of the T and Z characters are REQUIRED. 5. The T and Z characters in this syntax MUST be upper case. 6. The producer SHOULD include as much precision as its clock accuracy and performance permit. 7. Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision Examples: nanoseconds = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' Note: The additional precision field is still up for debate as we currently do not have consensus. What say everybody? Cheers Terry MacDonald Senior STIX Subject Matter Expert SOLTRA An FS-ISAC and DTCC Company +61 (407) 203 206 terry@soltra.com From: cti-stix@lists.oasis-open.org [ mailto:cti-stix@lists.oasis-open.org ] On Behalf Of Barnum, Sean D. Sent: Wednesday, 25 November 2015 5:34 AM To: Jordan, Bret < bret.jordan@bluecoat.com >; Wunder, John A. < jwunder@mitre.org > Cc: cti-stix@lists.oasis-open.org Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 I am fine with #1-4 I disagree with #5. Again, I will assert there IS a need for an explicit precision field. I have tried to explain the context a couple of time but apparently still have not done a good enough job. Unfortunately, I do not have the cycles today to attempt to explain the scenarios for precision on this topic in any further detail. Hopefully I will have a chance to do so tomorrow. I just wanted to make it clear for now that we do not have consensus on #5. Sean From: cti-stix@lists.oasis-open.org < cti-stix@lists.oasis-open.org > on behalf of Jordan, Bret < bret.jordan@bluecoat.com > Date: Tuesday, November 24, 2015 at 1:24 PM To: John Wunder < jwunder@mitre.org > Cc: cti-stix@lists.oasis-open.org < cti-stix@lists.oasis-open.org > Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 So I think we are finally getting somewhere..... Before we claim that we agree, let me paraphrase and summarize just to make sure: 1) A timestamp format of yyyy-mm-dd-Thh:mm:ssZ MUST be used Examples: 2015-11-23T13:35:12Z (for 1:35:12 in UTC format) 2) All timestamps MUST be in UTC a UI will change them as needed for an analyst 3) A timezone offset will NOT be used, all times will be recorded in UTC 4) Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision Examples: nanoseconds = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' etc. 5) There will be no extra precision field Open questions A) If the time of day is not known should it be: i) zeroed out Examples: '2015-11-24T11:00:00Z' (known only to the 11th hour) '2015-11-24T00:00:00Z (known only to the day) ii) not included, just like the sub-second (if we do this, is this a violation of RFC3339) Examples: '2015-11-24T11Z' (known only to the 11th hour) '2015-11-24TZ (known only to the day) Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050 Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg. On Nov 24, 2015, at 06:24, Wunder, John A. < jwunder@mitre.org > wrote: I have a slight preference for Trey’s approach but would also be fine with Terry’s. I prefer Trey’s because IMO support for a standard as-is is better than redefining the standard. RFC3339 allows for accuracy to any arbitrary (second) value and therefore the vast majority of date/time libraries will be able to support that. His approach allows us to use the standard as-is. In fact, the requirement for millisecond precision may make things more difficult for those who just want to use RFC3339 processing libraries because they have to get those libraries to produce *our* version of RFC 3339 rather than anything compatible with the standard. I don’t think it will allow accuracy to the day unless we move back to ISO 8601 with all the other openness that it brings. But, I tend to agree with Terry that it will matter very little in almost every circumstance so we shouldn’t worry about it. John On Nov 24, 2015, at 5:28 AM, Trey Darley < trey@soltra.com > wrote: On 24.11.2015 02:36:50, Terry MacDonald wrote: I think that we have to put this whole discussion in perspective. Most organizations have difficulty in discovering they have a breach within days and weeks, not within a second. So going with B) iii) and having the precision within 1 second realistically is good enough in my opinion. It is far more likely that all the clocks on the network are not synchronized, and all the tools are reporting different and unrelated times and that they are way more than a second out of alignment with each other. The zeroed out millisecond timestamp doesn’t impact us much when we have real-world problems such as that. I agree with Terry however... On 23.11.2015 20:51:09, Wunder, John A. wrote: This is not to say that we need a precision field, just that if we do it should be explicit rather than implicit. ...I also agree with John. On the one hand, I can't count the number of times I've seen investigations get thrown under the bus due to clock-sync issues. On the other hand, recent history has shown that historical assumptions made about datetime can come back to bite us with a vengeance. So I think we should just use the standard explicitly. The producer can use the level of precision they support and intend to communicate, the consumer can easily recognize the level of precision coming from the producer, and handle it appropriately. Doing this in the manner illustrated below imposes less burden on the consumer than handling an optional 'precision' field while accomplishing the same goal. <snip> #!/usr/bin/env python import re nanoseconds = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' seconds = '2015-11-24T09:42:54Z' minutes = '2015-11-24T09:42Z' # You get the idea... timestamps = [nanoseconds, microseconds, milliseconds, seconds, minutes] nano_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{9}Z') micro_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{6}Z') milli_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{3}Z') sec_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}Z') min_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}Z') for ts in timestamps: if nano_re.match(ts): print('Found nanosecond time...') elif micro_re.match(ts): print('Found microsecond time...') elif milli_re.match(ts): print('Found millisecond time...') elif sec_re.match(ts): print('Found second time...') elif min_re.match(ts): print('Found minute time...') </snip There! Entirely explicit, uses RFC 3339 as intended, and there's one clear way of doing things without resorting to optional fields. -- Cheers, Trey -- Trey Darley Senior Security Engineer 4DAA 0A88 34BC 27C9 FD2B A97E D3C6 5C74 0FB7 E430 Soltra An FS-ISAC & DTCC Company www.soltra.com -- It is always something. --RFC 1925 Attachment: signature.asc Description: Message signed with OpenPGP using GPGMail

“ What if a programmatic event occurred at precisely
"2015-11-24 00:00:00.000000" how would my logic discern the difference?  My only issue here is with assuming zero values imply range of uncertainty.  I am only arguing that there should be a mechanism (optional?) to state uncertainty ***explicitly***.  I'm
100% good with everything else here.”
 
Uncertainty should be described via other mechanisms in my opinion. Rather than precision I would rather see ‘confidence’ or suchlike
applied to describe the fact that what is described in the object is not high confidence. There has also been previous discussion about the admiralty code, intelligence source and information reliability which could be applicable here:

https://en.wikipedia.org/wiki/Intelligence_source_and_information_reliability

 
There are a number of Work Flow scenarios were this is relevant (User reports:  "Oh yeah I saw an alert from the antivirus thingie sometime last week that warned
me "ReallyBadStuff" will happen if I click OK to this, but I see these all the time and just clicked OK like I always do".  There are also legal implications for organizations in some communities for event reporting and time frames required under law to do
so.  In many cases you have to report before you can reasonably establish "when" something happened.  We need to be able to express "rough" findings rapidly to meet compliance regulation or contractual terms and then refine these dates and time as due diligence
investigation occurs.
 
If one is unsure of the exact time at the beginning of an event, maybe that information should be conveyed in the ‘investigation’ object
that was proposed? That object was proposed to be used for the uncertain part of initial investigations, before they turn into actual incidents. Maybe it’s better to describe the uncertainty via that one object rather than spread it through the complete model?
In most cases I would posit that a TI system would know the exact time that they did something, or that a close enough approximation would be available in logs. It seems to me that its only during the discovery/investigation/hypothesis phases you would need
to specify approximations in times, and that could be potentially handled by confidence (or something similar).
 
Cheers
 

Terry MacDonald
Senior STIX Subject Matter Expert
SOLTRA   An FS-ISAC and DTCC Company
+61 (407) 203 206
terry@soltra.com
 

 


From: Patrick Maroney [mailto:Pmaroney@Specere.org]

Sent: Wednesday, 25 November 2015 8:08 AM
To: Jordan, Bret <bret.jordan@bluecoat.com>; Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Cc: Terry MacDonald <terry@soltra.com>; Sean D. Barnum <sbarnum@mitre.org>; Wunder, John A. <jwunder@mitre.org>; cti-stix@lists.oasis-open.org
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000


 


What if a programmatic event occurred at precisely "2015-11-24 00:00:00.000000" how would my logic discern the difference?  My only issue here is with assuming zero
values imply range of uncertainty.  I am only arguing that there should be a mechanism (optional?) to state uncertainty ***explicitly***.  I'm 100% good with everything else here.


 


There are a number of Work Flow scenarios were this is relevant (User reports:  "Oh yeah I saw an alert from the antivirus thingie sometime last week that warned
me "ReallyBadStuff" will happen if I click OK to this, but I see these all the time and just clicked OK like I always do".  There are also legal implications for organizations in some communities for event reporting and time frames required under law to do
so.  In many cases you have to report before you can reasonably establish "when" something happened.  We need to be able to express "rough" findings rapidly to meet compliance regulation or contractual terms and then refine these dates and time as due diligence
investigation occurs.


 


 


re: "I too want to know this...  As the more I have thought about it, the more I am coming to the idea that the precision field is just theoretical.  It does not
have an effect in the actual work flow. "


 


It does have an effect on actual workflow for high Capability Maturity organizations that maintain strict device clock synchronization.  Knowing which block of time
to pull PCAPs from a Tape Library of PCAPs from a 10GBs channel for re=processing example can make the difference of hours in processing time (in the case of Netwitness it takes 2 Hours/Terabyte to reprocess raw data packets through a Decoder back into Metadata
and File Artifacts).  Every hour of delay in recovering Actionable IOCs and Malware Artifacts can result in 100's of additional APT compromised systems that need to be mitigated.  So yes, most organizations do not maintain strict infrastructure Time Synchronization,
but we should not presume that there is no value/requirement to support those who have learned these hard lessons and have addressed same in their policies and procedures.


 


Folks - We are real close on this one.  Is there a way to add Uncertainty as an optional parameter?


 






Patrick Maroney


President


Integrated Networking Technologies, Inc.


Office:  (856)983-0001


Cell:      (609)841-5104







 


From:
Bret Jordan < bret.jordan@bluecoat.com >
Date: Tuesday, November 24, 2015 at 3:38 PM
To: Jason Keirstead < Jason.Keirstead@ca.ibm.com >
Cc: Patrick Maroney < Pmaroney@Specere.org >, Terry MacDonald < terry@soltra.com >, Sean Barnum < sbarnum@mitre.org >, John Wunder < jwunder@mitre.org >,
" cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000


 



I too want to know this...  As the more I have thought about it, the more I am coming to the idea that the precision field is just theoretical.  It does not have
an effect in the actual work flow.  If it does, please give some concrete real world work flow examples of how this is going to help.







 


Thanks,


 


Bret



 


 


 



Bret Jordan CISSP


Director of Security Architecture and Standards Office of the CTO


Blue Coat Systems



PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050


"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." 









 



On Nov 24, 2015, at 13:25, Jason Keirstead < Jason.Keirstead@ca.ibm.com > wrote:

 


HI Patrick - please see my earlier email around this topic, specifically the part around end-to-end workflow
scenarios...

" "Well, were reporting this incident now as required by Law, but we are only certain at this point that the activity occurred as far back as yesterday.
"

So lets assume for a moment that in the above use case the timestamp said "2015-11-24 00:00:00.000000" and
did not indicate any type of "precision" indicating "+/- 24 hours"... why do I as the consumer of that information care.. how does that affect my consumption of that threat report? How can I make more effective use of that information in my system if
I have that precision indicator? I am not going to do low-level temporal analysis on this type of indicator, so it doesn't matter from that perspective. What is the workflow where it does matter?

-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security
www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


<graycol.gif> Patrick Maroney ---11/24/2015 04:17:59 PM---[+1] on use of syslog RFC and for support for Precision (in the literal sense). [+~] on being able t

From:
Patrick Maroney < Pmaroney@Specere.org >
To:
Terry MacDonald < terry@soltra.com >, "Barnum, Sean D." < sbarnum@mitre.org >, "Jordan, Bret" < bret.jordan@bluecoat.com >,
"Wunder, John A." < jwunder@mitre.org >
Cc:
" cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
Date:
11/24/2015 04:17 PM
Subject:
Re: [cti-stix] STIX timestamps and ISO 8601:2000
Sent by:
< cti-stix@lists.oasis-open.org >






[+1] on use of syslog RFC and for support for Precision (in the literal sense).
[+~] on being able to drive to consensus on this highly critical aspect "our thing" (even though we broke our priority list sequence)

But to Sean's point we do as a practical matter deal with "Uncertainty" (AKA "Precision") in Timestamps. There are a dozen use cases I can cite, but think I can simply use the
"Well, were reporting this incident now as required by Law, but we are only certain at this point that the activity occurred as far back as yesterday.


So we can clearly agree that there should be one and only one way of specifying temporal "values" (whether fixed, relative, or intervals). But his "one way of doing things" means
we still need a way to convey "uncertainty" where we need to communicate same.

Dose that makes sense?

Patrick Maroney
President
Integrated Networking Technologies, Inc.
Office: (856)983-0001
Cell: (609)841-5104

From:
< cti-stix@lists.oasis-open.org >
on behalf of Terry MacDonald < terry@soltra.com >
Date: Tuesday, November 24, 2015 at 2:37 PM
To: Sean Barnum < sbarnum@mitre.org >,
Bret Jordan < bret.jordan@bluecoat.com >,
John Wunder < jwunder@mitre.org >
Cc: " cti-stix@lists.oasis-open.org "
< cti-stix@lists.oasis-open.org >
Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000

I am fine with Trey’s proposal too. Specifically I support all 5 items on Bret’s email. I agree we do not have consensus on #5.

A few extra comments though, brought up by a review of the extra restrictions the
Syslog RFC (6.2.3 TIMESTAMP) places on implementations.


Is this an acceptable full list for the timestamp? (shamelessly borrowed bits from the syslog rfc).

The TIMESTAMP field is a formalized timestamp derived from [RFC3339].

Whereas [RFC3339] makes allowances for multiple syntaxes, this
document imposes further restrictions. The TIMESTAMP value MUST
follow these additional restrictions:
1.
An RFC3339 timestamp format MUST be used.

2.
All timestamps MUST be in UTC

3.
A timezone offset MUST NOT be used. All times MUST be recorded in UTC time.
4.
Usage of the "T" and "Z" characters are REQUIRED.
5.
The "T" and "Z" characters in this syntax MUST be upper case.
6.
The producer SHOULD include as much precision as its clock accuracy and performance permit.

7.
Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision
Examples:
nanoseconds = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'

Note: The additional precision field is still up for debate as we currently do not have consensus.

What say everybody?

Cheers

Terry MacDonald
Senior STIX Subject Matter Expert
SOLTRA An FS-ISAC and DTCC Company
+61 (407) 203 206 terry@soltra.com


From: cti-stix@lists.oasis-open.org [ mailto:cti-stix@lists.oasis-open.org ]
On Behalf Of Barnum, Sean D.
Sent: Wednesday, 25 November 2015 5:34 AM
To: Jordan, Bret < bret.jordan@bluecoat.com >; Wunder, John A. < jwunder@mitre.org >
Cc: cti-stix@lists.oasis-open.org
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000

I am fine with #1-4
I disagree with #5.
Again, I will assert there IS a need for an explicit precision field. I have tried to explain the context a couple of time but apparently still have not done a good enough job.
Unfortunately, I do not have the cycles today to attempt to explain the scenarios for precision on this topic in any further detail.
Hopefully I will have a chance to do so tomorrow.

I just wanted to make it clear for now that we do not have consensus on #5.

Sean

From:
" cti-stix@lists.oasis-open.org "
< cti-stix@lists.oasis-open.org >
on behalf of "Jordan, Bret" < bret.jordan@bluecoat.com >
Date: Tuesday, November 24, 2015 at 1:24 PM
To: John Wunder < jwunder@mitre.org >
Cc: " cti-stix@lists.oasis-open.org "
< cti-stix@lists.oasis-open.org >
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000

So I think we are finally getting somewhere..... Before we claim that we agree, let me paraphrase and summarize just to make sure:


1) A timestamp format of yyyy-mm-dd-Thh:mm:ssZ MUST be used
Examples:
2015-11-23T13:35:12Z (for 1:35:12 in UTC format)
2) All timestamps MUST be in UTC a UI will change them as needed for an analyst
3) A timezone offset will NOT be used, all times will be recorded in UTC
4) Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision
Examples:
nanoseconds = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'
etc.
5) There will be no extra precision field


Open questions
A) If the time of day is not known should it be:
i) zeroed out
Examples:
'2015-11-24T11:00:00Z' (known only to the 11th hour)
'2015-11-24T00:00:00Z (known only to the day)
ii) not included, just like the sub-second (if we do this, is this a violation of RFC3339)
Examples:
'2015-11-24T11Z' (known only to the 11th hour)
'2015-11-24TZ (known only to the day)
Thanks,

Bret



Bret Jordan CISSP
Director of Security Architecture and Standards Office of the CTO
Blue Coat Systems
PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."


On Nov 24, 2015, at 06:24, Wunder, John A. < jwunder@mitre.org > wrote:

I have a slight preference for Trey’s approach but would also be fine with Terry’s.

I prefer Trey’s because IMO support for a standard as-is is better than redefining the standard. RFC3339 allows for accuracy to any arbitrary (second) value and therefore the vast majority of date/time libraries will be able to support that. His approach allows
us to use the standard as-is. In fact, the requirement for millisecond precision may make things more difficult for those who just want to use RFC3339 processing libraries because they have to get those libraries to produce *our* version of RFC 3339 rather
than anything compatible with the standard.

I don’t think it will allow accuracy to the day unless we move back to ISO 8601 with all the other openness that it brings. But, I tend to agree with Terry that it will matter very little in almost every circumstance so we shouldn’t worry about it.

John
On Nov 24, 2015, at 5:28 AM, Trey Darley < trey@soltra.com > wrote:

On 24.11.2015 02:36:50, Terry MacDonald wrote:

I think that we have to put this whole discussion in perspective.
Most organizations have difficulty in discovering they have a
breach within days and weeks, not within a second. So going with
B) iii) and having the precision within 1 second realistically is
good enough in my opinion. It is far more likely that all the
clocks on the network are not synchronized, and all the tools are
reporting different and unrelated times and that they are way
more than a second out of alignment with each other. The zeroed
out millisecond timestamp doesn’t impact us much when we have
real-world problems such as that.

I agree with Terry however...

On 23.11.2015 20:51:09, Wunder, John A. wrote:

This is not to say that we need a precision field, just that if we
do it should be explicit rather than implicit.

...I also agree with John. On the one hand, I can't count the number
of times I've seen investigations get thrown under the bus due to
clock-sync issues. On the other hand, recent history has shown that
historical assumptions made about datetime can come back to bite us
with a vengeance.

So I think we should just use the standard explicitly. The producer
can use the level of precision they support and intend to communicate,
the consumer can easily recognize the level of precision coming from
the producer, and handle it appropriately. Doing this in the manner
illustrated below imposes less burden on the consumer than handling an
optional 'precision' field while accomplishing the same goal.

<snip>
#!/usr/bin/env python

import re

nanoseconds = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'
seconds = '2015-11-24T09:42:54Z'
minutes = '2015-11-24T09:42Z'
# You get the idea...

timestamps = [nanoseconds, microseconds, milliseconds, seconds, minutes]

nano_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{9}Z')
micro_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{6}Z')
milli_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{3}Z')
sec_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}Z')
min_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}Z')

for ts in timestamps:
if nano_re.match(ts):
print('Found nanosecond time...')
elif micro_re.match(ts):
print('Found microsecond time...')
elif milli_re.match(ts):
print('Found millisecond time...')
elif sec_re.match(ts):
print('Found second time...')
elif min_re.match(ts):
print('Found minute time...')
</snip

There! Entirely explicit, uses RFC 3339 as intended, and there's one
clear way of doing things without resorting to optional fields.

--
Cheers,
Trey
--
Trey Darley
Senior Security Engineer
4DAA 0A88 34BC 27C9 FD2B A97E D3C6 5C74 0FB7 E430
Soltra An FS-ISAC & DTCC Company
www.soltra.com
--
"It is always something." --RFC 1925







 

IMO we are rushing to the conclusion that we should have a single timestamp format and fill in any unknown levels of precision with 0s.
 
We have this EXACT problem with our threat analysis platform today, where timestamps with less precision (say, we know this happened on a given day) are converted
to an exact time with millisecond precision.  Our intelligence analysts can easily assume an event happened at the exact time listed (which is what the data point is saying) rather than defaulting to thinking that the data point may be inaccurate and really
may have happened some other second, minute or hour depending on the precision of the data source.
 
Please be careful about the potential confusion that might be introduced to a given threat analyst, as we have witnessed it firsthand.
 


Thanks,
 
Alex


 


From: cti-stix@lists.oasis-open.org [mailto:cti-stix@lists.oasis-open.org]
On Behalf Of Terry MacDonald
Sent: Tuesday, November 24, 2015 4:43 PM
To: Patrick Maroney; Jordan, Bret; Jason Keirstead
Cc: Sean D. Barnum; Wunder, John A.; cti-stix@lists.oasis-open.org
Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000


 
“ What if a programmatic event occurred at precisely
"2015-11-24 00:00:00.000000" how would my logic discern the difference?  My only issue here is with assuming zero values imply range of uncertainty.  I am only arguing that there should be a mechanism (optional?) to state uncertainty ***explicitly***.  I'm
100% good with everything else here.”
 
Uncertainty should be described via other mechanisms in my opinion. Rather than precision I would rather see ‘confidence’ or suchlike applied to
describe the fact that what is described in the object is not high confidence. There has also been previous discussion about the admiralty code, intelligence source and information reliability which could be applicable here:

https://en.wikipedia.org/wiki/Intelligence_source_and_information_reliability

 
There are a number of Work Flow scenarios were this is relevant (User reports:  "Oh yeah I saw an alert from the antivirus thingie sometime last week
that warned me "ReallyBadStuff" will happen if I click OK to this, but I see these all the time and just clicked OK like I always do".  There are also legal implications for organizations in some communities for event reporting and time frames required under
law to do so.  In many cases you have to report before you can reasonably establish "when" something happened.  We need to be able to express "rough" findings rapidly to meet compliance regulation or contractual terms and then refine these dates and time as
due diligence investigation occurs.
 
If one is unsure of the exact time at the beginning of an event, maybe that information should be conveyed in the ‘investigation’ object that was
proposed? That object was proposed to be used for the uncertain part of initial investigations, before they turn into actual incidents. Maybe it’s better to describe the uncertainty via that one object rather than spread it through the complete model? In most
cases I would posit that a TI system would know the exact time that they did something, or that a close enough approximation would be available in logs. It seems to me that its only during the discovery/investigation/hypothesis phases you would need to specify
approximations in times, and that could be potentially handled by confidence (or something similar).
 
Cheers
 

Terry MacDonald
Senior STIX Subject Matter Expert
SOLTRA   An FS-ISAC and DTCC Company
+61 (407) 203 206
terry@soltra.com
 

 


From: Patrick Maroney [ mailto:Pmaroney@Specere.org ]

Sent: Wednesday, 25 November 2015 8:08 AM
To: Jordan, Bret < bret.jordan@bluecoat.com >; Jason Keirstead < Jason.Keirstead@ca.ibm.com >
Cc: Terry MacDonald < terry@soltra.com >; Sean D. Barnum < sbarnum@mitre.org >; Wunder, John A. < jwunder@mitre.org >;
cti-stix@lists.oasis-open.org
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000


 


What if a programmatic event occurred at precisely "2015-11-24 00:00:00.000000" how would my logic discern the difference?  My only issue here is
with assuming zero values imply range of uncertainty.  I am only arguing that there should be a mechanism (optional?) to state uncertainty ***explicitly***.  I'm 100% good with everything else here.


 


There are a number of Work Flow scenarios were this is relevant (User reports:  "Oh yeah I saw an alert from the antivirus thingie sometime last week
that warned me "ReallyBadStuff" will happen if I click OK to this, but I see these all the time and just clicked OK like I always do".  There are also legal implications for organizations in some communities for event reporting and time frames required under
law to do so.  In many cases you have to report before you can reasonably establish "when" something happened.  We need to be able to express "rough" findings rapidly to meet compliance regulation or contractual terms and then refine these dates and time as
due diligence investigation occurs.


 


 


re: "I too want to know this...  As the more I have thought about it, the more I am coming to the idea that the precision field is just theoretical.
 It does not have an effect in the actual work flow. "


 


It does have an effect on actual workflow for high Capability Maturity organizations that maintain strict device clock synchronization.  Knowing which
block of time to pull PCAPs from a Tape Library of PCAPs from a 10GBs channel for re=processing example can make the difference of hours in processing time (in the case of Netwitness it takes 2 Hours/Terabyte to reprocess raw data packets through a Decoder
back into Metadata and File Artifacts).  Every hour of delay in recovering Actionable IOCs and Malware Artifacts can result in 100's of additional APT compromised systems that need to be mitigated.  So yes, most organizations do not maintain strict infrastructure
Time Synchronization, but we should not presume that there is no value/requirement to support those who have learned these hard lessons and have addressed same in their policies and procedures.


 


Folks - We are real close on this one.  Is there a way to add Uncertainty as an optional parameter?


 






Patrick Maroney


President


Integrated Networking Technologies, Inc.


Office:  (856)983-0001


Cell:      (609)841-5104







 


From:
Bret Jordan < bret.jordan@bluecoat.com >
Date: Tuesday, November 24, 2015 at 3:38 PM
To: Jason Keirstead < Jason.Keirstead@ca.ibm.com >
Cc: Patrick Maroney < Pmaroney@Specere.org >, Terry MacDonald < terry@soltra.com >, Sean Barnum < sbarnum@mitre.org >, John Wunder < jwunder@mitre.org >,
" cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000


 



I too want to know this...  As the more I have thought about it, the more I am coming to the idea that the precision field is just theoretical.  It
does not have an effect in the actual work flow.  If it does, please give some concrete real world work flow examples of how this is going to help.







 


Thanks,


 


Bret



 


 


 



Bret Jordan CISSP


Director of Security Architecture and Standards Office of the CTO


Blue Coat Systems



PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050


"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." 









 



On Nov 24, 2015, at 13:25, Jason Keirstead < Jason.Keirstead@ca.ibm.com > wrote:

 


HI Patrick - please see my earlier email around this topic, specifically the part around
end-to-end workflow scenarios...

" "Well, were reporting this incident now as required by Law, but we are only certain at this point that the activity occurred as far back as yesterday.
"

So lets assume for a moment that in the above use case the timestamp said "2015-11-24 00:00:00.000000" and
did not indicate any type of "precision" indicating "+/- 24 hours"... why do I as the consumer of that information care.. how does that affect my consumption of that threat report? How can I make more effective use of that information in my system if
I have that precision indicator? I am not going to do low-level temporal analysis on this type of indicator, so it doesn't matter from that perspective. What is the workflow where it does matter?

-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security
www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown


<graycol.gif> Patrick Maroney ---11/24/2015 04:17:59 PM---[+1] on use of syslog RFC and for support for Precision (in the literal sense). [+~] on being able t

From:
Patrick Maroney < Pmaroney@Specere.org >
To:
Terry MacDonald < terry@soltra.com >, "Barnum, Sean D." < sbarnum@mitre.org >, "Jordan,
Bret" < bret.jordan@bluecoat.com >, "Wunder, John A." < jwunder@mitre.org >
Cc:
" cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
Date:
11/24/2015 04:17 PM
Subject:
Re: [cti-stix] STIX timestamps and ISO 8601:2000
Sent by:
< cti-stix@lists.oasis-open.org >








[+1] on use of syslog RFC and for support for Precision (in the literal sense).
[+~] on being able to drive to consensus on this highly critical aspect "our thing" (even though we broke our priority list sequence)

But to Sean's point we do as a practical matter deal with "Uncertainty" (AKA "Precision") in Timestamps. There are a dozen use cases I can cite, but think I can
simply use the "Well, were reporting this incident now as required by Law, but we are only certain at this point that the activity occurred as far back as yesterday.


So we can clearly agree that there should be one and only one way of specifying temporal "values" (whether fixed, relative, or intervals). But his "one way of
doing things" means we still need a way to convey "uncertainty" where we need to communicate same.

Dose that makes sense?

Patrick Maroney
President
Integrated Networking Technologies, Inc.
Office: (856)983-0001
Cell: (609)841-5104

From:
< cti-stix@lists.oasis-open.org >
on behalf of Terry MacDonald < terry@soltra.com >
Date: Tuesday, November 24, 2015 at 2:37 PM
To: Sean Barnum < sbarnum@mitre.org >,
Bret Jordan < bret.jordan@bluecoat.com >,
John Wunder < jwunder@mitre.org >
Cc: " cti-stix@lists.oasis-open.org "
< cti-stix@lists.oasis-open.org >
Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000

I am fine with Trey’s proposal too. Specifically I support all 5 items on Bret’s email. I agree we do not have consensus on #5.

A few extra comments though, brought up by a review of the extra restrictions the
Syslog RFC (6.2.3 TIMESTAMP) places on implementations.


Is this an acceptable full list for the timestamp? (shamelessly borrowed bits from the syslog rfc).

The TIMESTAMP field is a formalized timestamp derived from [RFC3339].

Whereas [RFC3339] makes allowances for multiple syntaxes, this
document imposes further restrictions. The TIMESTAMP value MUST
follow these additional restrictions:
1.
An RFC3339 timestamp format MUST be used.

2.
All timestamps MUST be in UTC

3.
A timezone offset MUST NOT be used. All times MUST be recorded in UTC time.
4.
Usage of the "T" and "Z" characters are REQUIRED.
5.
The "T" and "Z" characters in this syntax MUST be upper case.
6.
The producer SHOULD include as much precision as its clock accuracy and performance permit.

7.
Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision
Examples:
nanoseconds = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'

Note: The additional precision field is still up for debate as we currently do not have consensus.

What say everybody?

Cheers

Terry MacDonald
Senior STIX Subject Matter Expert
SOLTRA An FS-ISAC and DTCC Company
+61 (407) 203 206 terry@soltra.com


From: cti-stix@lists.oasis-open.org [ mailto:cti-stix@lists.oasis-open.org ]
On Behalf Of Barnum, Sean D.
Sent: Wednesday, 25 November 2015 5:34 AM
To: Jordan, Bret < bret.jordan@bluecoat.com >; Wunder, John A. < jwunder@mitre.org >
Cc: cti-stix@lists.oasis-open.org
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000

I am fine with #1-4
I disagree with #5.
Again, I will assert there IS a need for an explicit precision field. I have tried to explain the context a couple of time but apparently still have not done a good enough job.
Unfortunately, I do not have the cycles today to attempt to explain the scenarios for precision on this topic in any further detail.
Hopefully I will have a chance to do so tomorrow.

I just wanted to make it clear for now that we do not have consensus on #5.

Sean

From:
" cti-stix@lists.oasis-open.org "
< cti-stix@lists.oasis-open.org >
on behalf of "Jordan, Bret" < bret.jordan@bluecoat.com >
Date: Tuesday, November 24, 2015 at 1:24 PM
To: John Wunder < jwunder@mitre.org >
Cc: " cti-stix@lists.oasis-open.org "
< cti-stix@lists.oasis-open.org >
Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000

So I think we are finally getting somewhere..... Before we claim that we agree, let me paraphrase and summarize just to make sure:


1) A timestamp format of yyyy-mm-dd-Thh:mm:ssZ MUST be used
Examples:
2015-11-23T13:35:12Z (for 1:35:12 in UTC format)
2) All timestamps MUST be in UTC a UI will change them as needed for an analyst
3) A timezone offset will NOT be used, all times will be recorded in UTC
4) Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision
Examples:
nanoseconds = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'
etc.
5) There will be no extra precision field


Open questions
A) If the time of day is not known should it be:
i) zeroed out
Examples:
'2015-11-24T11:00:00Z' (known only to the 11th hour)
'2015-11-24T00:00:00Z (known only to the day)
ii) not included, just like the sub-second (if we do this, is this a violation of RFC3339)
Examples:
'2015-11-24T11Z' (known only to the 11th hour)
'2015-11-24TZ (known only to the day)
Thanks,

Bret



Bret Jordan CISSP
Director of Security Architecture and Standards Office of the CTO
Blue Coat Systems
PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."


On Nov 24, 2015, at 06:24, Wunder, John A. < jwunder@mitre.org > wrote:

I have a slight preference for Trey’s approach but would also be fine with Terry’s.

I prefer Trey’s because IMO support for a standard as-is is better than redefining the standard. RFC3339 allows for accuracy to any arbitrary (second) value and therefore the vast majority of date/time libraries will be able to support that. His approach allows
us to use the standard as-is. In fact, the requirement for millisecond precision may make things more difficult for those who just want to use RFC3339 processing libraries because they have to get those libraries to produce *our* version of RFC 3339 rather
than anything compatible with the standard.

I don’t think it will allow accuracy to the day unless we move back to ISO 8601 with all the other openness that it brings. But, I tend to agree with Terry that it will matter very little in almost every circumstance so we shouldn’t worry about it.

John
On Nov 24, 2015, at 5:28 AM, Trey Darley < trey@soltra.com > wrote:

On 24.11.2015 02:36:50, Terry MacDonald wrote:

I think that we have to put this whole discussion in perspective.
Most organizations have difficulty in discovering they have a
breach within days and weeks, not within a second. So going with
B) iii) and having the precision within 1 second realistically is
good enough in my opinion. It is far more likely that all the
clocks on the network are not synchronized, and all the tools are
reporting different and unrelated times and that they are way
more than a second out of alignment with each other. The zeroed
out millisecond timestamp doesn’t impact us much when we have
real-world problems such as that.

I agree with Terry however...

On 23.11.2015 20:51:09, Wunder, John A. wrote:

This is not to say that we need a precision field, just that if we
do it should be explicit rather than implicit.

...I also agree with John. On the one hand, I can't count the number
of times I've seen investigations get thrown under the bus due to
clock-sync issues. On the other hand, recent history has shown that
historical assumptions made about datetime can come back to bite us
with a vengeance.

So I think we should just use the standard explicitly. The producer
can use the level of precision they support and intend to communicate,
the consumer can easily recognize the level of precision coming from
the producer, and handle it appropriately. Doing this in the manner
illustrated below imposes less burden on the consumer than handling an
optional 'precision' field while accomplishing the same goal.

<snip>
#!/usr/bin/env python

import re

nanoseconds = '2015-11-24T09:42:54.003259294Z'
microseconds = '2015-11-24T09:42:54.003259Z'
milliseconds = '2015-11-24T09:42:54.003Z'
seconds = '2015-11-24T09:42:54Z'
minutes = '2015-11-24T09:42Z'
# You get the idea...

timestamps = [nanoseconds, microseconds, milliseconds, seconds, minutes]

nano_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{9}Z')
micro_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{6}Z')
milli_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{3}Z')
sec_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}Z')
min_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}Z')

for ts in timestamps:
if nano_re.match(ts):
print('Found nanosecond time...')
elif micro_re.match(ts):
print('Found microsecond time...')
elif milli_re.match(ts):
print('Found millisecond time...')
elif sec_re.match(ts):
print('Found second time...')
elif min_re.match(ts):
print('Found minute time...')
</snip

There! Entirely explicit, uses RFC 3339 as intended, and there's one
clear way of doing things without resorting to optional fields.

--
Cheers,
Trey
--
Trey Darley
Senior Security Engineer
4DAA 0A88 34BC 27C9 FD2B A97E D3C6 5C74 0FB7 E430
Soltra An FS-ISAC & DTCC Company
www.soltra.com
--
"It is always something." --RFC 1925
 




 



This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.

Great points and sound words of caution. Bret  Sent from my Commodore 64 On Nov 24, 2015, at 4:17 PM, Foley, Alexander - GIS < alexander.foley@bankofamerica.com > wrote: IMO we are rushing to the conclusion that we should have a single timestamp format and fill in any unknown levels of precision with 0s.   We have this EXACT problem with our threat analysis platform today, where timestamps with less precision (say, we know this happened on a given day) are converted to an exact time with millisecond precision.  Our intelligence analysts can easily assume an event happened at the exact time listed (which is what the data point is saying) rather than defaulting to thinking that the data point may be inaccurate and really may have happened some other second, minute or hour depending on the precision of the data source.   Please be careful about the potential confusion that might be introduced to a given threat analyst, as we have witnessed it firsthand.   Thanks,   Alex   From: cti-stix@lists.oasis-open.org [ mailto:cti-stix@lists.oasis-open.org ] On Behalf Of Terry MacDonald Sent: Tuesday, November 24, 2015 4:43 PM To: Patrick Maroney; Jordan, Bret; Jason Keirstead Cc: Sean D. Barnum; Wunder, John A.; cti-stix@lists.oasis-open.org Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000   “ What if a programmatic event occurred at precisely "2015-11-24 00:00:00.000000" how would my logic discern the difference?  My only issue here is with assuming zero values imply range of uncertainty.  I am only arguing that there should be a mechanism (optional?) to state uncertainty ***explicitly***.  I'm 100% good with everything else here.”   Uncertainty should be described via other mechanisms in my opinion. Rather than precision I would rather see ‘confidence’ or suchlike applied to describe the fact that what is described in the object is not high confidence. There has also been previous discussion about the admiralty code, intelligence source and information reliability which could be applicable here: https://en.wikipedia.org/wiki/Intelligence_source_and_information_reliability   There are a number of Work Flow scenarios were this is relevant (User reports:  "Oh yeah I saw an alert from the antivirus thingie sometime last week that warned me "ReallyBadStuff" will happen if I click OK to this, but I see these all the time and just clicked OK like I always do".  There are also legal implications for organizations in some communities for event reporting and time frames required under law to do so.  In many cases you have to report before you can reasonably establish "when" something happened.  We need to be able to express "rough" findings rapidly to meet compliance regulation or contractual terms and then refine these dates and time as due diligence investigation occurs.   If one is unsure of the exact time at the beginning of an event, maybe that information should be conveyed in the ‘investigation’ object that was proposed? That object was proposed to be used for the uncertain part of initial investigations, before they turn into actual incidents. Maybe it’s better to describe the uncertainty via that one object rather than spread it through the complete model? In most cases I would posit that a TI system would know the exact time that they did something, or that a close enough approximation would be available in logs. It seems to me that its only during the discovery/investigation/hypothesis phases you would need to specify approximations in times, and that could be potentially handled by confidence (or something similar).   Cheers   Terry MacDonald Senior STIX Subject Matter Expert SOLTRA   An FS-ISAC and DTCC Company +61 (407) 203 206 terry@soltra.com     From: Patrick Maroney [ mailto:Pmaroney@Specere.org ] Sent: Wednesday, 25 November 2015 8:08 AM To: Jordan, Bret < bret.jordan@bluecoat.com >; Jason Keirstead < Jason.Keirstead@ca.ibm.com > Cc: Terry MacDonald < terry@soltra.com >; Sean D. Barnum < sbarnum@mitre.org >; Wunder, John A. < jwunder@mitre.org >; cti-stix@lists.oasis-open.org Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000   What if a programmatic event occurred at precisely "2015-11-24 00:00:00.000000" how would my logic discern the difference?  My only issue here is with assuming zero values imply range of uncertainty.  I am only arguing that there should be a mechanism (optional?) to state uncertainty ***explicitly***.  I'm 100% good with everything else here.   There are a number of Work Flow scenarios were this is relevant (User reports:  "Oh yeah I saw an alert from the antivirus thingie sometime last week that warned me "ReallyBadStuff" will happen if I click OK to this, but I see these all the time and just clicked OK like I always do".  There are also legal implications for organizations in some communities for event reporting and time frames required under law to do so.  In many cases you have to report before you can reasonably establish "when" something happened.  We need to be able to express "rough" findings rapidly to meet compliance regulation or contractual terms and then refine these dates and time as due diligence investigation occurs.     re: "I too want to know this...  As the more I have thought about it, the more I am coming to the idea that the precision field is just theoretical.  It does not have an effect in the actual work flow. "   It does have an effect on actual workflow for high Capability Maturity organizations that maintain strict device clock synchronization.  Knowing which block of time to pull PCAPs from a Tape Library of PCAPs from a 10GBs channel for re=processing example can make the difference of hours in processing time (in the case of Netwitness it takes 2 Hours/Terabyte to reprocess raw data packets through a Decoder back into Metadata and File Artifacts).  Every hour of delay in recovering Actionable IOCs and Malware Artifacts can result in 100's of additional APT compromised systems that need to be mitigated.  So yes, most organizations do not maintain strict infrastructure Time Synchronization, but we should not presume that there is no value/requirement to support those who have learned these hard lessons and have addressed same in their policies and procedures.   Folks - We are real close on this one.  Is there a way to add Uncertainty as an optional parameter?   Patrick Maroney President Integrated Networking Technologies, Inc. Office:  (856)983-0001 Cell:      (609)841-5104   From: Bret Jordan < bret.jordan@bluecoat.com > Date: Tuesday, November 24, 2015 at 3:38 PM To: Jason Keirstead < Jason.Keirstead@ca.ibm.com > Cc: Patrick Maroney < Pmaroney@Specere.org >, Terry MacDonald < terry@soltra.com >, Sean Barnum < sbarnum@mitre.org >, John Wunder < jwunder@mitre.org >, " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000   I too want to know this...  As the more I have thought about it, the more I am coming to the idea that the precision field is just theoretical.  It does not have an effect in the actual work flow.  If it does, please give some concrete real world work flow examples of how this is going to help.   Thanks,   Bret       Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."    On Nov 24, 2015, at 13:25, Jason Keirstead < Jason.Keirstead@ca.ibm.com > wrote:   HI Patrick - please see my earlier email around this topic, specifically the part around end-to-end workflow scenarios... " "Well, were reporting this incident now as required by Law, but we are only certain at this point that the activity occurred as far back as yesterday. " So lets assume for a moment that in the above use case the timestamp said "2015-11-24 00:00:00.000000" and did not indicate any type of "precision" indicating "+/- 24 hours"... why do I as the consumer of that information care.. how does that affect my consumption of that threat report? How can I make more effective use of that information in my system if I have that precision indicator? I am not going to do low-level temporal analysis on this type of indicator, so it doesn't matter from that perspective. What is the workflow where it does matter? - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown <graycol.gif> Patrick Maroney ---11/24/2015 04:17:59 PM---[+1] on use of syslog RFC and for support for Precision (in the literal sense). [+~] on being able t From: Patrick Maroney < Pmaroney@Specere.org > To: Terry MacDonald < terry@soltra.com >, "Barnum, Sean D." < sbarnum@mitre.org >, "Jordan, Bret" < bret.jordan@bluecoat.com >, "Wunder, John A." < jwunder@mitre.org > Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > Date: 11/24/2015 04:17 PM Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 Sent by: < cti-stix@lists.oasis-open.org > [+1] on use of syslog RFC and for support for Precision (in the literal sense). [+~] on being able to drive to consensus on this highly critical aspect "our thing" (even though we broke our priority list sequence) But to Sean's point we do as a practical matter deal with "Uncertainty" (AKA "Precision") in Timestamps. There are a dozen use cases I can cite, but think I can simply use the "Well, were reporting this incident now as required by Law, but we are only certain at this point that the activity occurred as far back as yesterday. So we can clearly agree that there should be one and only one way of specifying temporal "values" (whether fixed, relative, or intervals). But his "one way of doing things" means we still need a way to convey "uncertainty" where we need to communicate same. Dose that makes sense? Patrick Maroney President Integrated Networking Technologies, Inc. Office: (856)983-0001 Cell: (609)841-5104 From: < cti-stix@lists.oasis-open.org > on behalf of Terry MacDonald < terry@soltra.com > Date: Tuesday, November 24, 2015 at 2:37 PM To: Sean Barnum < sbarnum@mitre.org >, Bret Jordan < bret.jordan@bluecoat.com >, John Wunder < jwunder@mitre.org > Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000 I am fine with Trey’s proposal too. Specifically I support all 5 items on Bret’s email. I agree we do not have consensus on #5. A few extra comments though, brought up by a review of the extra restrictions the Syslog RFC (6.2.3 TIMESTAMP) places on implementations. Is this an acceptable full list for the timestamp? (shamelessly borrowed bits from the syslog rfc). The TIMESTAMP field is a formalized timestamp derived from [RFC3339]. Whereas [RFC3339] makes allowances for multiple syntaxes, this document imposes further restrictions. The TIMESTAMP value MUST follow these additional restrictions: 1. An RFC3339 timestamp format MUST be used. 2. All timestamps MUST be in UTC 3. A timezone offset MUST NOT be used. All times MUST be recorded in UTC time. 4. Usage of the "T" and "Z" characters are REQUIRED. 5. The "T" and "Z" characters in this syntax MUST be upper case. 6. The producer SHOULD include as much precision as its clock accuracy and performance permit. 7. Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision Examples: nanoseconds = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' Note: The additional precision field is still up for debate as we currently do not have consensus. What say everybody? Cheers Terry MacDonald Senior STIX Subject Matter Expert SOLTRA An FS-ISAC and DTCC Company +61 (407) 203 206 terry@soltra.com From: cti-stix@lists.oasis-open.org [ mailto:cti-stix@lists.oasis-open.org ] On Behalf Of Barnum, Sean D. Sent: Wednesday, 25 November 2015 5:34 AM To: Jordan, Bret < bret.jordan@bluecoat.com >; Wunder, John A. < jwunder@mitre.org > Cc: cti-stix@lists.oasis-open.org Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 I am fine with #1-4 I disagree with #5. Again, I will assert there IS a need for an explicit precision field. I have tried to explain the context a couple of time but apparently still have not done a good enough job. Unfortunately, I do not have the cycles today to attempt to explain the scenarios for precision on this topic in any further detail. Hopefully I will have a chance to do so tomorrow. I just wanted to make it clear for now that we do not have consensus on #5. Sean From: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > on behalf of "Jordan, Bret" < bret.jordan@bluecoat.com > Date: Tuesday, November 24, 2015 at 1:24 PM To: John Wunder < jwunder@mitre.org > Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 So I think we are finally getting somewhere..... Before we claim that we agree, let me paraphrase and summarize just to make sure: 1) A timestamp format of yyyy-mm-dd-Thh:mm:ssZ MUST be used Examples: 2015-11-23T13:35:12Z (for 1:35:12 in UTC format) 2) All timestamps MUST be in UTC a UI will change them as needed for an analyst 3) A timezone offset will NOT be used, all times will be recorded in UTC 4) Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision Examples: nanoseconds = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' etc. 5) There will be no extra precision field Open questions A) If the time of day is not known should it be: i) zeroed out Examples: '2015-11-24T11:00:00Z' (known only to the 11th hour) '2015-11-24T00:00:00Z (known only to the day) ii) not included, just like the sub-second (if we do this, is this a violation of RFC3339) Examples: '2015-11-24T11Z' (known only to the 11th hour) '2015-11-24TZ (known only to the day) Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." On Nov 24, 2015, at 06:24, Wunder, John A. < jwunder@mitre.org > wrote: I have a slight preference for Trey’s approach but would also be fine with Terry’s. I prefer Trey’s because IMO support for a standard as-is is better than redefining the standard. RFC3339 allows for accuracy to any arbitrary (second) value and therefore the vast majority of date/time libraries will be able to support that. His approach allows us to use the standard as-is. In fact, the requirement for millisecond precision may make things more difficult for those who just want to use RFC3339 processing libraries because they have to get those libraries to produce *our* version of RFC 3339 rather than anything compatible with the standard. I don’t think it will allow accuracy to the day unless we move back to ISO 8601 with all the other openness that it brings. But, I tend to agree with Terry that it will matter very little in almost every circumstance so we shouldn’t worry about it. John On Nov 24, 2015, at 5:28 AM, Trey Darley < trey@soltra.com > wrote: On 24.11.2015 02:36:50, Terry MacDonald wrote: I think that we have to put this whole discussion in perspective. Most organizations have difficulty in discovering they have a breach within days and weeks, not within a second. So going with B) iii) and having the precision within 1 second realistically is good enough in my opinion. It is far more likely that all the clocks on the network are not synchronized, and all the tools are reporting different and unrelated times and that they are way more than a second out of alignment with each other. The zeroed out millisecond timestamp doesn’t impact us much when we have real-world problems such as that. I agree with Terry however... On 23.11.2015 20:51:09, Wunder, John A. wrote: This is not to say that we need a precision field, just that if we do it should be explicit rather than implicit. ...I also agree with John. On the one hand, I can't count the number of times I've seen investigations get thrown under the bus due to clock-sync issues. On the other hand, recent history has shown that historical assumptions made about datetime can come back to bite us with a vengeance. So I think we should just use the standard explicitly. The producer can use the level of precision they support and intend to communicate, the consumer can easily recognize the level of precision coming from the producer, and handle it appropriately. Doing this in the manner illustrated below imposes less burden on the consumer than handling an optional 'precision' field while accomplishing the same goal. <snip> #!/usr/bin/env python import re nanoseconds = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' seconds = '2015-11-24T09:42:54Z' minutes = '2015-11-24T09:42Z' # You get the idea... timestamps = [nanoseconds, microseconds, milliseconds, seconds, minutes] nano_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{9}Z') micro_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{6}Z') milli_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{3}Z') sec_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}Z') min_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}Z') for ts in timestamps: if nano_re.match(ts): print('Found nanosecond time...') elif micro_re.match(ts): print('Found microsecond time...') elif milli_re.match(ts): print('Found millisecond time...') elif sec_re.match(ts): print('Found second time...') elif min_re.match(ts): print('Found minute time...') </snip There! Entirely explicit, uses RFC 3339 as intended, and there's one clear way of doing things without resorting to optional fields. -- Cheers, Trey -- Trey Darley Senior Security Engineer 4DAA 0A88 34BC 27C9 FD2B A97E D3C6 5C74 0FB7 E430 Soltra An FS-ISAC & DTCC Company www.soltra.com -- "It is always something." --RFC 1925     This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer . If you are not the intended recipient, please delete this message.

> "What if a programmatic event occurred at precisely "2015-11-24 00:00:00.000000" how would my logic discern the difference?" What is the use case for this though - so far all I have seen are theoretical. The only actual cases I have seen, are ones where the precision would actually already be present due to the type of indicator, so it's a moot point. > There are also legal implications for organizations in some communities for event reporting and time frames required under law to do so. In many cases you have to report before you can reasonably establish "when" something happened. We need to be able to express "rough" findings rapidly to meet compliance regulation or contractual terms and then refine these dates and time as due diligence investigation occurs. I understand this use case perfectly. But I still don't understand why the consumer of that report would care about the precision of the information. > " Knowing which block of time to pull PCAPs from a Tape Library of PCAPs from a 10GBs channel for re=processing example can make the difference of hours in processing time (in the case of Netwitness it takes 2 Hours/Terabyte to reprocess raw data packets through a Decoder back into Metadata and File Artifacts). Every hour of delay in recovering Actionable IOCs and Malware Artifacts can result in 100's of additional APT compromised systems that need to be mitigated " So the use case is that someone will a human-generated timestamp declared with a 24-48 hour time confidence and I will start a lengthy forensics reconstruction based on that time when I know it is not accurate.. ? " Every hour of delay in recovering Actionable IOCs and Malware Artifacts can result in 100's of additional APT compromised systems that need to be mitigated ", surely this means that I would not want to waste hours to days reconstructing inapplicable data? - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown Patrick Maroney ---11/24/2015 05:08:54 PM---What if a programmatic event occurred at precisely "2015-11-24 00:00:00.000000" how would my logic d From: Patrick Maroney <Pmaroney@Specere.org> To: "Jordan, Bret" <bret.jordan@bluecoat.com>, Jason Keirstead/CanEast/IBM@IBMCA Cc: Terry MacDonald <terry@soltra.com>, "Sean D. Barnum" <sbarnum@mitre.org>, "Wunder, John A." <jwunder@mitre.org>, "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org> Date: 11/24/2015 05:08 PM Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 Sent by: <cti-stix@lists.oasis-open.org> What if a programmatic event occurred at precisely "2015-11-24 00:00:00.000000" how would my logic discern the difference? My only issue here is with assuming zero values imply range of uncertainty. I am only arguing that there should be a mechanism (optional?) to state uncertainty ***explicitly***. I'm 100% good with everything else here. There are a number of Work Flow scenarios were this is relevant (User reports: "Oh yeah I saw an alert from the antivirus thingie sometime last week that warned me "ReallyBadStuff" will happen if I click OK to this, but I see these all the time and just clicked OK like I always do". There are also legal implications for organizations in some communities for event reporting and time frames required under law to do so. In many cases you have to report before you can reasonably establish "when" something happened. We need to be able to express "rough" findings rapidly to meet compliance regulation or contractual terms and then refine these dates and time as due diligence investigation occurs. re: "I too want to know this... As the more I have thought about it, the more I am coming to the idea that the precision field is just theoretical. It does not have an effect in the actual work flow. " It does have an effect on actual workflow for high Capability Maturity organizations that maintain strict device clock synchronization. Knowing which block of time to pull PCAPs from a Tape Library of PCAPs from a 10GBs channel for re=processing example can make the difference of hours in processing time (in the case of Netwitness it takes 2 Hours/Terabyte to reprocess raw data packets through a Decoder back into Metadata and File Artifacts). Every hour of delay in recovering Actionable IOCs and Malware Artifacts can result in 100's of additional APT compromised systems that need to be mitigated. So yes, most organizations do not maintain strict infrastructure Time Synchronization, but we should not presume that there is no value/requirement to support those who have learned these hard lessons and have addressed same in their policies and procedures. Folks - We are real close on this one. Is there a way to add Uncertainty as an optional parameter? Patrick Maroney President Integrated Networking Technologies, Inc. Office: (856)983-0001 Cell: (609)841-5104 From: Bret Jordan < bret.jordan@bluecoat.com > Date: Tuesday, November 24, 2015 at 3:38 PM To: Jason Keirstead < Jason.Keirstead@ca.ibm.com > Cc: Patrick Maroney < Pmaroney@Specere.org >, Terry MacDonald < terry@soltra.com >, Sean Barnum < sbarnum@mitre.org >, John Wunder < jwunder@mitre.org >, " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 I too want to know this... As the more I have thought about it, the more I am coming to the idea that the precision field is just theoretical. It does not have an effect in the actual work flow. If it does, please give some concrete real world work flow examples of how this is going to help. Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." On Nov 24, 2015, at 13:25, Jason Keirstead < Jason.Keirstead@ca.ibm.com > wrote: HI Patrick - please see my earlier email around this topic, specifically the part around end-to-end workflow scenarios... " "Well, were reporting this incident now as required by Law, but we are only certain at this point that the activity occurred as far back as yesterday. " So lets assume for a moment that in the above use case the timestamp said "2015-11-24 00:00:00.000000" and did not indicate any type of "precision" indicating "+/- 24 hours"... why do I as the consumer of that information care.. how does that affect my consumption of that threat report? How can I make more effective use of that information in my system if I have that precision indicator? I am not going to do low-level temporal analysis on this type of indicator, so it doesn't matter from that perspective. What is the workflow where it does matter? - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown <graycol.gif> Patrick Maroney ---11/24/2015 04:17:59 PM---[+1] on use of syslog RFC and for support for Precision (in the literal sense). [+~] on being able t From: Patrick Maroney < Pmaroney@Specere.org > To: Terry MacDonald < terry@soltra.com >, "Barnum, Sean D." < sbarnum@mitre.org >, "Jordan, Bret" < bret.jordan@bluecoat.com >, "Wunder, John A." < jwunder@mitre.org > Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > Date: 11/24/2015 04:17 PM Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 Sent by: < cti-stix@lists.oasis-open.org > [+1] on use of syslog RFC and for support for Precision (in the literal sense). [+~] on being able to drive to consensus on this highly critical aspect "our thing" (even though we broke our priority list sequence) But to Sean's point we do as a practical matter deal with "Uncertainty" (AKA "Precision") in Timestamps. There are a dozen use cases I can cite, but think I can simply use the "Well, were reporting this incident now as required by Law, but we are only certain at this point that the activity occurred as far back as yesterday. So we can clearly agree that there should be one and only one way of specifying temporal "values" (whether fixed, relative, or intervals). But his "one way of doing things" means we still need a way to convey "uncertainty" where we need to communicate same. Dose that makes sense? Patrick Maroney President Integrated Networking Technologies, Inc. Office: (856)983-0001 Cell: (609)841-5104 From: < cti-stix@lists.oasis-open.org > on behalf of Terry MacDonald < terry@soltra.com > Date: Tuesday, November 24, 2015 at 2:37 PM To: Sean Barnum < sbarnum@mitre.org >, Bret Jordan < bret.jordan@bluecoat.com >, John Wunder < jwunder@mitre.org > Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > Subject: RE: [cti-stix] STIX timestamps and ISO 8601:2000 I am fine with Trey’s proposal too. Specifically I support all 5 items on Bret’s email. I agree we do not have consensus on #5. A few extra comments though, brought up by a review of the extra restrictions the Syslog RFC (6.2.3 TIMESTAMP) places on implementations. Is this an acceptable full list for the timestamp? (shamelessly borrowed bits from the syslog rfc). The TIMESTAMP field is a formalized timestamp derived from [RFC3339]. Whereas [RFC3339] makes allowances for multiple syntaxes, this document imposes further restrictions. The TIMESTAMP value MUST follow these additional restrictions: 1. An RFC3339 timestamp format MUST be used. 2. All timestamps MUST be in UTC 3. A timezone offset MUST NOT be used. All times MUST be recorded in UTC time. 4. Usage of the "T" and "Z" characters are REQUIRED. 5. The "T" and "Z" characters in this syntax MUST be upper case. 6. The producer SHOULD include as much precision as its clock accuracy and performance permit. 7. Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision Examples: nanoseconds = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' Note: The additional precision field is still up for debate as we currently do not have consensus. What say everybody? Cheers Terry MacDonald Senior STIX Subject Matter Expert SOLTRA An FS-ISAC and DTCC Company +61 (407) 203 206 terry@soltra.com From: cti-stix@lists.oasis-open.org [ mailto:cti-stix@lists.oasis-open.org ] On Behalf Of Barnum, Sean D. Sent: Wednesday, 25 November 2015 5:34 AM To: Jordan, Bret < bret.jordan@bluecoat.com >; Wunder, John A. < jwunder@mitre.org > Cc: cti-stix@lists.oasis-open.org Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 I am fine with #1-4 I disagree with #5. Again, I will assert there IS a need for an explicit precision field. I have tried to explain the context a couple of time but apparently still have not done a good enough job. Unfortunately, I do not have the cycles today to attempt to explain the scenarios for precision on this topic in any further detail. Hopefully I will have a chance to do so tomorrow. I just wanted to make it clear for now that we do not have consensus on #5. Sean From: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > on behalf of "Jordan, Bret" < bret.jordan@bluecoat.com > Date: Tuesday, November 24, 2015 at 1:24 PM To: John Wunder < jwunder@mitre.org > Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > Subject: Re: [cti-stix] STIX timestamps and ISO 8601:2000 So I think we are finally getting somewhere..... Before we claim that we agree, let me paraphrase and summarize just to make sure: 1) A timestamp format of yyyy-mm-dd-Thh:mm:ssZ MUST be used Examples: 2015-11-23T13:35:12Z (for 1:35:12 in UTC format) 2) All timestamps MUST be in UTC a UI will change them as needed for an analyst 3) A timezone offset will NOT be used, all times will be recorded in UTC 4) Timestamps can have any level of sub-second precision that they support and a simple regex should be used in code to determine the precision Examples: nanoseconds = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' etc. 5) There will be no extra precision field Open questions A) If the time of day is not known should it be: i) zeroed out Examples: '2015-11-24T11:00:00Z' (known only to the 11th hour) '2015-11-24T00:00:00Z (known only to the day) ii) not included, just like the sub-second (if we do this, is this a violation of RFC3339) Examples: '2015-11-24T11Z' (known only to the 11th hour) '2015-11-24TZ (known only to the day) Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." On Nov 24, 2015, at 06:24, Wunder, John A. < jwunder@mitre.org > wrote: I have a slight preference for Trey’s approach but would also be fine with Terry’s. I prefer Trey’s because IMO support for a standard as-is is better than redefining the standard. RFC3339 allows for accuracy to any arbitrary (second) value and therefore the vast majority of date/time libraries will be able to support that. His approach allows us to use the standard as-is. In fact, the requirement for millisecond precision may make things more difficult for those who just want to use RFC3339 processing libraries because they have to get those libraries to produce *our* version of RFC 3339 rather than anything compatible with the standard. I don’t think it will allow accuracy to the day unless we move back to ISO 8601 with all the other openness that it brings. But, I tend to agree with Terry that it will matter very little in almost every circumstance so we shouldn’t worry about it. John On Nov 24, 2015, at 5:28 AM, Trey Darley < trey@soltra.com > wrote: On 24.11.2015 02:36:50, Terry MacDonald wrote: I think that we have to put this whole discussion in perspective. Most organizations have difficulty in discovering they have a breach within days and weeks, not within a second. So going with B) iii) and having the precision within 1 second realistically is good enough in my opinion. It is far more likely that all the clocks on the network are not synchronized, and all the tools are reporting different and unrelated times and that they are way more than a second out of alignment with each other. The zeroed out millisecond timestamp doesn’t impact us much when we have real-world problems such as that. I agree with Terry however... On 23.11.2015 20:51:09, Wunder, John A. wrote: This is not to say that we need a precision field, just that if we do it should be explicit rather than implicit. ...I also agree with John. On the one hand, I can't count the number of times I've seen investigations get thrown under the bus due to clock-sync issues. On the other hand, recent history has shown that historical assumptions made about datetime can come back to bite us with a vengeance. So I think we should just use the standard explicitly. The producer can use the level of precision they support and intend to communicate, the consumer can easily recognize the level of precision coming from the producer, and handle it appropriately. Doing this in the manner illustrated below imposes less burden on the consumer than handling an optional 'precision' field while accomplishing the same goal. <snip> #!/usr/bin/env python import re nanoseconds = '2015-11-24T09:42:54.003259294Z' microseconds = '2015-11-24T09:42:54.003259Z' milliseconds = '2015-11-24T09:42:54.003Z' seconds = '2015-11-24T09:42:54Z' minutes = '2015-11-24T09:42Z' # You get the idea... timestamps = [nanoseconds, microseconds, milliseconds, seconds, minutes] nano_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{9}Z') micro_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{6}Z') milli_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}.d{3}Z') sec_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}:d{2}Z') min_re = re.compile(r'd{4}-d{2}-d{2}Td{2}:d{2}Z') for ts in timestamps: if nano_re.match(ts): print('Found nanosecond time...') elif micro_re.match(ts): print('Found microsecond time...') elif milli_re.match(ts): print('Found millisecond time...') elif sec_re.match(ts): print('Found second time...') elif min_re.match(ts): print('Found minute time...') </snip There! Entirely explicit, uses RFC 3339 as intended, and there's one clear way of doing things without resorting to optional fields. -- Cheers, Trey -- Trey Darley Senior Security Engineer 4DAA 0A88 34BC 27C9 FD2B A97E D3C6 5C74 0FB7 E430 Soltra An FS-ISAC & DTCC Company www.soltra.com -- "It is always something." --RFC 1925