OASIS Cyber Threat Intelligence (CTI) TC

In the STIX spirit of implementing proposed specification additions, I have been implementing a script to generate deterministic IDs which I have commented on before.
 
I noticed something else which seems ambiguous the use of optional properties.  For example, let s take an email-message.  The three ID contributing properties are:
from_ref ,
subject ,
body .
Both are subject and
body optional.  Assuming both are not present, then only
from_ref will be used.  Therefore, any email message from the same email address, for which both the subject and body are empty (or not provided in
the email-message object), will have the same deterministic ID.  However, the email-message objects might be very different i.e., many non-contributing properties exists in both that are not the same.
 
So a collision will take place. Receiving two SCOs with the same id was supposed to a feature you didn t have to retain both,  but in this case you really have two different objects both of which should probably
be retained (e.g., different date
properties).
 
Additionally, this problem can also exist even with required only contributing properties.
 
I m aware that this is a known limitation, and perhaps it has already been taken into consideration, as I was not part of the subgroup that developed deterministic IDs.    But maybe we can do a little better
for instance:
 

Include more contributing properties.  For instance, adding the
date
property as a contributing property of the email-message object would help to avoid collisions.
Distinguish non-existent properties in the actual object from ones not provided in the STIX object i.e., if the subject was really empty provide the
subject property with a value of .  Some normative text ( SHOULD ) could be added. Use UUIDv4 if the number of non-contributing properties is over some threshold chances are there won t be another similar object.
 
Rich
-- 
Rich Piazza
The MITRE Corporation
781-271-3760
 

 
 
 
 

I think that email message should default to UUIDv4 in most cases. In general email messages shouldn t deduplicate with the use cases to deduplicate them being:   1.        It s the exact same email message and that s what we re tracking in which case message_id should be used as the basis of this determination if possible. 2.        We are tracking a more abstract layer in which case we want to build an ID based on the email s subject, body and attachments. When using email in this manner external relationships need to be used to point to the sender, and recipients since they weren t all included in one email, but instead received an email that from a CTI perspective was identical.   Jeffrey Mates, Civ DC3/TSD ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Computer Scientist Technical Solutions Development jeffrey.mates@dc3.mil 410-694-4335   From: cti@lists.oasis-open.org <cti@lists.oasis-open.org> On Behalf Of Piazza, Rich Sent: Friday, June 21, 2019 12:40 PM To: cti@lists.oasis-open.org Subject: [Non-DoD Source] [cti] More on ID Contributing Properties   In the STIX spirit of implementing proposed specification additions, I have been implementing a script to generate deterministic IDs which I have commented on before.   I noticed something else which seems ambiguous the use of optional properties.  For example, let s take an email-message.  The three ID contributing properties are: from_ref , subject , body . Both are subject and body optional.  Assuming both are not present, then only from_ref will be used.  Therefore, any email message from the same email address, for which both the subject and body are empty (or not provided in the email-message object), will have the same deterministic ID.  However, the email-message objects might be very different i.e., many non-contributing properties exists in both that are not the same.   So a collision will take place. Receiving two SCOs with the same id was supposed to a feature you didn t have to retain both,  but in this case you really have two different objects both of which should probably be retained (e.g., different date properties).   Additionally, this problem can also exist even with required only contributing properties.   I m aware that this is a known limitation, and perhaps it has already been taken into consideration, as I was not part of the subgroup that developed deterministic IDs.    But maybe we can do a little better for instance:   Include more contributing properties.  For instance, adding the date property as a contributing property of the email-message object would help to avoid collisions. Distinguish non-existent properties in the actual object from ones not provided in the STIX object i.e., if the subject was really empty provide the subject property with a value of .  Some normative text ( SHOULD ) could be added. Use UUIDv4 if the number of non-contributing properties is over some threshold chances are there won t be another similar object.   Rich --  Rich Piazza The MITRE Corporation 781-271-3760           Attachment: smime.p7s Description: S/MIME cryptographic signature