CTI STIX Subcommittee

 View Only
Expand all | Collapse all

Re: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier

  • 1.  Re: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier

    Posted 02-08-2019 19:17
      |   view attached




    UUIDv5 requires A namespace and so our consensus for SCOs that gave a SHOULD use UUIDv5 as the hash calculation mechanism does not conflict.
    I think the issue could be that for SCOs we intentionally did not restrict what that namespace must be. A producer could use their domain which would make their IDs consistent but unique to them. We could also specify a default namespace
    for STIX as a SHOULD or a MAY which would enable consistent IDs but not unique to any particular producer.
    I like this latter approach across ALL IDs as it yields both consistency but also flexibility where appropriate.
     
    It is challenging keeping up with what changes are being proposed to the wording.
    Is the wording change proposal asserting a MUST for which namespace to use?
    If so, I would strongly suggest changing that to a MAY and following the above approach for ALL IDs.
     

    Sean Barnum
    Principal Architect
    FireEye
    M: 703.473.8262

    E: sean.barnum@fireeye.com
     

    From: <cti-stix@lists.oasis-open.org> on behalf of "Kelley, Sarah E." <skelley@mitre.org>
    Date: Friday, February 8, 2019 at 2:08 PM
    To: Allan Thomson <athomson@lookingglasscyber.com>, Bret Jordan <Bret_Jordan@symantec.com>, "Wunder, John A." <jwunder@mitre.org>, Sergey Polzunov <sergey@eclecticiq.com>
    Cc: "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org>
    Subject: RE: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     

    I think I agree with Allan here. UUID5 requires you to namespace your ID, correct? That wasn t a part of the Cyber Observables proposal, and would, in fact, make it impossible for two different vendors to create the same deterministic ID.

     
    Am I missing something?
     

    Sarah Kelley
    Lead Cybersecurity Engineer, T8B2
    Defensive Operations
    The MITRE Corporation
    703-983-6242
    skelley@mitre.org


     


    From: cti-stix@lists.oasis-open.org <cti-stix@lists.oasis-open.org>
    On Behalf Of Allan Thomson
    Sent: Friday, February 8, 2019 1:32 PM
    To: Bret Jordan <Bret_Jordan@symantec.com>; Wunder, John A. <jwunder@mitre.org>; Sergey Polzunov <sergey@eclecticiq.com>
    Cc: cti-stix@lists.oasis-open.org
    Subject: Re: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     
    Bret There are multiple aspects of the SCO changes that require special attention particularly with IDs.
     
    For example, we * explicitly * stated that SCO can have different methods of determining IDs and there is an enumeration that calls out what is being used. There are also special rules on what constitutes an ID for SCO that do not
    apply to other objects.
     
    So it is incorrect to suggest that this one change covers everything that we did for SCO proposal and are waiting for that merge to happen.
     
    So until the complete merge occurs and we can review holistically we will not be able to say that the merged SCO feature and all changes are agreed to.
     

    Allan

     

    From: Bret Jordan < Bret_Jordan@symantec.com >
    Date: Friday, February 8, 2019 at 10:22 AM
    To: Allan Thomson < athomson@lookingglasscyber.com >, "Wunder, John" < jwunder@mitre.org >, Sergey Polzunov < sergey@eclecticiq.com >
    Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
    Subject: Re: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     


    I added these changes to 2.7 in suggestion mode.  Please review.   https://docs.google.com/document/d/1ShNq4c3e1CkfANmD9O--mdZ5H0O_GLnjN28a_yrEaco/edit#heading=h.ko24ggw4eq0q


     


    Also, please note, I added the term "Cyber Observables" to the list of objects that this ID will work for, that way we can delete the "extra" definition we had for just Cyber Observables.  This
    will then make all objects in STIX have the same definition for an ID.


     


     


    Bret


     





    From: Allan Thomson < athomson@lookingglasscyber.com >
    Sent: Friday, February 8, 2019 11:15 AM
    To: Wunder, John A.; Bret Jordan; Sergey Polzunov
    Cc: cti-stix@lists.oasis-open.org
    Subject: Re: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     




    Less is more.
     
    I agree with John. I didn t see that sentence on prior reading but agree its better to remove.
     

    Allan Thomson
    CTO ( +1-408-331-6646)
    LookingGlass Cyber Solutions

     

    From:
    "Wunder, John" < jwunder@mitre.org >
    Date: Friday, February 8, 2019 at 10:12 AM
    To: Bret Jordan < Bret_Jordan@symantec.com >, Allan Thomson < athomson@lookingglasscyber.com >, Sergey Polzunov < sergey@eclecticiq.com >
    Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
    Subject: Re: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     

    It s not a hill I would die on but the last sentence is very speculative: Some objects in STIX MAY define a list of properties that an organization MAY use for the name portion of the UUIDv5 hash, that MIGHT
    allow deterministic generation and verification of semantic equivalency.
     
    I think it just complicates things and IMO should be removed. If/when we choose to do that we could address it then.
     
    Other than that I like this change.
     
    John
     

    From:
    < cti-stix@lists.oasis-open.org > on behalf of "Bret Jordan (CS)" < Bret_Jordan@symantec.com >
    Date: Friday, February 8, 2019 at 1:08 PM
    To: Allan Thomson < athomson@lookingglasscyber.com >, Sergey Polzunov < sergey@eclecticiq.com >
    Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
    Subject: Re: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     


    Great... I will make that change.


     


    Any others?


     


    Bret





    From: Allan Thomson < athomson@lookingglasscyber.com >
    Sent: Friday, February 8, 2019 11:05 AM
    To: Bret Jordan; Sergey Polzunov
    Cc: cti-stix@lists.oasis-open.org
    Subject: Re: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     




    I suggest the statement on SHOULD regarding UUIDv5 be changed to MAY.
     
    There are multiple reasons why an organization may choose to use a different domain or some other identifier that is namespace unique. Simple reasons such as acquisition and M&A would cause the namespaces for a single entity to potentially
    be multiple values and an organization may choose to use a single name for the entire company even though they may have many sub-organizations with presence throughout the world.

     
    From
    For UUIDv5, the namespace SHOULD be the organization's fully qualified DNS name (example.com)
    To
    For UUIDv5, the namespace MAY be the organization's fully qualified DNS name (example.com) or some other organizational identifier that helps ensure uniqueness globally
     
     

    Allan Thomson
    CTO ( +1-408-331-6646)
    LookingGlass Cyber Solutions

     

    From:
    " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > on behalf of Bret Jordan < Bret_Jordan@symantec.com >
    Date: Friday, February 8, 2019 at 8:28 AM
    To: Sergey Polzunov < sergey@eclecticiq.com >
    Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
    Subject: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     


    Is there anyone that would object to making this change to Part 1 of STIX 2.1?   Also, I think the way this is worded, it would work for the definition of deterministic-ids in the Cyber Observable
    proposal.   


     


    What do people think?


     


    Bret


     





    From: Sergey Polzunov < sergey@eclecticiq.com >
    Sent: Friday, February 8, 2019 6:15 AM
    To: Bret Jordan
    Cc: cti-stix@lists.oasis-open.org
    Subject: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     




    +1, good change

    Thanks,
    Sergey Polzunov
    EclecticIQ



    > On 7 Feb 2019, at 19:01, Bret Jordan < bret_jordan@symantec.com > wrote:
    >
    > I have been thinking about this a lot over the past few weeks.  There is a lot of history here, and my concerns always were ensuring that two organizations did not produce two different objects with the same ID. 

    >
    > I personally believe that the ID of the object should not be used to find semantic equivalence.  If we want to add another property for that, then great. 

    >
    > I would support the following updated text:
    >
    > ### BEGIN FROM Part1 Section 2.7
    >
    > An identifier universally and uniquely identifies a SDO, SRO, Bundle, Language Content, or Marking Definition. Identifiers MUST follow the form object-type--hash, where object-type is the exact value (all type names are lowercase strings, by definition) from
    the type property of the object being identified or referenced and where the hash is either UUIDv4 is an RFC 4122-compliant Version 4 UUID or a Version 5 UUID. The UUID MUST be generated according to the algorithm(s) defined in RFC 4122, section 4.4 (Version
    4 UUID) or section 4.3 (Version 5 UUID) [RFC4122]. For UUIDv5, the namespace SHOULD be the organization's fully qualified DNS name (example.com) and the name should be a value consistent within the organizations that guarantees that two different objects do
    not generate the same ID. For example, an organization MAY choose to use all properties, a subset of all properties, an organizational content identifier, a STIX 1.x identifier, or something else as string data for the name. Some objects in STIX MAY define
    a list of properties that an organization MAY use for the name portion of the UUIDv5 hash, that MIGHT allow deterministic generation and verification of semantic equivalency.

    >
    > ### END
    >
    > If we did this definition right, we could use the SAME identifier for both STIX content and Cyber Observables, thus preventing a lot of future confusion in the market.

    >
    >
    > Bret
    > From: cti-stix@lists.oasis-open.org < cti-stix@lists.oasis-open.org > on behalf of Jason Keirstead < Jason.Keirstead@ca.ibm.com >
    > Sent: Tuesday, February 5, 2019 6:06 AM
    > To: Patrick Maroney
    > Cc: cti-stix@lists.oasis-open.org ; John-Mark Gurney; Wunder, John A.; Sean Barnum; Sergey Polzunov
    > Subject: Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier

    > UUIDv5 enables this.  Those who want to use a specific deterministic ID can just form a consortium of agreement, choose a namespace for this consortium to use, and use that namespace among all of their products.
    >
    > -
    > Jason Keirstead
    > Lead Architect - IBM Security Connect
    >
    https://clicktime.symantec.com/37uHtFjAfPoNTZLcQL3iqoS7Vc?u=www.ibm.com%2Fsecurity
    >
    > "Things may come to those who wait, but only the things left by those who hustle." - Unknown

    >
    >
    >
    >
    > From:        Patrick Maroney < pmaroney@darklight.ai >
    > To:        "Wunder, John A." < jwunder@mitre.org >, Jason Keirstead < Jason.Keirstead@ca.ibm.com >, Sean Barnum < sean.barnum@FireEye.com >
    > Cc:        " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >, John-Mark Gurney < jmg@newcontext.com >,
    Sergey Polzunov < sergey@eclecticiq.com >
    > Date:        02/04/2019 05:38 PM
    > Subject:        Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier
    >
    >
    > We want distinct Deterministic ID UUIDs to discriminate between Sources of same/similar information and to better detect/prevent parroting and/or leakage.

    > A given of community of trust can establish conventions within the UUIDv5 encoding as required (e.g., just Namespace, Namespace+Community_ID, Namespace+Key).   We should only be using and asserting immutable objects and using the canonical representations
    suggested.   A number of UUIDv5 use cases were proposed historically including non-attributional Source Traceability.

    > However, none of this impacts those not wishing to participate in Deterministic Reference ID use cases, but empowers those of us who do.

    > Patrick Maroney
    > DarkLight
    > Mobile: (609)841-5104
    > Email:  patrick.maroney@darklight.ai

    >
    https://clicktime.symantec.com/3BM8wpmh4d2rTufAQWwDRcR7Vc?u=www.darklight.ai


    > From: < cti-stix@lists.oasis-open.org > on behalf of John Wunder < jwunder@mitre.org >
    > Date: Monday, February 4, 2019 at 3:40 PM
    > To: Jason Keirstead < Jason.Keirstead@ca.ibm.com >, Sean Barnum < sean.barnum@FireEye.com >
    > Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >, John-Mark Gurney < jmg@newcontext.com >, Sergey
    Polzunov < sergey@eclecticiq.com >
    > Subject: Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier

    > Yeah I agree with Jason here.

    > I don t think that would work for UUIDv5 anyway, since everyone will be using different namespaces even if you hash the data the same way you end up with different (non-transparent) IDs:

    > >>> uid = uuid.uuid4()
    > >>> uuid.uuid5(uid, "data")
    > UUID('c5bb29ba-4d85-5280-b202-7085b7485b62')
    > >>> uid2 = uuid.uuid4()
    > >>> uuid.uuid5(uid2, "data")
    > UUID('d09fafa9-ead3-54f1-b1c7-1d1d55f66fb7')

    > You can see that with the same data (just the string literal data ) you end up with different IDs. You could of course use the namespace of a different producer to try to back generate a UUIDv5 for an object they ve presumably created, but IMO that s a recipe
    for trouble (did they actually create that object?). Instead you should just rely on having received that object before and use whatever UUID it had.

    > It seems like the simplest thing is to just allow UUIDv5 via the various proposals we ve seen. It meets the use cases below and is a very straightforward change.

    > John

    > From: < cti-stix@lists.oasis-open.org > on behalf of Jason Keirstead < Jason.Keirstead@ca.ibm.com >
    > Date: Monday, February 4, 2019 at 3:35 PM
    > To: Sean Barnum < sean.barnum@FireEye.com >
    > Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >, John-Mark Gurney < jmg@newcontext.com >, John
    Wunder < jwunder@mitre.org >, Sergey Polzunov < sergey@eclecticiq.com >
    > Subject: Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier

    > I don't think that figuring out semantic equivalence should be a goal here - referring back to Sergey's original email, that is not why people are trying to do this.

    >
    > People want to do this so they can have a bi-directional traceability method from a STIX ID back and forth into an already-existing ID system.
    >
    > Forcing them to generate the IDs based on properties, defeats the whole purpose.
    >
    > -
    > Jason Keirstead
    > Lead Architect - IBM Security Connect
    >
    https://clicktime.symantec.com/37uHtFjAfPoNTZLcQL3iqoS7Vc?u=www.ibm.com%2Fsecurity
    >
    > "Things may come to those who wait, but only the things left by those who hustle." - Unknown

    >
    >
    >
    >
    > From:        Sean Barnum < sean.barnum@FireEye.com >
    > To:        Jason Keirstead < Jason.Keirstead@ca.ibm.com >
    > Cc:        " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >, John-Mark Gurney < jmg@newcontext.com >,
    "Wunder, John A." < jwunder@mitre.org >, Sergey Polzunov < sergey@eclecticiq.com >
    > Date:        02/04/2019 04:30 PM
    > Subject:        Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier
    > Sent by:        < cti-stix@lists.oasis-open.org >
    >
    >
    >
    >
    > It is to assist in semantic equivalence normalization across producers. Just like we have done for SCOs.
    >
    > The reality is that objects such as Locations, Identities, etc are likely to be repeated and widely used about as much as many Observables.
    > The ability to inherently converge on equivalence of things like Locations and Identities via UUIDv5 calculation is extremely valuable.
    >
    > Sean Barnum
    > Principal Architect
    > FireEye
    > M: 703.473.8262
    > E: sean.barnum@fireeye.com
    >
    > From: Jason Keirstead < Jason.Keirstead@ca.ibm.com >
    > Date: Monday, February 4, 2019 at 3:25 PM
    > To: Sean Barnum < sean.barnum@FireEye.com >
    > Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >, John-Mark Gurney < jmg@newcontext.com >, "Wunder,
    John A." < jwunder@mitre.org >, Sergey Polzunov < sergey@eclecticiq.com >
    > Subject: Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier
    >
    > Actually, I don't agree with this part.
    >
    > The entire point of UUIDv5, is that I should not care what method you use to compute your IDs - because it's in your namespace, so its not my problem anymore. I don't think we want to codify it.
    >
    > -
    > Jason Keirstead
    > Lead Architect - IBM Security Connect
    >
    https://clicktime.symantec.com/37uHtFjAfPoNTZLcQL3iqoS7Vc?u=www.ibm.com%2Fsecurity
    >
    > "Things may come to those who wait, but only the things left by those who hustle." - Unknown

    >
    >
    >
    >
    > From:        Sean Barnum < sean.barnum@FireEye.com >
    > To:        Jason Keirstead < Jason.Keirstead@ca.ibm.com >, John-Mark Gurney < jmg@newcontext.com >
    > Cc:        " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >, "Wunder, John A." < jwunder@mitre.org >,
    Sergey Polzunov < sergey@eclecticiq.com >
    > Date:        02/04/2019 04:23 PM
    > Subject:        Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier
    > Sent by:        < cti-stix@lists.oasis-open.org >
    >
    >
    >
    >
    >
    > Agree.
    >
    > And I would suggest we DO want the calculation to be done based upon the data from the object . As that is how we get value from such an ID.
    > It should just not be done on ALL properties of the object.
    > We just need to define the semantically relevant properties to use for the calculation. This is exactly what we have just done for SCOs.
    >
    > Sean Barnum
    > Principal Architect
    > FireEye
    > M: 703.473.8262
    > E: sean.barnum@fireeye.com
    >
    > From: < cti-stix@lists.oasis-open.org > on behalf of Jason Keirstead < Jason.Keirstead@ca.ibm.com >
    > Date: Monday, February 4, 2019 at 3:20 PM
    > To: John-Mark Gurney < jmg@newcontext.com >
    > Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >, "Wunder, John A." < jwunder@mitre.org >, Sergey
    Polzunov < sergey@eclecticiq.com >
    > Subject: Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier
    >
    > I would think we would want to use a DNS or URL namespace, would we not?
    >
    > -
    > Jason Keirstead
    > Lead Architect - IBM Security Connect
    >
    https://clicktime.symantec.com/37uHtFjAfPoNTZLcQL3iqoS7Vc?u=www.ibm.com%2Fsecurity
    >
    > "Things may come to those who wait, but only the things left by those who hustle." - Unknown

    >
    >
    >
    >
    > From:        John-Mark Gurney < jmg@newcontext.com >
    > To:        Jason Keirstead < Jason.Keirstead@ca.ibm.com >
    > Cc:        "Wunder, John A." < jwunder@mitre.org >, " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >,
    Sergey Polzunov < sergey@eclecticiq.com >
    > Date:        02/04/2019 04:10 PM
    > Subject:        Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier
    >
    >
    >
    >
    >
    >
    > Jason Keirstead wrote this message on Mon, Feb 04, 2019 at 14:08 -0400:
    > > I would also support this.
    > >
    > > I have learned more about the inner workings of UUID4/5 and I don't have
    > > any reservations about it anymore. The odds of collision with a
    > > properly-implemented UUID5 are on-par with UUID4
    > >
    > > As far as John's comment below - all this means IMHO is the library has to
    > > force you to provide a namespace (ie make it a mandatory argument in your
    > > constructor or whatever).
    >
    > The one requirement I would like to make sure about UUIDv5 is that it
    > is NOT based upon the data from the object, otherwise versioning will
    > break.
    >
    > The reason we didn't use UUIDc4 as most of the proposals to use it was
    > to make it a hash of the contents, such as name and description, and
    > then update the UUID whenever the name and/or description changed..
    >
    > If we do this, the name space should probably be the identity of the
    > new STIX2 object.  This would prevent collisions from happening when
    > two entities try to create a "new" STIX2 object from a STIX1 object...
    >
    > > From:   "Wunder, John A." < jwunder@mitre.org >
    > > To:     Sergey Polzunov < sergey@eclecticiq.com >,

    > > " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
    > > Date:   02/04/2019 12:22 PM
    > > Subject:        [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in
    > > STIX2 identifier
    > > Sent by:        < cti-stix@lists.oasis-open.org >
    > >
    > >
    > >
    > > I've been thinking a lot about this and I think it makes sense.
    > >
    > > One of the concerns we had at the time we chose UUID4 is that users of
    > > libraries like python-stix would need to remember to set the UUID5
    > > namespace -- or, if they don't and python-stix has some default namespace,
    > > different tools using the libraries could have overlapping IDs. This would
    > > also apply to users of the new Java libraries that I've seen come out. It
    > > might mean these libraries requiring that people set a unique namespace
    > > before creating any objects, vs. now where it can just go ahead and create
    > > IDs by default. I'd be curious what other people think about this problem
    > > and how we can help avoid it becoming an issue (especially given how many
    > > people use those libraries).
    > >
    > > John
    > >
    > > On 2/4/19, 11:12 AM, " cti-stix@lists.oasis-open.org on behalf of Sergey

    > > Polzunov " < cti-stix@lists.oasis-open.org on behalf of

    > > sergey@eclecticiq.com > wrote:
    > >
    > >     Hey everybody!
    > > 
    > >     Current STIX2 spec definition of an`identifier` for STIX2 objects is
    > > as follows:
    > > 
    > >     > An identifier universally and uniquely identifies a SDO, SRO,
    > > Bundle, or Marking Definition. Identifiers MUST follow the form
    > > object-type--UUIDv4, where object-type is the exact value (all type names
    > > are lowercase strings, by definition) from the type property of the object
    > > being identified or referenced and where the UUIDv4 is an RFC
    > > 4122-compliant Version 4 UUID. The UUID MUST be generated according to the
    > > algorithm(s) defined in RFC 4122, section 4.4 (Version 4 UUID) [RFC4122].
    > >     from
    > >
    https://clicktime.symantec.com/3SNmD57f7Xxv1JXt4mbkz5S7Vc?u=http%3A%2F%2Fdocs.oasis-open.org%2Fcti%2Fstix%2Fv2.0%2Fcs01%2Fpart1-stix-core%2Fstix-v2.0-cs01-part1-stix-core.html%23_Toc496709265
    > >
    > > 
    > >     I think the requirement to have UUID4 brings more problems than
    > > benefits. It makes STIX1->STIX2 transition difficult, hurting existing
    > > STIX1 users.
    > >     I will try to show it in these 2 use cases.
    > > 
    > > 
    > >     Use case 1
    > >     ----------
    > >         Imagine that I'm a client of an intelligence provider A. I've been
    > > a client for a long time and I have received intelligence in STIX1.2,
    > > which I stored in my DB. I fetch new intelligence daily, downloading only
    > > fresh data. Often fresh data links to old objects for context.
    > >         Provider A decides to upgrade and switch to STIX2. In addition to
    > > an old STIX1.2 feed, provider creates new STIX2 feed with the same data.
    > > In STIX2 all objects have new identifiers and Provider A does not bother
    > > to supply a mapping of STIX1.2 ids to STIX2 ids. Now I, as a client, have
    > > 2 options:
    > >         - clean slate option: drop all old data from this provider and
    > > re-fetch everything. That will work if Provider A is the only provider I
    > > use or if I never referenced Provider A's data from my own intelligence.
    > > Not a great plan.
    > >         - new era option: leave my STIX1.2 data graph in place and start
    > > consuming new STIX2 feed from today. This option has one big issue: new
    > > STIX2 data will not be connected to STIX1.2 data I already have, because
    > > STIX2 ids are all different. If I want to deduce connection, I need to
    > > deduplicate the data against my existing STIX1.2 DB. This means my
    > > ingestion pipeline must be smart enough to compare STIX1.2 objects to
    > > STIX2 objects and be fast enough to do that for every new STIX2 object.
    > > This will be difficult to implement and will have a huge performance
    > > penalty.
    > > 
    > >     Use case 2
    > >     ----------
    > >         Imagine that I'm a NCSC. I receive intelligence from providers,
    > > combine it and distribute it to my clients. My providers are still on
    > > STIX1.2 but my clients want STIX2, so I must convert STIX1.2 I receive
    > > into STIX2. Full STIX1.2 entities I can transform easily but what do I do
    > > with IDREFs I have in my STIX1.2 data?
    > >         I can generate new STIX2 id every time I see new STIX1.2 IDREF in
    > > incoming data and store STIX1.2->STIX2 mapping somewhere to be used next
    > > time I see this IDREF. This is painful and will require additional
    > > resources, but it is doable. But it will only work until the moment my
    > > providers switch to STIX2 and start sending me full objects for those
    > > IDREFs with new random STIX2 identifiers! I can not predict these
    > > identifiers and I can't match them with the ones I generated. So my
    > > thinking is - what is the point in even bothering with old IDREFs? I will
    > > just drop them, sending my clients sometimes disconnected STIX2 entities,
    > > hoping that they will figure it out.
    > > 
    > > 
    > >     Proposed solutions if UUID5 is allowed in STIX2 identifiers:
    > > 
    > >     Use case 1 solution
    > >     -------------------
    > >         There can be a guideline that will recommend providers to use old
    > > STIX1.2 IDs as input for new STIX2 identifiers. If STIX2 identifiers are
    > > predictable I, as a client, can greatly simplify my deduplication logic. I
    > > can run DB migration once to calculate STIX2 identifiers for all my
    > > STIX1.2 objects and use these on ingestion for deduplication. Appending
    > > STIX2 data to my STIX1.2 DB will be much easier.
    > >         I'm also interested in pushing Provider A to adopting this STIX2
    > > identifier generation practice because it will save me money.
    > > 
    > >     Use case 2 solution
    > >     -------------------
    > >     WIth UUID5 I have a way out: I can generate new STIX2 ids from old
    > > STIX1.2 ids! I can parse IDREF value, that looks like `[ns
    > > prefix]:[construct type]-[GUID]`, and use provider's namespace / construct
    > > type to build new STIX2 identifier. The logic will be like this:
    > >         - full IDREF will be input for UUID5 function
    > >         - for STIX1.2 types that were split (like TTP), I do not know
    > > exact STIX2 type Provider would use for old TTP. My solution here would be
    > > to play safe and create relations for all possible types: for IDREF to
    > > TTP, I will create 4 relations: one to a possible Tool object, one to
    > > Malware, one to Attack Pattern and one to Identity. It is an overhead but
    > > it is a small price for keeping interconnected intelligence graph.
    > >         Again, when time comes and my providers move to STIX2, I'm
    > > interested in pushing them to adopt this id generation schema for old
    > > objects, because it will save me, as NCSC, money.
    > > 
    > > 
    > >     To reiterate, I would like to propose:
    > >     - a change in STIX2 spec to allow both UUID5 and UUID4 to be used in
    > > an identifier of SDO, SRO, MarkingDefinition and Custom Object entities;
    > >     - creating a guideline, complimentary to the spec, that would explain
    > > how STIX1.2 ids can be transformed into STIX2 for easier transition.
    > > 
    > > 
    > >     Practicalities:
    > > 
    > >     UUID5 ids require use of a namespace. UUID5 RFC (
    > >
    https://clicktime.symantec.com/3VHevgXCxDKCASQXfexTG2y7Vc?u=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc4122%23section-4.3
    > > ) defines some generic namespaces (
    > >
    https://clicktime.symantec.com/39DSizcAyzjYR5MNQsVcequ7Vc?u=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc4122%23appendix-C
    > > ) but does not prohibits the use of custom ones. I suggest this algorithm:
    > >         - namespace UUID5 is generated by using predefined `NameSpace_URL`
    > > namespace and producer's URL;
    > >         - for old objects, GUID part of STIX2 identifier is namespaced
    > > UUID5 generated from old STIX1.2 id
    > >         - for new objects, GUID part of STIX2 identifier is either
    > > namespaced UUID5 with random UUID4 string, or just random UUID4.
    > > 
    > >     Example python code for generating UUID5 with custom namespace:
    > > 
    > >         In [1]: import uuid
    > >            ...:
    > >            ...: stix12_id =
    > > 'eclecticiq:threat-actor-07fa8672-4bca-46e1-a60f-023882b4a473'
    > >            ...: namespace_uuid = uuid.uuid5(uuid.NAMESPACE_URL,
    > > ' https://clicktime.symantec.com/3Twt3zW8FEEWaBXtubfTCpc7Vc?u=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__eclecticiq.com_ns%26d%3DDwIGaQ%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3Dk6Q07xZDujljzkKqZUfupXFUDIHGIiq-Sl_u1bw0hyA%26m%3DcvP-VddGmd9zTZUjb6OSCUczFxCjDL1cA586YiCE8YI%26s%3DtQbiU4LJBfzo5lmgDPo4k6EjM9ZKKwE6AzhNphzBRcM%26e%3D%27 )
    > >            ...: stix2_uuid = uuid.uuid5(namespace_uuid, stix12_id)
    > >            ...: stix2_id = 'threat-actor--{}'.format(stix2_uuid)
    > >            ...:
    > >            ...: print("new STIX2 id: {}".format(stix2_id))
    > > 
    > >         new STIX2 id: threat-actor--adee573a-12e9-5dd3-958b-0040d32c6b3e
    > > 
    > > 
    > >     BONUS: python functions to convert STIX1.2 IDREFs into STIX2
    > > identifiers -
    > >
    https://clicktime.symantec.com/3TtVUzeNYqg3BBswQN7bhLt7Vc?u=https%3A%2F%2Fgist.github.com%2Ftraut%2Ffd4b9b8de3c2aa0e161d68c4099656e5
    > >
    > > 
    > > 
    > >     Thank you,
    > >     Sergey Polzunov
    > >     EclecticIQ
    > > 
    > > 
    > > 
    > > 
    > >
    > >
    > >
    > >
    > >
    > >
    >
    > --
    > John-Mark
    >
    > This email and any attachments thereto may contain private, confidential, and/or privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited.
    If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.

    > This email and any attachments thereto may contain private, confidential, and/or privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited.
    If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.

    >
    >







    This email and any attachments thereto may contain private, confidential, and/or privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited.
    If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.





  • 2.  Re: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier

    Posted 02-08-2019 19:36
      |   view attached




    Is the wording change proposal asserting a MUST for which namespace to use?
     

    No.
     

    But your question highlights why we need a merged proposal that we can all review in the doc instead of fragments over email.
     
    allan

     

    From: Sean Barnum <sean.barnum@FireEye.com>
    Date: Friday, February 8, 2019 at 11:17 AM
    To: "Kelley, Sarah E." <skelley@mitre.org>, Allan Thomson <athomson@lookingglasscyber.com>, Bret Jordan <Bret_Jordan@symantec.com>, "Wunder, John" <jwunder@mitre.org>, Sergey Polzunov <sergey@eclecticiq.com>
    Cc: "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org>
    Subject: Re: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     

    UUIDv5 requires A namespace and so our consensus for SCOs that gave a SHOULD use UUIDv5 as the hash calculation mechanism does not conflict.
    I think the issue could be that for SCOs we intentionally did not restrict what that namespace must be. A producer could use their domain which would make their IDs consistent but unique to them. We could also specify a default namespace
    for STIX as a SHOULD or a MAY which would enable consistent IDs but not unique to any particular producer.
    I like this latter approach across ALL IDs as it yields both consistency but also flexibility where appropriate.
     
    It is challenging keeping up with what changes are being proposed to the wording.
    Is the wording change proposal asserting a MUST for which namespace to use?
    If so, I would strongly suggest changing that to a MAY and following the above approach for ALL IDs.
     

    Sean Barnum
    Principal Architect
    FireEye
    M: 703.473.8262

    E: sean.barnum@fireeye.com
     

    From: <cti-stix@lists.oasis-open.org> on behalf of "Kelley, Sarah E." <skelley@mitre.org>
    Date: Friday, February 8, 2019 at 2:08 PM
    To: Allan Thomson <athomson@lookingglasscyber.com>, Bret Jordan <Bret_Jordan@symantec.com>, "Wunder, John A." <jwunder@mitre.org>, Sergey Polzunov <sergey@eclecticiq.com>
    Cc: "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org>
    Subject: RE: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     

    I think I agree with Allan here. UUID5 requires you to namespace your ID, correct? That wasn t a part of the Cyber Observables proposal, and would, in fact, make it impossible for two different vendors to create the same deterministic ID.

     
    Am I missing something?
     

    Sarah Kelley
    Lead Cybersecurity Engineer, T8B2
    Defensive Operations
    The MITRE Corporation
    703-983-6242
    skelley@mitre.org


     


    From: cti-stix@lists.oasis-open.org <cti-stix@lists.oasis-open.org>
    On Behalf Of Allan Thomson
    Sent: Friday, February 8, 2019 1:32 PM
    To: Bret Jordan <Bret_Jordan@symantec.com>; Wunder, John A. <jwunder@mitre.org>; Sergey Polzunov <sergey@eclecticiq.com>
    Cc: cti-stix@lists.oasis-open.org
    Subject: Re: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     
    Bret There are multiple aspects of the SCO changes that require special attention particularly with IDs.
     
    For example, we * explicitly * stated that SCO can have different methods of determining IDs and there is an enumeration that calls out what is being used. There are also special rules on what constitutes an ID for SCO that do not
    apply to other objects.
     
    So it is incorrect to suggest that this one change covers everything that we did for SCO proposal and are waiting for that merge to happen.
     
    So until the complete merge occurs and we can review holistically we will not be able to say that the merged SCO feature and all changes are agreed to.
     

    Allan

     

    From: Bret Jordan < Bret_Jordan@symantec.com >
    Date: Friday, February 8, 2019 at 10:22 AM
    To: Allan Thomson < athomson@lookingglasscyber.com >, "Wunder, John" < jwunder@mitre.org >, Sergey Polzunov < sergey@eclecticiq.com >
    Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
    Subject: Re: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     


    I added these changes to 2.7 in suggestion mode.  Please review.   https://docs.google.com/document/d/1ShNq4c3e1CkfANmD9O--mdZ5H0O_GLnjN28a_yrEaco/edit#heading=h.ko24ggw4eq0q


     


    Also, please note, I added the term "Cyber Observables" to the list of objects that this ID will work for, that way we can delete the "extra" definition we had for just Cyber Observables.  This
    will then make all objects in STIX have the same definition for an ID.


     


     


    Bret


     





    From: Allan Thomson < athomson@lookingglasscyber.com >
    Sent: Friday, February 8, 2019 11:15 AM
    To: Wunder, John A.; Bret Jordan; Sergey Polzunov
    Cc: cti-stix@lists.oasis-open.org
    Subject: Re: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     




    Less is more.
     
    I agree with John. I didn t see that sentence on prior reading but agree its better to remove.
     

    Allan Thomson
    CTO ( +1-408-331-6646)
    LookingGlass Cyber Solutions

     

    From:
    "Wunder, John" < jwunder@mitre.org >
    Date: Friday, February 8, 2019 at 10:12 AM
    To: Bret Jordan < Bret_Jordan@symantec.com >, Allan Thomson < athomson@lookingglasscyber.com >, Sergey Polzunov < sergey@eclecticiq.com >
    Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
    Subject: Re: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     

    It s not a hill I would die on but the last sentence is very speculative: Some objects in STIX MAY define a list of properties that an organization MAY use for the name portion of the UUIDv5 hash, that MIGHT
    allow deterministic generation and verification of semantic equivalency.
     
    I think it just complicates things and IMO should be removed. If/when we choose to do that we could address it then.
     
    Other than that I like this change.
     
    John
     

    From:
    < cti-stix@lists.oasis-open.org > on behalf of "Bret Jordan (CS)" < Bret_Jordan@symantec.com >
    Date: Friday, February 8, 2019 at 1:08 PM
    To: Allan Thomson < athomson@lookingglasscyber.com >, Sergey Polzunov < sergey@eclecticiq.com >
    Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
    Subject: Re: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     


    Great... I will make that change.


     


    Any others?


     


    Bret





    From: Allan Thomson < athomson@lookingglasscyber.com >
    Sent: Friday, February 8, 2019 11:05 AM
    To: Bret Jordan; Sergey Polzunov
    Cc: cti-stix@lists.oasis-open.org
    Subject: Re: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     




    I suggest the statement on SHOULD regarding UUIDv5 be changed to MAY.
     
    There are multiple reasons why an organization may choose to use a different domain or some other identifier that is namespace unique. Simple reasons such as acquisition and M&A would cause the namespaces for a single entity to potentially
    be multiple values and an organization may choose to use a single name for the entire company even though they may have many sub-organizations with presence throughout the world.

     
    From
    For UUIDv5, the namespace SHOULD be the organization's fully qualified DNS name (example.com)
    To
    For UUIDv5, the namespace MAY be the organization's fully qualified DNS name (example.com) or some other organizational identifier that helps ensure uniqueness globally
     
     

    Allan Thomson
    CTO ( +1-408-331-6646)
    LookingGlass Cyber Solutions

     

    From:
    " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org > on behalf of Bret Jordan < Bret_Jordan@symantec.com >
    Date: Friday, February 8, 2019 at 8:28 AM
    To: Sergey Polzunov < sergey@eclecticiq.com >
    Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
    Subject: [cti-stix] Re: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     


    Is there anyone that would object to making this change to Part 1 of STIX 2.1?   Also, I think the way this is worded, it would work for the definition of deterministic-ids in the Cyber Observable
    proposal.   


     


    What do people think?


     


    Bret


     





    From: Sergey Polzunov < sergey@eclecticiq.com >
    Sent: Friday, February 8, 2019 6:15 AM
    To: Bret Jordan
    Cc: cti-stix@lists.oasis-open.org
    Subject: [EXT] Re: [cti-stix] [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier


     




    +1, good change

    Thanks,
    Sergey Polzunov
    EclecticIQ



    > On 7 Feb 2019, at 19:01, Bret Jordan < bret_jordan@symantec.com > wrote:
    >
    > I have been thinking about this a lot over the past few weeks.  There is a lot of history here, and my concerns always were ensuring that two organizations did not produce two different objects with the same ID. 

    >
    > I personally believe that the ID of the object should not be used to find semantic equivalence.  If we want to add another property for that, then great. 

    >
    > I would support the following updated text:
    >
    > ### BEGIN FROM Part1 Section 2.7
    >
    > An identifier universally and uniquely identifies a SDO, SRO, Bundle, Language Content, or Marking Definition. Identifiers MUST follow the form object-type--hash, where object-type is the exact value (all type names are lowercase strings, by definition) from
    the type property of the object being identified or referenced and where the hash is either UUIDv4 is an RFC 4122-compliant Version 4 UUID or a Version 5 UUID. The UUID MUST be generated according to the algorithm(s) defined in RFC 4122, section 4.4 (Version
    4 UUID) or section 4.3 (Version 5 UUID) [RFC4122]. For UUIDv5, the namespace SHOULD be the organization's fully qualified DNS name (example.com) and the name should be a value consistent within the organizations that guarantees that two different objects do
    not generate the same ID. For example, an organization MAY choose to use all properties, a subset of all properties, an organizational content identifier, a STIX 1.x identifier, or something else as string data for the name. Some objects in STIX MAY define
    a list of properties that an organization MAY use for the name portion of the UUIDv5 hash, that MIGHT allow deterministic generation and verification of semantic equivalency.

    >
    > ### END
    >
    > If we did this definition right, we could use the SAME identifier for both STIX content and Cyber Observables, thus preventing a lot of future confusion in the market.

    >
    >
    > Bret
    > From: cti-stix@lists.oasis-open.org < cti-stix@lists.oasis-open.org > on behalf of Jason Keirstead < Jason.Keirstead@ca.ibm.com >
    > Sent: Tuesday, February 5, 2019 6:06 AM
    > To: Patrick Maroney
    > Cc: cti-stix@lists.oasis-open.org ; John-Mark Gurney; Wunder, John A.; Sean Barnum; Sergey Polzunov
    > Subject: Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier

    > UUIDv5 enables this.  Those who want to use a specific deterministic ID can just form a consortium of agreement, choose a namespace for this consortium to use, and use that namespace among all of their products.
    >
    > -
    > Jason Keirstead
    > Lead Architect - IBM Security Connect
    >
    https://clicktime.symantec.com/37uHtFjAfPoNTZLcQL3iqoS7Vc?u=www.ibm.com%2Fsecurity
    >
    > "Things may come to those who wait, but only the things left by those who hustle." - Unknown

    >
    >
    >
    >
    > From:        Patrick Maroney < pmaroney@darklight.ai >
    > To:        "Wunder, John A." < jwunder@mitre.org >, Jason Keirstead < Jason.Keirstead@ca.ibm.com >, Sean Barnum < sean.barnum@FireEye.com >
    > Cc:        " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >, John-Mark Gurney < jmg@newcontext.com >,
    Sergey Polzunov < sergey@eclecticiq.com >
    > Date:        02/04/2019 05:38 PM
    > Subject:        Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier
    >
    >
    > We want distinct Deterministic ID UUIDs to discriminate between Sources of same/similar information and to better detect/prevent parroting and/or leakage.

    > A given of community of trust can establish conventions within the UUIDv5 encoding as required (e.g., just Namespace, Namespace+Community_ID, Namespace+Key).   We should only be using and asserting immutable objects and using the canonical representations
    suggested.   A number of UUIDv5 use cases were proposed historically including non-attributional Source Traceability.

    > However, none of this impacts those not wishing to participate in Deterministic Reference ID use cases, but empowers those of us who do.

    > Patrick Maroney
    > DarkLight
    > Mobile: (609)841-5104
    > Email:  patrick.maroney@darklight.ai

    >
    https://clicktime.symantec.com/3BM8wpmh4d2rTufAQWwDRcR7Vc?u=www.darklight.ai


    > From: < cti-stix@lists.oasis-open.org > on behalf of John Wunder < jwunder@mitre.org >
    > Date: Monday, February 4, 2019 at 3:40 PM
    > To: Jason Keirstead < Jason.Keirstead@ca.ibm.com >, Sean Barnum < sean.barnum@FireEye.com >
    > Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >, John-Mark Gurney < jmg@newcontext.com >, Sergey
    Polzunov < sergey@eclecticiq.com >
    > Subject: Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier

    > Yeah I agree with Jason here.

    > I don t think that would work for UUIDv5 anyway, since everyone will be using different namespaces even if you hash the data the same way you end up with different (non-transparent) IDs:

    > >>> uid = uuid.uuid4()
    > >>> uuid.uuid5(uid, "data")
    > UUID('c5bb29ba-4d85-5280-b202-7085b7485b62')
    > >>> uid2 = uuid.uuid4()
    > >>> uuid.uuid5(uid2, "data")
    > UUID('d09fafa9-ead3-54f1-b1c7-1d1d55f66fb7')

    > You can see that with the same data (just the string literal data ) you end up with different IDs. You could of course use the namespace of a different producer to try to back generate a UUIDv5 for an object they ve presumably created, but IMO that s a recipe
    for trouble (did they actually create that object?). Instead you should just rely on having received that object before and use whatever UUID it had.

    > It seems like the simplest thing is to just allow UUIDv5 via the various proposals we ve seen. It meets the use cases below and is a very straightforward change.

    > John

    > From: < cti-stix@lists.oasis-open.org > on behalf of Jason Keirstead < Jason.Keirstead@ca.ibm.com >
    > Date: Monday, February 4, 2019 at 3:35 PM
    > To: Sean Barnum < sean.barnum@FireEye.com >
    > Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >, John-Mark Gurney < jmg@newcontext.com >, John
    Wunder < jwunder@mitre.org >, Sergey Polzunov < sergey@eclecticiq.com >
    > Subject: Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier

    > I don't think that figuring out semantic equivalence should be a goal here - referring back to Sergey's original email, that is not why people are trying to do this.

    >
    > People want to do this so they can have a bi-directional traceability method from a STIX ID back and forth into an already-existing ID system.
    >
    > Forcing them to generate the IDs based on properties, defeats the whole purpose.
    >
    > -
    > Jason Keirstead
    > Lead Architect - IBM Security Connect
    >
    https://clicktime.symantec.com/37uHtFjAfPoNTZLcQL3iqoS7Vc?u=www.ibm.com%2Fsecurity
    >
    > "Things may come to those who wait, but only the things left by those who hustle." - Unknown

    >
    >
    >
    >
    > From:        Sean Barnum < sean.barnum@FireEye.com >
    > To:        Jason Keirstead < Jason.Keirstead@ca.ibm.com >
    > Cc:        " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >, John-Mark Gurney < jmg@newcontext.com >,
    "Wunder, John A." < jwunder@mitre.org >, Sergey Polzunov < sergey@eclecticiq.com >
    > Date:        02/04/2019 04:30 PM
    > Subject:        Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier
    > Sent by:        < cti-stix@lists.oasis-open.org >
    >
    >
    >
    >
    > It is to assist in semantic equivalence normalization across producers. Just like we have done for SCOs.
    >
    > The reality is that objects such as Locations, Identities, etc are likely to be repeated and widely used about as much as many Observables.
    > The ability to inherently converge on equivalence of things like Locations and Identities via UUIDv5 calculation is extremely valuable.
    >
    > Sean Barnum
    > Principal Architect
    > FireEye
    > M: 703.473.8262
    > E: sean.barnum@fireeye.com
    >
    > From: Jason Keirstead < Jason.Keirstead@ca.ibm.com >
    > Date: Monday, February 4, 2019 at 3:25 PM
    > To: Sean Barnum < sean.barnum@FireEye.com >
    > Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >, John-Mark Gurney < jmg@newcontext.com >, "Wunder,
    John A." < jwunder@mitre.org >, Sergey Polzunov < sergey@eclecticiq.com >
    > Subject: Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier
    >
    > Actually, I don't agree with this part.
    >
    > The entire point of UUIDv5, is that I should not care what method you use to compute your IDs - because it's in your namespace, so its not my problem anymore. I don't think we want to codify it.
    >
    > -
    > Jason Keirstead
    > Lead Architect - IBM Security Connect
    >
    https://clicktime.symantec.com/37uHtFjAfPoNTZLcQL3iqoS7Vc?u=www.ibm.com%2Fsecurity
    >
    > "Things may come to those who wait, but only the things left by those who hustle." - Unknown

    >
    >
    >
    >
    > From:        Sean Barnum < sean.barnum@FireEye.com >
    > To:        Jason Keirstead < Jason.Keirstead@ca.ibm.com >, John-Mark Gurney < jmg@newcontext.com >
    > Cc:        " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >, "Wunder, John A." < jwunder@mitre.org >,
    Sergey Polzunov < sergey@eclecticiq.com >
    > Date:        02/04/2019 04:23 PM
    > Subject:        Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier
    > Sent by:        < cti-stix@lists.oasis-open.org >
    >
    >
    >
    >
    >
    > Agree.
    >
    > And I would suggest we DO want the calculation to be done based upon the data from the object . As that is how we get value from such an ID.
    > It should just not be done on ALL properties of the object.
    > We just need to define the semantically relevant properties to use for the calculation. This is exactly what we have just done for SCOs.
    >
    > Sean Barnum
    > Principal Architect
    > FireEye
    > M: 703.473.8262
    > E: sean.barnum@fireeye.com
    >
    > From: < cti-stix@lists.oasis-open.org > on behalf of Jason Keirstead < Jason.Keirstead@ca.ibm.com >
    > Date: Monday, February 4, 2019 at 3:20 PM
    > To: John-Mark Gurney < jmg@newcontext.com >
    > Cc: " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >, "Wunder, John A." < jwunder@mitre.org >, Sergey
    Polzunov < sergey@eclecticiq.com >
    > Subject: Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier
    >
    > I would think we would want to use a DNS or URL namespace, would we not?
    >
    > -
    > Jason Keirstead
    > Lead Architect - IBM Security Connect
    >
    https://clicktime.symantec.com/37uHtFjAfPoNTZLcQL3iqoS7Vc?u=www.ibm.com%2Fsecurity
    >
    > "Things may come to those who wait, but only the things left by those who hustle." - Unknown

    >
    >
    >
    >
    > From:        John-Mark Gurney < jmg@newcontext.com >
    > To:        Jason Keirstead < Jason.Keirstead@ca.ibm.com >
    > Cc:        "Wunder, John A." < jwunder@mitre.org >, " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >,
    Sergey Polzunov < sergey@eclecticiq.com >
    > Date:        02/04/2019 04:10 PM
    > Subject:        Re: [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in STIX2 identifier
    >
    >
    >
    >
    >
    >
    > Jason Keirstead wrote this message on Mon, Feb 04, 2019 at 14:08 -0400:
    > > I would also support this.
    > >
    > > I have learned more about the inner workings of UUID4/5 and I don't have
    > > any reservations about it anymore. The odds of collision with a
    > > properly-implemented UUID5 are on-par with UUID4
    > >
    > > As far as John's comment below - all this means IMHO is the library has to
    > > force you to provide a namespace (ie make it a mandatory argument in your
    > > constructor or whatever).
    >
    > The one requirement I would like to make sure about UUIDv5 is that it
    > is NOT based upon the data from the object, otherwise versioning will
    > break.
    >
    > The reason we didn't use UUIDc4 as most of the proposals to use it was
    > to make it a hash of the contents, such as name and description, and
    > then update the UUID whenever the name and/or description changed..
    >
    > If we do this, the name space should probably be the identity of the
    > new STIX2 object.  This would prevent collisions from happening when
    > two entities try to create a "new" STIX2 object from a STIX1 object...
    >
    > > From:   "Wunder, John A." < jwunder@mitre.org >
    > > To:     Sergey Polzunov < sergey@eclecticiq.com >,

    > > " cti-stix@lists.oasis-open.org " < cti-stix@lists.oasis-open.org >
    > > Date:   02/04/2019 12:22 PM
    > > Subject:        [cti-stix] Re: [EXT] [cti-stix] ability to use UUID5 in
    > > STIX2 identifier
    > > Sent by:        < cti-stix@lists.oasis-open.org >
    > >
    > >
    > >
    > > I've been thinking a lot about this and I think it makes sense.
    > >
    > > One of the concerns we had at the time we chose UUID4 is that users of
    > > libraries like python-stix would need to remember to set the UUID5
    > > namespace -- or, if they don't and python-stix has some default namespace,
    > > different tools using the libraries could have overlapping IDs. This would
    > > also apply to users of the new Java libraries that I've seen come out. It
    > > might mean these libraries requiring that people set a unique namespace
    > > before creating any objects, vs. now where it can just go ahead and create
    > > IDs by default. I'd be curious what other people think about this problem
    > > and how we can help avoid it becoming an issue (especially given how many
    > > people use those libraries).
    > >
    > > John
    > >
    > > On 2/4/19, 11:12 AM, " cti-stix@lists.oasis-open.org on behalf of Sergey

    > > Polzunov " < cti-stix@lists.oasis-open.org on behalf of

    > > sergey@eclecticiq.com > wrote:
    > >
    > >     Hey everybody!
    > > 
    > >     Current STIX2 spec definition of an`identifier` for STIX2 objects is
    > > as follows:
    > > 
    > >     > An identifier universally and uniquely identifies a SDO, SRO,
    > > Bundle, or Marking Definition. Identifiers MUST follow the form
    > > object-type--UUIDv4, where object-type is the exact value (all type names
    > > are lowercase strings, by definition) from the type property of the object
    > > being identified or referenced and where the UUIDv4 is an RFC
    > > 4122-compliant Version 4 UUID. The UUID MUST be generated according to the
    > > algorithm(s) defined in RFC 4122, section 4.4 (Version 4 UUID) [RFC4122].
    > >     from
    > >
    https://clicktime.symantec.com/3SNmD57f7Xxv1JXt4mbkz5S7Vc?u=http%3A%2F%2Fdocs.oasis-open.org%2Fcti%2Fstix%2Fv2.0%2Fcs01%2Fpart1-stix-core%2Fstix-v2.0-cs01-part1-stix-core.html%23_Toc496709265
    > >
    > > 
    > >     I think the requirement to have UUID4 brings more problems than
    > > benefits. It makes STIX1->STIX2 transition difficult, hurting existing
    > > STIX1 users.
    > >     I will try to show it in these 2 use cases.
    > > 
    > > 
    > >     Use case 1
    > >     ----------
    > >         Imagine that I'm a client of an intelligence provider A. I've been
    > > a client for a long time and I have received intelligence in STIX1.2,
    > > which I stored in my DB. I fetch new intelligence daily, downloading only
    > > fresh data. Often fresh data links to old objects for context.
    > >         Provider A decides to upgrade and switch to STIX2. In addition to
    > > an old STIX1.2 feed, provider creates new STIX2 feed with the same data.
    > > In STIX2 all objects have new identifiers and Provider A does not bother
    > > to supply a mapping of STIX1.2 ids to STIX2 ids. Now I, as a client, have
    > > 2 options:
    > >         - clean slate option: drop all old data from this provider and
    > > re-fetch everything. That will work if Provider A is the only provider I
    > > use or if I never referenced Provider A's data from my own intelligence.
    > > Not a great plan.
    > >         - new era option: leave my STIX1.2 data graph in place and start
    > > consuming new STIX2 feed from today. This option has one big issue: new
    > > STIX2 data will not be connected to STIX1.2 data I already have, because
    > > STIX2 ids are all different. If I want to deduce connection, I need to
    > > deduplicate the data against my existing STIX1.2 DB. This means my
    > > ingestion pipeline must be smart enough to compare STIX1.2 objects to
    > > STIX2 objects and be fast enough to do that for every new STIX2 object.
    > > This will be difficult to implement and will have a huge performance
    > > penalty.
    > > 
    > >     Use case 2
    > >     ----------
    > >         Imagine that I'm a NCSC. I receive intelligence from providers,
    > > combine it and distribute it to my clients. My providers are still on
    > > STIX1.2 but my clients want STIX2, so I must convert STIX1.2 I receive
    > > into STIX2. Full STIX1.2 entities I can transform easily but what do I do
    > > with IDREFs I have in my STIX1.2 data?
    > >         I can generate new STIX2 id every time I see new STIX1.2 IDREF in
    > > incoming data and store STIX1.2->STIX2 mapping somewhere to be used next
    > > time I see this IDREF. This is painful and will require additional
    > > resources, but it is doable. But it will only work until the moment my
    > > providers switch to STIX2 and start sending me full objects for those
    > > IDREFs with new random STIX2 identifiers! I can not predict these
    > > identifiers and I can't match them with the ones I generated. So my
    > > thinking is - what is the point in even bothering with old IDREFs? I will
    > > just drop them, sending my clients sometimes disconnected STIX2 entities,
    > > hoping that they will figure it out.
    > > 
    > > 
    > >     Proposed solutions if UUID5 is allowed in STIX2 identifiers:
    > > 
    > >     Use case 1 solution
    > >     -------------------
    > >         There can be a guideline that will recommend providers to use old
    > > STIX1.2 IDs as input for new STIX2 identifiers. If STIX2 identifiers are
    > > predictable I, as a client, can greatly simplify my deduplication logic. I
    > > can run DB migration once to calculate STIX2 identifiers for all my
    > > STIX1.2 objects and use these on ingestion for deduplication. Appending
    > > STIX2 data to my STIX1.2 DB will be much easier.
    > >         I'm also interested in pushing Provider A to adopting this STIX2
    > > identifier generation practice because it will save me money.
    > > 
    > >     Use case 2 solution
    > >     -------------------
    > >     WIth UUID5 I have a way out: I can generate new STIX2 ids from old
    > > STIX1.2 ids! I can parse IDREF value, that looks like `[ns
    > > prefix]:[construct type]-[GUID]`, and use provider's namespace / construct
    > > type to build new STIX2 identifier. The logic will be like this:
    > >         - full IDREF will be input for UUID5 function
    > >         - for STIX1.2 types that were split (like TTP), I do not know
    > > exact STIX2 type Provider would use for old TTP. My solution here would be
    > > to play safe and create relations for all possible types: for IDREF to
    > > TTP, I will create 4 relations: one to a possible Tool object, one to
    > > Malware, one to Attack Pattern and one to Identity. It is an overhead but
    > > it is a small price for keeping interconnected intelligence graph.
    > >         Again, when time comes and my providers move to STIX2, I'm
    > > interested in pushing them to adopt this id generation schema for old
    > > objects, because it will save me, as NCSC, money.
    > > 
    > > 
    > >     To reiterate, I would like to propose:
    > >     - a change in STIX2 spec to allow both UUID5 and UUID4 to be used in
    > > an identifier of SDO, SRO, MarkingDefinition and Custom Object entities;
    > >     - creating a guideline, complimentary to the spec, that would explain
    > > how STIX1.2 ids can be transformed into STIX2 for easier transition.
    > > 
    > > 
    > >     Practicalities:
    > > 
    > >     UUID5 ids require use of a namespace. UUID5 RFC (
    > >
    https://clicktime.symantec.com/3VHevgXCxDKCASQXfexTG2y7Vc?u=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc4122%23section-4.3
    > > ) defines some generic namespaces (
    > >
    https://clicktime.symantec.com/39DSizcAyzjYR5MNQsVcequ7Vc?u=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc4122%23appendix-C
    > > ) but does not prohibits the use of custom ones. I suggest this algorithm:
    > >         - namespace UUID5 is generated by using predefined `NameSpace_URL`
    > > namespace and producer's URL;
    > >         - for old objects, GUID part of STIX2 identifier is namespaced
    > > UUID5 generated from old STIX1.2 id
    > >         - for new objects, GUID part of STIX2 identifier is either
    > > namespaced UUID5 with random UUID4 string, or just random UUID4.
    > > 
    > >     Example python code for generating UUID5 with custom namespace:
    > > 
    > >         In [1]: import uuid
    > >            ...:
    > >            ...: stix12_id =
    > > 'eclecticiq:threat-actor-07fa8672-4bca-46e1-a60f-023882b4a473'
    > >            ...: namespace_uuid = uuid.uuid5(uuid.NAMESPACE_URL,
    > > ' https://clicktime.symantec.com/3Twt3zW8FEEWaBXtubfTCpc7Vc?u=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__eclecticiq.com_ns%26d%3DDwIGaQ%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3Dk6Q07xZDujljzkKqZUfupXFUDIHGIiq-Sl_u1bw0hyA%26m%3DcvP-VddGmd9zTZUjb6OSCUczFxCjDL1cA586YiCE8YI%26s%3DtQbiU4LJBfzo5lmgDPo4k6EjM9ZKKwE6AzhNphzBRcM%26e%3D%27 )
    > >            ...: stix2_uuid = uuid.uuid5(namespace_uuid, stix12_id)
    > >            ...: stix2_id = 'threat-actor--{}'.format(stix2_uuid)
    > >            ...:
    > >            ...: print("new STIX2 id: {}".format(stix2_id))
    > > 
    > >         new STIX2 id: threat-actor--adee573a-12e9-5dd3-958b-0040d32c6b3e
    > > 
    > > 
    > >     BONUS: python functions to convert STIX1.2 IDREFs into STIX2
    > > identifiers -
    > >
    https://clicktime.symantec.com/3TtVUzeNYqg3BBswQN7bhLt7Vc?u=https%3A%2F%2Fgist.github.com%2Ftraut%2Ffd4b9b8de3c2aa0e161d68c4099656e5
    > >
    > > 
    > > 
    > >     Thank you,
    > >     Sergey Polzunov
    > >     EclecticIQ
    > > 
    > > 
    > > 
    > > 
    > >
    > >
    > >
    > >
    > >
    > >
    >
    > --
    > John-Mark
    >
    > This email and any attachments thereto may contain private, confidential, and/or privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited.
    If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.

    > This email and any attachments thereto may contain private, confidential, and/or privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited.
    If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.

    >
    >






    This email and any attachments thereto may contain private, confidential, and/or privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is
    strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.