OASIS Cyber Threat Intelligence (CTI) TC

 View Only
Expand all | Collapse all

Observable Patterning

  • 1.  Observable Patterning

    Posted 09-21-2015 17:48






    Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around Bernd’s suggested refactoring.





    I think there are a few broad questions around this topic that we can touch upon here: 


    1.
    What capabilities should STIX/CybOX patterning have?
    2.
    Where should STIX/CybOX patterning live?
    3.     Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects?


    As far as 1) I would say the following, as a straw man:



    * Conditional operators
        * AND
        * OR
        * NOT
    * Basic temporal operators
        * FOLLOWED_BY
        * WITHIN
    * String matching
        * EQUALS
        * DOES_NOT_EQUAL
        * CONTAINS
        * DOES_NOT_CONTAIN
    * Basic arithmetic operators
        * >
        * <
        * >=
        * <=
    * Regular Expressions
        * PCRE compatible
    * Variable substitution











    Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator
    context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. 








    With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific
    patterning language (necessary for more complex pattern expressions), and also give us the flexibility to move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else).



    [1]  https://github.com/CybOXProject/schemas/issues/381


    Regards,
    Ivan





  • 2.  Re: [cti] Observable Patterning

    Posted 09-21-2015 18:06
    " Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. " While I tend to agree, I am curious if moving patterning to the Indicator level (which implies removing existing patterning from CybOX) would affect other non-STIX uses of CybOX (such as the DFAX standard I recently found out about). If the non-STIX users of CybOX also need patterning, then it seems to push for patterning to be part of CybOX, not STIX. - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown "Kirillov, Ivan A." ---09/21/2015 02:47:51 PM---Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the From: "Kirillov, Ivan A." <ikirillov@mitre.org> To: "cti@lists.oasis-open.org" <cti@lists.oasis-open.org> Date: 09/21/2015 02:47 PM Subject: [cti] Observable Patterning Sent by: <cti@lists.oasis-open.org> Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around Bernd’s suggested refactoring. I think there are a few broad questions around this topic that we can touch upon here: 1. What capabilities should STIX/CybOX patterning have? 2. Where should STIX/CybOX patterning live? 3. Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects? As far as 1) I would say the following, as a straw man: * Conditional operators * AND * OR * NOT * Basic temporal operators * FOLLOWED_BY * WITHIN * String matching * EQUALS * DOES_NOT_EQUAL * CONTAINS * DOES_NOT_CONTAIN * Basic arithmetic operators * > * < * >= * <= * Regular Expressions * PCRE compatible * Variable substitution Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific patterning language (necessary for more complex pattern expressions), and also give us the flexibility to move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else). [1] https://github.com/CybOXProject/schemas/issues/381 Regards, Ivan




  • 3.  Re: [cti] Observable Patterning

    Posted 09-21-2015 18:33




    Agreed, that’s definitely the question we need to be asking with regards to where patterning should live. I think this ultimately comes down to whether having a single patterning implementation
    is useful in multiple contexts, i.e., if a CybOX pattern is used in a MAEC or DFAX instance (both users of CybOX), does it (or can it) signify something else than if the pattern is used in STIX instance? For example, a pattern could potentially be used to
    describe a set of filenames created by a particular malware instance in MAEC. However, this could also be done in a MAEC-specific manner, without needing all of the heavyweight structure and semantics of an Indicator-based pattern. My feeling is that trying
    to support different contexts with a single Indicator-focused pattern structure is going to be messy and more trouble than it’s worth.


    Regards,
    Ivan









    From: < cti@lists.oasis-open.org > on behalf of Jason Keirstead
    Date: Monday, September 21, 2015 at 2:04 PM
    To: Ivan Kirillov
    Cc: " cti@lists.oasis-open.org "
    Subject: Re: [cti] Observable Patterning





    " Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context.
    This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data.

    "

    While I tend to agree, I am curious if moving patterning to the Indicator level (which implies removing existing patterning from CybOX) would affect other non-STIX uses of CybOX (such as the DFAX standard I recently found out about). If the non-STIX users of
    CybOX also need patterning, then it seems to push for patterning to be part of CybOX, not STIX.

    -
    Jason Keirstead
    Product Architect, Security Intelligence, IBM Security Systems
    www.ibm.com/security www.securityintelligence.com

    Without data, all you are is just another person with an opinion - Unknown


    "Kirillov,
    Ivan A." ---09/21/2015 02:47:51 PM---Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the

    From: "Kirillov, Ivan A." < ikirillov@mitre.org >
    To: " cti@lists.oasis-open.org " < cti@lists.oasis-open.org >
    Date: 09/21/2015 02:47 PM
    Subject: [cti] Observable Patterning
    Sent by: < cti@lists.oasis-open.org >





    Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around Bernd’s suggested refactoring.

    I think there are a few broad questions around this topic that we can touch upon here:


    1. What capabilities should STIX/CybOX patterning have?
    2. Where should STIX/CybOX patterning live?
    3. Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects?

    As far as 1) I would say the following, as a straw man:

    * Conditional operators
    * AND
    * OR
    * NOT
    * Basic temporal operators
    * FOLLOWED_BY
    * WITHIN
    * String matching
    * EQUALS
    * DOES_NOT_EQUAL
    * CONTAINS
    * DOES_NOT_CONTAIN
    * Basic arithmetic operators
    * >
    * <
    * >=
    * <=
    * Regular Expressions
    * PCRE compatible
    * Variable substitution

    Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context.
    This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data.


    With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific
    patterning language (necessary for more complex pattern expressions), and also give us the flexibility to move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else).

    [1] https://github.com/CybOXProject/schemas/issues/381

    Regards,
    Ivan









  • 4.  Re: [cti] Observable Patterning

    Posted 09-21-2015 18:49
    Agreed. Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg.   On Sep 21, 2015, at 12:33, Kirillov, Ivan A. < ikirillov@mitre.org > wrote: Agreed, that’s definitely the question we need to be asking with regards to where patterning should live. I think this ultimately comes down to whether having a single patterning implementation is useful in multiple contexts, i.e., if a CybOX pattern is used in a MAEC or DFAX instance (both users of CybOX), does it (or can it) signify something else than if the pattern is used in STIX instance? For example, a pattern could potentially be used to describe a set of filenames created by a particular malware instance in MAEC. However, this could also be done in a MAEC-specific manner, without needing all of the heavyweight structure and semantics of an Indicator-based pattern. My feeling is that trying to support different contexts with a single Indicator-focused pattern structure is going to be messy and more trouble than it’s worth. Regards, Ivan From: < cti@lists.oasis-open.org > on behalf of Jason Keirstead Date: Monday, September 21, 2015 at 2:04 PM To: Ivan Kirillov Cc: cti@lists.oasis-open.org Subject: Re: [cti] Observable Patterning Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. While I tend to agree, I am curious if moving patterning to the Indicator level (which implies removing existing patterning from CybOX) would affect other non-STIX uses of CybOX (such as the DFAX standard I recently found out about). If the non-STIX users of CybOX also need patterning, then it seems to push for patterning to be part of CybOX, not STIX. - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown <graycol.gif> Kirillov, Ivan A. ---09/21/2015 02:47:51 PM---Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the From: Kirillov, Ivan A. < ikirillov@mitre.org > To: cti@lists.oasis-open.org < cti@lists.oasis-open.org > Date: 09/21/2015 02:47 PM Subject: [cti] Observable Patterning Sent by: < cti@lists.oasis-open.org > Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around Bernd’s suggested refactoring. I think there are a few broad questions around this topic that we can touch upon here: 1. What capabilities should STIX/CybOX patterning have? 2. Where should STIX/CybOX patterning live? 3. Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects? As far as 1) I would say the following, as a straw man: * Conditional operators * AND * OR * NOT * Basic temporal operators * FOLLOWED_BY * WITHIN * String matching * EQUALS * DOES_NOT_EQUAL * CONTAINS * DOES_NOT_CONTAIN * Basic arithmetic operators * > * < * >= * <= * Regular Expressions * PCRE compatible * Variable substitution Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific patterning language (necessary for more complex pattern expressions), and also give us the flexibility to move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else). [1] https://github.com/CybOXProject/schemas/issues/381 Regards, Ivan <graycol.gif> --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail.  Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php Attachment: signature.asc Description: Message signed with OpenPGP using GPGMail


  • 5.  Re: [cti] Observable Patterning

    Posted 09-21-2015 22:40
    I would have to agree In principal this sounds like a sensible idea, but I share the same concerns that Jason has around how this will have knock-on effects with other downstream protocols. Ivan, I liked your issue #381 ( https://github.com/CybOXProject/schemas/issues/381 ). It does make sense to pull out the 'patterning' structure to a separate structure, and I agree that having it within STIX makes sense. I think a nice STIX alignment could be as follows: top-level STIX Indicator object that uses the patterning construct described in #381 to describe what people should look out for (describe clues to monitor for) new top-level STIX Sighting object that uses CyboX Observable Instances to describe what people have seen (describing history) This would give a nice separation of functionality between a STIX Indicator and Sighting, making it easier for people to add third-party relationships between sightings and indicators, and would make it much easier for people learning STIX to get their head around.  The patterning construct in #381 reminds me quite a bit of the STIX ObservableComposition - possibly further confirmation that STIX is the right place for it to sit? Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com On 22 September 2015 at 04:33, Kirillov, Ivan A. < ikirillov@mitre.org > wrote: Agreed, that’s definitely the question we need to be asking with regards to where patterning should live. I think this ultimately comes down to whether having a single patterning implementation is useful in multiple contexts, i.e., if a CybOX pattern is used in a MAEC or DFAX instance (both users of CybOX), does it (or can it) signify something else than if the pattern is used in STIX instance? For example, a pattern could potentially be used to describe a set of filenames created by a particular malware instance in MAEC. However, this could also be done in a MAEC-specific manner, without needing all of the heavyweight structure and semantics of an Indicator-based pattern. My feeling is that trying to support different contexts with a single Indicator-focused pattern structure is going to be messy and more trouble than it’s worth. Regards, Ivan From: < cti@lists.oasis-open.org > on behalf of Jason Keirstead Date: Monday, September 21, 2015 at 2:04 PM To: Ivan Kirillov Cc: " cti@lists.oasis-open.org " Subject: Re: [cti] Observable Patterning " Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. " While I tend to agree, I am curious if moving patterning to the Indicator level (which implies removing existing patterning from CybOX) would affect other non-STIX uses of CybOX (such as the DFAX standard I recently found out about). If the non-STIX users of CybOX also need patterning, then it seems to push for patterning to be part of CybOX, not STIX. - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown "Kirillov, Ivan A." ---09/21/2015 02:47:51 PM---Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the From: "Kirillov, Ivan A." < ikirillov@mitre.org > To: " cti@lists.oasis-open.org " < cti@lists.oasis-open.org > Date: 09/21/2015 02:47 PM Subject: [cti] Observable Patterning Sent by: < cti@lists.oasis-open.org > Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around Bernd’s suggested refactoring. I think there are a few broad questions around this topic that we can touch upon here: 1. What capabilities should STIX/CybOX patterning have? 2. Where should STIX/CybOX patterning live? 3. Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects? As far as 1) I would say the following, as a straw man: * Conditional operators * AND * OR * NOT * Basic temporal operators * FOLLOWED_BY * WITHIN * String matching * EQUALS * DOES_NOT_EQUAL * CONTAINS * DOES_NOT_CONTAIN * Basic arithmetic operators * > * < * >= * <= * Regular Expressions * PCRE compatible * Variable substitution Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific patterning language (necessary for more complex pattern expressions), and also give us the flexibility to move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else). [1] https://github.com/CybOXProject/schemas/issues/381 Regards, Ivan --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail.  Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php


  • 6.  Re: [cti] Observable Patterning

    Posted 09-21-2015 22:59
    The only concern I have is making sure the sighting object does not begin to look like an incident object.    Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg.   On Sep 21, 2015, at 16:39, Terry MacDonald < terry.macdonald@threatloop.com > wrote: I would have to agree In principal this sounds like a sensible idea, but I share the same concerns that Jason has around how this will have knock-on effects with other downstream protocols. Ivan, I liked your issue #381 ( https://github.com/CybOXProject/schemas/issues/381 ). It does make sense to pull out the 'patterning' structure to a separate structure, and I agree that having it within STIX makes sense. I think a nice STIX alignment could be as follows: top-level STIX Indicator object that uses the patterning construct described in #381 to describe what people should look out for (describe clues to monitor for) new top-level STIX Sighting object that uses CyboX Observable Instances to describe what people have seen (describing history) This would give a nice separation of functionality between a STIX Indicator and Sighting, making it easier for people to add third-party relationships between sightings and indicators, and would make it much easier for people learning STIX to get their head around.  The patterning construct in #381 reminds me quite a bit of the STIX ObservableComposition - possibly further confirmation that STIX is the right place for it to sit? Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com On 22 September 2015 at 04:33, Kirillov, Ivan A. < ikirillov@mitre.org > wrote: Agreed, that’s definitely the question we need to be asking with regards to where patterning should live. I think this ultimately comes down to whether having a single patterning implementation is useful in multiple contexts, i.e., if a CybOX pattern is used in a MAEC or DFAX instance (both users of CybOX), does it (or can it) signify something else than if the pattern is used in STIX instance? For example, a pattern could potentially be used to describe a set of filenames created by a particular malware instance in MAEC. However, this could also be done in a MAEC-specific manner, without needing all of the heavyweight structure and semantics of an Indicator-based pattern. My feeling is that trying to support different contexts with a single Indicator-focused pattern structure is going to be messy and more trouble than it’s worth. Regards, Ivan From: < cti@lists.oasis-open.org > on behalf of Jason Keirstead Date: Monday, September 21, 2015 at 2:04 PM To: Ivan Kirillov Cc: cti@lists.oasis-open.org Subject: Re: [cti] Observable Patterning Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. While I tend to agree, I am curious if moving patterning to the Indicator level (which implies removing existing patterning from CybOX) would affect other non-STIX uses of CybOX (such as the DFAX standard I recently found out about). If the non-STIX users of CybOX also need patterning, then it seems to push for patterning to be part of CybOX, not STIX. - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown <graycol.gif> Kirillov, Ivan A. ---09/21/2015 02:47:51 PM---Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the From: Kirillov, Ivan A. < ikirillov@mitre.org > To: cti@lists.oasis-open.org < cti@lists.oasis-open.org > Date: 09/21/2015 02:47 PM Subject: [cti] Observable Patterning Sent by: < cti@lists.oasis-open.org > Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around Bernd’s suggested refactoring. I think there are a few broad questions around this topic that we can touch upon here: 1. What capabilities should STIX/CybOX patterning have? 2. Where should STIX/CybOX patterning live? 3. Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects? As far as 1) I would say the following, as a straw man: * Conditional operators * AND * OR * NOT * Basic temporal operators * FOLLOWED_BY * WITHIN * String matching * EQUALS * DOES_NOT_EQUAL * CONTAINS * DOES_NOT_CONTAIN * Basic arithmetic operators * > * < * >= * <= * Regular Expressions * PCRE compatible * Variable substitution Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific patterning language (necessary for more complex pattern expressions), and also give us the flexibility to move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else). [1] https://github.com/CybOXProject/schemas/issues/381 Regards, Ivan --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail.  Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php Attachment: signature.asc Description: Message signed with OpenPGP using GPGMail


  • 7.  Re: [cti] Observable Patterning

    Posted 09-21-2015 23:20
    Hi Bret, A good point. The rationale behind the Sighting object is specifically just so that people can report what they've seen. The Incident object represents the fact that the producing organization officially responded in some way; that their Incident Response process was enacted. I see an Incident object as a collection of details describing a situation, and one of the details collected is the list of Sightings objects that are part of the Incident. The Sighting object would be related from the Incident object with the top-level relationship object I've been banging on about for the last year :). This would allow producers to easily add new Sightings to existing ongoing Incidents as simply as sending the Sighting object and a corresponding Relationship object describing the link. It would also open up the possibility of 'shared Incidents', where a group could all pool together their Sightings, patterns, etc to better describe and track a global incident of some kind (think Shellshock or some other vulnerability with a logo). Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com On 22 September 2015 at 08:58, Jordan, Bret < bret.jordan@bluecoat.com > wrote: The only concern I have is making sure the sighting object does not begin to look like an incident object.    Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."  On Sep 21, 2015, at 16:39, Terry MacDonald < terry.macdonald@threatloop.com > wrote: I would have to agree In principal this sounds like a sensible idea, but I share the same concerns that Jason has around how this will have knock-on effects with other downstream protocols. Ivan, I liked your issue #381 ( https://github.com/CybOXProject/schemas/issues/381 ). It does make sense to pull out the 'patterning' structure to a separate structure, and I agree that having it within STIX makes sense. I think a nice STIX alignment could be as follows: top-level STIX Indicator object that uses the patterning construct described in #381 to describe what people should look out for (describe clues to monitor for) new top-level STIX Sighting object that uses CyboX Observable Instances to describe what people have seen (describing history) This would give a nice separation of functionality between a STIX Indicator and Sighting, making it easier for people to add third-party relationships between sightings and indicators, and would make it much easier for people learning STIX to get their head around.  The patterning construct in #381 reminds me quite a bit of the STIX ObservableComposition - possibly further confirmation that STIX is the right place for it to sit? Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com On 22 September 2015 at 04:33, Kirillov, Ivan A. < ikirillov@mitre.org > wrote: Agreed, that’s definitely the question we need to be asking with regards to where patterning should live. I think this ultimately comes down to whether having a single patterning implementation is useful in multiple contexts, i.e., if a CybOX pattern is used in a MAEC or DFAX instance (both users of CybOX), does it (or can it) signify something else than if the pattern is used in STIX instance? For example, a pattern could potentially be used to describe a set of filenames created by a particular malware instance in MAEC. However, this could also be done in a MAEC-specific manner, without needing all of the heavyweight structure and semantics of an Indicator-based pattern. My feeling is that trying to support different contexts with a single Indicator-focused pattern structure is going to be messy and more trouble than it’s worth. Regards, Ivan From: < cti@lists.oasis-open.org > on behalf of Jason Keirstead Date: Monday, September 21, 2015 at 2:04 PM To: Ivan Kirillov Cc: " cti@lists.oasis-open.org " Subject: Re: [cti] Observable Patterning " Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. " While I tend to agree, I am curious if moving patterning to the Indicator level (which implies removing existing patterning from CybOX) would affect other non-STIX uses of CybOX (such as the DFAX standard I recently found out about). If the non-STIX users of CybOX also need patterning, then it seems to push for patterning to be part of CybOX, not STIX. - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown <graycol.gif> "Kirillov, Ivan A." ---09/21/2015 02:47:51 PM---Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the From: "Kirillov, Ivan A." < ikirillov@mitre.org > To: " cti@lists.oasis-open.org " < cti@lists.oasis-open.org > Date: 09/21/2015 02:47 PM Subject: [cti] Observable Patterning Sent by: < cti@lists.oasis-open.org > Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around Bernd’s suggested refactoring. I think there are a few broad questions around this topic that we can touch upon here: 1. What capabilities should STIX/CybOX patterning have? 2. Where should STIX/CybOX patterning live? 3. Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects? As far as 1) I would say the following, as a straw man: * Conditional operators * AND * OR * NOT * Basic temporal operators * FOLLOWED_BY * WITHIN * String matching * EQUALS * DOES_NOT_EQUAL * CONTAINS * DOES_NOT_CONTAIN * Basic arithmetic operators * > * < * >= * <= * Regular Expressions * PCRE compatible * Variable substitution Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific patterning language (necessary for more complex pattern expressions), and also give us the flexibility to move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else). [1] https://github.com/CybOXProject/schemas/issues/381 Regards, Ivan --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail.  Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php


  • 8.  Re: [cti] Observable Patterning

    Posted 09-22-2015 02:25
    Yes, that is my hope as well.  We will just need to be careful of scope creep on things that get added to the Sighting object.  If we are not careful, it could turn in to 70% of an incident object.   Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg.   On Sep 21, 2015, at 17:19, Terry MacDonald < terry.macdonald@threatloop.com > wrote: Hi Bret, A good point. The rationale behind the Sighting object is specifically just so that people can report what they've seen. The Incident object represents the fact that the producing organization officially responded in some way; that their Incident Response process was enacted. I see an Incident object as a collection of details describing a situation, and one of the details collected is the list of Sightings objects that are part of the Incident. The Sighting object would be related from the Incident object with the top-level relationship object I've been banging on about for the last year :). This would allow producers to easily add new Sightings to existing ongoing Incidents as simply as sending the Sighting object and a corresponding Relationship object describing the link. It would also open up the possibility of 'shared Incidents', where a group could all pool together their Sightings, patterns, etc to better describe and track a global incident of some kind (think Shellshock or some other vulnerability with a logo). Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com On 22 September 2015 at 08:58, Jordan, Bret < bret.jordan@bluecoat.com > wrote: The only concern I have is making sure the sighting object does not begin to look like an incident object.    Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg.   On Sep 21, 2015, at 16:39, Terry MacDonald < terry.macdonald@threatloop.com > wrote: I would have to agree In principal this sounds like a sensible idea, but I share the same concerns that Jason has around how this will have knock-on effects with other downstream protocols. Ivan, I liked your issue #381 ( https://github.com/CybOXProject/schemas/issues/381 ). It does make sense to pull out the 'patterning' structure to a separate structure, and I agree that having it within STIX makes sense. I think a nice STIX alignment could be as follows: top-level STIX Indicator object that uses the patterning construct described in #381 to describe what people should look out for (describe clues to monitor for) new top-level STIX Sighting object that uses CyboX Observable Instances to describe what people have seen (describing history) This would give a nice separation of functionality between a STIX Indicator and Sighting, making it easier for people to add third-party relationships between sightings and indicators, and would make it much easier for people learning STIX to get their head around.  The patterning construct in #381 reminds me quite a bit of the STIX ObservableComposition - possibly further confirmation that STIX is the right place for it to sit? Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com On 22 September 2015 at 04:33, Kirillov, Ivan A. < ikirillov@mitre.org > wrote: Agreed, that’s definitely the question we need to be asking with regards to where patterning should live. I think this ultimately comes down to whether having a single patterning implementation is useful in multiple contexts, i.e., if a CybOX pattern is used in a MAEC or DFAX instance (both users of CybOX), does it (or can it) signify something else than if the pattern is used in STIX instance? For example, a pattern could potentially be used to describe a set of filenames created by a particular malware instance in MAEC. However, this could also be done in a MAEC-specific manner, without needing all of the heavyweight structure and semantics of an Indicator-based pattern. My feeling is that trying to support different contexts with a single Indicator-focused pattern structure is going to be messy and more trouble than it’s worth. Regards, Ivan From: < cti@lists.oasis-open.org > on behalf of Jason Keirstead Date: Monday, September 21, 2015 at 2:04 PM To: Ivan Kirillov Cc: cti@lists.oasis-open.org Subject: Re: [cti] Observable Patterning Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. While I tend to agree, I am curious if moving patterning to the Indicator level (which implies removing existing patterning from CybOX) would affect other non-STIX uses of CybOX (such as the DFAX standard I recently found out about). If the non-STIX users of CybOX also need patterning, then it seems to push for patterning to be part of CybOX, not STIX. - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown <graycol.gif> Kirillov, Ivan A. ---09/21/2015 02:47:51 PM---Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the From: Kirillov, Ivan A. < ikirillov@mitre.org > To: cti@lists.oasis-open.org < cti@lists.oasis-open.org > Date: 09/21/2015 02:47 PM Subject: [cti] Observable Patterning Sent by: < cti@lists.oasis-open.org > Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around Bernd’s suggested refactoring. I think there are a few broad questions around this topic that we can touch upon here: 1. What capabilities should STIX/CybOX patterning have? 2. Where should STIX/CybOX patterning live? 3. Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects? As far as 1) I would say the following, as a straw man: * Conditional operators * AND * OR * NOT * Basic temporal operators * FOLLOWED_BY * WITHIN * String matching * EQUALS * DOES_NOT_EQUAL * CONTAINS * DOES_NOT_CONTAIN * Basic arithmetic operators * > * < * >= * <= * Regular Expressions * PCRE compatible * Variable substitution Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific patterning language (necessary for more complex pattern expressions), and also give us the flexibility to move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else). [1] https://github.com/CybOXProject/schemas/issues/381 Regards, Ivan --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail.  Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php Attachment: signature.asc Description: Message signed with OpenPGP using GPGMail


  • 9.  Re: [cti] Observable Patterning

    Posted 09-22-2015 03:41
    I suggest discussing Sighting in another Thread and Tracker# Meantime, I think that if we clearly differentiate Observable Instances and Patterns, that would make things easier/clearer to describe static things (Observables) and dynamic 'events' (described as patterns) observed happening on these static observables The sighting would become more interesting at the dynamic side Ref.  https://stixproject.github.io/data-model/1.2/indicator/SightingType/ (when it's not just a counter. Ref.  @sighting_count  https://stixproject.github.io/data-model/1.2/cybox/ObservableType/ ) maybe an argument to have the patterning at STIX level 2015-09-22 5:25 GMT+03:00 Jordan, Bret < bret.jordan@bluecoat.com > : Yes, that is my hope as well.  We will just need to be careful of scope creep on things that get added to the Sighting object.  If we are not careful, it could turn in to 70% of an incident object.   Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."  On Sep 21, 2015, at 17:19, Terry MacDonald < terry.macdonald@threatloop.com > wrote: Hi Bret, A good point. The rationale behind the Sighting object is specifically just so that people can report what they've seen. The Incident object represents the fact that the producing organization officially responded in some way; that their Incident Response process was enacted. I see an Incident object as a collection of details describing a situation, and one of the details collected is the list of Sightings objects that are part of the Incident. The Sighting object would be related from the Incident object with the top-level relationship object I've been banging on about for the last year :). This would allow producers to easily add new Sightings to existing ongoing Incidents as simply as sending the Sighting object and a corresponding Relationship object describing the link. It would also open up the possibility of 'shared Incidents', where a group could all pool together their Sightings, patterns, etc to better describe and track a global incident of some kind (think Shellshock or some other vulnerability with a logo). Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com On 22 September 2015 at 08:58, Jordan, Bret < bret.jordan@bluecoat.com > wrote: The only concern I have is making sure the sighting object does not begin to look like an incident object.    Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."  On Sep 21, 2015, at 16:39, Terry MacDonald < terry.macdonald@threatloop.com > wrote: I would have to agree In principal this sounds like a sensible idea, but I share the same concerns that Jason has around how this will have knock-on effects with other downstream protocols. Ivan, I liked your issue #381 ( https://github.com/CybOXProject/schemas/issues/381 ). It does make sense to pull out the 'patterning' structure to a separate structure, and I agree that having it within STIX makes sense. I think a nice STIX alignment could be as follows: top-level STIX Indicator object that uses the patterning construct described in #381 to describe what people should look out for (describe clues to monitor for) new top-level STIX Sighting object that uses CyboX Observable Instances to describe what people have seen (describing history) This would give a nice separation of functionality between a STIX Indicator and Sighting, making it easier for people to add third-party relationships between sightings and indicators, and would make it much easier for people learning STIX to get their head around.  The patterning construct in #381 reminds me quite a bit of the STIX ObservableComposition - possibly further confirmation that STIX is the right place for it to sit? Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com On 22 September 2015 at 04:33, Kirillov, Ivan A. < ikirillov@mitre.org > wrote: Agreed, that’s definitely the question we need to be asking with regards to where patterning should live. I think this ultimately comes down to whether having a single patterning implementation is useful in multiple contexts, i.e., if a CybOX pattern is used in a MAEC or DFAX instance (both users of CybOX), does it (or can it) signify something else than if the pattern is used in STIX instance? For example, a pattern could potentially be used to describe a set of filenames created by a particular malware instance in MAEC. However, this could also be done in a MAEC-specific manner, without needing all of the heavyweight structure and semantics of an Indicator-based pattern. My feeling is that trying to support different contexts with a single Indicator-focused pattern structure is going to be messy and more trouble than it’s worth. Regards, Ivan From: < cti@lists.oasis-open.org > on behalf of Jason Keirstead Date: Monday, September 21, 2015 at 2:04 PM To: Ivan Kirillov Cc: " cti@lists.oasis-open.org " Subject: Re: [cti] Observable Patterning " Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. " While I tend to agree, I am curious if moving patterning to the Indicator level (which implies removing existing patterning from CybOX) would affect other non-STIX uses of CybOX (such as the DFAX standard I recently found out about). If the non-STIX users of CybOX also need patterning, then it seems to push for patterning to be part of CybOX, not STIX. - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown <graycol.gif> "Kirillov, Ivan A." ---09/21/2015 02:47:51 PM---Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the From: "Kirillov, Ivan A." < ikirillov@mitre.org > To: " cti@lists.oasis-open.org " < cti@lists.oasis-open.org > Date: 09/21/2015 02:47 PM Subject: [cti] Observable Patterning Sent by: < cti@lists.oasis-open.org > Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around Bernd’s suggested refactoring. I think there are a few broad questions around this topic that we can touch upon here: 1. What capabilities should STIX/CybOX patterning have? 2. Where should STIX/CybOX patterning live? 3. Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects? As far as 1) I would say the following, as a straw man: * Conditional operators * AND * OR * NOT * Basic temporal operators * FOLLOWED_BY * WITHIN * String matching * EQUALS * DOES_NOT_EQUAL * CONTAINS * DOES_NOT_CONTAIN * Basic arithmetic operators * > * < * >= * <= * Regular Expressions * PCRE compatible * Variable substitution Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific patterning language (necessary for more complex pattern expressions), and also give us the flexibility to move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else). [1] https://github.com/CybOXProject/schemas/issues/381 Regards, Ivan --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail.  Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php


  • 10.  Re: [cti] Observable Patterning

    Posted 09-24-2015 08:26
    Um, nobody is suggesting that the CybOX 2.x / STIX 1.x standards are going to be deleted from the internet, guys. DFAX, MAEC, et al are *not* OASIS standards. If we're going to constrain what with do inside OASIS based on the perceived needs of non-OASIS standards, then we're doomed. I could just as easily make up a standard (call it FooBAR), define it in such a way that it relies on some totally whacky STIX construct that everybody else wants to eliminate in STIX 2.0, and suddenly, what, OASIS-CTI is going to be magically constrained because my random standard uses something weird nobody else cares about? No thanks! The DFAX, MAEC, et al communities are more than welcome to a) join OASIS and voice their own arguments or b) continue using the CybOX 2.x / STIX 1.x standards as the exist today. Cheers, Trey -- Trey Darley Senior Security Engineer Soltra An FS-ISAC & DTCC Company www.soltra.com From: cti@lists.oasis-open.org <cti@lists.oasis-open.org> on behalf of Jason Keirstead <Jason.Keirstead@ca.ibm.com> Sent: Monday, September 21, 2015 20:04 To: Kirillov, Ivan A. Cc: cti@lists.oasis-open.org Subject: Re: [cti] Observable Patterning   " Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. " While I tend to agree, I am curious if moving patterning to the Indicator level (which implies removing existing patterning from CybOX) would affect other non-STIX uses of CybOX (such as the DFAX standard I recently found out about). If the non-STIX users of CybOX also need patterning, then it seems to push for patterning to be part of CybOX, not STIX. - Jason Keirstead Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown "Kirillov, Ivan A." ---09/21/2015 02:47:51 PM---Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the From: "Kirillov, Ivan A." <ikirillov@mitre.org> To: "cti@lists.oasis-open.org" <cti@lists.oasis-open.org> Date: 09/21/2015 02:47 PM Subject: [cti] Observable Patterning Sent by: <cti@lists.oasis-open.org> Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around Bernd’s suggested refactoring. I think there are a few broad questions around this topic that we can touch upon here: 1. What capabilities should STIX/CybOX patterning have? 2. Where should STIX/CybOX patterning live? 3. Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects? As far as 1) I would say the following, as a straw man: * Conditional operators * AND * OR * NOT * Basic temporal operators * FOLLOWED_BY * WITHIN * String matching * EQUALS * DOES_NOT_EQUAL * CONTAINS * DOES_NOT_CONTAIN * Basic arithmetic operators * > * < * >= * <= * Regular Expressions * PCRE compatible * Variable substitution Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific patterning language (necessary for more complex pattern expressions), and also give us the flexibility to move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else). [1] https://github.com/CybOXProject/schemas/issues/381 Separate Patterns and Instances in CybOX Observables and Objects · Issue #381 · CybOXProject/schemas Currently, a CybOX Observable, in combination with an Object, can define either an observed instance of some data (e.g., a file), or a pattern for detecting some data. While this duality has its us... Read more... Regards, Ivan


  • 11.  Re: [cti] Observable Patterning

    Posted 09-21-2015 18:10
    Thanks for the new thread and tracker. We already have the CybOX Object Relationships Vocabulary https://cyboxproject.github.io/documentation/object-relationships/ and ConditionTypeEnum Equals DoesNotEqual Contains DoesNotContain StartsWith EndsWith GreaterThan GreaterThanOrEqual LessThan LessThanOrEqual InclusiveBetween ExclusiveBetween FitsPattern BitwiseAnd BitwiseOr BitwiseXor ConditionApplicationEnum ANY ALL NONE I think we should focus on the temporal elements, and when/where/how to put them. 2015-09-21 20:47 GMT+03:00 Kirillov, Ivan A. <ikirillov@mitre.org>: > Relaunching this as a separate thread based on Jerome’s suggestion, so as to > not further hijack the discussion around Bernd’s suggested refactoring. > > I think there are a few broad questions around this topic that we can touch > upon here: > > 1. What capabilities should STIX/CybOX patterning have? > 2. Where should STIX/CybOX patterning live? > 3. Should patterns have their own implementation/structure, or continue > to be embedded in Observables/Objects? > > As far as 1) I would say the following, as a straw man: > > * Conditional operators > * AND > * OR > * NOT > * Basic temporal operators > * FOLLOWED_BY > * WITHIN > * String matching > * EQUALS > * DOES_NOT_EQUAL > * CONTAINS > * DOES_NOT_CONTAIN > * Basic arithmetic operators > * > > * < > * >= > * <= > * Regular Expressions > * PCRE compatible > * Variable substitution > > Regarding 2), it seems to me that patterning is inherent to Indicators, and > thus should be part of STIX. I just can’t see the use case for a CybOX > Observable with a pattern that stands on its own, or is used in a > non-Indicator context. This would also serve to greatly simplify CybOX, as > it would mean that CybOX can ONLY capture instances of cyber data. > > With regards to 3), I see lots of advantages to having a separate structure > for patterns, which I’ve documented in the corresponding CybOX issue [1]. > The short of it is that it will get rid of the instance/pattern duality, > allow for the creation of a domain-specific patterning language (necessary > for more complex pattern expressions), and also give us the flexibility to > move patterning out to wherever makes the most sense (whether it’s STIX or > somewhere else). > > [1] https://github.com/CybOXProject/schemas/issues/381 > > Regards, > Ivan


  • 12.  Re: [cti] Observable Patterning

    Posted 09-22-2015 04:01
    I just would like to highlight (again) the similarities between the example described on github and the OVAL language see end of this page https://oval.mitre.org/language/about/definition.html while an Observable is an 'object with a state', the test/check/pattern is separated from the list of 'static observables' (Note that I am not suggesting using OVAL, but we could inspire from some of its mechanisms) 2015-09-21 20:47 GMT+03:00 Kirillov, Ivan A. <ikirillov@mitre.org>: > Relaunching this as a separate thread based on Jerome’s suggestion, so as to > not further hijack the discussion around Bernd’s suggested refactoring. > > I think there are a few broad questions around this topic that we can touch > upon here: > > 1. What capabilities should STIX/CybOX patterning have? > 2. Where should STIX/CybOX patterning live? > 3. Should patterns have their own implementation/structure, or continue > to be embedded in Observables/Objects? > > As far as 1) I would say the following, as a straw man: > > * Conditional operators > * AND > * OR > * NOT > * Basic temporal operators > * FOLLOWED_BY > * WITHIN > * String matching > * EQUALS > * DOES_NOT_EQUAL > * CONTAINS > * DOES_NOT_CONTAIN > * Basic arithmetic operators > * > > * < > * >= > * <= > * Regular Expressions > * PCRE compatible > * Variable substitution > > Regarding 2), it seems to me that patterning is inherent to Indicators, and > thus should be part of STIX. I just can’t see the use case for a CybOX > Observable with a pattern that stands on its own, or is used in a > non-Indicator context. This would also serve to greatly simplify CybOX, as > it would mean that CybOX can ONLY capture instances of cyber data. > > With regards to 3), I see lots of advantages to having a separate structure > for patterns, which I’ve documented in the corresponding CybOX issue [1]. > The short of it is that it will get rid of the instance/pattern duality, > allow for the creation of a domain-specific patterning language (necessary > for more complex pattern expressions), and also give us the flexibility to > move patterning out to wherever makes the most sense (whether it’s STIX or > somewhere else). > > [1] https://github.com/CybOXProject/schemas/issues/381 > > Regards, > Ivan


  • 13.  Re: Observable Patterning

    Posted 09-24-2015 08:30
    +100, Ivan. Patterning should live in _one_ place and have _one_ syntax. I tend to think the STIX Indicator level is the right place for this to live. Cheers, Trey -- Trey Darley Senior Security Engineer Soltra An FS-ISAC & DTCC Company www.soltra.com From: cti@lists.oasis-open.org <cti@lists.oasis-open.org> on behalf of Kirillov, Ivan A. <ikirillov@mitre.org> Sent: Monday, September 21, 2015 19:47 To: cti@lists.oasis-open.org Subject: [cti] Observable Patterning   Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around Bernd’s suggested refactoring. I think there are a few broad questions around this topic that we can touch upon here:  1. What capabilities should STIX/CybOX patterning have? 2. Where should STIX/CybOX patterning live? 3.     Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects? As far as 1) I would say the following, as a straw man: * Conditional operators     * AND     * OR     * NOT * Basic temporal operators     * FOLLOWED_BY     * WITHIN * String matching     * EQUALS     * DOES_NOT_EQUAL     * CONTAINS     * DOES_NOT_CONTAIN * Basic arithmetic operators     * >     * <     * >=     * <= * Regular Expressions     * PCRE compatible * Variable substitution Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data.  With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific patterning language (necessary for more complex pattern expressions), and also give us the flexibility to move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else). [1]  https://github.com/CybOXProject/schemas/issues/381 Regards, Ivan


  • 14.  RE: Observable Patterning

    Posted 09-24-2015 13:06
    What implication would patterning in STIX Indicators have for the relationship between STIX Indicators and CybOX objects?   In order for a pattern to be specified in a STIX Indicator (e.g., ‘IP Address in 1.2.3.4/24’), would the pattern would need to use fields from a CybOX object? If the answer is yes, does that create a hard dependency between STIX Indicator and a particular CybOX object and version?   What about consistency with other uses of STIX Indicator? Consider a SNORT signature or YARA rule – patterning is contained in those signatures and wouldn’t be at the STIX Indicator level, which seems inconsistent to me.   > Patterning should live in _one_ place and have _one_ syntax. I really like this idea, and I’m asking questions that I hope will lead us in this direction.   Thank you. -Mark   From: cti@lists.oasis-open.org [mailto:cti@lists.oasis-open.org] On Behalf Of Trey Darley Sent: Thursday, September 24, 2015 4:30 AM To: Kirillov, Ivan A. <ikirillov@mitre.org>; cti@lists.oasis-open.org Subject: [cti] Re: Observable Patterning   +100, Ivan.   Patterning should live in _one_ place and have _one_ syntax.   I tend to think the STIX Indicator level is the right place for this to live.   Cheers, Trey -- Trey Darley Senior Security Engineer Soltra An FS-ISAC & DTCC Company www.soltra.com   From: cti@lists.oasis-open.org < cti@lists.oasis-open.org > on behalf of Kirillov, Ivan A. < ikirillov@mitre.org > Sent: Monday, September 21, 2015 19:47 To: cti@lists.oasis-open.org Subject: [cti] Observable Patterning   Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around Bernd’s suggested refactoring.   I think there are a few broad questions around this topic that we can touch upon here:    1. What capabilities should STIX/CybOX patterning have? 2. Where should STIX/CybOX patterning live? 3.     Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects?   As far as 1) I would say the following, as a straw man:   * Conditional operators     * AND     * OR     * NOT * Basic temporal operators     * FOLLOWED_BY     * WITHIN * String matching     * EQUALS     * DOES_NOT_EQUAL     * CONTAINS     * DOES_NOT_CONTAIN * Basic arithmetic operators     * >     * <     * >=     * <= * Regular Expressions     * PCRE compatible * Variable substitution   Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data.  With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific patterning language (necessary for more complex pattern expressions), and also give us the flexibility to move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else).   [1]  https://github.com/CybOXProject/schemas/issues/381   Regards, Ivan


  • 15.  Re: Observable Patterning

    Posted 09-24-2015 13:31




    >  In order for a pattern to be specified in a STIX Indicator (e.g., ‘IP Address in 1.2.3.4/24’), would the pattern would need to use fields from a CybOX
    object? If the answer is yes, does that create a hard dependency between STIX Indicator and a particular CybOX object and version?


    Having a separate patterning structure would mean that we would somehow need to reference CybOX Object fields, and unless we completely decouple CybOX from STIX Indicators (this seems unlikely, though I believe there have
    been a few suggestions to this effect), there would indeed be a hard dependency between a STIX Indicator and a particular CybOX Object version. This is necessary to explicitly specify the semantics of “what” the Indicator is looking for. 






    >  What about consistency with other uses of STIX Indicator? Consider a SNORT signature or YARA rule – patterning is contained in those signatures and wouldn’t
    be at the STIX Indicator level, which seems inconsistent to me.


    This seems to be more of an issue with how different types of patterns are captured in STIX Indicators. CybOX-based patterns are currently represented as a separate entity, whereas other types
    of patterns (SNORT, YARA, etc.) are captured as Test_Mechanisms. Given their functional overlap, it may make more sense to have a single “Pattern” field that can be extended to capture CybOX-based patterns, SNORT rules, YARA sigs, etc.


    Regards,
    Ivan




    From: Mark Davidson
    Date: Thursday, September 24, 2015 at 9:06 AM
    To: Trey Darley, Ivan Kirillov, " cti@lists.oasis-open.org "
    Subject: RE: Observable Patterning








    What implication would patterning in STIX Indicators have for the relationship between STIX Indicators and CybOX objects?
     
    In order for a pattern to be specified in a STIX Indicator (e.g., ‘IP Address in 1.2.3.4/24’), would the pattern would need to use fields from a CybOX
    object? If the answer is yes, does that create a hard dependency between STIX Indicator and a particular CybOX object and version?
     
    What about consistency with other uses of STIX Indicator? Consider a SNORT signature or YARA rule – patterning is contained in those signatures and
    wouldn’t be at the STIX Indicator level, which seems inconsistent to me.
     
    > Patterning should live in _one_ place and have _one_ syntax.
    I really like this idea, and I’m asking questions that I hope will lead us in this direction.
     
    Thank you.
    -Mark
     


    From:
    cti@lists.oasis-open.org [ mailto:cti@lists.oasis-open.org ]
    On Behalf Of Trey Darley
    Sent: Thursday, September 24, 2015 4:30 AM
    To: Kirillov, Ivan A. < ikirillov@mitre.org >;
    cti@lists.oasis-open.org
    Subject: [cti] Re: Observable Patterning


     

    +100, Ivan.
     
    Patterning should live in _one_ place and have _one_ syntax.
     
    I tend to think the STIX Indicator level is the right place for this to live.
     




    Cheers,


    Trey


    --


    Trey Darley


    Senior Security Engineer


    Soltra An FS-ISAC & DTCC Company


    www.soltra.com





     








    From: cti@lists.oasis-open.org
    < cti@lists.oasis-open.org > on behalf of Kirillov, Ivan
    A. < ikirillov@mitre.org >
    Sent: Monday, September 21, 2015 19:47
    To: cti@lists.oasis-open.org
    Subject: [cti] Observable Patterning

     







    Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around
    Bernd’s suggested refactoring.





     


    I think there are a few broad questions around this topic that we can touch upon here: 


     


    1.
    What capabilities should STIX/CybOX patterning have?


    2.
    Where should STIX/CybOX patterning live?


    3.     Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects?


     


    As far as 1) I would say the following, as a straw man:


     



    * Conditional operators


        * AND


        * OR


        * NOT


    * Basic temporal operators


        * FOLLOWED_BY


        * WITHIN


    * String matching


        * EQUALS


        * DOES_NOT_EQUAL


        * CONTAINS


        * DOES_NOT_CONTAIN


    * Basic arithmetic operators


        * >


        * <


        * >=


        * <=


    * Regular Expressions


        * PCRE compatible


    * Variable substitution



     










    Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that
    stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. 







    With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in
    the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific patterning language (necessary for more complex pattern expressions), and also give us the flexibility to
    move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else).


     


    [1]  https://github.com/CybOXProject/schemas/issues/381


     


    Regards,


    Ivan













  • 16.  Re: [cti] Re: Observable Patterning

    Posted 09-25-2015 00:31
    I don't really see Test_Mechanisms as a candidate for reusing the patterning that we've been discussing. Each Test_Mechanism rule is a rule derived from the Indicators, and written in the tools native language. It doesn't make sense to me to have  patterning in the Test_Mechanism unless it is natively in the tool that uses the rules. I do think that patterning makes more sense residing within the STIX Indicator object (and moving Observable Instances to a new top-level Sighting object), and then using a top-level relationship object to record the relationship between the patterned STIX Indicator object and the STIX Test_Mechanism. Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers. On 24 September 2015 at 23:30, Kirillov, Ivan A. < ikirillov@mitre.org > wrote: >  In order for a pattern to be specified in a STIX Indicator (e.g., ‘IP Address in 1.2.3.4/24 ’), would the pattern would need to use fields from a CybOX object? If the answer is yes, does that create a hard dependency between STIX Indicator and a particular CybOX object and version? Having a separate patterning structure would mean that we would somehow need to reference CybOX Object fields, and unless we completely decouple CybOX from STIX Indicators (this seems unlikely, though I believe there have been a few suggestions to this effect), there would indeed be a hard dependency between a STIX Indicator and a particular CybOX Object version. This is necessary to explicitly specify the semantics of “what” the Indicator is looking for.  >  What about consistency with other uses of STIX Indicator? Consider a SNORT signature or YARA rule – patterning is contained in those signatures and wouldn’t be at the STIX Indicator level, which seems inconsistent to me. This seems to be more of an issue with how different types of patterns are captured in STIX Indicators. CybOX-based patterns are currently represented as a separate entity, whereas other types of patterns (SNORT, YARA, etc.) are captured as Test_Mechanisms. Given their functional overlap, it may make more sense to have a single “Pattern” field that can be extended to capture CybOX-based patterns, SNORT rules, YARA sigs, etc. Regards, Ivan From: Mark Davidson Date: Thursday, September 24, 2015 at 9:06 AM To: Trey Darley, Ivan Kirillov, " cti@lists.oasis-open.org " Subject: RE: Observable Patterning What implication would patterning in STIX Indicators have for the relationship between STIX Indicators and CybOX objects?   In order for a pattern to be specified in a STIX Indicator (e.g., ‘IP Address in 1.2.3.4/24 ’), would the pattern would need to use fields from a CybOX object? If the answer is yes, does that create a hard dependency between STIX Indicator and a particular CybOX object and version?   What about consistency with other uses of STIX Indicator? Consider a SNORT signature or YARA rule – patterning is contained in those signatures and wouldn’t be at the STIX Indicator level, which seems inconsistent to me.   > Patterning should live in _one_ place and have _one_ syntax. I really like this idea, and I’m asking questions that I hope will lead us in this direction.   Thank you. -Mark   From: cti@lists.oasis-open.org [ mailto:cti@lists.oasis-open.org ] On Behalf Of Trey Darley Sent: Thursday, September 24, 2015 4:30 AM To: Kirillov, Ivan A. < ikirillov@mitre.org >; cti@lists.oasis-open.org Subject: [cti] Re: Observable Patterning   +100, Ivan.   Patterning should live in _one_ place and have _one_ syntax.   I tend to think the STIX Indicator level is the right place for this to live.   Cheers, Trey -- Trey Darley Senior Security Engineer Soltra An FS-ISAC & DTCC Company www.soltra.com   From: cti@lists.oasis-open.org < cti@lists.oasis-open.org > on behalf of Kirillov, Ivan A. < ikirillov@mitre.org > Sent: Monday, September 21, 2015 19:47 To: cti@lists.oasis-open.org Subject: [cti] Observable Patterning   Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around Bernd’s suggested refactoring.   I think there are a few broad questions around this topic that we can touch upon here:    1. What capabilities should STIX/CybOX patterning have? 2. Where should STIX/CybOX patterning live? 3.     Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects?   As far as 1) I would say the following, as a straw man:   * Conditional operators     * AND     * OR     * NOT * Basic temporal operators     * FOLLOWED_BY     * WITHIN * String matching     * EQUALS     * DOES_NOT_EQUAL     * CONTAINS     * DOES_NOT_CONTAIN * Basic arithmetic operators     * >     * <     * >=     * <= * Regular Expressions     * PCRE compatible * Variable substitution   Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data.  With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific patterning language (necessary for more complex pattern expressions), and also give us the flexibility to move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else).   [1]  https://github.com/CybOXProject/schemas/issues/381   Regards, Ivan


  • 17.  RE: [cti] Re: Observable Patterning

    Posted 09-25-2015 11:02
    Hi, > I don't really see Test_Mechanisms as a candidate for reusing the > patterning that we've been discussing. Each Test_Mechanism rule is a > rule derived from the Indicators, and written in the tools native > language. The principle of having a test-mechanism derive from what is described within an indicator using observable patterns is the intended meaning, but this principle is not always followed: it may be quite complicated to describe in STIX/CybOX, what can be succinctly described in a certain pattern language or the authoring party simply does not want to bother with this -- so the STIX indicator is also used as a vehicle to exchange such a test mechanism, enriched with contextual information contained in the indicator and referenced/referencing STIX objects, but not containing corresponding observable patterns. I would guess that we will continue to see such use of a TestMechanism within an otherwise rather empty indicator (empty with respect to Observables), even if the pattern language is evolved such that in theory it could describe everything that is expressible in any thinkable test mechanism. > It doesn't make sense to me to have patterning in the > Test_Mechanism unless it is natively in the tool that uses the rules. I had made that proposal in one of my emails with the intent of encapsulating complexity. I can see why one would not want to use a test mechanism as container for STIX/CybOX containers -- after all, no other test mechanism would have references out of the test-mechanism blob, whereas a STIX pattern could have many references to objects/object properties defined elsewhere. But I really would like to get rid of AND/OR/...-composition on the level of indicators and push the whole pattern language into (oh no, here it comes) yet another STIX top level object STIX_Pattern or whatever. Then allow STIX_Patterns to reference other patterns for composition, but not Indicators. The reason is that the whole logical and temporal composition is really complicated. The Indicator is absolutely central to STIX, but like I said above, there may be use-cases to communicate things to look out for *without* the use of CybOX and CybOX patterning but via a test mechanism (or, I haven't given up on that idea, a STIX-standardized key/value-pair mechanism for communicating patterns). Now, if we encapsulate the complications of the pattern language in something that is removed from the Indicator level, then one can implement the non-patterning-part of STIX much easier in order to use it for use cases as described above. Also, using STIX for other domains (fraud and what not) as has been mentioned on the list yesterday, could become easier, since for these other domains, the full might of STIX patterns may not be needed ... or such domains may even need something else entirely. (Hm, I just realize that this reasoning would actually support making patterning part of CybOX as a CybOX-pattern-top-level object.) In closing, because I have proposed yet another top-level STIX entity: in one of his mails about a possible top-level targeting object, Aharon was concerned about making things too complicated by adding more and more top-level objects. That is a valid concern, but on the other hand, having huge schema definitions for objects that encompass all kinds of substructures, is also quite complicated. So, while we certainly should have good reasons for introducing new top-level objects, I would say that given the right circumstances, new top-level objects will actually decrease complexity. So, in short: - I fully agree that decoupling patterning and instances is the right way to go. - I would like to encapsulate patterns in a top-level construct that may be referenced by STIX indicators while removing not/and/or composition for indicators. - I am not quite sure yet whether the 'Pattern' top-level object would better be part of STIX or part of CybOX. If we think the patterning is reusable also for extended domains such as a possible SocEngOX (Social Engineering Observable EXchange) language and we think we will want to write things like (SocEng-Observation "Phone call from Mr. Evil") THEN (CybOX Email Object with sender "lucifer.evil@hell.com) then patterning is probably part of STIX rather than CybOX; otherwise, as a cyber-focused language, it might well be part of CybOX. Kind regards, Bernd ------ Bernd Grobauer, Siemens CERT


  • 18.  RE: [cti] Re: Observable Patterning

    Posted 09-25-2015 12:02
    Adding my thoughts to the Test_Mechanism idea. In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers. If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of: Title: Send a signature Description: An analyst creates a signature/indicator and chooses to share it with partner organizations. Main Success Scenario: 1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern) 2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator) 3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and 3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator) 4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action) Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing. Thank you. -Mark [1] http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html


  • 19.  Re: [cti] Re: Observable Patterning

    Posted 09-25-2015 13:14





    To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the
    case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms)
    – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot. 






    I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability
    should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they
    do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern.




    Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints
    against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning
    (GLUP), anyone? :-) 




    So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable:






    pattern example_1 



    {



      objects:

    $OBJ1 = {type = AddressObject,
             fields = [{ “ category ” : ” ipv4-addr ”},
                       { “ address_value ” : ” 192.168.2.3 ” }]}  

       

    $OBJ2 =  {type = AddressObject,


             fields = [{ “ category ” : ” ipv4-addr ”},
                       { “ address_value ” : ” 192.168.4.7 ” }]}  

    condition:

        OBJ1 and OBJ2

    }

     



    Regards,

    Ivan




    On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote:




    Adding my thoughts to the Test_Mechanism idea.


    In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have
    today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers.


    If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of:


    Title: Send a signature


    Description: An analyst creates a signature/indicator and chooses to share it with partner organizations.


    Main Success Scenario:


    1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern)
    2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator)
    3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and
    3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator)
    4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action)


    Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing.


    Thank you.
    -Mark


    [1]
    http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html





  • 20.  Re: [cti] Re: Observable Patterning

    Posted 09-25-2015 15:12
    A general [+1] for everything @Ivan said below (although agnostic on some of the specifics) To this "flexible" patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:  "a" followed by "b" , "a" during "b" ) along with compound boolean and set theory _expression_. Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 6:13 AM -0700, "Kirillov, Ivan A." < ikirillov@mitre.org > wrote: To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot.  I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern. Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-)  So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable: pattern example_1  {   objects: $OBJ1 = {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.2.3 ” }]}       $OBJ2 =  {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.4.7 ” }]}   condition:     OBJ1 and OBJ2 }   Regards, Ivan On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote: Adding my thoughts to the Test_Mechanism idea. In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers. If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of: Title: Send a signature Description: An analyst creates a signature/indicator and chooses to share it with partner organizations. Main Success Scenario: 1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern) 2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator) 3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and 3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator) 4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action) Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing. Thank you. -Mark [1] http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html


  • 21.  Re: [cti] Re: Observable Patterning

    Posted 09-25-2015 15:33
    I would like to add a couple of conceptual ideas to this discussion (along with a specific illustrative reference): (1)  If we architect our flexible "patterning" model correctly it 'should' be directly applicable to our TAXII Query Language. (2) Graph data models provide some interesting correlations/visualizations.  As previously suggested neo4j's Cypher Query language might serve as a good conceptual model for how to express "patterns" http://assets.neo4j.org/download/Neo4j_CheatSheet_v3.pdf Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 8:12 AM -0700, "Patrick Maroney" < Pmaroney@Specere.org > wrote: A general [+1] for everything @Ivan said below (although agnostic on some of the specifics) To this "flexible" patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:  "a" followed by "b" , "a" during "b" ) along with compound boolean and set theory _expression_. Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 6:13 AM -0700, "Kirillov, Ivan A." < ikirillov@mitre.org > wrote: To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot.  I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern. Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-)  So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable: pattern example_1  {   objects: $OBJ1 = {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.2.3 ” }]}       $OBJ2 =  {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.4.7 ” }]}   condition:     OBJ1 and OBJ2 }   Regards, Ivan On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote: Adding my thoughts to the Test_Mechanism idea. In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers. If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of: Title: Send a signature Description: An analyst creates a signature/indicator and chooses to share it with partner organizations. Main Success Scenario: 1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern) 2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator) 3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and 3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator) 4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action) Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing. Thank you. -Mark [1] http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html


  • 22.  Re: [cti] Re: Observable Patterning

    Posted 09-26-2015 01:02
    I agree with Patrick's comments about ensuring pattern support for querying, but not the idea querying should be done in TAXII. Querying and answering should be done within STIX, so that the querying and answering done by the language contained within TAXII, and so that TAXII is free to concentrate on only delivering content. We should instead develop a STIX query object, and a STIX reply object in order to support question and answer directly within STIX. I won't go into the specific argument why here and clog this thread, but it has been discussed on the TAXII list recently.   Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers. On 26 September 2015 at 01:33, Patrick Maroney < Pmaroney@specere.org > wrote: I would like to add a couple of conceptual ideas to this discussion (along with a specific illustrative reference): (1)  If we architect our flexible "patterning" model correctly it 'should' be directly applicable to our TAXII Query Language. (2) Graph data models provide some interesting correlations/visualizations.  As previously suggested neo4j's Cypher Query language might serve as a good conceptual model for how to express "patterns" http://assets.neo4j.org/download/Neo4j_CheatSheet_v3.pdf Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 8:12 AM -0700, "Patrick Maroney" < Pmaroney@Specere.org > wrote: A general [+1] for everything @Ivan said below (although agnostic on some of the specifics) To this "flexible" patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:  "a" followed by "b" , "a" during "b" ) along with compound boolean and set theory _expression_. Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 6:13 AM -0700, "Kirillov, Ivan A." < ikirillov@mitre.org > wrote: To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot.  I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern. Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-)  So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable: pattern example_1  {   objects: $OBJ1 = {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.2.3 ” }]}       $OBJ2 =  {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.4.7 ” }]}   condition:     OBJ1 and OBJ2 }   Regards, Ivan On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote: Adding my thoughts to the Test_Mechanism idea. In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers. If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of: Title: Send a signature Description: An analyst creates a signature/indicator and chooses to share it with partner organizations. Main Success Scenario: 1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern) 2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator) 3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and 3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator) 4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action) Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing. Thank you. -Mark [1] http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html


  • 23.  Re: [cti] Re: Observable Patterning

    Posted 09-26-2015 14:58
    Probably not clear that I've similarly argued to keep TAXII focused on the transport of packages.  In the query method suggested for consideration, the Pattern would be passed as a STIX (Query/RFI) package and results similarly returned (again as a STIX package). It's important to visualize this a Data Graph that maps to the Conceptual CTI model (which is the basis for suggesting Cypher as an example representation).  One can describe/query precise graphs of Objects/Relationships, any Nodes/Edges off of a Root Node, etc. Hope this makes sense (at least in terms of framing the proposal). Sent from Outlook On Fri, Sep 25, 2015 at 6:01 PM -0700, "Terry MacDonald" < terry.macdonald@threatloop.com > wrote: I agree with Patrick's comments about ensuring pattern support for querying, but not the idea querying should be done in TAXII. Querying and answering should be done within STIX, so that the querying and answering done by the language contained within TAXII, and so that TAXII is free to concentrate on only delivering content. We should instead develop a STIX query object, and a STIX reply object in order to support question and answer directly within STIX. I won't go into the specific argument why here and clog this thread, but it has been discussed on the TAXII list recently.   Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers. On 26 September 2015 at 01:33, Patrick Maroney < Pmaroney@specere.org > wrote: I would like to add a couple of conceptual ideas to this discussion (along with a specific illustrative reference): (1)  If we architect our flexible "patterning" model correctly it 'should' be directly applicable to our TAXII Query Language. (2) Graph data models provide some interesting correlations/visualizations.  As previously suggested neo4j's Cypher Query language might serve as a good conceptual model for how to express "patterns" http://assets.neo4j.org/download/Neo4j_CheatSheet_v3.pdf Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 8:12 AM -0700, "Patrick Maroney" < Pmaroney@Specere.org > wrote: A general [+1] for everything @Ivan said below (although agnostic on some of the specifics) To this "flexible" patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:  "a" followed by "b" , "a" during "b" ) along with compound boolean and set theory _expression_. Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 6:13 AM -0700, "Kirillov, Ivan A." < ikirillov@mitre.org > wrote: To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot.  I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern. Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-)  So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable: pattern example_1  {   objects: $OBJ1 = {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.2.3 ” }]}       $OBJ2 =  {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.4.7 ” }]}   condition:     OBJ1 and OBJ2 }   Regards, Ivan On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote: Adding my thoughts to the Test_Mechanism idea. In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers. If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of: Title: Send a signature Description: An analyst creates a signature/indicator and chooses to share it with partner organizations. Main Success Scenario: 1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern) 2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator) 3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and 3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator) 4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action) Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing. Thank you. -Mark [1] http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html


  • 24.  Re: [cti] Re: Observable Patterning

    Posted 09-26-2015 16:39
    so should we try to capture the requirements in one place for this change request? Quickly: - Human readable - Machine friendly (parsing/computation, algorithmically friendly) - Data graph friendly 2015-09-26 17:57 GMT+03:00 Patrick Maroney < Pmaroney@specere.org > : Probably not clear that I've similarly argued to keep TAXII focused on the transport of packages.  In the query method suggested for consideration, the Pattern would be passed as a STIX (Query/RFI) package and results similarly returned (again as a STIX package). It's important to visualize this a Data Graph that maps to the Conceptual CTI model (which is the basis for suggesting Cypher as an example representation).  One can describe/query precise graphs of Objects/Relationships, any Nodes/Edges off of a Root Node, etc. Hope this makes sense (at least in terms of framing the proposal). Sent from Outlook On Fri, Sep 25, 2015 at 6:01 PM -0700, "Terry MacDonald" < terry.macdonald@threatloop.com > wrote: I agree with Patrick's comments about ensuring pattern support for querying, but not the idea querying should be done in TAXII. Querying and answering should be done within STIX, so that the querying and answering done by the language contained within TAXII, and so that TAXII is free to concentrate on only delivering content. We should instead develop a STIX query object, and a STIX reply object in order to support question and answer directly within STIX. I won't go into the specific argument why here and clog this thread, but it has been discussed on the TAXII list recently.   Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers. On 26 September 2015 at 01:33, Patrick Maroney < Pmaroney@specere.org > wrote: I would like to add a couple of conceptual ideas to this discussion (along with a specific illustrative reference): (1)  If we architect our flexible "patterning" model correctly it 'should' be directly applicable to our TAXII Query Language. (2) Graph data models provide some interesting correlations/visualizations.  As previously suggested neo4j's Cypher Query language might serve as a good conceptual model for how to express "patterns" http://assets.neo4j.org/download/Neo4j_CheatSheet_v3.pdf Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 8:12 AM -0700, "Patrick Maroney" < Pmaroney@Specere.org > wrote: A general [+1] for everything @Ivan said below (although agnostic on some of the specifics) To this "flexible" patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:  "a" followed by "b" , "a" during "b" ) along with compound boolean and set theory _expression_. Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 6:13 AM -0700, "Kirillov, Ivan A." < ikirillov@mitre.org > wrote: To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot.  I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern. Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-)  So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable: pattern example_1  {   objects: $OBJ1 = {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.2.3 ” }]}       $OBJ2 =  {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.4.7 ” }]}   condition:     OBJ1 and OBJ2 }   Regards, Ivan On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote: Adding my thoughts to the Test_Mechanism idea. In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers. If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of: Title: Send a signature Description: An analyst creates a signature/indicator and chooses to share it with partner organizations. Main Success Scenario: 1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern) 2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator) 3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and 3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator) 4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action) Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing. Thank you. -Mark [1] http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html


  • 25.  Re: [cti] Re: Observable Patterning

    Posted 09-27-2015 07:02
    We should consider reuse/review of the PoolsType, as per used in the  Parameter_Observables, coming from the COAs constructs https://stixproject.github.io/data-model/1.2/cybox/PoolsType/ (Referenced by  Parameter_Observables... in  https://stixproject.github.io/data-model/1.2/coa/CourseOfActionType/ ) Potentially, reajusting the FrequencyType in STIX https://stixproject.github.io/data-model/1.2/cybox/FrequencyType/ 2015-09-26 19:38 GMT+03:00 Jerome Athias < athiasjerome@gmail.com > : so should we try to capture the requirements in one place for this change request? Quickly: - Human readable - Machine friendly (parsing/computation, algorithmically friendly) - Data graph friendly 2015-09-26 17:57 GMT+03:00 Patrick Maroney < Pmaroney@specere.org > : Probably not clear that I've similarly argued to keep TAXII focused on the transport of packages.  In the query method suggested for consideration, the Pattern would be passed as a STIX (Query/RFI) package and results similarly returned (again as a STIX package). It's important to visualize this a Data Graph that maps to the Conceptual CTI model (which is the basis for suggesting Cypher as an example representation).  One can describe/query precise graphs of Objects/Relationships, any Nodes/Edges off of a Root Node, etc. Hope this makes sense (at least in terms of framing the proposal). Sent from Outlook On Fri, Sep 25, 2015 at 6:01 PM -0700, "Terry MacDonald" < terry.macdonald@threatloop.com > wrote: I agree with Patrick's comments about ensuring pattern support for querying, but not the idea querying should be done in TAXII. Querying and answering should be done within STIX, so that the querying and answering done by the language contained within TAXII, and so that TAXII is free to concentrate on only delivering content. We should instead develop a STIX query object, and a STIX reply object in order to support question and answer directly within STIX. I won't go into the specific argument why here and clog this thread, but it has been discussed on the TAXII list recently.   Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers. On 26 September 2015 at 01:33, Patrick Maroney < Pmaroney@specere.org > wrote: I would like to add a couple of conceptual ideas to this discussion (along with a specific illustrative reference): (1)  If we architect our flexible "patterning" model correctly it 'should' be directly applicable to our TAXII Query Language. (2) Graph data models provide some interesting correlations/visualizations.  As previously suggested neo4j's Cypher Query language might serve as a good conceptual model for how to express "patterns" http://assets.neo4j.org/download/Neo4j_CheatSheet_v3.pdf Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 8:12 AM -0700, "Patrick Maroney" < Pmaroney@Specere.org > wrote: A general [+1] for everything @Ivan said below (although agnostic on some of the specifics) To this "flexible" patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:  "a" followed by "b" , "a" during "b" ) along with compound boolean and set theory _expression_. Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 6:13 AM -0700, "Kirillov, Ivan A." < ikirillov@mitre.org > wrote: To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot.  I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern. Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-)  So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable: pattern example_1  {   objects: $OBJ1 = {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.2.3 ” }]}       $OBJ2 =  {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.4.7 ” }]}   condition:     OBJ1 and OBJ2 }   Regards, Ivan On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote: Adding my thoughts to the Test_Mechanism idea. In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers. If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of: Title: Send a signature Description: An analyst creates a signature/indicator and chooses to share it with partner organizations. Main Success Scenario: 1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern) 2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator) 3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and 3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator) 4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action) Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing. Thank you. -Mark [1] http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html


  • 26.  Re: [cti] Re: Observable Patterning

    Posted 09-27-2015 07:30
    this was just to point out, that Test_Mechanisms for both "Detection" and "Remediation" (COA) could (SHOULD) use the same description mechanisms (language) (Ref.  https://stixproject.github.io/data-model/1.2/genericStructuredCOA/GenericStructuredCOAType/ Specification) 2015-09-27 10:01 GMT+03:00 Jerome Athias < athiasjerome@gmail.com > : We should consider reuse/review of the PoolsType, as per used in the  Parameter_Observables, coming from the COAs constructs https://stixproject.github.io/data-model/1.2/cybox/PoolsType/ (Referenced by  Parameter_Observables... in  https://stixproject.github.io/data-model/1.2/coa/CourseOfActionType/ ) Potentially, reajusting the FrequencyType in STIX https://stixproject.github.io/data-model/1.2/cybox/FrequencyType/ 2015-09-26 19:38 GMT+03:00 Jerome Athias < athiasjerome@gmail.com > : so should we try to capture the requirements in one place for this change request? Quickly: - Human readable - Machine friendly (parsing/computation, algorithmically friendly) - Data graph friendly 2015-09-26 17:57 GMT+03:00 Patrick Maroney < Pmaroney@specere.org > : Probably not clear that I've similarly argued to keep TAXII focused on the transport of packages.  In the query method suggested for consideration, the Pattern would be passed as a STIX (Query/RFI) package and results similarly returned (again as a STIX package). It's important to visualize this a Data Graph that maps to the Conceptual CTI model (which is the basis for suggesting Cypher as an example representation).  One can describe/query precise graphs of Objects/Relationships, any Nodes/Edges off of a Root Node, etc. Hope this makes sense (at least in terms of framing the proposal). Sent from Outlook On Fri, Sep 25, 2015 at 6:01 PM -0700, "Terry MacDonald" < terry.macdonald@threatloop.com > wrote: I agree with Patrick's comments about ensuring pattern support for querying, but not the idea querying should be done in TAXII. Querying and answering should be done within STIX, so that the querying and answering done by the language contained within TAXII, and so that TAXII is free to concentrate on only delivering content. We should instead develop a STIX query object, and a STIX reply object in order to support question and answer directly within STIX. I won't go into the specific argument why here and clog this thread, but it has been discussed on the TAXII list recently.   Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers. On 26 September 2015 at 01:33, Patrick Maroney < Pmaroney@specere.org > wrote: I would like to add a couple of conceptual ideas to this discussion (along with a specific illustrative reference): (1)  If we architect our flexible "patterning" model correctly it 'should' be directly applicable to our TAXII Query Language. (2) Graph data models provide some interesting correlations/visualizations.  As previously suggested neo4j's Cypher Query language might serve as a good conceptual model for how to express "patterns" http://assets.neo4j.org/download/Neo4j_CheatSheet_v3.pdf Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 8:12 AM -0700, "Patrick Maroney" < Pmaroney@Specere.org > wrote: A general [+1] for everything @Ivan said below (although agnostic on some of the specifics) To this "flexible" patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:  "a" followed by "b" , "a" during "b" ) along with compound boolean and set theory _expression_. Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 6:13 AM -0700, "Kirillov, Ivan A." < ikirillov@mitre.org > wrote: To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot.  I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern. Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-)  So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable: pattern example_1  {   objects: $OBJ1 = {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.2.3 ” }]}       $OBJ2 =  {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.4.7 ” }]}   condition:     OBJ1 and OBJ2 }   Regards, Ivan On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote: Adding my thoughts to the Test_Mechanism idea. In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers. If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of: Title: Send a signature Description: An analyst creates a signature/indicator and chooses to share it with partner organizations. Main Success Scenario: 1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern) 2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator) 3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and 3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator) 4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action) Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing. Thank you. -Mark [1] http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html


  • 27.  Re: [cti] Re: Observable Patterning

    Posted 09-28-2015 13:36






    >- Human readable
         Agreed.
    >- Machine friendly (parsing/computation, algorithmically friendly)
         Agreed.
    >- Data graph friendly
         Not so sure on this one. 


    While I agree that STIX/TAXII querying (wherever it ends up) is inherently graph-based, I don’t believe the same is true for Indicator patterning. The main difference is that, in my view, the majority of Indicator patterns are meant to be parsed and executed
    against data that is inherently flat – lists of files, lists of processes, lists of IP addresses, etc. Thus while the two may overlap in certain places, I’d be very hesitant about overloading an Indicator patterning structure to support graph based querying,
    though perhaps it could be extended to do so. Again, I think a primary focus here should be to keep Indicator patterns SIMPLE, so that they can easily be written and consumed by analysts.


    Regards,
    Ivan







    From: Jerome Athias
    Date: Saturday, September 26, 2015 at 12:38 PM
    To: Patrick Maroney
    Cc: Terry MacDonald, " cti@lists.oasis-open.org ", Ivan Kirillov
    Subject: Re: [cti] Re: Observable Patterning





    so should we try to capture the requirements in one place for this change request?
    Quickly:
    - Human readable
    - Machine friendly (parsing/computation, algorithmically friendly)
    - Data graph friendly


    2015-09-26 17:57 GMT+03:00 Patrick Maroney
    < Pmaroney@specere.org > :


    Probably not clear that I've similarly argued to keep TAXII focused on the transport of packages.  In the query method suggested for consideration, the Pattern would be passed as a STIX (Query/RFI) package and results similarly returned (again as a STIX
    package).


    It's important to visualize this a Data Graph that maps to the Conceptual CTI model (which is the basis for suggesting Cypher as an example representation).  One can describe/query precise graphs of Objects/Relationships, any Nodes/Edges off of a Root
    Node, etc.


    Hope this makes sense (at least in terms of framing the proposal).

    Sent from Outlook






    On Fri, Sep 25, 2015 at 6:01 PM -0700, "Terry MacDonald"
    < terry.macdonald@threatloop.com > wrote:




    I agree with Patrick's comments about ensuring pattern support for querying, but not the idea querying should be done in TAXII.
    Querying and answering should be done within STIX, so that the querying and answering done by the language contained within TAXII, and so that TAXII is free to concentrate on only delivering content. We
    should instead develop a STIX query object, and a STIX reply object in order to support question and answer directly within STIX. I won't go into the specific argument why here and clog this thread, but it has been discussed on the TAXII list recently.  
    Cheers








    Terry MacDonald STIX, TAXII, CybOX Consultant



    M: +61-407-203-026
    E:  terry.macdonald@threatloop.com
    W:  www.threatloop.com






    Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those
    of my employers.






    On 26 September 2015 at 01:33, Patrick Maroney
    < Pmaroney@specere.org > wrote:


    I would like to add a couple of conceptual ideas to this discussion (along with a specific illustrative reference):


    (1)  If we architect our flexible "patterning" model correctly it 'should' be directly applicable to our TAXII Query Language.


    (2) Graph data models provide some interesting correlations/visualizations.  As previously suggested neo4j's Cypher Query language might serve as a good conceptual model for how to express "patterns"


    http://assets.neo4j.org/download/Neo4j_CheatSheet_v3.pdf

    Patrick Maroney
    President
    Integrated Networking Technologies, Inc.
    Desk: (856)983-0001
    Cell: (609)841-5104
    Email: pmaroney@specere.org






    On Fri, Sep 25, 2015 at 8:12 AM -0700, "Patrick Maroney"
    < Pmaroney@Specere.org > wrote:



    A general [+1] for everything @Ivan said below (although agnostic on some of the specifics)


    To this "flexible" patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:  "a" followed by "b" , "a" during "b" ) along with compound boolean and set theory _expression_.

    Patrick Maroney
    President
    Integrated Networking Technologies, Inc.
    Desk: (856)983-0001
    Cell: (609)841-5104
    Email: pmaroney@specere.org





    On Fri, Sep 25, 2015 at 6:13 AM -0700, "Kirillov, Ivan A."
    < ikirillov@mitre.org > wrote:





    To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the
    case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms)
    – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot. 




    I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar
    to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think
    having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through
    a CybOX pattern.


    Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately,
    I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even
    live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-) 


    So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy
    for humans to read and write, and also machine-parseable:





    pattern example_1 



    {



      objects:

    $OBJ1 = {type = AddressObject,
             fields = [{ “ category ” : ” ipv4-addr ”},
                       { “ address_value ” : ” 192.168.2.3 ” }]}  

       

    $OBJ2 =  {type = AddressObject,


             fields = [{ “ category ” : ” ipv4-addr ”},
                       { “ address_value ” : ” 192.168.4.7 ” }]}  

    condition:

        OBJ1 and OBJ2

    }

     


    Regards,
    Ivan


    On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote:



    Adding my thoughts to the Test_Mechanism idea.


    In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have
    today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers.


    If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of:


    Title: Send a signature


    Description: An analyst creates a signature/indicator and chooses to share it with partner organizations.


    Main Success Scenario:


    1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern)
    2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator)
    3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and
    3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator)
    4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action)


    Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing.


    Thank you.
    -Mark


    [1]
    http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html





  • 28.  Re: [cti] Re: Observable Patterning

    Posted 09-28-2015 15:58
    Hi Ivan, I’d be interested what the team feels about the relationship between these patterns and test mechanism concept. Please correct me if wrong; - indicator patterns described a pattern of potentially malicious activity that is intended to convey the pattern in a way that a human can understand and that machines can translate into action-based use-cases (like detection) - test mechanisms are a pre-computed or created pattern specific to a certain use-case (like snort detection, yara, etc.) Since in both cases that might be conditional relationships and nested conditional calculations (e.g. If you don’t see X, then the previous variable in de flow is +1, if you do seeX, then the previous variable in the flow is –1) - it makes it ill-suited for graph representation in the context of graph querying. J- From: <Kirillov>, "Ivan A." < ikirillov@mitre.org > Date: Monday, September 28, 2015 at 3:36 PM To: Jerome Athias < athiasjerome@gmail.com >, Patrick Maroney < Pmaroney@specere.org > Cc: Terry MacDonald < terry.macdonald@threatloop.com >, " cti@lists.oasis-open.org " < cti@lists.oasis-open.org > Subject: Re: [cti] Re: Observable Patterning >- Human readable      Agreed. >- Machine friendly (parsing/computation, algorithmically friendly)      Agreed. >- Data graph friendly      Not so sure on this one.  While I agree that STIX/TAXII querying (wherever it ends up) is inherently graph-based, I don’t believe the same is true for Indicator patterning. The main difference is that, in my view, the majority of Indicator patterns are meant to be parsed and executed against data that is inherently flat – lists of files, lists of processes, lists of IP addresses, etc. Thus while the two may overlap in certain places, I’d be very hesitant about overloading an Indicator patterning structure to support graph based querying, though perhaps it could be extended to do so. Again, I think a primary focus here should be to keep Indicator patterns SIMPLE, so that they can easily be written and consumed by analysts. Regards, Ivan From: Jerome Athias Date: Saturday, September 26, 2015 at 12:38 PM To: Patrick Maroney Cc: Terry MacDonald, " cti@lists.oasis-open.org ", Ivan Kirillov Subject: Re: [cti] Re: Observable Patterning so should we try to capture the requirements in one place for this change request? Quickly: - Human readable - Machine friendly (parsing/computation, algorithmically friendly) - Data graph friendly 2015-09-26 17:57 GMT+03:00 Patrick Maroney < Pmaroney@specere.org > : Probably not clear that I've similarly argued to keep TAXII focused on the transport of packages.  In the query method suggested for consideration, the Pattern would be passed as a STIX (Query/RFI) package and results similarly returned (again as a STIX package). It's important to visualize this a Data Graph that maps to the Conceptual CTI model (which is the basis for suggesting Cypher as an example representation).  One can describe/query precise graphs of Objects/Relationships, any Nodes/Edges off of a Root Node, etc. Hope this makes sense (at least in terms of framing the proposal). Sent from Outlook On Fri, Sep 25, 2015 at 6:01 PM -0700, "Terry MacDonald" < terry.macdonald@threatloop.com > wrote: I agree with Patrick's comments about ensuring pattern support for querying, but not the idea querying should be done in TAXII. Querying and answering should be done within STIX, so that the querying and answering done by the language contained within TAXII, and so that TAXII is free to concentrate on only delivering content. We should instead develop a STIX query object, and a STIX reply object in order to support question and answer directly within STIX. I won't go into the specific argument why here and clog this thread, but it has been discussed on the TAXII list recently.   Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers. On 26 September 2015 at 01:33, Patrick Maroney < Pmaroney@specere.org > wrote: I would like to add a couple of conceptual ideas to this discussion (along with a specific illustrative reference): (1)  If we architect our flexible "patterning" model correctly it 'should' be directly applicable to our TAXII Query Language. (2) Graph data models provide some interesting correlations/visualizations.  As previously suggested neo4j's Cypher Query language might serve as a good conceptual model for how to express "patterns" http://assets.neo4j.org/download/Neo4j_CheatSheet_v3.pdf Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 8:12 AM -0700, "Patrick Maroney" < Pmaroney@Specere.org > wrote: A general [+1] for everything @Ivan said below (although agnostic on some of the specifics) To this "flexible" patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:  "a" followed by "b" , "a" during "b" ) along with compound boolean and set theory _expression_. Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 6:13 AM -0700, "Kirillov, Ivan A." < ikirillov@mitre.org > wrote: To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot.  I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern. Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-)  So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable: pattern example_1  {   objects: $OBJ1 = {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.2.3 ” }]}       $OBJ2 =  {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.4.7 ” }]}   condition:     OBJ1 and OBJ2 }   Regards, Ivan On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote: Adding my thoughts to the Test_Mechanism idea. In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers. If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of: Title: Send a signature Description: An analyst creates a signature/indicator and chooses to share it with partner organizations. Main Success Scenario: 1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern) 2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator) 3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and 3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator) 4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action) Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing. Thank you. -Mark [1] http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html


  • 29.  RE: [cti] Re: Observable Patterning

    Posted 09-28-2015 16:35
    I think there’s two intertwined topics to disambiguate:   ·          Querying STIX information, where some see STIX as a graph-based model o    E.g., Tell me all the Indicators related to this TTP ·          CybOX patterns, which are much more signature-like o    E.g., Find the file with Hash=0x1234   When I think of a CybOX pattern, I tend to think of something that is a peer to signature/Test Mechanism.   Thank you. -Mark   From: cti@lists.oasis-open.org [mailto:cti@lists.oasis-open.org] On Behalf Of Joep Gommers Sent: Monday, September 28, 2015 11:58 AM To: Kirillov, Ivan A. <ikirillov@mitre.org>; Jerome Athias <athiasjerome@gmail.com>; Patrick Maroney <Pmaroney@specere.org> Cc: Terry MacDonald <terry.macdonald@threatloop.com>; cti@lists.oasis-open.org Subject: Re: [cti] Re: Observable Patterning   Hi Ivan,   I’d be interested what the team feels about the relationship between these patterns and test mechanism concept. Please correct me if wrong; - indicator patterns described a pattern of potentially malicious activity that is intended to convey the pattern in a way that a human can understand and that machines can translate into action-based use-cases (like detection) - test mechanisms are a pre-computed or created pattern specific to a certain use-case (like snort detection, yara, etc.)   Since in both cases that might be conditional relationships and nested conditional calculations (e.g. If you don’t see X, then the previous variable in de flow is +1, if you do seeX, then the previous variable in the flow is –1) - it makes it ill-suited for graph representation in the context of graph querying.   J-   From: <Kirillov>, "Ivan A." < ikirillov@mitre.org > Date: Monday, September 28, 2015 at 3:36 PM To: Jerome Athias < athiasjerome@gmail.com >, Patrick Maroney < Pmaroney@specere.org > Cc: Terry MacDonald < terry.macdonald@threatloop.com >, " cti@lists.oasis-open.org " < cti@lists.oasis-open.org > Subject: Re: [cti] Re: Observable Patterning   >- Human readable      Agreed. >- Machine friendly (parsing/computation, algorithmically friendly)      Agreed. >- Data graph friendly      Not so sure on this one.    While I agree that STIX/TAXII querying (wherever it ends up) is inherently graph-based, I don’t believe the same is true for Indicator patterning. The main difference is that, in my view, the majority of Indicator patterns are meant to be parsed and executed against data that is inherently flat – lists of files, lists of processes, lists of IP addresses, etc. Thus while the two may overlap in certain places, I’d be very hesitant about overloading an Indicator patterning structure to support graph based querying, though perhaps it could be extended to do so. Again, I think a primary focus here should be to keep Indicator patterns SIMPLE, so that they can easily be written and consumed by analysts.   Regards, Ivan   From: Jerome Athias Date: Saturday, September 26, 2015 at 12:38 PM To: Patrick Maroney Cc: Terry MacDonald, " cti@lists.oasis-open.org ", Ivan Kirillov Subject: Re: [cti] Re: Observable Patterning   so should we try to capture the requirements in one place for this change request? Quickly: - Human readable - Machine friendly (parsing/computation, algorithmically friendly) - Data graph friendly   2015-09-26 17:57 GMT+03:00 Patrick Maroney < Pmaroney@specere.org >: Probably not clear that I've similarly argued to keep TAXII focused on the transport of packages.  In the query method suggested for consideration, the Pattern would be passed as a STIX (Query/RFI) package and results similarly returned (again as a STIX package).   It's important to visualize this a Data Graph that maps to the Conceptual CTI model (which is the basis for suggesting Cypher as an example representation).  One can describe/query precise graphs of Objects/Relationships, any Nodes/Edges off of a Root Node, etc.   Hope this makes sense (at least in terms of framing the proposal). Sent from Outlook   On Fri, Sep 25, 2015 at 6:01 PM -0700, "Terry MacDonald" < terry.macdonald@threatloop.com > wrote: I agree with Patrick's comments about ensuring pattern support for querying, but not the idea querying should be done in TAXII. Querying and answering should be done within STIX, so that the querying and answering done by the language contained within TAXII, and so that TAXII is free to concentrate on only delivering content. We should instead develop a STIX query object, and a STIX reply object in order to support question and answer directly within STIX. I won't go into the specific argument why here and clog this thread, but it has been discussed on the TAXII list recently.   Cheers Terry MacDonald STIX, TAXII, CybOX Consultant   M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com     Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers.   On 26 September 2015 at 01:33, Patrick Maroney < Pmaroney@specere.org > wrote: I would like to add a couple of conceptual ideas to this discussion (along with a specific illustrative reference):   (1)  If we architect our flexible "patterning" model correctly it 'should' be directly applicable to our TAXII Query Language.   (2) Graph data models provide some interesting correlations/visualizations.  As previously suggested neo4j's Cypher Query language might serve as a good conceptual model for how to express "patterns"   http://assets.neo4j.org/download/Neo4j_CheatSheet_v3.pdf Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org   On Fri, Sep 25, 2015 at 8:12 AM -0700, "Patrick Maroney" < Pmaroney@Specere.org > wrote: A general [+1] for everything @Ivan said below (although agnostic on some of the specifics)   To this "flexible" patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:  "a" followed by "b" , "a" during "b" ) along with compound boolean and set theory _expression_. Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org   On Fri, Sep 25, 2015 at 6:13 AM -0700, "Kirillov, Ivan A." < ikirillov@mitre.org > wrote: To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot.    I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern.   Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-)    So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable:   pattern example_1  {   objects: $OBJ1 = {type = AddressObject,          fields = [{“category”:”ipv4-addr”},                    {“address_value”:”192.168.2.3”}]}      $OBJ2 = {type = AddressObject,          fields = [{“category”:”ipv4-addr”},                    {“address_value”:”192.168.4.7”}]}  condition:     OBJ1 and OBJ2 }   Regards, Ivan   On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote:   Adding my thoughts to the Test_Mechanism idea.   In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers.   If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of:   Title: Send a signature   Description: An analyst creates a signature/indicator and chooses to share it with partner organizations.   Main Success Scenario:   1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern) 2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator) 3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and 3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator) 4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action)   Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing.   Thank you. -Mark   [1] http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html  


  • 30.  Re: [cti] Re: Observable Patterning

    Posted 09-29-2015 06:31
    I would agree Mark, I do think the topics are intertwined. Even after looking at this for some time, I still couldn't decide whether patterning should sit within CybOX, STIX or both. So I decided to try and take a step back to look at what the bigger picture. I decided to look at the type of scenarios that one would want to describe, and see which patterning option would best cover those scenarios. As I see it there are a few major types of scenarios one would want to capture using patterns: A single fully defined pattern to look for exactly the same thing someone else found (e.g. www.badsite.com ) A single regex pattern to look for somethign similar to what others found (e.g. xx3[a-b]{3}.*.com) Send a group of patterns describing a single event (happened at the same time), each pattern different yet related in some way (e.g. email with a sender address bob@notevil.com and subject 'found any cats') Send groups of patterns describing a multiple sequential events that together indicate something bad is happening (e.g. dns lookup to *. evilsite.com followed by an email with a sender address bob@notevil.com and subject 'found any cats') I used the above scenarios along with the following guidance found within the STIX and CybOX sites: "In STIX, an Indicator is used to represent detection guidance: it captures how to detect something (i.e. a pattern of what to look for) and what it means if you find it. For example, "If you see 192.168.1.1 in your network traffic it means you might have detected C2 traffic from a ZeuS trojan."" "CybOX is a structured language for the specification, capture, characterization, and communication of events or stateful properties that are observable in the operational domain." I then tried to pull these together to describe the 3 different options we have for where patterns sit.  At this point, I prefer Option 1, but I would like to see what others think. ----- Option 1: Patterns only in CybOX, removed from STIX This option changes the Indicator object to only have a single Cybox Pattern description in it. That single pattern contains as little or as much information in it to describe the scenarios listed above. Using the great posts within the Github issue ( https://github.com/CybOXProject/schemas/issues/381 ) I've attempted to describe the 4 scenarios I listed earlier using a JSON style notation, as if the patterns and sequencing was done within CybOX. THe resulting psuedo code is below. Please note: The cybox patterns would all sit within a STIX Indicator object. I haven't put the STIX Indicator in the psuedo code. Single item: pattern: {         id: id1,         objects: [{                 id: obj1                  type: DomainObject                 properties: [{                         path: /type                         condition: EQUALS                           value: FQDN                 },{                         path: /value                         condition: EQUALS                           value: www.badsite.com                 }]         }],         condition {                 obj1         }  } Single regex: pattern: {         id: id2,         objects: [{                 id: obj1                  type: DomainObject                 properties: [{                         path: /type                         condition: EQUALS                           value: FQDN                 },{                         path: /value                         condition: MATCHES                           value: xx3[a-b]{3}.*.com                 }]         }],         condition {                 obj1         }  } Group of Patterns: pattern: {         id: id3,         objects: [{                 id: obj1                  type: FileObject                 properties: [{                         path: /hashes/hash/simple_hash_value                         condition: EQUALS                           value: c38862b4835729d979e7940d72a48172         },{                 id: obj2                  type: FileObject                 properties: [{                         path: /filename                         condition: CONTAINS                           value: malicious.dll                 }]         }],         condition {                 obj1 OR obj2         }  } Sequence of Group of Patterns: pattern: {         id: id4,         objects: [{                 id: obj1                  type: DomainObject                 properties: [{                         path: /type                         condition: EQUALS                           value: FQDN                 },{                         path: /value                         condition: EQUALS                           value: www.badsite.com                 }]         }],         condition {                 obj1         }  }, pattern: {         id: id5,         objects: [{                 id: obj1                  type: FileObject                 properties: [{                         path: /hashes/hash/simple_hash_value                         condition: EQUALS                           value: c38862b4835729d979e7940d72a48172         },{                 id: obj2                  type: FileObject                 properties: [{                         path: /filename                         condition: CONTAINS                           value: malicious.dll                 }]         }],         condition {                 obj1 OR obj2         }  }, pattern_sequence: {         id: seqid1,         sequence: [{                 idref: id4,                 order: 1         },{                 idref: id5,                 order: 2,                 after: PT5S   #ISO8601 duration standard format (after 5 secs)                 within: PT1M   #ISO8601 duration standard format (within          }] } Pros: patterns are all described within one area - CybOX patterns are available to all upstream languages that use CybOX by keeping patterns within CybOX they are easy to keep in step with the version of CybOX. If they lived in STIX then changes to the objects would potentially require changes to STIX. simplifies STIX by removing the need for Indicator Observable_Composition (although that would potentially still be needed for a Sighting Object's observable instances) potentially simplifies STIX by reducing the need for Composite_Indicator_Expression (although this needs more thought) Cons:  Doesn't align with the stated purpose of CybOX to describe only observable properties ----- Option 2: Patterns only in STIX, uses CybOX as it is used now This option reuses the Indicator object's Observable Composition as the thing that offers the patterning. This is trying to use the existing way of doing things but in a pattern style. Using a JSON style notation, the resulting psuedo code is below. Please note, there isn't any way of sequencing indicators easily at present within STIX, so I took the liberty of adding the indicator_sequence section (although there is bound to be a better way of doing it). There are Observable Events/Actions that could be used instead to potentially describe the temporal nature of the indicator sequence but I couldnt work out how to make that easily discernable. Single item: indicator: {         id: id1,         observable: {                 object: [{                         id: obj1                          type: DomainObject                         properties: [{                                 path: /type                                 condition: EQUALS                                   value: FQDN                         },{                                 path: /value                                 condition: EQUALS                                   value: www.badsite.com                         }]                 }],         }  } Single regex: indicator: {         id: id2,         observable: {                 object: [{                         id: obj1                          type: DomainObject                         properties: [{                                 path: /type                                 condition: EQUALS                                   value: FQDN                         },{                                 path: /value                                 condition: EQUALS                                   value: xx3[a-b]{3}.*.com                         }]                 }],         }  } Group of Patterns: indicator: {         id: id3,         observable: {                 observable_composition: {                         operator: OR                         observable: {                                 object: {                                         id: obj1                                          type: FileObject                                         properties: [{                                                 path: /hashes/hash/simple_hash_value                                                 condition: EQUALS                                                   value: c38862b4835729d979e7940d72a48172                                         }]                                 }                         },                         observable: {                                 object: {                                         id: obj1                                          type: FileObject                                         properties: [{                                                 path: /hashes/hash/simple_hash_value                                                 condition: EQUALS                                                   value: c38862b4835729d979e7940d72a48172                                         }]                                 }                         }                 }         } } Sequence of Group of Patterns: indicator: {         id: id4,         observable: {                 object: [{                         id: obj1                          type: DomainObject                         properties: [{                                 path: /type                                 condition: EQUALS                                   value: FQDN                         },{                                 path: /value                                 condition: EQUALS                                   value: xx3[a-b]{3}.*.com                         }]                 }],         }  }, indicator: {         id: id3,         observable: {                 observable_composition: {                         operator: OR                         observable: {                                 object: {                                         id: obj1                                          type: FileObject                                         properties: [{                                                 path: /hashes/hash/simple_hash_value                                                 condition: EQUALS                                                   value: c38862b4835729d979e7940d72a48172                                         }]                                 }                         },                         observable: {                                 object: {                                         id: obj1                                          type: FileObject                                         properties: [{                                                 path: /hashes/hash/simple_hash_value                                                 condition: EQUALS                                                   value: c38862b4835729d979e7940d72a48172                                         }]                                 }                         }                 }         } }, indicator_sequence: {         id: seqid1,         sequence: [{                 idref: id4,                 order: 1         },{                 idref: id5,                 order: 2,                 after: PT5S   #ISO8601 duration standard format (after 5 secs)                 within: PT1M   #ISO8601 duration standard format (within          }] } Pros: reuses existing Indicator Observable_Composition objects Cons:  patterns are only described within STIX. Not available to any other languages such as MAEC or CAPEC Patterns don't live within cybox packages, so might be harder to keep STIX and CybOX in step. ----- Option 3: Patterns only in STIX, no CybOX This option would make patterns only in the STIX Indicator object and not use CybOX objects at all, which in my opinion is wrong. So I didn't even bother fleshing out that scenario :). Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers. On 29 September 2015 at 02:34, Davidson II, Mark S < mdavidson@mitre.org > wrote: I think there’s two intertwined topics to disambiguate:   ·          Querying STIX information, where some see STIX as a graph-based model o    E.g., Tell me all the Indicators related to this TTP ·          CybOX patterns, which are much more signature-like o    E.g., Find the file with Hash=0x1234   When I think of a CybOX pattern, I tend to think of something that is a peer to signature/Test Mechanism.   Thank you. -Mark   From: cti@lists.oasis-open.org [mailto: cti@lists.oasis-open.org ] On Behalf Of Joep Gommers Sent: Monday, September 28, 2015 11:58 AM To: Kirillov, Ivan A. < ikirillov@mitre.org >; Jerome Athias < athiasjerome@gmail.com >; Patrick Maroney < Pmaroney@specere.org > Cc: Terry MacDonald < terry.macdonald@threatloop.com >; cti@lists.oasis-open.org Subject: Re: [cti] Re: Observable Patterning   Hi Ivan,   I’d be interested what the team feels about the relationship between these patterns and test mechanism concept. Please correct me if wrong; - indicator patterns described a pattern of potentially malicious activity that is intended to convey the pattern in a way that a human can understand and that machines can translate into action-based use-cases (like detection) - test mechanisms are a pre-computed or created pattern specific to a certain use-case (like snort detection, yara, etc.)   Since in both cases that might be conditional relationships and nested conditional calculations (e.g. If you don’t see X, then the previous variable in de flow is +1, if you do seeX, then the previous variable in the flow is –1) - it makes it ill-suited for graph representation in the context of graph querying.   J-   From: <Kirillov>, "Ivan A." < ikirillov@mitre.org > Date: Monday, September 28, 2015 at 3:36 PM To: Jerome Athias < athiasjerome@gmail.com >, Patrick Maroney < Pmaroney@specere.org > Cc: Terry MacDonald < terry.macdonald@threatloop.com >, " cti@lists.oasis-open.org " < cti@lists.oasis-open.org > Subject: Re: [cti] Re: Observable Patterning   >- Human readable      Agreed. >- Machine friendly (parsing/computation, algorithmically friendly)      Agreed. >- Data graph friendly      Not so sure on this one.    While I agree that STIX/TAXII querying (wherever it ends up) is inherently graph-based, I don’t believe the same is true for Indicator patterning. The main difference is that, in my view, the majority of Indicator patterns are meant to be parsed and executed against data that is inherently flat – lists of files, lists of processes, lists of IP addresses, etc. Thus while the two may overlap in certain places, I’d be very hesitant about overloading an Indicator patterning structure to support graph based querying, though perhaps it could be extended to do so. Again, I think a primary focus here should be to keep Indicator patterns SIMPLE, so that they can easily be written and consumed by analysts.   Regards, Ivan   From: Jerome Athias Date: Saturday, September 26, 2015 at 12:38 PM To: Patrick Maroney Cc: Terry MacDonald, " cti@lists.oasis-open.org ", Ivan Kirillov Subject: Re: [cti] Re: Observable Patterning   so should we try to capture the requirements in one place for this change request? Quickly: - Human readable - Machine friendly (parsing/computation, algorithmically friendly) - Data graph friendly   2015-09-26 17:57 GMT+03:00 Patrick Maroney < Pmaroney@specere.org >: Probably not clear that I've similarly argued to keep TAXII focused on the transport of packages.  In the query method suggested for consideration, the Pattern would be passed as a STIX (Query/RFI) package and results similarly returned (again as a STIX package).   It's important to visualize this a Data Graph that maps to the Conceptual CTI model (which is the basis for suggesting Cypher as an example representation).  One can describe/query precise graphs of Objects/Relationships, any Nodes/Edges off of a Root Node, etc.   Hope this makes sense (at least in terms of framing the proposal). Sent from Outlook   On Fri, Sep 25, 2015 at 6:01 PM -0700, "Terry MacDonald" < terry.macdonald@threatloop.com > wrote: I agree with Patrick's comments about ensuring pattern support for querying, but not the idea querying should be done in TAXII. Querying and answering should be done within STIX, so that the querying and answering done by the language contained within TAXII, and so that TAXII is free to concentrate on only delivering content. We should instead develop a STIX query object, and a STIX reply object in order to support question and answer directly within STIX. I won't go into the specific argument why here and clog this thread, but it has been discussed on the TAXII list recently.   Cheers Terry MacDonald STIX, TAXII, CybOX Consultant   M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com     Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers.   On 26 September 2015 at 01:33, Patrick Maroney < Pmaroney@specere.org > wrote: I would like to add a couple of conceptual ideas to this discussion (along with a specific illustrative reference):   (1)  If we architect our flexible "patterning" model correctly it 'should' be directly applicable to our TAXII Query Language.   (2) Graph data models provide some interesting correlations/visualizations.  As previously suggested neo4j's Cypher Query language might serve as a good conceptual model for how to express "patterns"   http://assets.neo4j.org/download/Neo4j_CheatSheet_v3.pdf Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org   On Fri, Sep 25, 2015 at 8:12 AM -0700, "Patrick Maroney" < Pmaroney@Specere.org > wrote: A general [+1] for everything @Ivan said below (although agnostic on some of the specifics)   To this "flexible" patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:  "a" followed by "b" , "a" during "b" ) along with compound boolean and set theory _expression_. Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org   On Fri, Sep 25, 2015 at 6:13 AM -0700, "Kirillov, Ivan A." < ikirillov@mitre.org > wrote: To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot.    I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern.   Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-)    So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable:   pattern example_1  {   objects: $OBJ1 = {type = AddressObject,          fields = [{“category”:”ipv4-addr”},                    {“address_value”:”192.168.2.3”}]}      $OBJ2 = {type = AddressObject,          fields = [{“category”:”ipv4-addr”},                    {“address_value”:”192.168.4.7”}]}  condition:     OBJ1 and OBJ2 }   Regards, Ivan   On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote:   Adding my thoughts to the Test_Mechanism idea.   In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers.   If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of:   Title: Send a signature   Description: An analyst creates a signature/indicator and chooses to share it with partner organizations.   Main Success Scenario:   1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern) 2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator) 3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and 3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator) 4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action)   Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing.   Thank you. -Mark   [1] http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html  


  • 31.  Re: [cti] Re: Observable Patterning

    Posted 09-29-2015 13:53




    Great post, Terry (I really like the JSON examples :)).  A few thoughts from my perspective:





    A single fully defined pattern to look for exactly the same thing someone else found (e.g.  www.badsite.com ) A single regex pattern to look for somethign similar to what others found (e.g. xx3[a-b]{3}.*.com) Send a group of patterns describing a single event (happened at the same time), each pattern different yet related in some way (e.g. email with a sender address  bob@notevil.com  and
    subject 'found any cats') Send groups of patterns describing a multiple sequential events that together indicate something bad is happening (e.g. dns lookup to *. evilsite.com  followed
    by an email with a sender address  bob@notevil.com  and subject 'found any cats')






    These are all valid scenarios for Indicator-based patterns, for detecting some particular flavor of badness. This is an important distinction, as in my view patterns should be used only for Indicators – in other places, their
    semantics are less clear (what do they mean? how should they be used?). It’s also worth noting that in such cases when we’re referring to events, we’re really referring to the corresponding detectable artifacts of the events, such as emails, DNS queries, and
    the like. 




    Option 1: Patterns only in CybOX, removed from STIX




    Pros:


    patterns are all described within one area – CybOX patterns are available to all upstream languages that use CybOX

    [Ivan] I’m still on the fence as to whether this is useful or not. Patterns have very clear semantics in Indicators, but I’m not sure that’s the case elsewhere. What would a digital forensics pattern mean, and how it would
    be used (as an example)?
    by keeping patterns within CybOX they are easy to keep in step with the version of CybOX. If they lived in STIX then changes to the objects would
    potentially require changes to STIX.

    [Ivan] True, but STIX would still have a hard dependency on CybOX. If CybOX were to be updated, STIX would have to follow step to use any new features/changes in CybOX. 
    simplifies STIX by removing the need for Indicator Observable_Composition (although that would potentially still be needed for a Sighting Object's
    observable instances)

    [Ivan] Agreed! I think separate Patterns can indeed subsume the capabilities of/need for a separate Observable_Composition structure.
    potentially simplifies STIX by reducing the need for Composite_Indicator_Expression (although this needs more thought)


    Cons: 


    Doesn't align with the stated purpose of CybOX to describe only observable properties

    [Ivan] I think this is a big one; I think a large chunk of the inherent complexity in CybOX stems from its duality of expressing observable properties (instances) and also patterns. My feeling
    is that CybOX should focus on observable instances and the corresponding Object data models, while patterning and other expressions against these data models should be defined elsewhere (likely STIX). To me at least this seems like a logical delineation of
    functionality.


    Option 3: Patterns only in STIX, no CybOX





    Personally, this is my preferred option :-) I like to think about it this way:

    CybOX defines the set of Object data models and the necessary corresponding structures around them

    Used for capturing only observable instances
    STIX defines a patterning implementation with a
    binding to the CybOX data models

    Other bindings could be developed for other data models, for expressing patterns against different domains, such as fraud (as an example)

    A binding to CybOX would essentially be a flattened list of CybOX Object fields, for use in patterning. This would be defined against a specific version of CybOX, and would likely exclude fields
    (and even possibly Objects) that don’t make sense for detection, such as entities like Process IDs that are temporal and/or non-deterministic in nature. It would also allow us to state exactly what a particular field means in a pattern and how it should be
    detected.


    Other bindings could then be developed against other, non-CybOX data models. For instance, I know there’s been some discussion in the past on detecting money mule activity – if we had a domain
    independent patterning implementation, we could develop a binding against an ACH bank transaction data model (for instance), and use it to express patterns against it. 


    So as a strawman, in JSON this could look something like (where the type field indicates the CybOX binding):







    indicator: {
            id: id1,
            pattern: {object: {
                                id: obj1,
                                type: cybox2.1.1:FileObject,
                                properties: [{
                                        path: /hashes/hash/simple_hash_value
                                        condition: EQUALS 
                                        value: c38862b4835729d979e7940d72a48172
                                }]},
                     object: {
                                id: obj2,
                                type: cybox2.1.1:FileObject,
                                properties: [{
                                        path: /hashes/hash/simple_hash_value
                                        condition: EQUALS 
                                        value: c38862b4835729d979e7940d72a48172
                                }]}
                            },
                     condition: "obj1 AND obj2"
    }




    Pros:


    Clear delineation in function between CybOX Observables and STIX Indicators (single responsibility principle)

    CybOX Objects & Observables: instances of “what” was seen STIX Indicators: patterns of “what” to look for and “how” to look for it (conditional/temporal logic/etc.)
    Extensibility for non-CybOX based patterning

    Cons: 


    Data-model specific bindings for patterning are another document/specification that must be written

    Must be updated for every CybOX release where changes are made to the Object data models

    Regards,


    Ivan




    From: Terry MacDonald
    Date: Tuesday, September 29, 2015 at 2:30 AM
    To: Mark Davidson
    Cc: Joep Gommers, Ivan Kirillov, Jerome Athias, Patrick Maroney, Terry MacDonald, " cti@lists.oasis-open.org "
    Subject: Re: [cti] Re: Observable Patterning





    I would agree Mark, I do think the topics are intertwined. Even after looking at this for some time, I still couldn't decide whether patterning should sit within CybOX, STIX or both. So I decided to try and take a step back to look at what the
    bigger picture.


    I decided to look at the type of scenarios that one would want to describe, and see which patterning option would best cover those scenarios. As I see it there are a few major types of scenarios one would want to capture using patterns:


    A single fully defined pattern to look for exactly the same thing someone else found (e.g.
    www.badsite.com ) A single regex pattern to look for somethign similar to what others found (e.g. xx3[a-b]{3}.*.com) Send a group of patterns describing a single event (happened at the same time), each pattern different yet related in some way (e.g. email with a sender address
    bob@notevil.com and subject 'found any cats') Send groups of patterns describing a multiple sequential events that together indicate something bad is happening (e.g. dns lookup to *. evilsite.com followed by an email with a sender address
    bob@notevil.com and subject 'found any cats')
    I used the above scenarios along with the following guidance found within the STIX and CybOX sites:


    "In STIX, an Indicator is used to represent detection guidance: it captures how to detect something (i.e. a pattern of what to look for) and what it means if you find it. For example, "If you see 192.168.1.1 in your network traffic it means you might have
    detected C2 traffic from a ZeuS trojan.""
    "CybOX is a structured language for the specification, capture, characterization, and communication of events or stateful properties that are observable in the operational domain."



    I then tried to pull these together to describe the 3 different options we have for where patterns sit. 


    At this point, I prefer Option 1, but I would like to see what others think.


    -----



    Option 1: Patterns only in CybOX, removed from STIX


    This option changes the Indicator object to only have a single Cybox Pattern description in it. That single pattern contains as little or as much information in it to describe the scenarios listed above. Using the great posts within the Github issue ( https://github.com/CybOXProject/schemas/issues/381 )
    I've attempted to describe the 4 scenarios I listed earlier using a JSON style notation, as if the patterns and sequencing was done within CybOX. THe resulting psuedo code is below. Please note: The cybox patterns would all sit within a STIX Indicator object.
    I haven't put the STIX Indicator in the psuedo code.


    Single item:


    pattern: {
            id: id1,
            objects: [{
                    id: obj1 
                    type: DomainObject
                    properties: [{
                            path: /type
                            condition: EQUALS  
                            value: FQDN
                    },{
                            path: /value


                            condition: EQUALS  
                            value:
    www.badsite.com
                    }]
            }],

            condition {
                    obj1
            }
     }


    Single regex:




    pattern: {
            id: id2,
            objects: [{
                    id: obj1 
                    type: DomainObject
                    properties: [{
                            path: /type
                            condition: EQUALS  
                            value: FQDN
                    },{
                            path: /value


                            condition: MATCHES  
                            value: xx3[a-b]{3}.*.com
                    }]
            }],

            condition {
                    obj1
            }
     }




    Group of Patterns:





    pattern: {
            id: id3,
            objects: [{
                    id: obj1 
                    type: FileObject
                    properties: [{
                            path: /hashes/hash/simple_hash_value
                            condition: EQUALS  
                            value: c38862b4835729d979e7940d72a48172
            },{

                    id: obj2 
                    type: FileObject
                    properties: [{
                            path: /filename
                            condition: CONTAINS  
                            value: malicious.dll
                    }]



            }],

            condition {
                    obj1 OR obj2
            }
     }





    Sequence of Group of Patterns:





    pattern: {
            id: id4,
            objects: [{
                    id: obj1 
                    type: DomainObject
                    properties: [{
                            path: /type
                            condition: EQUALS  
                            value: FQDN
                    },{
                            path: /value


                            condition: EQUALS  
                            value:
    www.badsite.com
                    }]
            }],

            condition {
                    obj1
            }
     },


    pattern: {
            id: id5,
            objects: [{
                    id: obj1 
                    type: FileObject
                    properties: [{
                            path: /hashes/hash/simple_hash_value
                            condition: EQUALS  
                            value: c38862b4835729d979e7940d72a48172
            },{

                    id: obj2 
                    type: FileObject
                    properties: [{
                            path: /filename
                            condition: CONTAINS  
                            value: malicious.dll
                    }]


            }],
            condition {
                    obj1 OR obj2
            }
     },

    pattern_sequence: {
            id: seqid1,
            sequence: [{
                    idref: id4,
                    order: 1
            },{
                    idref: id5,
                    order: 2,
                    after: PT5S   #ISO8601 duration standard format (after 5 secs)

                    within: PT1M   #ISO8601 duration standard format (within 
            }]


    }



    Pros:


    patterns are all described within one area - CybOX patterns are available to all upstream languages that use CybOX by keeping patterns within CybOX they are easy to keep in step with the version of CybOX. If they lived in STIX then changes to the objects would potentially require changes to STIX. simplifies STIX by removing the need for Indicator Observable_Composition (although that would potentially still be needed for a Sighting Object's observable instances) potentially simplifies STIX by reducing the need for Composite_Indicator_Expression (although this needs more thought)


    Cons: 


    Doesn't align with the stated purpose of CybOX to describe only observable properties



    -----


    Option 2: Patterns only in STIX, uses CybOX as it is used now


    This option reuses the Indicator object's Observable Composition as the thing that offers the patterning. This is trying to use the existing way of doing things but in a pattern style. Using a JSON style notation, the resulting psuedo code is below. Please
    note, there isn't any way of sequencing indicators easily at present within STIX, so I took the liberty of adding the indicator_sequence section (although there is bound to be a better way of doing it). There are Observable Events/Actions that could be used
    instead to potentially describe the temporal nature of the indicator sequence but I couldnt work out how to make that easily discernable.


    Single item:


    indicator: {
            id: id1,
            observable: {
                    object: [{
                            id: obj1 
                            type: DomainObject
                            properties: [{
                                    path: /type
                                    condition: EQUALS  
                                    value: FQDN
                            },{
                                    path: /value


                                    condition: EQUALS  
                                    value:
    www.badsite.com
                            }]
                    }],

            }
     }



    Single regex:




    indicator: {
            id: id2,

            observable: {
                    object: [{
                            id: obj1 
                            type: DomainObject
                            properties: [{
                                    path: /type
                                    condition: EQUALS  
                                    value: FQDN
                            },{
                                    path: /value


                                    condition: EQUALS  
                                    value: xx3[a-b]{3}.*.com
                            }]
                    }],

            }
     }





    Group of Patterns:





    indicator: {
            id: id3,

            observable: {

                    observable_composition: {

                            operator: OR

                            observable: {


                                    object: {


                                            id: obj1 
                                            type: FileObject
                                            properties: [{
                                                    path: /hashes/hash/simple_hash_value
                                                    condition: EQUALS  
                                                    value: c38862b4835729d979e7940d72a48172
                                            }]
                                    }
                            },



                            observable: {


                                    object: {


                                            id: obj1 
                                            type: FileObject
                                            properties: [{
                                                    path: /hashes/hash/simple_hash_value
                                                    condition: EQUALS  
                                                    value: c38862b4835729d979e7940d72a48172
                                            }]

                                    }
                            }
                    }
            }
    }







    Sequence of Group of Patterns:





    indicator: {
            id: id4,

            observable: {
                    object: [{
                            id: obj1 
                            type: DomainObject
                            properties: [{
                                    path: /type
                                    condition: EQUALS  
                                    value: FQDN
                            },{
                                    path: /value


                                    condition: EQUALS  
                                    value: xx3[a-b]{3}.*.com
                            }]
                    }],

            }
     },


    indicator: {
            id: id3,

            observable: {
                    observable_composition: {
                            operator: OR

                            observable: {


                                    object: {


                                            id: obj1 
                                            type: FileObject
                                            properties: [{
                                                    path: /hashes/hash/simple_hash_value
                                                    condition: EQUALS  
                                                    value: c38862b4835729d979e7940d72a48172
                                            }]
                                    }
                            },



                            observable: {


                                    object: {


                                            id: obj1 
                                            type: FileObject
                                            properties: [{
                                                    path: /hashes/hash/simple_hash_value
                                                    condition: EQUALS  
                                                    value: c38862b4835729d979e7940d72a48172
                                            }]

                                    }
                            }
                    }
            }
    },

    indicator_sequence: {


            id: seqid1,
            sequence: [{
                    idref: id4,
                    order: 1
            },{
                    idref: id5,
                    order: 2,
                    after: PT5S   #ISO8601 duration standard format (after 5 secs)

                    within: PT1M   #ISO8601 duration standard format (within 
            }]


    }



    Pros:


    reuses existing Indicator Observable_Composition objects

    Cons: 


    patterns are only described within STIX. Not available to any other languages such as MAEC or CAPEC
    Patterns don't live within cybox packages, so might be harder to keep STIX and CybOX in step.




    -----



    Option 3: Patterns only in STIX, no CybOX


    This option would make patterns only in the STIX Indicator object and not use CybOX objects at all, which in my opinion is wrong. So I didn't even bother fleshing out that scenario :).













    Cheers

    Terry MacDonald STIX, TAXII, CybOX Consultant



    M: +61-407-203-026
    E:  terry.macdonald@threatloop.com
    W:  www.threatloop.com






    Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those
    of my employers.








    On 29 September 2015 at 02:34, Davidson II, Mark S
    < mdavidson@mitre.org > wrote:



    I think there’s two intertwined topics to disambiguate:
     
    ·         
    Querying STIX information, where some see STIX as a graph-based model
    o   
    E.g., Tell me all the Indicators related to this TTP
    ·         
    CybOX patterns, which are much more signature-like
    o   
    E.g., Find the file with Hash=0x1234
     
    When I think of a CybOX pattern, I tend to think of something that is a peer to signature/Test Mechanism.

     
    Thank you.
    -Mark
     


    From: cti@lists.oasis-open.org
    [mailto: cti@lists.oasis-open.org ]
    On Behalf Of Joep Gommers
    Sent: Monday, September 28, 2015 11:58 AM
    To: Kirillov, Ivan A. < ikirillov@mitre.org >; Jerome Athias < athiasjerome@gmail.com >; Patrick Maroney < Pmaroney@specere.org >
    Cc: Terry MacDonald < terry.macdonald@threatloop.com >;
    cti@lists.oasis-open.org


    Subject: Re: [cti] Re: Observable Patterning






     

    Hi Ivan,


     


    I’d be interested what the team feels about the relationship between these patterns and test mechanism concept. Please correct me
    if wrong;


    - indicator patterns described a pattern of potentially malicious activity that is intended to convey the pattern in a way that a
    human can understand and that machines can translate into action-based use-cases (like detection)


    - test mechanisms are a pre-computed or created pattern specific to a certain use-case (like snort detection, yara, etc.)


     


    Since in both cases that might be conditional relationships and nested conditional calculations (e.g. If you don’t see X, then the
    previous variable in de flow is +1, if you do seeX, then the previous variable in the flow is –1) - it makes it ill-suited for graph representation in the context of graph querying.


     


    J-


     


    From:
    <Kirillov>, "Ivan A." < ikirillov@mitre.org >
    Date: Monday, September 28, 2015 at 3:36 PM
    To: Jerome Athias < athiasjerome@gmail.com >, Patrick Maroney < Pmaroney@specere.org >
    Cc: Terry MacDonald < terry.macdonald@threatloop.com >, " cti@lists.oasis-open.org " < cti@lists.oasis-open.org >
    Subject: Re: [cti] Re: Observable Patterning


     







    >- Human readable


         Agreed.


    >- Machine friendly (parsing/computation, algorithmically friendly)


         Agreed.


    >- Data graph friendly


         Not so sure on this one. 


     


    While I agree that STIX/TAXII querying (wherever it ends up) is inherently graph-based, I don’t believe the same is true for Indicator
    patterning. The main difference is that, in my view, the majority of Indicator patterns are meant to be parsed and executed against data that is inherently flat – lists of files, lists of processes, lists of IP addresses, etc. Thus while the two may overlap
    in certain places, I’d be very hesitant about overloading an Indicator patterning structure to support graph based querying, though perhaps it could be extended to do so. Again, I think a primary focus here should be to keep Indicator patterns SIMPLE, so that
    they can easily be written and consumed by analysts.


     


    Regards,


    Ivan


     





    From:
    Jerome Athias
    Date: Saturday, September 26, 2015 at 12:38 PM
    To: Patrick Maroney
    Cc: Terry MacDonald, " cti@lists.oasis-open.org ", Ivan Kirillov
    Subject: Re: [cti] Re: Observable Patterning


     




    so should we try to capture the requirements in one place for this change request?


    Quickly:


    - Human readable


    - Machine friendly (parsing/computation, algorithmically friendly)


    - Data graph friendly



     

    2015-09-26 17:57 GMT+03:00 Patrick Maroney < Pmaroney@specere.org >:



    Probably not clear that I've similarly argued to keep TAXII focused on the transport of packages.  In the query method suggested for
    consideration, the Pattern would be passed as a STIX (Query/RFI) package and results similarly returned (again as a STIX package).


     


    It's important to visualize this a Data Graph that maps to the Conceptual CTI model (which is the basis for suggesting Cypher as an
    example representation).  One can describe/query precise graphs of Objects/Relationships, any Nodes/Edges off of a Root Node, etc.


     



    Hope this makes sense (at least in terms of framing the proposal).

    Sent from
    Outlook

     









    On Fri, Sep 25, 2015 at 6:01 PM -0700, "Terry MacDonald" < terry.macdonald@threatloop.com > wrote:



    I agree with Patrick's comments about ensuring pattern support for querying, but not the idea querying should be done in TAXII.
    Querying and answering should be done within STIX, so that the querying and answering done by the language contained within TAXII, and so that TAXII
    is free to concentrate on only delivering content. We should instead develop a STIX query object, and a STIX reply object in order to support question and answer directly within STIX. I won't go into the specific argument why here and clog this thread, but
    it has been discussed on the TAXII list recently.  
    Cheers








    Terry MacDonald STIX, TAXII, CybOX Consultant


     


    M:
    +61-407-203-026


    E:  terry.macdonald@threatloop.com


    W:  www.threatloop.com


     





     


    Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views
    do not necessarily reflect those of my employers.






     

    On 26 September 2015 at 01:33, Patrick Maroney < Pmaroney@specere.org > wrote:



    I would like to add a couple of conceptual ideas to this discussion (along with a specific illustrative reference):


     


    (1)  If we architect our flexible "patterning" model correctly it 'should' be directly applicable to our TAXII Query Language.


     


    (2) Graph data models provide some interesting correlations/visualizations.  As previously suggested neo4j's Cypher Query language
    might serve as a good conceptual model for how to express "patterns"


     



    http://assets.neo4j.org/download/Neo4j_CheatSheet_v3.pdf

    Patrick Maroney
    President
    Integrated Networking Technologies, Inc.
    Desk: (856)983-0001
    Cell: (609)841-5104
    Email: pmaroney@specere.org

     









    On Fri, Sep 25, 2015 at 8:12 AM -0700, "Patrick Maroney" < Pmaroney@Specere.org > wrote:



    A general [+1] for everything @Ivan said below (although agnostic on some of the specifics)


     



    To this "flexible" patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:  "a" followed by "b" , "a" during "b"
    ) along with compound boolean and set theory _expression_.

    Patrick Maroney
    President
    Integrated Networking Technologies, Inc.
    Desk: (856)983-0001
    Cell: (609)841-5104
    Email: pmaroney@specere.org

     







    On Fri, Sep 25, 2015 at 6:13 AM -0700, "Kirillov, Ivan A." < ikirillov@mitre.org > wrote:





    To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there
    is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below,
    I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot. 




     


    I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects
    and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I
    think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can
    be better expressed, say, through a YARA rule than through a CybOX pattern.


     


    Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being
    used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account
    (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-) 


     


    So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge
    on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable:


     



    pattern example_1 


    {


      objects:
    $OBJ1 = {type = AddressObject,
             fields = [{“category”:”ipv4-addr”},
                       {“address_value”:”192.168.2.3”}]} 
       
    $OBJ2 = {type = AddressObject,
             fields = [{“category”:”ipv4-addr”},
                       {“address_value”:”192.168.4.7”}]} 
    condition:
        OBJ1 and OBJ2
    }
     



    Regards,


    Ivan


     


    On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote:


     



    Adding my thoughts to the Test_Mechanism idea.


     


    In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my
    mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The
    way I'm thinking would have all signature-ish things become peers.


     


    If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of:


     


    Title: Send a signature


     


    Description: An analyst creates a signature/indicator and chooses to share it with partner organizations.


     


    Main Success Scenario:


     


    1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern)


    2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator)


    3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and


    3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or
    STIX Indicator)


    4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action)


     


    Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement),
    but at least for me, this is how I understand the workflow from Indicator creation to information sharing.


     


    Thank you.


    -Mark


     


    [1]

    http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html


     





  • 32.  Re: [cti] Observable Patterning

    Posted 09-28-2015 17:22
    Joep, et all, I am struggling with this.  The way I understand this discussion to date, over the past several months is: 1) We need a way of saying someone saw something and their assertion about that is it was bad.   a) We need some way of tracking temporal aspects and chains of events.  Ex. PDF got downloaded, within 20 seconds 2 EXE get downloaded, 1 minute later a connection to x.y.z.a gets opened up and we assert that it is bad. b) Need a way to track sightings of said events c) Need a way to assert relationships about said events and add to or remove relationships of said events.  Or even give +1s or -1s to said assertions.   2) Test Mechanisms will have a lot of the same assertions and constructs but I see them, as you point out, as being specific patters for certain things.  So I can see these as being name value pairs where it might look like: { test_mechanisms : [ { key : snort , value: some snort signature in native snort syntax } ] } Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg.   On Sep 28, 2015, at 09:57, Joep Gommers < joep@intelworks.com > wrote: Hi Ivan, I’d be interested what the team feels about the relationship between these patterns and test mechanism concept. Please correct me if wrong; - indicator patterns described a pattern of potentially malicious activity that is intended to convey the pattern in a way that a human can understand and that machines can translate into action-based use-cases (like detection) - test mechanisms are a pre-computed or created pattern specific to a certain use-case (like snort detection, yara, etc.) Since in both cases that might be conditional relationships and nested conditional calculations (e.g. If you don’t see X, then the previous variable in de flow is +1, if you do seeX, then the previous variable in the flow is –1) - it makes it ill-suited for graph representation in the context of graph querying. J- From: <Kirillov>, Ivan A. < ikirillov@mitre.org > Date: Monday, September 28, 2015 at 3:36 PM To: Jerome Athias < athiasjerome@gmail.com >, Patrick Maroney < Pmaroney@specere.org > Cc: Terry MacDonald < terry.macdonald@threatloop.com >, cti@lists.oasis-open.org < cti@lists.oasis-open.org > Subject: Re: [cti] Re: Observable Patterning >- Human readable      Agreed. >- Machine friendly (parsing/computation, algorithmically friendly)      Agreed. >- Data graph friendly      Not so sure on this one.  While I agree that STIX/TAXII querying (wherever it ends up) is inherently graph-based, I don’t believe the same is true for Indicator patterning. The main difference is that, in my view, the majority of Indicator patterns are meant to be parsed and executed against data that is inherently flat – lists of files, lists of processes, lists of IP addresses, etc. Thus while the two may overlap in certain places, I’d be very hesitant about overloading an Indicator patterning structure to support graph based querying, though perhaps it could be extended to do so. Again, I think a primary focus here should be to keep Indicator patterns SIMPLE, so that they can easily be written and consumed by analysts. Regards, Ivan From: Jerome Athias Date: Saturday, September 26, 2015 at 12:38 PM To: Patrick Maroney Cc: Terry MacDonald, cti@lists.oasis-open.org , Ivan Kirillov Subject: Re: [cti] Re: Observable Patterning so should we try to capture the requirements in one place for this change request? Quickly: - Human readable - Machine friendly (parsing/computation, algorithmically friendly) - Data graph friendly 2015-09-26 17:57 GMT+03:00 Patrick Maroney < Pmaroney@specere.org > : Probably not clear that I've similarly argued to keep TAXII focused on the transport of packages.  In the query method suggested for consideration, the Pattern would be passed as a STIX (Query/RFI) package and results similarly returned (again as a STIX package). It's important to visualize this a Data Graph that maps to the Conceptual CTI model (which is the basis for suggesting Cypher as an example representation).  One can describe/query precise graphs of Objects/Relationships, any Nodes/Edges off of a Root Node, etc. Hope this makes sense (at least in terms of framing the proposal). Sent from Outlook On Fri, Sep 25, 2015 at 6:01 PM -0700, Terry MacDonald < terry.macdonald@threatloop.com > wrote: I agree with Patrick's comments about ensuring pattern support for querying, but not the idea querying should be done in TAXII. Querying and answering should be done within STIX, so that the querying and answering done by the language contained within TAXII, and so that TAXII is free to concentrate on only delivering content. We should instead develop a STIX query object, and a STIX reply object in order to support question and answer directly within STIX. I won't go into the specific argument why here and clog this thread, but it has been discussed on the TAXII list recently.   Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers. On 26 September 2015 at 01:33, Patrick Maroney < Pmaroney@specere.org > wrote: I would like to add a couple of conceptual ideas to this discussion (along with a specific illustrative reference): (1)  If we architect our flexible patterning model correctly it 'should' be directly applicable to our TAXII Query Language. (2) Graph data models provide some interesting correlations/visualizations.  As previously suggested neo4j's Cypher Query language might serve as a good conceptual model for how to express patterns http://assets.neo4j.org/download/Neo4j_CheatSheet_v3.pdf Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 8:12 AM -0700, Patrick Maroney < Pmaroney@Specere.org > wrote: A general [+1] for everything @Ivan said below (although agnostic on some of the specifics) To this flexible patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:   a followed by b , a during b ) along with compound boolean and set theory _expression_. Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 6:13 AM -0700, Kirillov, Ivan A. < ikirillov@mitre.org > wrote: To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot.  I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern. Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-)  So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable: pattern example_1  {   objects: $OBJ1 = {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.2.3 ” }]}       $OBJ2 =  {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.4.7 ” }]}   condition:     OBJ1 and OBJ2 }   Regards, Ivan On 9/25/15, 8:02 AM, Davidson II, Mark S < mdavidson@mitre.org > wrote: Adding my thoughts to the Test_Mechanism idea. In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, signature-ish things includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers. If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of: Title: Send a signature Description: An analyst creates a signature/indicator and chooses to share it with partner organizations. Main Success Scenario: 1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern) 2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator) 3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and 3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator) 4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action) Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing. Thank you. -Mark [1] http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html


  • 33.  Re: [cti] Re: Observable Patterning

    Posted 09-29-2015 18:20
    Joep, I don’t think there is a clear dichotomy as you suggest between the Observable (observable pattern) of an Indicator and any test mechanisms defined for it. They are really semantically the same thing. They are both simply observable patterns for identifying whether some particular factual observable conditions exist. These patterns are devoid of any context as to “malicious activity” or other wise. It is the enclosing Indicator that assigns such context to the pattern. You are correct in that test mechanisms would most likely be used (rather than Observable) based on some certain use case where the other format provides expressivity that CybOX does not or the fact that the targeted recipient of the Indicator has particular tooling capabilities that you are looking to serve. sean From: < cti@lists.oasis-open.org > on behalf of Joep Gommers < joep@intelworks.com > Date: Monday, September 28, 2015 at 11:57 AM To: Steve Cell < ikirillov@mitre.org >, Jerome Athias < athiasjerome@gmail.com >, Patrick Maroney < Pmaroney@Specere.org > Cc: Terry MacDonald < terry.macdonald@threatloop.com >, " cti@lists.oasis-open.org " < cti@lists.oasis-open.org > Subject: Re: [cti] Re: Observable Patterning Hi Ivan, I’d be interested what the team feels about the relationship between these patterns and test mechanism concept. Please correct me if wrong; - indicator patterns described a pattern of potentially malicious activity that is intended to convey the pattern in a way that a human can understand and that machines can translate into action-based use-cases (like detection) - test mechanisms are a pre-computed or created pattern specific to a certain use-case (like snort detection, yara, etc.) Since in both cases that might be conditional relationships and nested conditional calculations (e.g. If you don’t see X, then the previous variable in de flow is +1, if you do seeX, then the previous variable in the flow is –1) - it makes it ill-suited for graph representation in the context of graph querying. J- From: <Kirillov>, "Ivan A." < ikirillov@mitre.org > Date: Monday, September 28, 2015 at 3:36 PM To: Jerome Athias < athiasjerome@gmail.com >, Patrick Maroney < Pmaroney@specere.org > Cc: Terry MacDonald < terry.macdonald@threatloop.com >, " cti@lists.oasis-open.org " < cti@lists.oasis-open.org > Subject: Re: [cti] Re: Observable Patterning >- Human readable      Agreed. >- Machine friendly (parsing/computation, algorithmically friendly)      Agreed. >- Data graph friendly      Not so sure on this one.  While I agree that STIX/TAXII querying (wherever it ends up) is inherently graph-based, I don’t believe the same is true for Indicator patterning. The main difference is that, in my view, the majority of Indicator patterns are meant to be parsed and executed against data that is inherently flat – lists of files, lists of processes, lists of IP addresses, etc. Thus while the two may overlap in certain places, I’d be very hesitant about overloading an Indicator patterning structure to support graph based querying, though perhaps it could be extended to do so. Again, I think a primary focus here should be to keep Indicator patterns SIMPLE, so that they can easily be written and consumed by analysts. Regards, Ivan From: Jerome Athias Date: Saturday, September 26, 2015 at 12:38 PM To: Patrick Maroney Cc: Terry MacDonald, " cti@lists.oasis-open.org ", Ivan Kirillov Subject: Re: [cti] Re: Observable Patterning so should we try to capture the requirements in one place for this change request? Quickly: - Human readable - Machine friendly (parsing/computation, algorithmically friendly) - Data graph friendly 2015-09-26 17:57 GMT+03:00 Patrick Maroney < Pmaroney@specere.org > : Probably not clear that I've similarly argued to keep TAXII focused on the transport of packages.  In the query method suggested for consideration, the Pattern would be passed as a STIX (Query/RFI) package and results similarly returned (again as a STIX package). It's important to visualize this a Data Graph that maps to the Conceptual CTI model (which is the basis for suggesting Cypher as an example representation).  One can describe/query precise graphs of Objects/Relationships, any Nodes/Edges off of a Root Node, etc. Hope this makes sense (at least in terms of framing the proposal). Sent from Outlook On Fri, Sep 25, 2015 at 6:01 PM -0700, "Terry MacDonald" < terry.macdonald@threatloop.com > wrote: I agree with Patrick's comments about ensuring pattern support for querying, but not the idea querying should be done in TAXII. Querying and answering should be done within STIX, so that the querying and answering done by the language contained within TAXII, and so that TAXII is free to concentrate on only delivering content. We should instead develop a STIX query object, and a STIX reply object in order to support question and answer directly within STIX. I won't go into the specific argument why here and clog this thread, but it has been discussed on the TAXII list recently.   Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers. On 26 September 2015 at 01:33, Patrick Maroney < Pmaroney@specere.org > wrote: I would like to add a couple of conceptual ideas to this discussion (along with a specific illustrative reference): (1)  If we architect our flexible "patterning" model correctly it 'should' be directly applicable to our TAXII Query Language. (2) Graph data models provide some interesting correlations/visualizations.  As previously suggested neo4j's Cypher Query language might serve as a good conceptual model for how to express "patterns" http://assets.neo4j.org/download/Neo4j_CheatSheet_v3.pdf Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 8:12 AM -0700, "Patrick Maroney" < Pmaroney@Specere.org > wrote: A general [+1] for everything @Ivan said below (although agnostic on some of the specifics) To this "flexible" patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:  "a" followed by "b" , "a" during "b" ) along with compound boolean and set theory _expression_. Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 6:13 AM -0700, "Kirillov, Ivan A." < ikirillov@mitre.org > wrote: To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot.  I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern. Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-)  So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable: pattern example_1  {   objects: $OBJ1 = {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.2.3 ” }]}       $OBJ2 =  {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.4.7 ” }]}   condition:     OBJ1 and OBJ2 }   Regards, Ivan On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote: Adding my thoughts to the Test_Mechanism idea. In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers. If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of: Title: Send a signature Description: An analyst creates a signature/indicator and chooses to share it with partner organizations. Main Success Scenario: 1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern) 2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator) 3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and 3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator) 4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action) Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing. Thank you. -Mark [1] http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html


  • 34.  Re: [cti] Re: Observable Patterning

    Posted 09-29-2015 18:14
    Comments inline From: < cti@lists.oasis-open.org > on behalf of Steve Cell < ikirillov@mitre.org > Date: Monday, September 28, 2015 at 9:36 AM To: Jerome Athias < athiasjerome@gmail.com >, Patrick Maroney < Pmaroney@Specere.org > Cc: Terry MacDonald < terry.macdonald@threatloop.com >, " cti@lists.oasis-open.org " < cti@lists.oasis-open.org > Subject: Re: [cti] Re: Observable Patterning >- Human readable      Agreed. [sean]As human readable as possible I would agree with. >- Machine friendly (parsing/computation, algorithmically friendly)      Agreed. [sean]Agree >- Data graph friendly      Not so sure on this one.  While I agree that STIX/TAXII querying (wherever it ends up) is inherently graph-based, I don’t believe the same is true for Indicator patterning. The main difference is that, in my view, the majority of Indicator patterns are meant to be parsed and executed against data that is inherently flat – lists of files, lists of processes, lists of IP addresses, etc. Thus while the two may overlap in certain places, I’d be very hesitant about overloading an Indicator patterning structure to support graph based querying, though perhaps it could be extended to do so. Again, I think a primary focus here should be to keep Indicator patterns SIMPLE, so that they can easily be written and consumed by analysts. [sean]I would agree with Ivan here that we may be mixing up topics again by assuming that patterning needs are all the same for observables, indicators, COA parameters, infrastructure characterization, affected software characterization, victim targeting characterization, and for gross-level querying for STIX content. I think there may be inherent differences in expressing patterns IN the content and expressing patterns ON the content. The latter certainly seems to have value in considering graph-based capabilities. The former might but it is far less clear. Regards, Ivan From: Jerome Athias Date: Saturday, September 26, 2015 at 12:38 PM To: Patrick Maroney Cc: Terry MacDonald, " cti@lists.oasis-open.org ", Ivan Kirillov Subject: Re: [cti] Re: Observable Patterning so should we try to capture the requirements in one place for this change request? Quickly: - Human readable - Machine friendly (parsing/computation, algorithmically friendly) - Data graph friendly 2015-09-26 17:57 GMT+03:00 Patrick Maroney < Pmaroney@specere.org > : Probably not clear that I've similarly argued to keep TAXII focused on the transport of packages.  In the query method suggested for consideration, the Pattern would be passed as a STIX (Query/RFI) package and results similarly returned (again as a STIX package). It's important to visualize this a Data Graph that maps to the Conceptual CTI model (which is the basis for suggesting Cypher as an example representation).  One can describe/query precise graphs of Objects/Relationships, any Nodes/Edges off of a Root Node, etc. Hope this makes sense (at least in terms of framing the proposal). Sent from Outlook On Fri, Sep 25, 2015 at 6:01 PM -0700, "Terry MacDonald" < terry.macdonald@threatloop.com > wrote: I agree with Patrick's comments about ensuring pattern support for querying, but not the idea querying should be done in TAXII. Querying and answering should be done within STIX, so that the querying and answering done by the language contained within TAXII, and so that TAXII is free to concentrate on only delivering content. We should instead develop a STIX query object, and a STIX reply object in order to support question and answer directly within STIX. I won't go into the specific argument why here and clog this thread, but it has been discussed on the TAXII list recently.   Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers. On 26 September 2015 at 01:33, Patrick Maroney < Pmaroney@specere.org > wrote: I would like to add a couple of conceptual ideas to this discussion (along with a specific illustrative reference): (1)  If we architect our flexible "patterning" model correctly it 'should' be directly applicable to our TAXII Query Language. (2) Graph data models provide some interesting correlations/visualizations.  As previously suggested neo4j's Cypher Query language might serve as a good conceptual model for how to express "patterns" http://assets.neo4j.org/download/Neo4j_CheatSheet_v3.pdf Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 8:12 AM -0700, "Patrick Maroney" < Pmaroney@Specere.org > wrote: A general [+1] for everything @Ivan said below (although agnostic on some of the specifics) To this "flexible" patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:  "a" followed by "b" , "a" during "b" ) along with compound boolean and set theory _expression_. Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 6:13 AM -0700, "Kirillov, Ivan A." < ikirillov@mitre.org > wrote: To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot.  I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern. Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-)  So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable: pattern example_1  {   objects: $OBJ1 = {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.2.3 ” }]}       $OBJ2 =  {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.4.7 ” }]}   condition:     OBJ1 and OBJ2 }   Regards, Ivan On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote: Adding my thoughts to the Test_Mechanism idea. In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers. If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of: Title: Send a signature Description: An analyst creates a signature/indicator and chooses to share it with partner organizations. Main Success Scenario: 1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern) 2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator) 3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and 3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator) 4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action) Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing. Thank you. -Mark [1] http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html


  • 35.  Re: [cti] Re: Observable Patterning

    Posted 09-29-2015 18:08
    Comments inline From: < cti@lists.oasis-open.org > on behalf of Patrick Maroney < Pmaroney@Specere.org > Date: Saturday, September 26, 2015 at 10:57 AM To: Terry MacDonald < terry.macdonald@threatloop.com > Cc: " cti@lists.oasis-open.org " < cti@lists.oasis-open.org >, Steve Cell < ikirillov@mitre.org > Subject: Re: [cti] Re: Observable Patterning Probably not clear that I've similarly argued to keep TAXII focused on the transport of packages.  In the query method suggested for consideration, the Pattern would be passed as a STIX (Query/RFI) package and results similarly returned (again as a STIX package). [sean]I also think we should keep TAXII focused on the transport of packages and I think I amy agree with the approach Pat is suggesting here but I am not sure. I think it will take quite a bit more discussion and detail to reach any sort of conclusions on appropriateness and practicality. It's important to visualize this a Data Graph that maps to the Conceptual CTI model (which is the basis for suggesting Cypher as an example representation).  One can describe/query precise graphs of Objects/Relationships, any Nodes/Edges off of a Root Node, etc. Hope this makes sense (at least in terms of framing the proposal). Sent from Outlook On Fri, Sep 25, 2015 at 6:01 PM -0700, "Terry MacDonald" < terry.macdonald@threatloop.com > wrote: I agree with Patrick's comments about ensuring pattern support for querying, but not the idea querying should be done in TAXII. Querying and answering should be done within STIX, so that the querying and answering done by the language contained within TAXII, and so that TAXII is free to concentrate on only delivering content. We should instead develop a STIX query object, and a STIX reply object in order to support question and answer directly within STIX. I won't go into the specific argument why here and clog this thread, but it has been discussed on the TAXII list recently.   Cheers Terry MacDonald STIX, TAXII, CybOX Consultant M: +61-407-203-026 E:  terry.macdonald@threatloop.com W:  www.threatloop.com Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers. On 26 September 2015 at 01:33, Patrick Maroney < Pmaroney@specere.org > wrote: I would like to add a couple of conceptual ideas to this discussion (along with a specific illustrative reference): (1)  If we architect our flexible "patterning" model correctly it 'should' be directly applicable to our TAXII Query Language. (2) Graph data models provide some interesting correlations/visualizations.  As previously suggested neo4j's Cypher Query language might serve as a good conceptual model for how to express "patterns" http://assets.neo4j.org/download/Neo4j_CheatSheet_v3.pdf Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 8:12 AM -0700, "Patrick Maroney" < Pmaroney@Specere.org > wrote: A general [+1] for everything @Ivan said below (although agnostic on some of the specifics) To this "flexible" patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:  "a" followed by "b" , "a" during "b" ) along with compound boolean and set theory _expression_. Patrick Maroney President Integrated Networking Technologies, Inc. Desk: (856)983-0001 Cell: (609)841-5104 Email: pmaroney@specere.org On Fri, Sep 25, 2015 at 6:13 AM -0700, "Kirillov, Ivan A." < ikirillov@mitre.org > wrote: To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot.  I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern. Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-)  So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable: pattern example_1  {   objects: $OBJ1 = {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.2.3 ” }]}       $OBJ2 =  {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.4.7 ” }]}   condition:     OBJ1 and OBJ2 }   Regards, Ivan On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote: Adding my thoughts to the Test_Mechanism idea. In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers. If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of: Title: Send a signature Description: An analyst creates a signature/indicator and chooses to share it with partner organizations. Main Success Scenario: 1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern) 2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator) 3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and 3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator) 4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action) Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing. Thank you. -Mark [1] http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html


  • 36.  Re: [cti] Re: Observable Patterning

    Posted 09-29-2015 18:04
    Comments inline From: < cti@lists.oasis-open.org > on behalf of Steve Cell < ikirillov@mitre.org > Date: Friday, September 25, 2015 at 9:13 AM To: " cti@lists.oasis-open.org " < cti@lists.oasis-open.org > Subject: Re: [cti] Re: Observable Patterning To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot.  [sean]I would concur. Relevant part of my previous post: "It looks like the difference between Observable and Test_Mechanism within STIX Indicators may be confusing for some. There really is not a strong semantic difference between these sub-structures of Indicator. They both are intended to express the observable pattern relevant to that Indicator. The only difference is that Observable is built on the presumption that the pattern is expressed using CybOX while Test_Mechanism is an extension point for expressing the pattern is any other format desired (e.g. YARA, OVAL, OpenIOC). They are currently separate structures because at the time they were designed there was a clear desire to demonstrate a preference for CybOX  over other approaches for expressing observable patterns due to the value of consistency as CybOX is used in many other areas of STIX (malware characterization, incident investigation, digital forensics, etc.) where these observable patterns would likely come from anyway. It may be possible (potentially desirable) in STIX 2.0 to remove the separation between STIX/Indicator/Observable and STIX/Indicator/Test_Mechanism with a single Observable Pattern structure that is basically what Test_Mechanism is today but with a default implementation of a CybOX based observable pattern. So basically the current Observable field would be renamed Observable Pattern and changed to TestMechanismType, and a new CybOXTestMechanismType would be created and asserted as the suggested default extension. It would also probably make sense to change the wording of Test Mechanism to Observable Pattern." I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), [sean]I would assert that it is more than just Objects but also Actions as well. and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern. [sean]I agree that a monolithic pattern language is impractical. Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-)  [sean]At this point, I would neither agree or disagree with the idea of a top-level Pattern structure. I would have so see more of what is being proposed. At the level of detail provided so far I am not seeing how it would serve the relevant patterning needs that exist within STIX. So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable: pattern example_1  {   objects: $OBJ1 = {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.2.3 ” }]}       $OBJ2 =  {type = AddressObject,          fields = [{ “ category ” : ” ipv4-addr ”},                    { “ address_value ” : ” 192.168.4.7 ” }]}   condition:     OBJ1 and OBJ2 }   Regards, Ivan On 9/25/15, 8:02 AM, "Davidson II, Mark S" < mdavidson@mitre.org > wrote: Adding my thoughts to the Test_Mechanism idea. In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers. If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of: Title: Send a signature Description: An analyst creates a signature/indicator and chooses to share it with partner organizations. Main Success Scenario: 1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern) 2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator) 3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and 3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator) 4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action) Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing. Thank you. -Mark [1] http://stix.mitre.org/language/version1.2/xsddocs/XMLSchema/indicator/2.2/indicator.html


  • 37.  Re: [cti] Re: Observable Patterning

    Posted 09-29-2015 17:56
    Comments inline On 9/25/15, 7:02 AM, " cti@lists.oasis-open.org on behalf of Grobauer, Bernd" < cti@lists.oasis-open.org on behalf of Bernd.Grobauer@siemens.com > wrote: Hi, I don't really see Test_Mechanisms as a candidate for reusing the patterning that we've been discussing. Each Test_Mechanism rule is a rule derived from the Indicators, and written in the tools native language. The principle of having a test-mechanism derive from what is described within an indicator using observable patterns is the intended meaning, but this principle is not always followed: it may be quite complicated to describe in STIX/CybOX, what can be succinctly described in a certain pattern language or the authoring party simply does not want to bother with this -- so the STIX indicator is also used as a vehicle to exchange such a test mechanism, enriched with contextual information contained in the indicator and referenced/referencing STIX objects, but not containing corresponding observable patterns. I would guess that we will continue to see such use of a TestMechanism within an otherwise rather empty indicator (empty with respect to Observables), even if the pattern language is evolved such that in theory it could describe everything that is expressible in any thinkable test mechanism. [sean]Spot on. Test Mechanisms are not necessarily “derived from” the CybOX observable patterns in Indicators. An Indicator can have just an Observable, just a Test_Mechanism or both. As Bernd points out, sometimes a Text_Mechanism can the most appropriate way to express the pattern either due to inherent capabilities of the external patterning format or due to the available tooling of the recipient. The only real key requirement is that if they are both present the patterns should not be in conflict. It doesn't make sense to me to have patterning in the Test_Mechanism unless it is natively in the tool that uses the rules. I had made that proposal in one of my emails with the intent of encapsulating complexity. I can see why one would not want to use a test mechanism as container for STIX/CybOX containers -- after all, no other test mechanism would have references out of the test-mechanism blob, whereas a STIX pattern could have many references to objects/object properties defined elsewhere. But I really would like to get rid of AND/OR/...-composition on the level of indicators and push the whole pattern language into (oh no, here it comes) yet another STIX top level object STIX_Pattern or whatever. Then allow STIX_Patterns to reference other patterns for composition, but not Indicators. The reason is that the whole logical and temporal composition is really complicated. The Indicator is absolutely central to STIX, but like I said above, there may be use-cases to communicate things to look out for *without* the use of CybOX and CybOX patterning but via a test mechanism (or, I haven't given up on that idea, a STIX-standardized key/value-pair mechanism for communicating patterns). [sean]for reasons described in my first post on this thread and in the documentation it links to, I don’t think that this will be practical Now, if we encapsulate the complications of the pattern language in something that is removed from the Indicator level, then one can implement the non-patterning-part of STIX much easier in order to use it for use cases as described above. Also, using STIX for other domains (fraud and what not) as has been mentioned on the list yesterday, could become easier, since for these other domains, the full might of STIX patterns may not be needed ... or such domains may even need something else entirely. (Hm, I just realize that this reasoning would actually support making patterning part of CybOX as a CybOX-pattern-top-level object.) [sean] I think your last parenthetical nailed it. It is exactly this need for differing context-specific patterning that argues against trying to define one ring … erm I mean pattern language to rule them all. In closing, because I have proposed yet another top-level STIX entity: in one of his mails about a possible top-level targeting object, Aharon was concerned about making things too complicated by adding more and more top-level objects. That is a valid concern, but on the other hand, having huge schema definitions for objects that encompass all kinds of substructures, is also quite complicated.  So, while we certainly should have good reasons for introducing new top-level objects, I would say that given the right circumstances, new top-level objects will actually decrease complexity. So, in short: - I fully agree that decoupling patterning and instances   is the right way to go. - I would like to encapsulate patterns in a top-level construct   that may be referenced by STIX indicators while removing   not/and/or composition for indicators. - I am not quite sure yet whether the 'Pattern' top-level object   would better be part of STIX or part of CybOX.   If we think the patterning is reusable also for extended domains such   as a possible SocEngOX (Social Engineering Observable EXchange)   language and we think we will want to write things like   (SocEng-Observation "Phone call from Mr. Evil") THEN   (CybOX Email Object with sender " lucifer.evil@hell.com )   then patterning is probably part of STIX rather than CybOX; otherwise,   as a cyber-focused language, it might well be part of CybOX. Kind regards, Bernd ------ Bernd Grobauer, Siemens CERT