OASIS Cyber Threat Intelligence (CTI) TC

 View Only
  • 1.  Re: [cti] Re: Observable Patterning

    Posted 10-01-2015 19:41





    >Where I get confused is whether the intent is to pass “observed” or “observable” patterns.

    >I see the latter term used to mean both what is seen and what could hypothetically be seen 

    >(but may not have been in recent history).

     

    >The former means I am seeing an attack ongoing here and now.

    >The latter means one could see this pattern, look for it, and look to see how complete the pattern is.

    >Furthermore, if you see it, it means this.



    Great questions, Michael, and I think you characterized things quite well in your description.  The intent is to support both characterization of “observed”
    data (which we refer to as observable  “ instances" in CybOX parlance)  and  also
    observable “patterns” (captured as STIX Indicators) that could hypothetically be seen and represent some specific thing if seen (e.g., the presence of a particular malware instance). From my perspective, one of the potential issues with our existing implementation
    of this approach is that observable instances and patterns are captured in the same place and using the same structure (the CybOX ObservableType); the primary difference between the two is a “condition” property that is present on patterns and not on instances. 


    >One more question, do we expect patterns to span multiple organizations?

    >(where a pattern is only observable when events across organizations are assembled)


    From a practical standpoint, I don’t see this happening, is it would require shared visibility into each organization’s sensor network (SIEM/IDS/HIDS/etc.). However, this is not to say
    that our pattern implementation will necessarily preclude this. 


    Regards,
    Ivan






    From: Michael Hammer
    Date: Thursday, October 1, 2015 at 1:21 PM
    To: Sean Barnum, Ivan Kirillov, Mark Davidson, " trey@soltra.com ", " cti@lists.oasis-open.org "
    Subject: RE: [cti] Re: Observable Patterning








    Please pardon me if this is such a basic question.
     
    I understand that there are atomic events that are observable.
    I understand that those events may be constructed into either simplex or complex patterns of observables.
    (where the atomic observables can appear across multiple possible patterns)
    The above are things that can be observed as facts or inputs to some logic machine.
     
    I understand that there can be patterns of assertions that can lead to a set of conclusions.
    Those assertion patterns could be simple or complex.  Such patterns might represent hypothesis or experience.
    (Reminds me of the design combinational or even sequential digital logic hardware,
      Which as we know, any hardware can be converted into software.)
     
    Where I get confused is whether the intent is to pass “observed” or “observable” patterns.
    I see the latter term used to mean both what is seen and what could hypothetically be seen

    (but may not have been in recent history).
     
    The former means I am seeing an attack ongoing here and now.
    The latter means one could see this pattern, look for it, and look to see how complete the pattern is.
    Furthermore, if you see it, it means this.
     
    I am seeing this as one thing is a set of inputs to a logic machine,
    While the other is really just a blueprint of a logic machine.
     
    One more question, do we expect patterns to span multiple organizations?
    (where a pattern is only observable when events across organizations are assembled)
     

    ________________________________
    Michael Hammer
    Principal Engineer
    michael.hammer@yaanatech.com
    Mobile:
    +1
    408 202 9291
    542 Gibraltar Drive
    Milpitas, CA 95035 USA
    www.yaanatech.com

     


    From:
    cti@lists.oasis-open.org [ mailto:cti@lists.oasis-open.org ]
    On Behalf Of Barnum, Sean D.
    Sent: Tuesday, September 29, 2015 1:43 PM
    To: Kirillov, Ivan A. < ikirillov@mitre.org >; Davidson II, Mark S < mdavidson@mitre.org >; Trey Darley < trey@soltra.com >;
    cti@lists.oasis-open.org
    Subject: Re: [cti] Re: Observable Patterning


     

    Hi all,


     


    I am trying to catch up on this thread and offer some thoughts.


     


    I will start here but will likely add some thoughts farther along the thread as well.


     


    Let me start by saying that I think we are talking about several different topics here and kind of munging them all together. I think we are going to need
    to separate them to deal with them appropriately.


    So far I see the following topics:



    Should observable patterns be expressed using a dedicated patterning syntax separate from the structures used to express observable instances (observations)?
    If expressed separately, where should such an observable patterning language be defined (in CybOX, in STIX, in STIX/Indicator, etc.)?
    Are observable patterns the same thing as indicator patterns?
    Should there be one patterning language for both observable patterns and indicator patterns?
    Are STIX Indicators the only place observable patterns are used?
    Is there a semantic difference between STIX/Indicator/Observable (where a CybOX observable pattern is expressed) and STIX/Indicator/Test_Mechanism (where an observable pattern is expressed using
    some external syntax)?

     


    My thoughts on these topics:



    I think this may be possible but will need to ensure that the appropriate expressivity is maintained. This is not as simple as it may seem on the surface (e.g. simply flat key-value pairs are
    definitely insufficient) and will take some work to get right.
    I would strongly assert that an observable patterning language (mechanism for expressing observable patterns) should remain as part of CybOX. CybOX is a language for characterizing the “facts”
    about cyber observables separate from the context of their meaning. This is due to the fact that many different domains have a need to express context about cyber observables but the contexts are typically different in nature. In this way, CybOX can act as
    a common language for cyber observables among a diverse set of domain-specific languages. The relevant “facts” can either be very specific and detailed values of events, objects and properties or can be a bit more fuzzy patterns on events, objects and properties.
    These more fuzzy patterns can be used to express a targeted pattern to look for among a set of detailed observable instances but they can also be simply less precise characterizations of cyber observations where  more detail is not possible  due to the precision
    of the sensors  making the observations, of the tools used to capture, analyze or communicate the observation, or the nature of the sharing source for the observation. No matter which way you cut it, I would assert that observable patterning should stay within
    the realm of CybOX and not be artificially lifted to any particular domain-specfic use of it which would hinder its ability to be used by other domain-specific uses that had a need. You may want to take a look at my thoughts on #3, #4 and #5 for more context.
    Observable patterns and indicator patterns are NOT the same thing. Observable patterns are a (arguably “the”) key part of any indicator as they express what patterns should be looked for in
    operational data to identify a sighting of that indicator. However, indicators can also have logical compositional patterning of their own that is separate from the underlying observable patterning in any particular indicator. There are specific reasons for
    this that are described
    here . It is important not to confuse these two forms of patterning. Indicator logical composition patterning (AND, OR, NOT) is done through the same approach as observable logical composition patterning but they are composing very different things and in
    addition, observable patterning also includes relational patterning which is not relevant for indicators.
    It may be possible to create one language for observable and indicator patterning but it may be difficult or impractical. I am not asserting that it will be but only that it could be. It would
    need to be structured in an abstract way that allowed assertions of logical composition on different types of things (e.g. Observables or indicators) and could not be tied directly to one or the other. On the other hand the requisite capabilities that would
    need to be included for defining relational compositions or even just property-based patterns for observables would need to be tied pretty closely with the appropriate event, object and property structures of CybOX which would be completely out of scope for
    STIX. In other words using a single patterning language would require STIX Indicators (and other areas discussed in #5 below) to leverage a language of which the large majority of its capability is tied to CybOX. This does not seem like good separation of
    concerns to me. Maybe it would make sense to structure a single patterning language in three parts: logical composition, relational composition and direct assertions (on events, objects or properties). The logical composition part could be designed abstractly
    to be leverageable by either STIX Indicators or by CybOX and the other two parts could be CybOX specific. That way CybOX could leverage all three parts together but STIX could just use the logical composition part. Just spitballing here.
    I think it is important that people remember that Indicators are not the only parts of STIX to leverage observable patterns. There are actually 5 different places within STIX that leverage observable
    patterns to express what needs to be expressed. Indicators are only 1 of these 5. The others are below with more detail and examples available

    here :


    Specify particular structured technical details for explicit characterization
    of a  Course of Action 's
    Parameter Observables.
    Specify what software is known to be affected by a given vulnerability
    via the Exploit Target  AffectedSoftware 's
    Observable characterization.
    Characterize specific technical infrastructure utilized for cyber attack
    via a TTP  Infrastructure 's
    Observable Characterization
    Characterize specific victim technical context details being targeted
    by an attacker in instances of TTP  VictimTargeting 's
    Technical Targeting Details.

    It looks like the difference between Observable and Test_Mechanism within STIX Indicators may be confusing for some. There really is not a strong semantic difference between these sub-structures
    of Indicator. They both are intended to express the observable pattern relevant to that Indicator. The only difference is that Observable is built on the presumption that the pattern is expressed using CybOX while Test_Mechanism is an extension point for expressing
    the pattern is any other format desired (e.g. YARA, OVAL, OpenIOC). They are currently separate structures because at the time they were designed there was a clear desire to demonstrate a preference for CybOX  over other approaches for expressing observable
    patterns due to the value of consistency as CybOX is used in many other areas of STIX (malware characterization, incident investigation, digital forensics, etc.) where these observable patterns would likely come from anyway. It may be possible (potentially
    desirable) in STIX 2.0 to remove the separation between STIX/Indicator/Observable and STIX/Indicator/Test_Mechanism with a single Observable Pattern structure that is basically what Test_Mechanism is today but with a default implementation of a CybOX based
    observable pattern. So basically the current Observable field would be renamed Observable Pattern and changed to TestMechanismType, and a new CybOXTestMechanismType would be created and asserted as the suggested default extension. It would also probably make
    sense to change the wording of Test Mechanism to Observable Pattern.

     


    Sorry for the long email but wanted to comment at least briefly on all the various topics being combined in this thread.


    Again, I would suggest we separate them out.


     


    sean


     


     


    From:
    < cti@lists.oasis-open.org > on behalf of Steve Cell < ikirillov@mitre.org >
    Date: Thursday, September 24, 2015 at 9:30 AM
    To: Mark Davidson < mdavidson@mitre.org >, Trey Darley < trey@soltra.com >, " cti@lists.oasis-open.org " < cti@lists.oasis-open.org >
    Subject: [cti] Re: Observable Patterning


     





    >  In order for a pattern to be specified in a STIX Indicator
    (e.g., ‘IP Address in 1.2.3.4/24’), would the pattern would need to use fields from a CybOX object? If the answer is yes, does that create a hard dependency between STIX Indicator and a particular CybOX object and version?


     


    Having a separate patterning structure would mean that we would somehow need to reference CybOX Object fields, and unless we completely decouple CybOX from
    STIX Indicators (this seems unlikely, though I believe there have been a few suggestions to this effect), there would indeed be a hard dependency between a STIX Indicator and a particular CybOX Object version. This is necessary to explicitly specify the semantics
    of “what” the Indicator is looking for. 



     


    >  What about consistency with other uses of STIX Indicator?
    Consider a SNORT signature or YARA rule – patterning is contained in those signatures and wouldn’t be at the STIX Indicator level, which seems inconsistent to me.


     


    This seems to be more of an issue with how different types of patterns are captured in STIX Indicators. CybOX-based patterns are currently represented as a
    separate entity, whereas other types of patterns (SNORT, YARA, etc.) are captured as Test_Mechanisms. Given their functional overlap, it may make more sense to have a single “Pattern” field that can be extended to capture CybOX-based patterns, SNORT rules,
    YARA sigs, etc.


     


    Regards,


    Ivan


     


    From:
    Mark Davidson
    Date: Thursday, September 24, 2015 at 9:06 AM
    To: Trey Darley, Ivan Kirillov, " cti@lists.oasis-open.org "
    Subject: RE: Observable Patterning


     



    What implication would patterning in STIX Indicators have for the relationship between STIX Indicators and CybOX objects?
     
    In order for a pattern to be specified in a STIX Indicator (e.g., ‘IP Address in 1.2.3.4/24’), would the pattern would need to use fields from a CybOX
    object? If the answer is yes, does that create a hard dependency between STIX Indicator and a particular CybOX object and version?
     
    What about consistency with other uses of STIX Indicator? Consider a SNORT signature or YARA rule – patterning is contained in those signatures and
    wouldn’t be at the STIX Indicator level, which seems inconsistent to me.
     
    > Patterning should live in _one_ place and have _one_ syntax.
    I really like this idea, and I’m asking questions that I hope will lead us in this direction.
     
    Thank you.
    -Mark
     


    From: cti@lists.oasis-open.org
    [ mailto:cti@lists.oasis-open.org ]
    On Behalf Of Trey Darley
    Sent: Thursday, September 24, 2015 4:30 AM
    To: Kirillov, Ivan A. < ikirillov@mitre.org >;
    cti@lists.oasis-open.org
    Subject: [cti] Re: Observable Patterning


     

    +100, Ivan.
     
    Patterning should live in _one_ place and have _one_ syntax.
     
    I tend to think the STIX Indicator level is the right place for this to live.
     




    Cheers,


    Trey


    --


    Trey Darley


    Senior Security Engineer


    Soltra An FS-ISAC & DTCC Company


    www.soltra.com





     








    From: cti@lists.oasis-open.org
    < cti@lists.oasis-open.org >
    on behalf of Kirillov, Ivan A. < ikirillov@mitre.org >
    Sent: Monday, September 21, 2015 19:47
    To: cti@lists.oasis-open.org
    Subject: [cti] Observable Patterning

     







    Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around
    Bernd’s suggested refactoring.





     


    I think there are a few broad questions around this topic that we can touch upon here: 


     


    1.
    What capabilities should STIX/CybOX patterning have?


    2.
    Where should STIX/CybOX patterning live?


    3.     Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects?


     


    As far as 1) I would say the following, as a straw man:


     



    * Conditional operators


        * AND


        * OR


        * NOT


    * Basic temporal operators


        * FOLLOWED_BY


        * WITHIN


    * String matching


        * EQUALS


        * DOES_NOT_EQUAL


        * CONTAINS


        * DOES_NOT_CONTAIN


    * Basic arithmetic operators


        * >


        * <


        * >=


        * <=


    * Regular Expressions


        * PCRE compatible


    * Variable substitution



     










    Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that
    stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data. 







    With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in
    the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific patterning language (necessary for more complex pattern expressions), and also give us the flexibility to
    move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else).


     


    [1]  https://github.com/CybOXProject/schemas/issues/381


     


    Regards,


    Ivan

















  • 2.  Re: [cti] Re: Observable Patterning

    Posted 10-02-2015 14:47
    [...One more question, do we expect patterns to span multiple organizations? >(where a pattern is only observable when events across organizations are assembled) From a practical standpoint, I don’t see this happening, is it would require shared visibility into each organization’s sensor network (SIEM/IDS/HIDS/etc.)....] Actually this is already happening with products like BrightPoint Security Sentinel.  The collection and analysis of multi-organization targeting/attack patterns has also been happening manually for some time in some communities and serves as a reliable method of attribution and prediction. This missing piece is common tokenization methods to strip specific attribution while retaining context. Patrick Maroney _____________________________ From: Kirillov, Ivan A. < ikirillov@mitre.org > Sent: Thursday, October 1, 2015 2:40 PM Subject: Re: [cti] Re: Observable Patterning To: < cti@lists.oasis-open.org >, Davidson II, Mark S < mdavidson@mitre.org >, Barnum, Sean D. < sbarnum@mitre.org >, Michael Hammer < michael.hammer@yaanatech.com >, < trey@soltra.com > >Where I get confused is whether the intent is to pass “observed” or “observable” patterns. >I see the latter term used to mean both what is seen and what could hypothetically be seen  >(but may not have been in recent history).   >The former means I am seeing an attack ongoing here and now. >The latter means one could see this pattern, look for it, and look to see how complete the pattern is. >Furthermore, if you see it, it means this. Great questions, Michael, and I think you characterized things quite well in your description.  The intent is to support both characterization of “observed” data (which we refer to as observable  “ instances" in CybOX parlance)  and  also observable “patterns” (captured as STIX Indicators) that could hypothetically be seen and represent some specific thing if seen (e.g., the presence of a particular malware instance). From my perspective, one of the potential issues with our existing implementation of this approach is that observable instances and patterns are captured in the same place and using the same structure (the CybOX ObservableType); the primary difference between the two is a “condition” property that is present on patterns and not on instances.  >One more question, do we expect patterns to span multiple organizations? >(where a pattern is only observable when events across organizations are assembled) From a practical standpoint, I don’t see this happening, is it would require shared visibility into each organization’s sensor network (SIEM/IDS/HIDS/etc.). However, this is not to say that our pattern implementation will necessarily preclude this.  Regards, Ivan From: Michael Hammer Date: Thursday, October 1, 2015 at 1:21 PM To: Sean Barnum, Ivan Kirillov, Mark Davidson, " trey@soltra.com ", " cti@lists.oasis-open.org " Subject: RE: [cti] Re: Observable Patterning Please pardon me if this is such a basic question.   I understand that there are atomic events that are observable. I understand that those events may be constructed into either simplex or complex patterns of observables. (where the atomic observables can appear across multiple possible patterns) The above are things that can be observed as facts or inputs to some logic machine.   I understand that there can be patterns of assertions that can lead to a set of conclusions. Those assertion patterns could be simple or complex.  Such patterns might represent hypothesis or experience. (Reminds me of the design combinational or even sequential digital logic hardware,   Which as we know, any hardware can be converted into software.)   Where I get confused is whether the intent is to pass “observed” or “observable” patterns. I see the latter term used to mean both what is seen and what could hypothetically be seen (but may not have been in recent history).   The former means I am seeing an attack ongoing here and now. The latter means one could see this pattern, look for it, and look to see how complete the pattern is. Furthermore, if you see it, it means this.   I am seeing this as one thing is a set of inputs to a logic machine, While the other is really just a blueprint of a logic machine.   One more question, do we expect patterns to span multiple organizations? (where a pattern is only observable when events across organizations are assembled)   ________________________________ Michael Hammer Principal Engineer michael.hammer@yaanatech.com Mobile: +1 408 202 9291 542 Gibraltar Drive Milpitas, CA 95035 USA www.yaanatech.com   From: cti@lists.oasis-open.org [ mailto:cti@lists.oasis-open.org ] On Behalf Of Barnum, Sean D. Sent: Tuesday, September 29, 2015 1:43 PM To: Kirillov, Ivan A. < ikirillov@mitre.org >; Davidson II, Mark S < mdavidson@mitre.org >; Trey Darley < trey@soltra.com >; cti@lists.oasis-open.org Subject: Re: [cti] Re: Observable Patterning   Hi all,   I am trying to catch up on this thread and offer some thoughts.   I will start here but will likely add some thoughts farther along the thread as well.   Let me start by saying that I think we are talking about several different topics here and kind of munging them all together. I think we are going to need to separate them to deal with them appropriately. So far I see the following topics: Should observable patterns be expressed using a dedicated patterning syntax separate from the structures used to express observable instances (observations)? If expressed separately, where should such an observable patterning language be defined (in CybOX, in STIX, in STIX/Indicator, etc.)? Are observable patterns the same thing as indicator patterns? Should there be one patterning language for both observable patterns and indicator patterns? Are STIX Indicators the only place observable patterns are used? Is there a semantic difference between STIX/Indicator/Observable (where a CybOX observable pattern is expressed) and STIX/Indicator/Test_Mechanism (where an observable pattern is expressed using some external syntax)?   My thoughts on these topics: I think this may be possible but will need to ensure that the appropriate expressivity is maintained. This is not as simple as it may seem on the surface (e.g. simply flat key-value pairs are definitely insufficient) and will take some work to get right. I would strongly assert that an observable patterning language (mechanism for expressing observable patterns) should remain as part of CybOX. CybOX is a language for characterizing the “facts” about cyber observables separate from the context of their meaning. This is due to the fact that many different domains have a need to express context about cyber observables but the contexts are typically different in nature. In this way, CybOX can act as a common language for cyber observables among a diverse set of domain-specific languages. The relevant “facts” can either be very specific and detailed values of events, objects and properties or can be a bit more fuzzy patterns on events, objects and properties. These more fuzzy patterns can be used to express a targeted pattern to look for among a set of detailed observable instances but they can also be simply less precise characterizations of cyber observations where  more detail is not possible  due to the precision of the sensors  making the observations, of the tools used to capture, analyze or communicate the observation, or the nature of the sharing source for the observation. No matter which way you cut it, I would assert that observable patterning should stay within the realm of CybOX and not be artificially lifted to any particular domain-specfic use of it which would hinder its ability to be used by other domain-specific uses that had a need. You may want to take a look at my thoughts on #3, #4 and #5 for more context. Observable patterns and indicator patterns are NOT the same thing. Observable patterns are a (arguably “the”) key part of any indicator as they express what patterns should be looked for in operational data to identify a sighting of that indicator. However, indicators can also have logical compositional patterning of their own that is separate from the underlying observable patterning in any particular indicator. There are specific reasons for this that are described here . It is important not to confuse these two forms of patterning. Indicator logical composition patterning (AND, OR, NOT) is done through the same approach as observable logical composition patterning but they are composing very different things and in addition, observable patterning also includes relational patterning which is not relevant for indicators. It may be possible to create one language for observable and indicator patterning but it may be difficult or impractical. I am not asserting that it will be but only that it could be. It would need to be structured in an abstract way that allowed assertions of logical composition on different types of things (e.g. Observables or indicators) and could not be tied directly to one or the other. On the other hand the requisite capabilities that would need to be included for defining relational compositions or even just property-based patterns for observables would need to be tied pretty closely with the appropriate event, object and property structures of CybOX which would be completely out of scope for STIX. In other words using a single patterning language would require STIX Indicators (and other areas discussed in #5 below) to leverage a language of which the large majority of its capability is tied to CybOX. This does not seem like good separation of concerns to me. Maybe it would make sense to structure a single patterning language in three parts: logical composition, relational composition and direct assertions (on events, objects or properties). The logical composition part could be designed abstractly to be leverageable by either STIX Indicators or by CybOX and the other two parts could be CybOX specific. That way CybOX could leverage all three parts together but STIX could just use the logical composition part. Just spitballing here. I think it is important that people remember that Indicators are not the only parts of STIX to leverage observable patterns. There are actually 5 different places within STIX that leverage observable patterns to express what needs to be expressed. Indicators are only 1 of these 5. The others are below with more detail and examples available here : Specify particular structured technical details for explicit characterization of a  Course of Action 's Parameter Observables. Specify what software is known to be affected by a given vulnerability via the Exploit Target  AffectedSoftware 's Observable characterization. Characterize specific technical infrastructure utilized for cyber attack via a TTP  Infrastructure 's Observable Characterization Characterize specific victim technical context details being targeted by an attacker in instances of TTP  VictimTargeting 's Technical Targeting Details. It looks like the difference between Observable and Test_Mechanism within STIX Indicators may be confusing for some. There really is not a strong semantic difference between these sub-structures of Indicator. They both are intended to express the observable pattern relevant to that Indicator. The only difference is that Observable is built on the presumption that the pattern is expressed using CybOX while Test_Mechanism is an extension point for expressing the pattern is any other format desired (e.g. YARA, OVAL, OpenIOC). They are currently separate structures because at the time they were designed there was a clear desire to demonstrate a preference for CybOX  over other approaches for expressing observable patterns due to the value of consistency as CybOX is used in many other areas of STIX (malware characterization, incident investigation, digital forensics, etc.) where these observable patterns would likely come from anyway. It may be possible (potentially desirable) in STIX 2.0 to remove the separation between STIX/Indicator/Observable and STIX/Indicator/Test_Mechanism with a single Observable Pattern structure that is basically what Test_Mechanism is today but with a default implementation of a CybOX based observable pattern. So basically the current Observable field would be renamed Observable Pattern and changed to TestMechanismType, and a new CybOXTestMechanismType would be created and asserted as the suggested default extension. It would also probably make sense to change the wording of Test Mechanism to Observable Pattern.   Sorry for the long email but wanted to comment at least briefly on all the various topics being combined in this thread. Again, I would suggest we separate them out.   sean     From: < cti@lists.oasis-open.org > on behalf of Steve Cell < ikirillov@mitre.org > Date: Thursday, September 24, 2015 at 9:30 AM To: Mark Davidson < mdavidson@mitre.org >, Trey Darley < trey@soltra.com >, " cti@lists.oasis-open.org " < cti@lists.oasis-open.org > Subject: [cti] Re: Observable Patterning   >  In order for a pattern to be specified in a STIX Indicator (e.g., ‘IP Address in 1.2.3.4/24’), would the pattern would need to use fields from a CybOX object? If the answer is yes, does that create a hard dependency between STIX Indicator and a particular CybOX object and version?   Having a separate patterning structure would mean that we would somehow need to reference CybOX Object fields, and unless we completely decouple CybOX from STIX Indicators (this seems unlikely, though I believe there have been a few suggestions to this effect), there would indeed be a hard dependency between a STIX Indicator and a particular CybOX Object version. This is necessary to explicitly specify the semantics of “what” the Indicator is looking for.    >  What about consistency with other uses of STIX Indicator? Consider a SNORT signature or YARA rule – patterning is contained in those signatures and wouldn’t be at the STIX Indicator level, which seems inconsistent to me.   This seems to be more of an issue with how different types of patterns are captured in STIX Indicators. CybOX-based patterns are currently represented as a separate entity, whereas other types of patterns (SNORT, YARA, etc.) are captured as Test_Mechanisms. Given their functional overlap, it may make more sense to have a single “Pattern” field that can be extended to capture CybOX-based patterns, SNORT rules, YARA sigs, etc.   Regards, Ivan   From: Mark Davidson Date: Thursday, September 24, 2015 at 9:06 AM To: Trey Darley, Ivan Kirillov, " cti@lists.oasis-open.org " Subject: RE: Observable Patterning   What implication would patterning in STIX Indicators have for the relationship between STIX Indicators and CybOX objects?   In order for a pattern to be specified in a STIX Indicator (e.g., ‘IP Address in 1.2.3.4/24’), would the pattern would need to use fields from a CybOX object? If the answer is yes, does that create a hard dependency between STIX Indicator and a particular CybOX object and version?   What about consistency with other uses of STIX Indicator? Consider a SNORT signature or YARA rule – patterning is contained in those signatures and wouldn’t be at the STIX Indicator level, which seems inconsistent to me.   > Patterning should live in _one_ place and have _one_ syntax. I really like this idea, and I’m asking questions that I hope will lead us in this direction.   Thank you. -Mark   From: cti@lists.oasis-open.org [ mailto:cti@lists.oasis-open.org ] On Behalf Of Trey Darley Sent: Thursday, September 24, 2015 4:30 AM To: Kirillov, Ivan A. < ikirillov@mitre.org >; cti@lists.oasis-open.org Subject: [cti] Re: Observable Patterning   +100, Ivan.   Patterning should live in _one_ place and have _one_ syntax.   I tend to think the STIX Indicator level is the right place for this to live.   Cheers, Trey -- Trey Darley Senior Security Engineer Soltra An FS-ISAC & DTCC Company www.soltra.com   From: cti@lists.oasis-open.org < cti@lists.oasis-open.org > on behalf of Kirillov, Ivan A. < ikirillov@mitre.org > Sent: Monday, September 21, 2015 19:47 To: cti@lists.oasis-open.org Subject: [cti] Observable Patterning   Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around Bernd’s suggested refactoring.   I think there are a few broad questions around this topic that we can touch upon here:    1. What capabilities should STIX/CybOX patterning have? 2. Where should STIX/CybOX patterning live? 3.     Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects?   As far as 1) I would say the following, as a straw man:   * Conditional operators     * AND     * OR     * NOT * Basic temporal operators     * FOLLOWED_BY     * WITHIN * String matching     * EQUALS     * DOES_NOT_EQUAL     * CONTAINS     * DOES_NOT_CONTAIN * Basic arithmetic operators     * >     * <     * >=     * <= * Regular Expressions     * PCRE compatible * Variable substitution   Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data.  With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific patterning language (necessary for more complex pattern expressions), and also give us the flexibility to move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else).   [1]  https://github.com/CybOXProject/schemas/issues/381   Regards, Ivan


  • 3.  Re: [cti] Observable Patterning

    Posted 10-02-2015 22:00
    We do this today in our Global Intelligence Network and have, in fact, been doing it for years.  So yes, this would be valuable.  Even if it was looked at as multiple sites within one large global company.   Thanks, Bret Bret Jordan CISSP Director of Security Architecture and Standards Office of the CTO Blue Coat Systems PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050 Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg.   On Oct 2, 2015, at 08:47, Patrick Maroney < Pmaroney@Specere.org > wrote: [...One more question, do we expect patterns to span multiple organizations? >(where a pattern is only observable when events across organizations are assembled) From a practical standpoint, I don’t see this happening, is it would require shared visibility into each organization’s sensor network (SIEM/IDS/HIDS/etc.)....] Actually this is already happening with products like BrightPoint Security Sentinel.  The collection and analysis of multi-organization targeting/attack patterns has also been happening manually for some time in some communities and serves as a reliable method of attribution and prediction. This missing piece is common tokenization methods to strip specific attribution while retaining context. Patrick Maroney _____________________________ From: Kirillov, Ivan A. < ikirillov@mitre.org > Sent: Thursday, October 1, 2015 2:40 PM Subject: Re: [cti] Re: Observable Patterning To: < cti@lists.oasis-open.org >, Davidson II, Mark S < mdavidson@mitre.org >, Barnum, Sean D. < sbarnum@mitre.org >, Michael Hammer < michael.hammer@yaanatech.com >, < trey@soltra.com > >Where I get confused is whether the intent is to pass “observed” or “observable” patterns. >I see the latter term used to mean both what is seen and what could hypothetically be seen  >(but may not have been in recent history).   >The former means I am seeing an attack ongoing here and now. >The latter means one could see this pattern, look for it, and look to see how complete the pattern is. >Furthermore, if you see it, it means this. Great questions, Michael, and I think you characterized things quite well in your description.  The intent is to support both characterization of “observed” data (which we refer to as observable  “ instances in CybOX parlance)  and  also observable “patterns” (captured as STIX Indicators) that could hypothetically be seen and represent some specific thing if seen (e.g., the presence of a particular malware instance). From my perspective, one of the potential issues with our existing implementation of this approach is that observable instances and patterns are captured in the same place and using the same structure (the CybOX ObservableType); the primary difference between the two is a “condition” property that is present on patterns and not on instances.  >One more question, do we expect patterns to span multiple organizations? >(where a pattern is only observable when events across organizations are assembled) From a practical standpoint, I don’t see this happening, is it would require shared visibility into each organization’s sensor network (SIEM/IDS/HIDS/etc.). However, this is not to say that our pattern implementation will necessarily preclude this.  Regards, Ivan From:   Michael Hammer   Date:   Thursday, October 1, 2015 at 1:21 PM   To:   Sean Barnum, Ivan Kirillov, Mark Davidson,   trey@soltra.com ,   cti@lists.oasis-open.org   Subject:   RE: [cti] Re: Observable Patterning   Please pardon me if this is such a basic question.   I understand that there are atomic events that are observable. I understand that those events may be constructed into either simplex or complex patterns of observables. (where the atomic observables can appear across multiple possible patterns) The above are things that can be observed as facts or inputs to some logic machine.   I understand that there can be patterns of assertions that can lead to a set of conclusions. Those assertion patterns could be simple or complex.  Such patterns might represent hypothesis or experience. (Reminds me of the design combinational or even sequential digital logic hardware,   Which as we know, any hardware can be converted into software.)   Where I get confused is whether the intent is to pass “observed” or “observable” patterns. I see the latter term used to mean both what is seen and what could hypothetically be seen (but may not have been in recent history).   The former means I am seeing an attack ongoing here and now. The latter means one could see this pattern, look for it, and look to see how complete the pattern is. Furthermore, if you see it, it means this.   I am seeing this as one thing is a set of inputs to a logic machine, While the other is really just a blueprint of a logic machine.   One more question, do we expect patterns to span multiple organizations? (where a pattern is only observable when events across organizations are assembled)   ________________________________ Michael Hammer Principal Engineer michael.hammer@yaanatech.com Mobile:   +1   408 202 9291 542 Gibraltar Drive Milpitas, CA 95035 USA www.yaanatech.com   From:   cti@lists.oasis-open.org   [ mailto:cti@lists.oasis-open.org ]   On Behalf Of   Barnum, Sean D. Sent:   Tuesday, September 29, 2015 1:43 PM To:   Kirillov, Ivan A. < ikirillov@mitre.org >; Davidson II, Mark S < mdavidson@mitre.org >; Trey Darley < trey@soltra.com >;   cti@lists.oasis-open.org Subject:   Re: [cti] Re: Observable Patterning   Hi all,   I am trying to catch up on this thread and offer some thoughts.   I will start here but will likely add some thoughts farther along the thread as well.   Let me start by saying that I think we are talking about several different topics here and kind of munging them all together. I think we are going to need to separate them to deal with them appropriately. So far I see the following topics: Should observable patterns be expressed using a dedicated patterning syntax separate from the structures used to express observable instances (observations)? If expressed separately, where should such an observable patterning language be defined (in CybOX, in STIX, in STIX/Indicator, etc.)? Are observable patterns the same thing as indicator patterns? Should there be one patterning language for both observable patterns and indicator patterns? Are STIX Indicators the only place observable patterns are used? Is there a semantic difference between STIX/Indicator/Observable (where a CybOX observable pattern is expressed) and STIX/Indicator/Test_Mechanism (where an observable pattern is expressed using some external syntax)?   My thoughts on these topics: I think this may be possible but will need to ensure that the appropriate expressivity is maintained. This is not as simple as it may seem on the surface (e.g. simply flat key-value pairs are definitely insufficient) and will take some work to get right. I would strongly assert that an observable patterning language (mechanism for expressing observable patterns) should remain as part of CybOX. CybOX is a language for characterizing the “facts” about cyber observables separate from the context of their meaning. This is due to the fact that many different domains have a need to express context about cyber observables but the contexts are typically different in nature. In this way, CybOX can act as a common language for cyber observables among a diverse set of domain-specific languages. The relevant “facts” can either be very specific and detailed values of events, objects and properties or can be a bit more fuzzy patterns on events, objects and properties. These more fuzzy patterns can be used to express a targeted pattern to look for among a set of detailed observable instances but they can also be simply less precise characterizations of cyber observations where  more detail is not possible  due to the precision of the sensors  making the observations, of the tools used to capture, analyze or communicate the observation, or the nature of the sharing source for the observation. No matter which way you cut it, I would assert that observable patterning should stay within the realm of CybOX and not be artificially lifted to any particular domain-specfic use of it which would hinder its ability to be used by other domain-specific uses that had a need. You may want to take a look at my thoughts on #3, #4 and #5 for more context. Observable patterns and indicator patterns are NOT the same thing. Observable patterns are a (arguably “the”) key part of any indicator as they express what patterns should be looked for in operational data to identify a sighting of that indicator. However, indicators can also have logical compositional patterning of their own that is separate from the underlying observable patterning in any particular indicator. There are specific reasons for this that are described   here . It is important not to confuse these two forms of patterning. Indicator logical composition patterning (AND, OR, NOT) is done through the same approach as observable logical composition patterning but they are composing very different things and in addition, observable patterning also includes relational patterning which is not relevant for indicators. It may be possible to create one language for observable and indicator patterning but it may be difficult or impractical. I am not asserting that it will be but only that it could be. It would need to be structured in an abstract way that allowed assertions of logical composition on different types of things (e.g. Observables or indicators) and could not be tied directly to one or the other. On the other hand the requisite capabilities that would need to be included for defining relational compositions or even just property-based patterns for observables would need to be tied pretty closely with the appropriate event, object and property structures of CybOX which would be completely out of scope for STIX. In other words using a single patterning language would require STIX Indicators (and other areas discussed in #5 below) to leverage a language of which the large majority of its capability is tied to CybOX. This does not seem like good separation of concerns to me. Maybe it would make sense to structure a single patterning language in three parts: logical composition, relational composition and direct assertions (on events, objects or properties). The logical composition part could be designed abstractly to be leverageable by either STIX Indicators or by CybOX and the other two parts could be CybOX specific. That way CybOX could leverage all three parts together but STIX could just use the logical composition part. Just spitballing here. I think it is important that people remember that Indicators are not the only parts of STIX to leverage observable patterns. There are actually 5 different places within STIX that leverage observable patterns to express what needs to be expressed. Indicators are only 1 of these 5. The others are below with more detail and examples available   here :   Specify particular structured technical details for explicit characterization of a  Course of Action 's Parameter Observables. Specify what software is known to be affected by a given vulnerability via the Exploit Target  AffectedSoftware 's Observable characterization. Characterize specific technical infrastructure utilized for cyber attack via a TTP  Infrastructure 's Observable Characterization Characterize specific victim technical context details being targeted by an attacker in instances of TTP  VictimTargeting 's Technical Targeting Details. It looks like the difference between Observable and Test_Mechanism within STIX Indicators may be confusing for some. There really is not a strong semantic difference between these sub-structures of Indicator. They both are intended to express the observable pattern relevant to that Indicator. The only difference is that Observable is built on the presumption that the pattern is expressed using CybOX while Test_Mechanism is an extension point for expressing the pattern is any other format desired (e.g. YARA, OVAL, OpenIOC). They are currently separate structures because at the time they were designed there was a clear desire to demonstrate a preference for CybOX  over other approaches for expressing observable patterns due to the value of consistency as CybOX is used in many other areas of STIX (malware characterization, incident investigation, digital forensics, etc.) where these observable patterns would likely come from anyway. It may be possible (potentially desirable) in STIX 2.0 to remove the separation between STIX/Indicator/Observable and STIX/Indicator/Test_Mechanism with a single Observable Pattern structure that is basically what Test_Mechanism is today but with a default implementation of a CybOX based observable pattern. So basically the current Observable field would be renamed Observable Pattern and changed to TestMechanismType, and a new CybOXTestMechanismType would be created and asserted as the suggested default extension. It would also probably make sense to change the wording of Test Mechanism to Observable Pattern.   Sorry for the long email but wanted to comment at least briefly on all the various topics being combined in this thread. Again, I would suggest we separate them out.   sean     From:   < cti@lists.oasis-open.org > on behalf of Steve Cell < ikirillov@mitre.org > Date:   Thursday, September 24, 2015 at 9:30 AM To:   Mark Davidson < mdavidson@mitre.org >, Trey Darley < trey@soltra.com >, cti@lists.oasis-open.org < cti@lists.oasis-open.org > Subject:   [cti] Re: Observable Patterning   >  In order for a pattern to be specified in a STIX Indicator (e.g., ‘IP Address in 1.2.3.4/24’), would the pattern would need to use fields from a CybOX object? If the answer is yes, does that create a hard dependency between STIX Indicator and a particular CybOX object and version?   Having a separate patterning structure would mean that we would somehow need to reference CybOX Object fields, and unless we completely decouple CybOX from STIX Indicators (this seems unlikely, though I believe there have been a few suggestions to this effect), there would indeed be a hard dependency between a STIX Indicator and a particular CybOX Object version. This is necessary to explicitly specify the semantics of “what” the Indicator is looking for.    >  What about consistency with other uses of STIX Indicator? Consider a SNORT signature or YARA rule – patterning is contained in those signatures and wouldn’t be at the STIX Indicator level, which seems inconsistent to me.   This seems to be more of an issue with how different types of patterns are captured in STIX Indicators. CybOX-based patterns are currently represented as a separate entity, whereas other types of patterns (SNORT, YARA, etc.) are captured as Test_Mechanisms. Given their functional overlap, it may make more sense to have a single “Pattern” field that can be extended to capture CybOX-based patterns, SNORT rules, YARA sigs, etc.   Regards, Ivan   From:   Mark Davidson Date:   Thursday, September 24, 2015 at 9:06 AM To:   Trey Darley, Ivan Kirillov, cti@lists.oasis-open.org Subject:   RE: Observable Patterning   What implication would patterning in STIX Indicators have for the relationship between STIX Indicators and CybOX objects?   In order for a pattern to be specified in a STIX Indicator (e.g., ‘IP Address in 1.2.3.4/24’), would the pattern would need to use fields from a CybOX object? If the answer is yes, does that create a hard dependency between STIX Indicator and a particular CybOX object and version?   What about consistency with other uses of STIX Indicator? Consider a SNORT signature or YARA rule – patterning is contained in those signatures and wouldn’t be at the STIX Indicator level, which seems inconsistent to me.   > Patterning should live in _one_ place and have _one_ syntax. I really like this idea, and I’m asking questions that I hope will lead us in this direction.   Thank you. -Mark   From: cti@lists.oasis-open.org   [ mailto:cti@lists.oasis-open.org ]   On Behalf Of   Trey Darley Sent:   Thursday, September 24, 2015 4:30 AM To:   Kirillov, Ivan A. < ikirillov@mitre.org >;   cti@lists.oasis-open.org Subject:   [cti] Re: Observable Patterning   +100, Ivan.   Patterning should live in _one_ place and have _one_ syntax.   I tend to think the STIX Indicator level is the right place for this to live.   Cheers, Trey -- Trey Darley Senior Security Engineer Soltra An FS-ISAC & DTCC Company www.soltra.com   From: cti@lists.oasis-open.org   < cti@lists.oasis-open.org > on behalf of Kirillov, Ivan A. < ikirillov@mitre.org > Sent:   Monday, September 21, 2015 19:47 To:   cti@lists.oasis-open.org Subject:   [cti] Observable Patterning   Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around Bernd’s suggested refactoring.   I think there are a few broad questions around this topic that we can touch upon here:    1. What capabilities should STIX/CybOX patterning have? 2. Where should STIX/CybOX patterning live? 3.     Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects?   As far as 1) I would say the following, as a straw man:   * Conditional operators     * AND     * OR     * NOT * Basic temporal operators     * FOLLOWED_BY     * WITHIN * String matching     * EQUALS     * DOES_NOT_EQUAL     * CONTAINS     * DOES_NOT_CONTAIN * Basic arithmetic operators     * >     * <     * >=     * <= * Regular Expressions     * PCRE compatible * Variable substitution   Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data.  With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific patterning language (necessary for more complex pattern expressions), and also give us the flexibility to move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else).   [1]  https://github.com/CybOXProject/schemas/issues/381   Regards, Ivan Attachment: signature.asc Description: Message signed with OpenPGP using GPGMail