>Where I get confused is whether the intent is to pass “observed” or “observable” patterns.
>I see the latter term used to mean both what is seen and what could hypothetically be seen
>(but may not have been in recent history).
>The former means I am seeing an attack ongoing here and now.
>The latter means one could see this pattern, look for it, and look to see how complete the pattern is.
>Furthermore, if you see it, it means this.
Great questions, Michael, and I think you characterized things quite well in your description. The intent is to support both characterization of “observed”
data (which we refer to as observable “ instances" in CybOX parlance) and also
observable “patterns” (captured as STIX Indicators) that could hypothetically be seen and represent some specific thing if seen (e.g., the presence of a particular malware instance). From my perspective, one of the potential issues with our existing implementation
of this approach is that observable instances and patterns are captured in the same place and using the same structure (the CybOX ObservableType); the primary difference between the two is a “condition” property that is present on patterns and not on instances.
>One more question, do we expect patterns to span multiple organizations?
>(where a pattern is only observable when events across organizations are assembled)
From a practical standpoint, I don’t see this happening, is it would require shared visibility into each organization’s sensor network (SIEM/IDS/HIDS/etc.). However, this is not to say
that our pattern implementation will necessarily preclude this.
Regards,
Ivan
From: Michael Hammer
Date: Thursday, October 1, 2015 at 1:21 PM
To: Sean Barnum, Ivan Kirillov, Mark Davidson, "
trey@soltra.com ", "
cti@lists.oasis-open.org "
Subject: RE: [cti] Re: Observable Patterning
Please pardon me if this is such a basic question.
I understand that there are atomic events that are observable.
I understand that those events may be constructed into either simplex or complex patterns of observables.
(where the atomic observables can appear across multiple possible patterns)
The above are things that can be observed as facts or inputs to some logic machine.
I understand that there can be patterns of assertions that can lead to a set of conclusions.
Those assertion patterns could be simple or complex. Such patterns might represent hypothesis or experience.
(Reminds me of the design combinational or even sequential digital logic hardware,
Which as we know, any hardware can be converted into software.)
Where I get confused is whether the intent is to pass “observed” or “observable” patterns.
I see the latter term used to mean both what is seen and what could hypothetically be seen
(but may not have been in recent history).
The former means I am seeing an attack ongoing here and now.
The latter means one could see this pattern, look for it, and look to see how complete the pattern is.
Furthermore, if you see it, it means this.
I am seeing this as one thing is a set of inputs to a logic machine,
While the other is really just a blueprint of a logic machine.
One more question, do we expect patterns to span multiple organizations?
(where a pattern is only observable when events across organizations are assembled)
________________________________
Michael Hammer
Principal Engineer
michael.hammer@yaanatech.com Mobile:
+1
408 202 9291
542 Gibraltar Drive
Milpitas, CA 95035 USA
www.yaanatech.com From:
cti@lists.oasis-open.org [ mailto:
cti@lists.oasis-open.org ]
On Behalf Of Barnum, Sean D.
Sent: Tuesday, September 29, 2015 1:43 PM
To: Kirillov, Ivan A. <
ikirillov@mitre.org >; Davidson II, Mark S <
mdavidson@mitre.org >; Trey Darley <
trey@soltra.com >;
cti@lists.oasis-open.org Subject: Re: [cti] Re: Observable Patterning
Hi all,
I am trying to catch up on this thread and offer some thoughts.
I will start here but will likely add some thoughts farther along the thread as well.
Let me start by saying that I think we are talking about several different topics here and kind of munging them all together. I think we are going to need
to separate them to deal with them appropriately.
So far I see the following topics:
Should observable patterns be expressed using a dedicated patterning syntax separate from the structures used to express observable instances (observations)?
If expressed separately, where should such an observable patterning language be defined (in CybOX, in STIX, in STIX/Indicator, etc.)?
Are observable patterns the same thing as indicator patterns?
Should there be one patterning language for both observable patterns and indicator patterns?
Are STIX Indicators the only place observable patterns are used?
Is there a semantic difference between STIX/Indicator/Observable (where a CybOX observable pattern is expressed) and STIX/Indicator/Test_Mechanism (where an observable pattern is expressed using
some external syntax)?
My thoughts on these topics:
I think this may be possible but will need to ensure that the appropriate expressivity is maintained. This is not as simple as it may seem on the surface (e.g. simply flat key-value pairs are
definitely insufficient) and will take some work to get right.
I would strongly assert that an observable patterning language (mechanism for expressing observable patterns) should remain as part of CybOX. CybOX is a language for characterizing the “facts”
about cyber observables separate from the context of their meaning. This is due to the fact that many different domains have a need to express context about cyber observables but the contexts are typically different in nature. In this way, CybOX can act as
a common language for cyber observables among a diverse set of domain-specific languages. The relevant “facts” can either be very specific and detailed values of events, objects and properties or can be a bit more fuzzy patterns on events, objects and properties.
These more fuzzy patterns can be used to express a targeted pattern to look for among a set of detailed observable instances but they can also be simply less precise characterizations of cyber observations where more detail is not possible due to the precision
of the sensors making the observations, of the tools used to capture, analyze or communicate the observation, or the nature of the sharing source for the observation. No matter which way you cut it, I would assert that observable patterning should stay within
the realm of CybOX and not be artificially lifted to any particular domain-specfic use of it which would hinder its ability to be used by other domain-specific uses that had a need. You may want to take a look at my thoughts on #3, #4 and #5 for more context.
Observable patterns and indicator patterns are NOT the same thing. Observable patterns are a (arguably “the”) key part of any indicator as they express what patterns should be looked for in
operational data to identify a sighting of that indicator. However, indicators can also have logical compositional patterning of their own that is separate from the underlying observable patterning in any particular indicator. There are specific reasons for
this that are described
here . It is important not to confuse these two forms of patterning. Indicator logical composition patterning (AND, OR, NOT) is done through the same approach as observable logical composition patterning but they are composing very different things and in
addition, observable patterning also includes relational patterning which is not relevant for indicators.
It may be possible to create one language for observable and indicator patterning but it may be difficult or impractical. I am not asserting that it will be but only that it could be. It would
need to be structured in an abstract way that allowed assertions of logical composition on different types of things (e.g. Observables or indicators) and could not be tied directly to one or the other. On the other hand the requisite capabilities that would
need to be included for defining relational compositions or even just property-based patterns for observables would need to be tied pretty closely with the appropriate event, object and property structures of CybOX which would be completely out of scope for
STIX. In other words using a single patterning language would require STIX Indicators (and other areas discussed in #5 below) to leverage a language of which the large majority of its capability is tied to CybOX. This does not seem like good separation of
concerns to me. Maybe it would make sense to structure a single patterning language in three parts: logical composition, relational composition and direct assertions (on events, objects or properties). The logical composition part could be designed abstractly
to be leverageable by either STIX Indicators or by CybOX and the other two parts could be CybOX specific. That way CybOX could leverage all three parts together but STIX could just use the logical composition part. Just spitballing here.
I think it is important that people remember that Indicators are not the only parts of STIX to leverage observable patterns. There are actually 5 different places within STIX that leverage observable
patterns to express what needs to be expressed. Indicators are only 1 of these 5. The others are below with more detail and examples available
here :
Specify particular structured technical details for explicit characterization
of a Course of Action 's
Parameter Observables.
Specify what software is known to be affected by a given vulnerability
via the Exploit Target AffectedSoftware 's
Observable characterization.
Characterize specific technical infrastructure utilized for cyber attack
via a TTP Infrastructure 's
Observable Characterization
Characterize specific victim technical context details being targeted
by an attacker in instances of TTP VictimTargeting 's
Technical Targeting Details.
It looks like the difference between Observable and Test_Mechanism within STIX Indicators may be confusing for some. There really is not a strong semantic difference between these sub-structures
of Indicator. They both are intended to express the observable pattern relevant to that Indicator. The only difference is that Observable is built on the presumption that the pattern is expressed using CybOX while Test_Mechanism is an extension point for expressing
the pattern is any other format desired (e.g. YARA, OVAL, OpenIOC). They are currently separate structures because at the time they were designed there was a clear desire to demonstrate a preference for CybOX over other approaches for expressing observable
patterns due to the value of consistency as CybOX is used in many other areas of STIX (malware characterization, incident investigation, digital forensics, etc.) where these observable patterns would likely come from anyway. It may be possible (potentially
desirable) in STIX 2.0 to remove the separation between STIX/Indicator/Observable and STIX/Indicator/Test_Mechanism with a single Observable Pattern structure that is basically what Test_Mechanism is today but with a default implementation of a CybOX based
observable pattern. So basically the current Observable field would be renamed Observable Pattern and changed to TestMechanismType, and a new CybOXTestMechanismType would be created and asserted as the suggested default extension. It would also probably make
sense to change the wording of Test Mechanism to Observable Pattern.
Sorry for the long email but wanted to comment at least briefly on all the various topics being combined in this thread.
Again, I would suggest we separate them out.
sean
From:
<
cti@lists.oasis-open.org > on behalf of Steve Cell <
ikirillov@mitre.org >
Date: Thursday, September 24, 2015 at 9:30 AM
To: Mark Davidson <
mdavidson@mitre.org >, Trey Darley <
trey@soltra.com >, "
cti@lists.oasis-open.org " <
cti@lists.oasis-open.org >
Subject: [cti] Re: Observable Patterning
> In order for a pattern to be specified in a STIX Indicator
(e.g., ‘IP Address in 1.2.3.4/24’), would the pattern would need to use fields from a CybOX object? If the answer is yes, does that create a hard dependency between STIX Indicator and a particular CybOX object and version?
Having a separate patterning structure would mean that we would somehow need to reference CybOX Object fields, and unless we completely decouple CybOX from
STIX Indicators (this seems unlikely, though I believe there have been a few suggestions to this effect), there would indeed be a hard dependency between a STIX Indicator and a particular CybOX Object version. This is necessary to explicitly specify the semantics
of “what” the Indicator is looking for.
> What about consistency with other uses of STIX Indicator?
Consider a SNORT signature or YARA rule – patterning is contained in those signatures and wouldn’t be at the STIX Indicator level, which seems inconsistent to me.
This seems to be more of an issue with how different types of patterns are captured in STIX Indicators. CybOX-based patterns are currently represented as a
separate entity, whereas other types of patterns (SNORT, YARA, etc.) are captured as Test_Mechanisms. Given their functional overlap, it may make more sense to have a single “Pattern” field that can be extended to capture CybOX-based patterns, SNORT rules,
YARA sigs, etc.
Regards,
Ivan
From:
Mark Davidson
Date: Thursday, September 24, 2015 at 9:06 AM
To: Trey Darley, Ivan Kirillov, "
cti@lists.oasis-open.org "
Subject: RE: Observable Patterning
What implication would patterning in STIX Indicators have for the relationship between STIX Indicators and CybOX objects?
In order for a pattern to be specified in a STIX Indicator (e.g., ‘IP Address in 1.2.3.4/24’), would the pattern would need to use fields from a CybOX
object? If the answer is yes, does that create a hard dependency between STIX Indicator and a particular CybOX object and version?
What about consistency with other uses of STIX Indicator? Consider a SNORT signature or YARA rule – patterning is contained in those signatures and
wouldn’t be at the STIX Indicator level, which seems inconsistent to me.
> Patterning should live in _one_ place and have _one_ syntax.
I really like this idea, and I’m asking questions that I hope will lead us in this direction.
Thank you.
-Mark
From:
cti@lists.oasis-open.org [ mailto:
cti@lists.oasis-open.org ]
On Behalf Of Trey Darley
Sent: Thursday, September 24, 2015 4:30 AM
To: Kirillov, Ivan A. <
ikirillov@mitre.org >;
cti@lists.oasis-open.org Subject: [cti] Re: Observable Patterning
+100, Ivan.
Patterning should live in _one_ place and have _one_ syntax.
I tend to think the STIX Indicator level is the right place for this to live.
Cheers,
Trey
--
Trey Darley
Senior Security Engineer
Soltra An FS-ISAC & DTCC Company
www.soltra.com From:
cti@lists.oasis-open.org <
cti@lists.oasis-open.org >
on behalf of Kirillov, Ivan A. <
ikirillov@mitre.org >
Sent: Monday, September 21, 2015 19:47
To:
cti@lists.oasis-open.org Subject: [cti] Observable Patterning
Relaunching this as a separate thread based on Jerome’s suggestion, so as to not further hijack the discussion around
Bernd’s suggested refactoring.
I think there are a few broad questions around this topic that we can touch upon here:
1.
What capabilities should STIX/CybOX patterning have?
2.
Where should STIX/CybOX patterning live?
3. Should patterns have their own implementation/structure, or continue to be embedded in Observables/Objects?
As far as 1) I would say the following, as a straw man:
* Conditional operators
* AND
* OR
* NOT
* Basic temporal operators
* FOLLOWED_BY
* WITHIN
* String matching
* EQUALS
* DOES_NOT_EQUAL
* CONTAINS
* DOES_NOT_CONTAIN
* Basic arithmetic operators
* >
* <
* >=
* <=
* Regular Expressions
* PCRE compatible
* Variable substitution
Regarding 2), it seems to me that patterning is inherent to Indicators, and thus should be part of STIX. I just can’t see the use case for a CybOX Observable with a pattern that
stands on its own, or is used in a non-Indicator context. This would also serve to greatly simplify CybOX, as it would mean that CybOX can ONLY capture instances of cyber data.
With regards to 3), I see lots of advantages to having a separate structure for patterns, which I’ve documented in
the corresponding CybOX issue [1]. The short of it is that it will get rid of the instance/pattern duality, allow for the creation of a domain-specific patterning language (necessary for more complex pattern expressions), and also give us the flexibility to
move patterning out to wherever makes the most sense (whether it’s STIX or somewhere else).
[1]
https://github.com/CybOXProject/schemas/issues/381 Regards,
Ivan