I think this would be most usefully discussed in the context of normative text. I’ve added some text to the STIX pre-draft [1] to help get us going. Here is the text I have added, all comments (including: “I hate that please remove it!”) are welcome.
Extension Fields in STIX 2.0
The authors of this specification recognize that there will be cases
where certain information exchanges can be improved by adding fields that are not specified in this document; these fields are called an
Extension Field .
An Extension Field is a field that is present in a STIX document and is not defined in this specification. This section provides guidance and requirements for extending STIX in an interoperable manner. A STIX document MAY contain any number of Extension Fields.
Extension Fields must follow these rules:
Extension Fields MUST NOT have the same name as
any field in
this specification.
Extension Fields SHOULD be uniquely named. This specification recommends that maintainers
of Extension Fields use a consistent prefix (e.g., a namespace). This reduces the likelihood that multiple Extension Fields will have the same name.
Rules for processing Extension Fields SHOULD be well defined and accessible to any consumer
that would be reasonably expected to parse them.
Any consumer that receives a STIX document with one or more Extension
Fields that are not understood MAY:
refuse to process the document further;
silently ignore non-understood Extension Fields;
or continue processing the document
Producers of STIX documents that contain Extension Fields should be well
aware of the variability of consumer behavior depending on whether or not the consumer understands the Extension Fields present in the STIX document.
The reporting of errors originating from the processing of Extension
Fields depends heavily on the technology used to transport the STIX document and is therefore not covered in this specification.
Consumers that receives a STIX document that contains one or more Extension Fields that are understood must process the Extension Fields according
to the rules for that Extension Field.
Thank you.
-Mark
[1]
https://docs.google.com/document/d/1U48DOJzh2qELOEhhVWz_G6hL0Bazx1Y52wpOeR8jaVk/edit# From: <
cti@lists.oasis-open.org > on behalf of "Wunder, John A." <
jwunder@mitre.org >
Date: Sunday, January 31, 2016 at 4:04 PM
To: Eric Burger <
Eric.Burger@georgetown.edu >, "
cti@lists.oasis-open.org " <
cti@lists.oasis-open.org >
Subject: Re: [cti] Missing MTI - what to do?
As an example, our data markings processing rules need requirements for what to do when you can’t understand the marking model (you only do TLP, they send you something different) or marking application (you only do level 1, they send you level 2).
It sounds like people are on the same page here…unlike our STIX 1.2 schemas (which failed validation on extra fields), we can be more open to “extra” content in the documents.
From: <
cti@lists.oasis-open.org >, Eric Burger <
ewb25@georgetown.edu > on behalf of Eric Burger <
Eric.Burger@georgetown.edu >
Date: Sunday, January 31, 2016 at 3:51 PM
To: "
cti@lists.oasis-open.org " <
cti@lists.oasis-open.org >
Subject: Re: [cti] Missing MTI - what to do?
Whoa, whoa, whoa! Major misunderstanding here.
I was not suggesting that we should chose what to do in all cases when confronted with JSON that you do not understand. Terry here gives a perfect example where ignoring fields is plain wrong. In fact, the ‘feature’ of JSON parsers silently dropping
keys it does not understand is a very, very, very serious, potentially fatal, flaw.
If I need you do understand a field to understand my document, and you do not understand that field, in fact, you cannot understand or, worse, will misunderstand, my message. In fact, imagine:
{
“delightful” : “names”,
“delete” : “*”
}
You don’t understand ‘delightful’, but you do understand ‘delete’. Wow, you just deleted your whole data base. That sucks - all I wanted you do to was take out the delightful names in your ACL list. You
would have known that if only you knew what my ‘delightful’ message was about.
What I am advocating for is:
Specify what to do in general when you get a field you do not understand
Consensus seems to be ignore That is a reasonable default action as it is more likely to interoperate into the future
Have a mechanism for specifying in specific when you need the receiver to understand either the whole message or a particular message element
RFC 3459 does this for email
On Jan 30, 2016, at 1:23 AM, Terry MacDonald <
terry@soltra.com > wrote:
I was meaning that if we had this object:
{
"name ": "exploit",
"name_ext": {
"value": "super-duper-malware-variant",
"vocab": "
https://bigbank.com/superdooper-malware-vocab "
}
}
And the object was corrupted in transit and became
{
"name ": "exploit",
"name_ext": {
"value": "super-duper-malware",
"vocab": "
https://bigbank.com/superdooper-malware-vocab }
}
Having the illegal fields ignored means that the vocab field is invalid. The whole name_ext object is invalid as all the required fields aren’t present, so that
ultimately the only bit that gets through the parser is:
{
"name ": "exploit"
}
Which is different to what the producer was meaning to have.
I know it is an edge case and one that only warrants a small amount of thought, but we will need to ensure that the ambiguity is removed in some way in a way
that negates this from happening. It could be as simple as saying exactly what Bret suggested – that the default vocab chosen must be as close approximation as possible to the custom vocab selection.
Cheers
Terry MacDonald
Senior STIX Subject Matter Expert
SOLTRA An FS-ISAC and DTCC Company
+61 (407) 203 206
terry@soltra.com From: Jordan, Bret
[ mailto:
bret.jordan@bluecoat.com ]
Sent: Saturday, 30 January 2016 8:40 AM
To: Terry MacDonald <
terry@soltra.com >
Cc: Jason Keirstead <
Jason.Keirstead@ca.ibm.com >;
cti@lists.oasis-open.org ; Eric Burger <
Eric.Burger@georgetown.edu >
Subject: Re: [cti] Missing MTI - what to do?
Not sure I follow. If your tool does not support custom vocabs then you will not have the data in them. I would suggest as a best practice people should always, when possible, pick a fallback option instead of using "other".
Bret
Sent from my Commodore 64
On Jan 29, 2016, at 2:36 PM, Terry MacDonald <
terry@soltra.com > wrote:
This could be problematic for the vocabs extensions if the _ext object is used (custom vocab) and the extension object field is corrupted and ignored. It means that the logic will not see that the extension object exists, and will just use the value selected
within the default vocab...
Maybe we could avert this by specifying a 'custom-vocab' option for all enums to explicitly specify that there is an extension to the custom vocab?
Cheers
Terry MacDonald
On 30/01/2016 02:15, "Jordan, Bret" <
bret.jordan@bluecoat.com > wrote:
I can see this happening a lot. And the nice thing is most JSON parsers will just silently drop fields it does not understand. I am thinking it might actually be hard to figure out if there is in fact extra data there that it does not know what to do with.
The way I see this working is vendors communications in side a closed system (network / eco-system) may use all kinds of extra stuff. Then at the board TAXII server, it would get stripped out.
Thanks,
Bret
Bret Jordan CISSP
Director of Security Architecture and Standards Office of the CTO
Blue Coat Systems
PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."
On Jan 29, 2016, at 07:59, Jason Keirstead <
Jason.Keirstead@ca.ibm.com > wrote:
As someone who is *extremely likely* to have to make use of both provisional portions of future versions of our specifications and also proprietary enhancements for internal use - I would strongly advocate for option (b) (ignore the field). We need to be able
to add fields to the JSON and have them be silently ignored if unsupported by the receiver.
-
Jason Keirstead
STSM, Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown
<graycol.gif> Eric Burger ---01/29/2016 08:28:38 AM---There are only three choices if one is missing or has an odd MTI field: Fail the transaction / rejec
From: Eric Burger <
Eric.Burger@georgetown.edu >
To: "
cti@lists.oasis-open.org "
<
cti@lists.oasis-open.org >
Date: 01/29/2016 08:28 AM
Subject: [cti] Missing MTI - what to do?
Sent by: <
cti@lists.oasis-open.org >
There are only three choices if one is missing or has an odd MTI field:
Fail the transaction / reject to document
Ignore the field
Fix the field if you can
I would offer we say what we mean. There are two places to do that. The first is in the spec itself. For example, if “id” is missing, I think we would say in the spec the serialization is malformed and return an error
(or silently go away). The second is to tag the transaction, perhaps at the TAXII level, with what you want to have happen. For example, if I have private extensions, I can tag them as something you can freely ignore or drop. Conversely, if I have extensions
or optional fields that you really need to understand to process the message, I can tag them as something that if you do not recognize them, fail the document so I know you do not know.
For an example of this working in email system gateways, see
https://tools.ietf.org/html/rfc3459 On Jan 27, 2016, at 10:57 AM, Paul Patrick <
ppatrick@isightpartners.com >
wrote:
As a side note, I’ve not seen a statement about what the appropriate behavior should be when a JSON MTI compliant consumer receives a document with a field it doesn’t understand (maybe due to a typo or somebody tried
to make a private extension). But if the specified behavior was to ignore the field rather than fail the processing of the entire document, then under the situation where a JSON-LD language binding document was inadvertently sent to a consumer that only excepts
JSON MTI language binding, the behavior would still allow the document to be processed as a compliant JSON MTI document since the @nodes and @fields would be ignored or skipped.