Well,
“ So how can I say if they violated the spec or not, to know if the content is valid? How can an auditor know this? ”
If the spec says
“ don ’ t pass x on ”
and you do, that would seem to be a clear – and testable
– violation, no?
But I agree
– it ’ s easier said than done. My main point
is that several projects have gotten so bogged down in determining whether something is
“ testable ” or
“ normative ” that they actually miss the
bigger prize of gaining buy-in to the use of the spec.
Cheers,
Peter
From: Jason Keirstead [mailto:
Jason.Keirstead@ca.ibm.com]
Sent: 22 November 2016 15:58
To: Peter F Brown <
peter@peterfbrown.com>
Cc: Bret Jordan (CS) <
Bret_Jordan@symantec.com>;
duncan@sfractal.com;
cti@lists.oasis-open.org Subject: RE: [cti] Normative Statements
I dunno - I guess we have to agree to disagree, because I am unconvinced.
If entity X hands me a bundle that has a bunch of pieces of data inside it - absent any other external information, I have no idea as to their intent as-per the creation of that bundle. So how can I say if they violated the spec or not, to know if the content
is valid? How can an auditor know this? What value is having this statement in the specification if no one can know?
-
Jason Keirstead
STSM, Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown
Peter
F Brown ---11/22/2016 10:43:29 AM---I don’t agree Jason, Using Duncan’s example, you can say "you violated the spec" - That is clearly t
From: Peter F Brown <
peter@peterfbrown.com >
To: Jason Keirstead/CanEast/IBM@IBMCA, "Bret Jordan (CS)" <
Bret_Jordan@symantec.com >
Cc: "
duncan@sfractal.com " <
duncan@sfractal.com >, "
cti@lists.oasis-open.org "
<
cti@lists.oasis-open.org >
Date: 11/22/2016 10:43 AM
Subject: RE: [cti] Normative Statements
Sent by: <
cti@lists.oasis-open.org >
I don ’ t agree Jason,
Using Duncan ’ s example, you can say "you violated the spec" - That is clearly testable
Peter
From: Jason Keirstead [ mailto:
Jason.Keirstead@ca.ibm.com ]
Sent: 22 November 2016 15:39
To: Bret Jordan (CS) <
Bret_Jordan@symantec.com >
Cc: Peter F Brown <
peter@peterfbrown.com >;
duncan@sfractal.com ;
cti@lists.oasis-open.org Subject: Re: [cti] Normative Statements
The thing with this statement however is you can't even procedurally test it (as I pointed out - this is neither testable by a human nor a machine). The only person who could test this statement is a psychic.
-
Jason Keirstead
STSM, Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown
"Bret
Jordan (CS)" ---11/21/2016 01:56:34 PM---Agreed... Too often we look at these documents through the lens of a developer and what a product c
From: "Bret Jordan (CS)" <
Bret_Jordan@symantec.com >
To: Peter F Brown <
peter@peterfbrown.com >, "
duncan@sfractal.com " <
duncan@sfractal.com >, "
cti@lists.oasis-open.org "
<
cti@lists.oasis-open.org >
Date: 11/21/2016 01:56 PM
Subject: Re: [cti] Normative Statements
Sent by: <
cti@lists.oasis-open.org >
Agreed... Too often we look at these documents through the lens of a developer and what a product can and can not do.
Bret
From:
cti@lists.oasis-open.org <
cti@lists.oasis-open.org >
on behalf of Peter F Brown <
peter@peterfbrown.com >
Sent: Monday, November 21, 2016 10:28:42 AM
To:
duncan@sfractal.com ;
cti@lists.oasis-open.org Subject: RE: [cti] Normative Statements
+1
Many TCs get wrapped around the axle on this.
“Testing” compliance is not limited to testing some piece of running code, it can ask be adored as here to conforming with some procedural requirement (as often happens in many ISO standards, such as ISO 9000, 27000, etc.
Regards,
Peter
Sent from my phone. Apologies for brevity, levity, and laxity: it’s hard to write on a moving planet
From:
duncan@sfractal.com Sent: Monday, 21 November 2016 17:05
To:
cti@lists.oasis-open.org Subject: RE: [cti] Normative Statements
> " I would argue that if a normative statement can not be tested then it is not actually normative and is just a guideline. "
> "MUST all normative statements be testable? "
I disagree. Using the example below " Implementations of TAXII servers that offer TLP MUST NOT forward STIX documents marked TLP Red to non-trusted destinations ".
This is untestable BUT when the it does occur - you can say "you violated the spec". If it's non-normative, then it is not a violation if you do it. I vote normative wording if we require it, even it if not testable in all cases.
Duncan Sparrell
s-Fractal Consulting LLC
iPhone, iTypo, iApologize
Original Message --------
Subject: Re: [cti] Normative Statements
From: "Wunder, John A." < jwunder@mitre.org >
Date: Wed, November 16, 2016 8:47 am
To: Jason Keirstead < Jason.Keirstead@ca.ibm.com >, Eric Burger
< Eric.Burger@georgetown.edu >
Cc: " cti@lists.oasis-open.org " < cti@lists.oasis-open.org >
As you guys are reviewing the documents can you be checking for this? I just looked through all of the MUST requirements across the documents and while there might be a couple ones in a gray area (it’s testable if you have the source data, but you can’t look
at content absent the source data and validate it) but for the most part I think we’re in good shape.
The SHOULD requirements are obviously a bit harder to evaluate and we could probably debate for years about them but if you see anything especially bad definitely bring it up.
John
From: < cti@lists.oasis-open.org >
on behalf of Jason Keirstead < Jason.Keirstead@ca.ibm.com >
Date: Wednesday, November 16, 2016 at 8:07 AM
To: Eric Burger < Eric.Burger@georgetown.edu >
Cc: " cti@lists.oasis-open.org " < cti@lists.oasis-open.org >
Subject: Re: [cti] Normative Statements
I would argue that if a normative statement can not be tested then it is not actually normative and is just a guideline.
It should be noted that we aren't even talking about "automated testing" - the proposed normative statement is not even testable in the mind of a human reading the document, because they have no idea if the things in the bundle were intended by the producer
to be related or not.
As such, I agree with Alan that such statements serve little purpose in a spec and belong more in a set of implementor guidelines.
--
Sent from my mobile device, please excuse any typos.
Eric Burger --- [cti] Normative Statements ---
From:
"Eric Burger" < Eric.Burger@georgetown.edu >
To:
cti@lists.oasis-open.org
Date:
Tue, Nov 15, 2016 7:28 PM
Subject:
[cti] Normative Statements
MUST all normative statements be testable?
I suppose it depends on what we mean by testable. A few have said that not everything that we have as normative statements are not, in fact, testable. I would offer that is a proof point of something that cannot be normative.
Let us take an example:
Implementations of TAXII servers that offer TLP MUST NOT forward STIX documents marked TLP Red to non-trusted destinations.
This sounds like a fantastic requirement. However, this is what that requirement translates to when we write code:
Implementations of TAXII servers that offer TLP MUST NOT forward STIX documents marked TLP Red to non-trusted destinations, unless they feel like it because it is impossible for the sender to know what the recipient does once they receive and decode the document.
Now we can have such statements in requirements documents or system conformance documents. However, they are meaningless in protocol or document definition documents. In fact, I would offer they are dangerous. Let us consider this example. I am a consumer of
CTI technology. I read the specs, and a TAXII server MUST NOT forward a STIX document marked TLP Red to non-trusted destinations. I am looking at a vendor, and their product “is fully compliant with the TAXII specification.” Too bad for me there is no way
to hold them to the fire if they do improper forwarding. It’s way too late to call the Protocol Police.
While I am on my soapbox, since I just saw a dialog here along the lines of “Bundles SHOULD not have related objects in them,” I would like to reiterate the best practice for MUST/SHOULD/MAY.
MUST is something that the implementation must do. If it is something the implementation must do, it should be possible to test for it, because if it is something it must do, one clearly
can check to see if it does not do it.
SHOULD is something that the implementation MUST do, UNLESS there is an enumerated reason not to do it. That is the formulation for SHOULD: “The implementation SHOULD implement X, unless
Y or Z are present.” [This highlights the Bundle argument: “Bundles SHOULD NOT have related objects, UNLESS they are related.” In English: the spec says NOTHING about the relatedness of objects in Bundles.] If you cannot enumerate when the SHOULD is not a
MUST, then the SHOULD is a MAY.
MAY is something that might be nice, and if it is present, please don’t barf on it.
Note that given the formulation of SHOULD, specifically that the conditions under which the implementation does not do the SHOULD, leads us to a clearer formulation of SHOULD, namely the conditional MUST. Using the above example, instead of:
The implementation SHOULD implement X, unless Y or Z are present.
It is clearer to say:
If Y and Z are not present, the implementation MUST do X.
Beating the dead horse: every SHOULD and MAY in the specification non-linearly increases the likelihood of implementation errors and interoperability failures. The spec is already extremely hard to implement - JSON does not eliminate cyclomatic complexity!
There is no reason to hand our adversaries which is supposed to make or more resilient to attack an infrastructure begging for attack.
--------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php