Hi Bret,
I appreciate the fact you want to take specifics and get to work on them, this is what has brought the specification already this far. I’m not trying to be in
your way in taken in further, yet it would exactly my point not to go into this. There is a range of topics that many people have already solved in developing REST based APIs and it is my recommendation that the group strongly considers passionately _not_
wanting to pursue getting into things like this. That was one of the two key points I was trying to make earlier.
(low hanging fruit:)
With regards to pagination:
-
Perhaps we could consider move on from limit/offset based pagination, considering big and changing collections, often causing incorrect or non-performant
server implementations. (solved in jsonapi)
-
Caching might experience issues with http headers for pagination (solved in jsonapi)
-
With regards to the ability to debug, it would be ideal to have as much important information in the URL (solved in jsonapi)
With regards to filtering:
-
You’re right in that RC1 is fairly aligned after I believe one of the team brought it up being inspired by jsonapi.
Misc
-
In the current specification, sometimes responses are arrays and not an object, which can be challenging since it cannot be extended in a non-compat
breaking way; also known to have caused security issues with browsers in the past (solved in jsonapi)
-
Not mixing layers in the stack continues to be rough edge. From TLS requirements, stix json embedded in taxi, etc. (jsonapi would ensure strict layering)
-
Hard to extent
Again, _ please _ don’t take the above practical opportunities for improvement – forwarded to me by our team – as the topic of conversation. Simply to illustrate.
I hope that we can continue to identify opportunities for improvement and continue to fix them. The topic I’m trying to raise is that some of these opportunities for improvement will be discovering after implementation. It would be a shame if we could have
avoided some of them.
Also not my intent to keep on making the same point over and over, I’ll leave it in all your capable hands. Please be so kind as to discuss any technical details
with my team.
Best regards,
Joep
From:
Bret Jordan <
Bret_Jordan@symantec.com>
Date: Monday, 13 March 2017 at 18:57
To: Joep Gommers <
joep@eclecticiq.com>, "cti@lists.oasis-open.org" <
cti@lists.oasis-open.org>
Subject: Re: On current TAXII discussions
Joep,
Another thing that is called out is that Filtering in JSON-API is more robust. Can you clarify EclecticIQ's view here? The JSON API says the following about filtering:
Source:
http://jsonapi.org/format/#fetching-filtering ##BEGIN
Filtering
The filter query parameter is reserved for filtering data. Servers and clients SHOULD use this key for filtering operations.
Note: JSON API is agnostic about the strategies supported by a server. The filter query parameter can be used as the basis for any number of filtering strategies.
##END
So it seems like JSON-API is "agnostic about the strategies" around filtering. And thus gives not additional context. So how is it better than what we are doing? In fact, our filtering currently
mimics this design. We just call it "match" instead of "filter". But that could easily be changed if the community decided on it.
Bret
From: Joep Gommers <
joep@eclecticiq.com>
Sent: Sunday, March 12, 2017 2:10 PM
To: Bret Jordan;
cti@lists.oasis-open.org Subject: Re: On current TAXII discussions
Bret,
My apologies if I’ve misspoken, it was not the intent to call anything broken. I make two remarks on opportunities for improvement and de-risking.
The first is about using RC1 (roots, collections, objects) and using an existing API to de-risk the path from unproven specification to robust implementation (pagination, filtering, syntax) etc.
JSON-API as an example as shown in our draft, but there might be others.
The other point is about the separation of STIX and TAXII. TAXII query is a good example (and an awesome idea). I’ll defer to any technical commentary and specifics to your discussions with the
team, that’s beyond my expertise.
Best regards,
Joep
From: Bret Jordan <
Bret_Jordan@symantec.com>
Date: Sunday, 12 March 2017 at 16:24
To: Joep Gommers <
joep@eclecticiq.com>, "cti@lists.oasis-open.org" <
cti@lists.oasis-open.org>
Subject: Re: On current TAXII discussions
Joep,
After reading your email and comparing it to what has been said by Sergey and Wouter of your team it sounds like the real issue at play is the fact that you do not want TAXII to have TAXII Query functionality.
Aka, the fact that RC1 of TAXII 2.0 has the beginnings of TAXII Query and allows a TAXII client to query for STIX data. Yes this means that the TAXII server needs to know how to look in to the STIX data and find the right records. This seems to be the real
issue.
As I have said all along, if there are things that RC1 does not do or things that it does that are broken, please bring them up. But saying it is broken without concrete examples or illustrating where it is
broken, is hard to follow.
Bret
From:
cti@lists.oasis-open.org <
cti@lists.oasis-open.org> on behalf of Joep Gommers <
joep@eclecticiq.com>
Sent: Sunday, March 12, 2017 6:13:07 AM
To:
cti@lists.oasis-open.org Subject: [cti] On current TAXII discussions
Team,
We’re in the business of exchanging CTI and being able to use it per the use-cases of analysts and subsequent intelligence-led security efforts. Over the past three years we’ve had to be business of many other things to get there; protocols, work-arounds to
bad implementations, extending specs, ignoring specs – all in the pursuit of robustness, stability and moving on to what really mattered. We’ve only recently had the luxury of time in being able to look holistically at RC1 in the backdrop of only know being
able to reflect on our real-world experiences the past 18-months. I apologize if this cadence is not ideal for the community. This is part on us and part of the reality of where the market stands today in being able to give it real-world insights.
Our company’s big bet for scalability is STIX and TAXII and we have and will continue to invest millions in its advancement. Open-source implementations, storage, distribution, gateways, uni-directional data diodes, workflows, exchange and so forth. I can assure
the community that we’re both invested in the effort and grounded in the reality of its many gaps, use-cases and non-functional requirements. I’m sure there are many technical pro’s and con’s to our proposal and spec draft and I assume the community with give
it the consideration and discussion it deserves. The underlying message though is that the journey from unproven specification to proven and robust implementation is long and costly. This does not seem to be a key consideration and from a business perspective
I see disproportional risk to reward not choosing for existing APIs with mature existing communities of developers; JSON-API is an example, so is Odata. And if I’m completely honest, I’m not too keen on having to pick up that slack as one of the few that will
invest in fully implementing and supporting the specification if it can be avoided. We will if the community decides so though.
Reviewing the substance of discussion, there is allot of brain allocated to technical nuances that many others have solved before where I think it would be better spend on CTI and a wider set of use-cases. I realize that, not having had the luxury of participating
much more than we did, this hypocrisy is forgiven and seen in the light in which it is intended; positive criticism/reality check from the outside.
Lastly, it’s worthwhile to point out that in the current spec we require TAXII servers/clients to understand STIX. STIX is non-trivial and manipulating it (considering its many features) in this way puts a disproportioned burden on TAXII servers and their implementation.
Although less complicated for IOC-only data, that’s not what TAXII is all about. We would highly recommend making a much cleaner cut.
Combined we believe there is a notable risk that 2.1 or 3.0 will already require significant protocol changes. Our stance is:
- There exists significant avoidable risk in the current specification and recommend to seriously consider APIs with existing and mature communities (our short-cut with the JSON-API proposal and draft or another);
- Our proposed draft is not equal to ‘a suggestion’ or ‘brought up before’ and comes with a draft, takes into account RC1 functionality and adds some. We hoped this would have been enough to separate this discussions from some of the previous. If by its own
merit is not seen as sufficiently clear in terms of added value, that’s fine – but is not equal to an opening to put everything back on the table.
- Putting a burden of parsing STIX on TAXII as per RC1 is strongly advised to re-consider. Considerably impacts barrier to entry in implementing and for those implementing, might lead to less open code (due to size investment).
Hope this helps in the discussions around our proposal, but perhaps even more so for the future.
Best regards,
Joep
Founder & CEO
EclecticIQ
From: <
cti@lists.oasis-open.org> on behalf of Jason Keirstead <
Jason.Keirstead@ca.ibm.com>
Date: Saturday, 11 March 2017 at 01:29
To: "pmaroney@wapacklabs.com" <
pmaroney@wapacklabs.com>
Cc: "Bret_Jordan@symantec.com" <
Bret_Jordan@symantec.com>, "cti@lists.oasis-open.org" <
cti@lists.oasis-open.org>
Subject: Re: [cti] TAXII
Hello all - I have been making numerous arguments against this JSON-API proposal on Slack, but figured I had better send them to the mailing list as the whole TC does not monitor Slack.
- First, let me preface this by saying personally I do not agree at all with how this proposal has come about. We have been working on TAXII 2 as a community for over a year, and many, many TC members have contributed to it. To come in at the RC stage with
an "out-of-the-blue" ground-up re-proposal is far from ideal. It is in my opinion very hard to consider a proposal solely on its merits with timing such as this.
- In my opinion, in order to make an argument to re-visit all of TAXII at this late stage, one should have a compelling set of reasons for this re-visit - ie a list of problems that exist in the RC proposal that either do not exist or are solved in the new
proposal. There is no such set of reasons presented here - it is simply a ground-up re-think with no compelling arguments presented as to why it is superior to the current RC TAXII proposal. As such, to me this JSON-API proposal to me is very much a solution
in search of a problem... what are the specific problems it is solving?
- The new JSON-API based proposal is **extremely verbose**. It is difficult to put an exact number on the "data bloat" occurring here, but I would conservatively place it at around 10x vs. the RC1 proposal. We must keep in mind that most vendors are potentially
dealing with extremely large data set sizes when it comes to CTI production and consumption, and size matters - both on the wire, and in memory, and at parse time. The argument "CPU/Memory/Disk is cheap" does not hold water in the world of big data. Data size
still matters. In return for this data bloat, I would hope that this proposal came with a large set of concrete benefits, but I don't see them.
- Finally - If we were actually going to re-open everything and spend another 3/6/9/12 months on TAXII 2, then I would humbly also request time to submit a proposal for OData as well -
http://www.odata.org/ -
as this is an existing OASIS standard developed over a long period of time, and backs large scale enterprise services already, so I have a lot more confidence in it than JSON-API for our use cases, and it would also give us TAXII Query as well as many other
capabilities out of the box.
-
Jason Keirstead
STSM, Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security www.securityintelligence.com Without data, all you are is just another person with an opinion - Unknown
----- Original message -----
From: Patrick Maroney <
pmaroney@wapacklabs.com>
Sent by: <
cti@lists.oasis-open.org>
To: Bret Jordan <
Bret_Jordan@symantec.com>
Cc: "cti@lists.oasis-open.org" <
cti@lists.oasis-open.org>
Subject: Re: [cti] TAXII
Date: Wed, Mar 8, 2017 11:22 AM
The Eclectic-IQ slide on Objects (slide 8) is a compelling argument (as are arguments for json-api, etc.). As is the general argument for looking to and fully vetting mature transport mechanisms like XMPP.
My .02: With all due respect to the efforts of those folks who've expended significant effort to get us to where we are, I don't think one week is adequate for the community to fully assess and understand these counter-proposals.
Patrick Maroney
Principle Engineer - Data Science & Analytics
Wapack Labs
pmaroney@wapacklabs.com (609)841-5104
On Mar 7, 2017, at 12:24 PM, Bret Jordan <
Bret_Jordan@symantec.com> wrote:
All,
On the working call today, we had two presentations about possible additions / changes / modifications to the current TAXII RC 1 specification. The first presentation was from EclecticIQ on the possible use of JSON-API. The second presentation was from Cisco
on the use of XMPP-Grid. At a minimum, I would like this community to use these presentations as a sanity check for what we have done.
Call to action: As a member of this open community, if you are in support of either of these technologies, or would like the TAXII Community to focus more time on either of them, please speak up. If you have questions about what these technologies would mean
for TAXII, please ask here on the email list and we will defer to either EclecticIQ or Cisco to answer.
With all suggestions, it is up to the sponsor of that idea to gather support, drive the discussion, and help write any and all normative text that would support it.
As a reminder, the current TAXII specification is nearing completing. So if we need to adopt or change anything based on these presentations, we really need this community to identify that within the next week.
Bret
--------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php