The current TAXII spec is addressing all of the use cases that have been brought forward. If you would like some additional functionality, please write up a proposal.
In regards to what you are doing. You have said that need to write a solution to aggregate the data from multiple sensors. Since you are writing the solution, you can write it how ever you need. If you need to paginate data from that solution you are
writing, then by definition you will need to store that data for some period of time. So once again, since you are writing the solution and needing to store the data in a data store, you can easily solve this problem.
If TAXII is missing support for some use cases, please write up a proposal for these use cases and how you would propose solving them, and why current solutions do not meet those needs. I think that would really help the technical committee understand
what you need and why.
Bret
Sent from my Commodore 128D
PGP
Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050
On Sep 10, 2019, at 2:42 PM, Andras Iklody <
andras.iklody@circl.lu > wrote:
The taxii service would internally query a set of other services and
provide these as a response. There is no centralised database for this
data. Again, TAxII is only there for the transport layer, behind that
we'd have a collection of tools with their own data-sets in various
formats - the only timestamp we have is what they provide.
I am not sure where this data-point should come from. We could create
timestamps on the fly, based on the time when the request hits the
service and add incrementing values in the micro-second part, but that
seems like a dirty hack.
Hope this makes sense, but I have a feeling that the current TaXII
design fits a limited type of tools whilst ignoring other use-cases and
it seems to break the original paradigm of it being purely a transport
layer protocol and actually requires data to be in a static database
behind it (if you want to be able to paginate). Or am I missing
something obvious?
Best regards,
Andras
On 10.09.19 14:26, Bret Jordan wrote:
You keep talking about systems that emit or use second precision data.
The TAXII date_added value is not related to the data or content. It
is not related to the STIX data at all and it should not be kept. It is
an arbitrary value that does not need to be kept. It is purely a
tracking point for the TAXII server itself. Two TAXII servers that have
the same object will more than likely not have the same value for the
date added.
So said another way, if I sync content from you, I will not keep the
date added values from your server. I will assign my own, because this
meta data is tied to the TAXII server itself. An organize may choose
to keep it, but it does not need to. It should be considered a throw
away value.
So just like you have an auto incrementing index value, you would have
an auto incrementing date value. So in your solution you could just
store the date in a Unix time stamp format with microsecond precision.
This auto incrementing index could just be translated back and fort has
between RFC3339 format.
Bret
Sent from my Commodore 128D
PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050
On Sep 10, 2019, at 10:43 AM, Andras Iklody <
andras.iklody@circl.lu < mailto:
andras.iklody@circl.lu >> wrote:
Hi Bret,
So, if this is the recommended hack to use for TAXII, this advice could
perhaps be in the final text to avoid other implementers falling into
the trap of not being able to fetch subsets of the data as in the
situation I've explained before.
So to reiterate, date_added should be used as the iterator for
pagination, use consecutively incremented micro precision for systems
that emit second precision data.
As for writing a proposal for changes, I think based on our previous
experiences it's a route that we're not willing to go down - fighting
for absolutely minor changes for years is out of scope for small open
source teams. We're just looking for guidance how to implement what has
been agreed upon by the TC at this point.
Best regards,
Andras
On 09.09.19 17:46, Bret Jordan wrote:
As I stated in a previous email, you should be able to easily do a
date_added+.000001 for each record. You are going to have to store
these records on your system, and thus you will need some basic
meta-data to store them anyway. You probably are going to store
which device you got them from, and a bunch of other things. A
simple timestamp should not be a significant challenge.
If you want to write up a proposal for TAXII 2.2, I am sure others,
including me, would love to see it. Please just make sure your
proposal does not break existing use-cases.
Thanks
Bret
On Sep 9, 2019, at 5:14 PM, Andras Iklody <
andras.iklody@circl.lu < mailto:
andras.iklody@circl.lu >> wrote:
The data is collected in bulk from the other sources. We could indeed
attach a timestamp to the bulk collection, but that would be the same
for all data collected in one shot exceeding any sane pagination limits.
I am somewhat confused that we're back at the data storage question
though, TAXII/STIX should NOT have inherent needs when it comes to how
data is stored as from what I understood it's purely a
transport/exchange format / protocol. If TAXII basically requires us to
move all the stored data to true micro second precision (instead of
padding it on fetch for lower precision tools), then that will kill a
lot of interactions with other tools out there.
Best regards,
Andras
On 09.09.19 17:07, Bret Jordan wrote:
So if you are building a streaming collector that gathers data from
other sensors, you have control over how that data gets added /
stored on that intermediary streaming server. Since you may need
to paginate the data, you must be definition then need to
temporarily store that data. If you need to store data, then you
can define what ever data structures you want. Thus you can define
the date added property with microsecond precision. Keep in mind
that the data added is NOT the date that the end / origin sensor
used for its timestamp.
I just do not see how this is broken or why it would not work.
Please help me
Bret