OASIS Open Document Format for Office Applications (OpenDocument) TC

 View Only
  • 1.  Defect severity

    Posted 03-12-2009 17:00
    It is probably worth getting to a common understanding of defects and how 
    much we're going to commit to resolving for ODF 1.2.  OASIS doesn't 
    mandate any particular taxonomy for defects, and ISO/IEC has only a 
    technical/editorial two-bucket classification system which is a bit too 
    course for our needs.
    
    I'll toss out this classification scheme for defects, a scale of 6 
    severity levels:
    
    0. Defects which could cause the standard to be used in ways which are 
    unsafe to persons or the environment, or which violate civil or human 
    rights.  For example, defects related to privacy, safety tolerance, 
    encryption, etc. 
    
    1. Defects whose existence would likely cause direct business or financial 
    loss to users of the standard.  For example, spreadsheet financial 
    functions which are defined incorrectly,
    
    2. Defects which prevent the standard from being used in the way which it 
    was intended.  This severity level requires a likelihood of misapplication 
    of the standard, not merely a remote potential for misapplication. 
    
    3. Defects which violate drafting requirements from OASIS or ISO/IEC.  For 
    example, lack of a scope statement or misuse of control vocabulary.
    
    4. De minimis defects , i.e., trivial defects, hardly worth fixing. 
    Obviously, even the smallest defect related to health and safety must be 
    given considerable regard.  However, a typographical error where the 
    meaning is otherwise clear from the context may be considered trivial. 
    Similarly, a grammatical error which does not change the meaning of a 
    clause or a terminology question where the meaning is clear and 
    unambiguous to a person having ordinary skill in the art to which the 
    clause most closely pertains, i.e., 2D vector graphics.
    
    5. Matters of personal style.  For example, a request to use "do not" 
    rather than the contraction "don't".  These are opinions, but not defects.
    
    Obviously the above are not rigid mathematical definitions.  Most are 
    judgement calls.  We are fortunate to have so many ODF implementors on the 
    TC to help us accurately evaluate defects and set appropriate severity 
    levels.
    
    If something like the above meets with the TC's approval, then we could 
    elaborate on the definitions and agree to apply them in our decision 
    making. 
    
    For example, we could say that defects reported on published ODF standards 
    would be processed as follows:
    
    Severity 0-1 would trigger the TC to immediately prepare an errata 
    document.
    Severity 2 would be resolved in the next-scheduled errata document.
    Severity 3-4 will be resolved in the next technical revision of the ODF 
    standard, i.e., pushed into the next version.
    
    We could also agree to treat this severity levels differently depending on 
    where we are in the drafting process for ODF 1.2.  For example, once we 
    send out for public review, we might commit to fix all defects of severity 
    0-3, but not 4 & 5.
    
    Let me know if this is is a useful model for thinking of defects.
    
    -Rob
    


  • 2.  RE: [office] Defect severity

    Posted 03-15-2009 19:13
    1. I have been mulling on these and there are a couple of cases that I don't
    know how to qualify.  
    
       1.1 Also, the dependence on subjective assessment seems to be a problem
    for some of these.
    
       1.2 In addition, in practical situations, it seems to me the problem is
    not about fixing the standard quickly but getting implementations that may
    exhibit the problem to come up with important fixes by some sort of
    pre-standard agreement that can be rolled into the standard as needed.
    Putting together a coalition around a remedy, for something with
    criticality, seems far more likely and relevant than cranking on the
    standard.
    
    2. (3) is easy.  (1) is pretty easy too, since there is likely to be a
    pretty good objective demonstration of the defect.  Correcting or clarifying
    the specification will matter for (1), but agreeing on the problem and its
    repair for documents and implementations "in the wild" will have more
    urgency. 
    
    3. (0) is difficult. 
       3.1 I am aware at least two security exposures around the use of
    encryption and digest algorithms, but I would not put them in (0).  They
    demonstrate the absence of a worked-through threat model more than
    representing an immediate public-safety matter.  
       3.2 In some cases, lack of accessibility provisions comes under civil
    rights protections and also statutory requirements in certain domains of
    use.  I still bet these are in the (2-3) level, in some sense, depending on
    any claims of scope, and to the extent that it is essential that provisions
    be made in the format itself.
       3.3 Actually, the definition, for packages, of the *required*
    manifest:checksum element is a security disaster, since it basically allows
    the file to be decrypted without knowing the passphrase from which the
    digest is computed.  (See 5.5, below.  Even when the key generation
    algorithm is chosen to use a different digest method, presence of the SHA1
    digest considerably simplifies plaintext attack on the passphrase in order
    to crack the document.)  It intrigues me even more that this schema-required
    attribute does not appear in the sample manifest and, I trust, not in any
    actually-encrypted files.  That the provision has still not been removed
    says something, but I am unsure what.  
       3.4 [STOP THE PRESSES: I was so curious about manifest:checksum that I
    used the OO.o save-with-password option on the 1.2 Package draft itself.
    Surprise, surprise, not only are manifest:checksum attributes present, their
    values are different for each encrypted item, even though I only entered one
    pass phrase and the same digest algorithm is identified each time.  This is
    clearly not reproducible from any information provided in the specification
    and anything understood about the "intent."] 
    
    4. For (4)-(5) there is not much to quarrel with, although I think where
    there is an issue around ambiguity and understandability (especially in the
    face of international translations), so we need to be careful about "trivial
    for whom?" 
       4.1  I also think we should not collapse skilled-in-the-art and
    skilled-in-language and also skilled-in-understanding-specifications.  
       4.2  It's my experience that those who are skilled-in-the-art of
    specification with regard to a particular subject matter have a heightened
    sensitivity to ambiguity and under-specification as well as sharpened
    recognition of careless use of their specialty nomenclature and concepts.
    If we are not those experts, we should not be saying "oh, the ones skilled
    in the art will understand this precisely."
       4.3 Furthermore, a specification for one audience that depends on skills
    of an allied art needs to make that dependency very clear so that (1) those
    not so specialized are led to the required resources (2) those not so
    specialized are led to recognize that they require specialized knowledge,
    and (3) those who are so specialized recognize that the dependency is
    precisely-expressed.
    
    5. (2) is the most problematic for me.  The difficulty I see is over how one
    ascertains intention on the parts of the standard writers and whether tacit
    understanding is a reliable quality.  
       5.1 I suggest that any time it is necessary to inspect implementations,
    and the standard does not say implementation-determined, that should be an
    automatic (2).  
       5.2 Also, when a comment claims that the intention is not apparent, I
    think we should accept that statement on face value, at least provisionally,
    until there is enough deeper analysis to understand what the disconnect is.
    There is probably some remedy required, even if it is not the one the
    commenter suggests. 
       5.3 For (2), the lack of a measurable quality is particularly
    troublesome.  I don't think "likelihood of misapplication" is any help here.
    What does it mean to "misapply" a standard?  Is that the appropriate
    criterion?  
       5.4 Here's an example.  There are places where the specification provides
    that digest algorithms to be applied in the handling of pass phrases that
    are used for various document-protection purposes.  The referenced
    specifications for those digest algorithms (to the extent references are
    provided) simply describe the input as a string of bits in storage.
    Clearly, for the algorithm to work in a given application, there needs to be
    an agreement on how that string of bits is coded before the digest algorithm
    is applied, lest the result not be reproducible.  Nowhere in the ODF
    specifications is this missing piece of agreement specified.  (I suspect the
    same is true for many other applications that appeal to the digest algorithm
    specifications.)  In an international setting, this is a particularly
    troubling situation, since there is a very big difference even when assuming
    single-character octets (e.g., 7-bit ASCII versus ISO 8859-1) or
    multi-byte/double-byte characters (coded in SHIFT-JIS versus UTF-8 or UTF-16
    or UTF-32).  There are, for complex languages, normalization rules that must
    be applied as well to ensure that the "same" composed/combined characters
    lead to the same encoding before the digest algorithm is applied.  Finally,
    there are white-space rules and other rules to be considered in order to
    have the digest be reproducible among different products working with the
    same document, with different default language settings, platform defaults,
    etc., etc.  I'd love to see how the intent is to be understood here.   
       5.5 Here's another on the same pet topic. The ODF-specific encryption
    technique for package items applies a key derivation and encryption process
    to each encrypted item in the package, and the manifest entry for each such
    item contains the necessary parameters by which the key derivation can be
    repeated and decryption accomplished.  Nowhere is it required that all of
    these encryptions start with the same pass phrase.  In fact, it is
    conceivable that a different pass phrase is used for each of the
    encryptions.  It is highly unlikely that is the case in any current
    implementations, since it makes no sense to suppose that an user who
    requests encryption and supplies a pass phrase (of the kind wondered about
    in 5.4, above) has any idea which parts of the ODF package will be encrypted
    as a result, and an user who knows or has been conveyed that pass phrase
    certainly doesn't know which parts of the package have been encrypted.  I
    would claim, in this case, after having invested considerable study to this
    part of the specification and the sample manifest (the only place to learn
    certain things), but without even bothering to try it with OO.o, that there
    is a single pass phrase and it (appropriately-digested) is used as part of
    the key-generation procedure for each encrypted item in the package.  You
    could convince me that this single pass-phrase, symmetric key-derivation
    procedure is "the way the standard is intended."  I'm not sure what the
    value of leaving this underspecified is other than to make developers work
    harder to figure out what only makes sense, after verifying that with an
    implementation or two.  [Addendum: I have checked this with OO.o and have
    found a baffling aspect to it, noted in 3.4, above.]
    
     - Dennis
    
    


  • 3.  RE: [office] Defect severity

    Posted 03-15-2009 20:04
    Dennis,
    
    This is not an attempt to remove subjective risk assessment from the 
    defect categorization process.  It is merely an attempt to provide a 
    framework for us to use when subjectively evaluating defects, hopefully 
    something more useful that ISO's binary technical/editorial 
    classification.  In the end it is still subjective.  Implicitly, the 
    scheme is "Sev 0, where the ODF TC agrees that the defect is such that..." 
    and "Sev 1, where the ODF TC agrees that the defect is such that...", etc.
    
    But to the extent we have the input of all of the major ODF stakeholders, 
    vendors as well as user and government adopters of ODF, I think we'll do 
    well. 
    
    I throw these definitions up on the Wiki and we can elaborate on them 
    there.  I'll also add a severity column to the comment registry, so we can 
    start tracking them there as well. 
    
    -Rob
    
    
    
    


  • 4.  Discussion: Public Comment and Errata Urgency/Priority/Action Levels

    Posted 04-27-2009 16:32
    In the time since we discussed severity levels, I have found that I am
    uncomfortable attempting to apply the severity levels.
    
    I would rather use is an indication of the priority/precedence/urgency that
    we attach to a reported issue in terms of the action we propose to take.  I
    am assuming that the issue passes categorization and it is agreed that the
    matter discussed does occur (or there is a demonstrated likelihood of
    misunderstanding that a repair could avoid).  
    
    For me, this is different than attempting to create a template for inherent
    severity, and more flexible.  That is why I am asking your impressions and
    discussion on the list.
    
    Here are suggested action levels.  Note that these can be assigned
    differently with respect to the same issue but different versions of
    specifications and in-progress development of committee drafts.  These are
    assigned as the judgment of the TC on how to proceed with an issue.  This
    seems to provide the discretion we require and that we can apply on a
    case-by-case basis.
    
    0. No Action.  Although the situation occurs, it is concluded that no action
    is to be taken.
    
    1. Elective.  The issue may be considered in work in progress; it is not
    considered material and not worthy of errata or corrigendum to existing
    work.
    
    2. Desirable.  The issue will be addressed in work in progress.  It may be
    reflected in a future errata or corrigendum, if such are being produced
    anyway. 
    
    3. Important.  The issue shall be addressed in work in progress.  It shall
    be reflected in an erratum or corrigendum at the next opportunity.
    
    4. Urgent.  The issue shall be addressed at the earliest possible time.  It
    will be addressed in work in progress.  It is important enough to trigger
    production of an erratum or corrigendum or advance notice of a defect to be
    remedied. 
    
    5. Critical.  There is an emergency involving risk to implementation and use
    of a specified feature that must be addressed immediately.  Extraordinary
    out-of-cycle measures will be taken to issue advisories and other
    announcements of the issue and recommended work-around until it can be
    handled as part of an urgent response.
    
    
     - Dennis
    
    
    
    ADDITIONAL BACKGROUND
    
    1. I chose to number lowest from 0 because it is easy to remember that 0
    corresponds to "no action."
    
    2. There may be a fine distinction between No Action and Elective.  In the
    case of elective items, there is the possibility that there may be some
    improvement in addressing the issue but it is left to the discretion of
    those working on future materials.  This level is designed to ensure that
    there is consideration, but there is no commitment with regard to the
    outcome.  This is a place for severity level 5 items that are thought worth
    considering but not required.
    
    3. The "Critical" category is for some "the standard is on fire" situation.
    I can't imagine what that might be, unless it is related to security issues
    or a feature of the specification that is the target of some sort of exploit
    or failure if implemented.  It is here just for completeness.
    
    4. I tried fewer than 6 levels but I kept needing to put another back in
    every time.
    
    THE CURRENT SEVERITY LEVELS
    
    [In my experience with defect/incident reporting systems, it is the
    (external) submitter of an issue who assesses the severity level, not the
    responding organization, which assigns priority and other disposition
    categories.  It took me a while to notice that difference.]
    
    Severity Levels as currently summarized on the Wiki
    (http://wiki.oasis-open.org/office/Severity_Levels):
    
     0. Defects which could cause the standard to be used in ways which are
    unsafe to persons or the environment, or which violate civil or human
    rights.  For example, severe related to privacy, safety tolerance,
    encryption, etc. 
    
     1. Defects whose existence would likely cause direct business or financial
    loss to users of the standard.  For example, spreadsheet financial functions
    which are defined incorrectly,
    
     2. Defects which prevent the standard from being used in the way which it
    was intended.  This severity level requires a likelihood of misapplication
    of the standard, not merely a remote potential for misapplication. 
     
     3. Defects which violate drafting requirements from OASIS or ISO/IEC.  For
    example, lack of a scope statement or misuse of control vocabulary.
    
     4. De minimis defects , i.e., trivial defects, hardly worth fixing.
    Obviously, even the smallest defect related to health and safety must be
    given considerable regard.  However, a typographical error where the meaning
    is otherwise clear from the context may be considered trivial. Similarly, a
    grammatical error which does not change the meaning of a clause or a
    terminology question where the meaning is clear and unambiguous to a person
    having ordinary skill in the art to which the clause most closely pertains,
    i.e., 2D vector graphics.
    
     5. Matters of personal style.  For example, a request to use "do not"
    rather than the contraction "don't".  These are opinions, but not defects. 
    
    


  • 5.  Re: [office] Discussion: Public Comment and Errata Urgency/Priority/ActionLevels

    Posted 04-28-2009 02:28
    Dennis, all,
    
    I would go for action levels, that represent the priorities
    predetermined by the JIRA bug tracker, in order to ensure consistent
    work flow.
    Starting at 'Create Isuue' [1], I have been proceeding to step 2, where
    I find 'Blocker/Critical/Major/Minor/Trivial' on the 'Priority'
    drop-down. Both your proposal and the severity levels defined at the ODF
    wiki [2] fail in mapping to the 'Create Issue' form, because they break
    severity down into six levels, rather than five predefined in the bug
    tracker. This seems to be configurable with JIRA [3], however I would
    guess, that tweaks have a global effect on the JIRA installation at
    OASIS. Did someone dive deeper into this yet?
    
    Best regards,
    Peter
    
    [1] http://tools.oasis-open.org/issues/secure/CreateIssue!default.jspa
    [2] http://wiki.oasis-open.org/office/Severity_Levels
    [3] http://www.atlassian.com/software/jira/docs/v3.13.1/priorities.html
    
    Dennis E. Hamilton wrote:
    > In the time since we discussed severity levels, I have found that I am
    > uncomfortable attempting to apply the severity levels.
    > 
    > I would rather use is an indication of the priority/precedence/urgency that
    > we attach to a reported issue in terms of the action we propose to take.  I
    > am assuming that the issue passes categorization and it is agreed that the
    > matter discussed does occur (or there is a demonstrated likelihood of
    > misunderstanding that a repair could avoid).  
    > 
    > For me, this is different than attempting to create a template for inherent
    > severity, and more flexible.  That is why I am asking your impressions and
    > discussion on the list.
    > 
    > Here are suggested action levels.  Note that these can be assigned
    > differently with respect to the same issue but different versions of
    > specifications and in-progress development of committee drafts.  These are
    > assigned as the judgment of the TC on how to proceed with an issue.  This
    > seems to provide the discretion we require and that we can apply on a
    > case-by-case basis.
    > 
    > 0. No Action.  Although the situation occurs, it is concluded that no action
    > is to be taken.
    > 
    > 1. Elective.  The issue may be considered in work in progress; it is not
    > considered material and not worthy of errata or corrigendum to existing
    > work.
    > 
    > 2. Desirable.  The issue will be addressed in work in progress.  It may be
    > reflected in a future errata or corrigendum, if such are being produced
    > anyway. 
    > 
    > 3. Important.  The issue shall be addressed in work in progress.  It shall
    > be reflected in an erratum or corrigendum at the next opportunity.
    > 
    > 4. Urgent.  The issue shall be addressed at the earliest possible time.  It
    > will be addressed in work in progress.  It is important enough to trigger
    > production of an erratum or corrigendum or advance notice of a defect to be
    > remedied. 
    > 
    > 5. Critical.  There is an emergency involving risk to implementation and use
    > of a specified feature that must be addressed immediately.  Extraordinary
    > out-of-cycle measures will be taken to issue advisories and other
    > announcements of the issue and recommended work-around until it can be
    > handled as part of an urgent response.
    > 
    > 
    >  - Dennis
    > 
    > 
    > 
    > ADDITIONAL BACKGROUND
    > 
    > 1. I chose to number lowest from 0 because it is easy to remember that 0
    > corresponds to "no action."
    > 
    > 2. There may be a fine distinction between No Action and Elective.  In the
    > case of elective items, there is the possibility that there may be some
    > improvement in addressing the issue but it is left to the discretion of
    > those working on future materials.  This level is designed to ensure that
    > there is consideration, but there is no commitment with regard to the
    > outcome.  This is a place for severity level 5 items that are thought worth
    > considering but not required.
    > 
    > 3. The "Critical" category is for some "the standard is on fire" situation.
    > I can't imagine what that might be, unless it is related to security issues
    > or a feature of the specification that is the target of some sort of exploit
    > or failure if implemented.  It is here just for completeness.
    > 
    > 4. I tried fewer than 6 levels but I kept needing to put another back in
    > every time.
    > 
    > THE CURRENT SEVERITY LEVELS
    > 
    > [In my experience with defect/incident reporting systems, it is the
    > (external) submitter of an issue who assesses the severity level, not the
    > responding organization, which assigns priority and other disposition
    > categories.  It took me a while to notice that difference.]
    > 
    > Severity Levels as currently summarized on the Wiki
    > (http://wiki.oasis-open.org/office/Severity_Levels):
    > 
    >  0. Defects which could cause the standard to be used in ways which are
    > unsafe to persons or the environment, or which violate civil or human
    > rights.  For example, severe related to privacy, safety tolerance,
    > encryption, etc. 
    > 
    >  1. Defects whose existence would likely cause direct business or financial
    > loss to users of the standard.  For example, spreadsheet financial functions
    > which are defined incorrectly,
    > 
    >  2. Defects which prevent the standard from being used in the way which it
    > was intended.  This severity level requires a likelihood of misapplication
    > of the standard, not merely a remote potential for misapplication. 
    >  
    >  3. Defects which violate drafting requirements from OASIS or ISO/IEC.  For
    > example, lack of a scope statement or misuse of control vocabulary.
    > 
    >  4. De minimis defects , i.e., trivial defects, hardly worth fixing.
    > Obviously, even the smallest defect related to health and safety must be
    > given considerable regard.  However, a typographical error where the meaning
    > is otherwise clear from the context may be considered trivial. Similarly, a
    > grammatical error which does not change the meaning of a clause or a
    > terminology question where the meaning is clear and unambiguous to a person
    > having ordinary skill in the art to which the clause most closely pertains,
    > i.e., 2D vector graphics.
    > 
    >  5. Matters of personal style.  For example, a request to use "do not"
    > rather than the contraction "don't".  These are opinions, but not defects. 
    > 
    > 


  • 6.  RE: [office] Discussion: Public Comment and Errata Urgency/Priority/Action Levels

    Posted 05-08-2009 04:45
    Hi Peter,
    
    The JIRA Priorities are clearly defined around software deliverables and also support after deployment.  There are other deficiencies, but I don't suppose to go into battle over that.
    
    I do recommend that we come up with an understanding of these level that correspond in clear-cut ways to creation of specifications and documentation of other kinds.  Likewise, we need to understand how to calibrate the importance of responding to defect reports on that scale.
    
    I don't have any specific recommendation.  We should think it over.
    
     - Dennis
    
    MORE BACKGROUND
    
    My first thought was that the action/urgency levels I had in mind interleaved around the five JIRA "priorities."
    
    I now see it is more complicated than that.  First the JIRA priorities are not priorities ("trivial" is not a priority) and the tool-tips that go with them reveal even more about them as being judgments about the issue.
    
    "Blocker" is an interesting one, and it applies to in-progress work, of course.  I can see ways to use that.
    
    Here's a crude mapping (very crude, I am just playing here):
        Urgency	     JIRA Priority    Severity
    
    0: no action (an actual resolution in JIRA)
                     trivial        5: matter of style 
                     (cosmetic)     4: de minimis
    1. elective
                     minor (can workaround)
    2. desirable                    3. mandated
    3. important                    2. insufficient 
    4. urgent        major          1. serious
                     (loss of function)     
    5. critical      critical       0. critical
                     (fails)
                     blocker  
                     (cannot proceed)   
    
    You can slide the Urgency and Severity notions around the JIRA Priorities in different ways.  I'm not going to fuss over that.  I think making something sensible about the JIRA Priorities that work for us in terms of urgency and level of response is good enough.
    
    


  • 7.  Re: [office] Discussion: Public Comment and Errata Urgency/Priority/ActionLevels

    Posted 05-13-2009 13:28
    Dennis,
    
    Dennis E. Hamilton wrote:
    > Hi Peter,
    > 
    > [...]  There are other deficiencies, but I don't suppose to go into battle over that.
    
    Thanks :-)
    
    > 
    > I do recommend that we come up with an understanding of these level that correspond in clear-cut ways to creation of specifications and documentation of other kinds.  Likewise, we need to understand how to calibrate the importance of responding to defect reports on that scale.
    > 
    
    Sounds reasonable.
    
    [...]
    
    > 
    > Here's a crude mapping (very crude, I am just playing here):
    >     Urgency	     JIRA Priority    Severity
    > 
    > 0: no action (an actual resolution in JIRA)
    >                  trivial        5: matter of style 
    >                  (cosmetic)     4: de minimis
    > 1. elective
    >                  minor (can workaround)
    > 2. desirable                    3. mandated
    > 3. important                    2. insufficient 
    > 4. urgent        major          1. serious
    >                  (loss of function)     
    > 5. critical      critical       0. critical
    >                  (fails)
    >                  blocker  
    >                  (cannot proceed)   
    > 
    > You can slide the Urgency and Severity notions around the JIRA Priorities in different ways.  I'm not going to fuss over that.  I think making something sensible about the JIRA Priorities that work for us in terms of urgency and level of response is good enough.
    
    I totally agree here. It's probably more effective to adopt ourselves to
    the JIRA 'priorities', which seem match our requirements for the most
    part, rather than starting a long discussion on the perfect mapping.
    
    Best regards,
    Peter
    
    > 
    > 


  • 8.  Re: [office] Discussion: Public Comment and Errata Urgency/Priority/Action Levels

    Posted 05-15-2009 09:36
    On Friday 08 May 2009, Dennis E. Hamilton wrote:
    > This seems to be configurable with JIRA [3], however I would
    > guess, that tweaks have a global effect on the JIRA installation at
    > OASIS.
    
    No, JIRA is very customizable, you can configure everything per
    project where necessary.
    
    -- 
    David Faure, faure@kde.org, sponsored by Qt Software @ Nokia to work on KDE,
    Konqueror (http://www.konqueror.org), and KOffice (http://www.koffice.org).
    


  • 9.  Re: [office] Discussion: Public Comment and Errata Urgency/Priority/ActionLevels

    Posted 05-15-2009 21:54
    David Faure