OASIS Open Document Format for Office Applications (OpenDocument) TC

Back to discussions
Expand all | Collapse all

Directories in Zip packages - was RE: XAdES support in ODF

Dennis Hamilton

Dennis Hamilton09-26-2010 20:48

Dennis Hamilton

Dennis Hamilton09-27-2010 00:30

  • 1.  Directories in Zip packages - was RE: XAdES support in ODF

    Posted 09-26-2010 20:48
      |   view attached



  • 2.  RE: Directories in Zip packages - was RE: XAdES support in ODF

    Posted 09-26-2010 21:13 
    Dennis, 

the ZIP appnote  (any version, at least since 4.50) says

" external file attributes: (4 bytes)

          The mapping of the external attributes is
          host-system dependent (see 'version made by').  For
          MS-DOS, the low order byte is the MS-DOS directory
          attribute byte.  If input came from standard input, this
          field is set to zero.
...
file name: (Variable)

          The name of the file, with optional relative path.
          The path stored should not contain a drive or
          device letter, or a leading slash.  All slashes
          should be forward slashes '/' as opposed to
          backwards slashes '\' for compatibility with Amiga
          and UNIX file systems etc.
..."

(Winzip probably 'translates' this for viewing purposes)


But the real questions are:

a) is adding (empty) directories, for whatever reason, allowed per ODF 1.2 spec
(since it requires a somewhat special attribute)

b) if so, should it be signed (I would say yes)

c) in that case, should the ODF spec text "sign every file" be changed in "every entry" 
(or similar)


Best regards

Bart

________________________________________
From: Dennis E. Hamilton [dennis.hamilton@acm.org]
Sent: Sunday, September 26, 2010 10:48 PM
To: Hanssens Bart; 'David LeBlanc'; office@lists.oasis-open.org
Cc: Cornelis Frank
Subject: Directories in Zip packages - was RE: XAdES support in ODF

For further amusement, the attachment is what PKZip for Windows version
12.50.0013 says about helloworld-signed.odt.

It seems that my (1-4) below have more evidence to deal with.

Re (1), PKWare prefers to show "/" as the segment separator.

Re (2), I have no idea at this point whether this is covered in the APPNOTE
or an extension

Re (3), the observation about no directory entries for Configurations2/,
Thumbnails/, etc., also applies to Configuration2//images as well as
Configurations2/accelerator.

Re (4), the "\" business may be answered.  I know where to find a reasonable
hex editor. Back soon.

 - Dennis



    
                                                
    
					
                                                    

        
                                                
				
    
                                                
    
                                                    
                                                        
                                                        Original Message


  • 3.  RE: [office] RE: Directories in Zip packages

    Posted 09-26-2010 23:20 
    My additional analysis follows in a separate note.  

Here is what I have:

1. Yes, directories/folders can be represented by file entries.  I was
completely off base about that.  I have confirmed that in APPNOTE 6.2.0.
But I don't think it happens the way you think in this particular file.  I
will check the central directory records I reviewed more closely. 

2. There is no reason that ODF 1.2 ever needs such things in a package,
since the manifest is independent of that stuff.

3. It is not possible to sign a directory/folder entry in the Zip because
there is nothing to sign.  There is no data in such an entry and there
should be no way to insert data in such an entry without it being a file
instead of a folder.  We'll have to see if that is actually a hole in Zip.
(Oddly enough, the 0-length file Configurations2\accelerator\current.xml
*has* data, and it is compressed, but the uncompressed size is 0.  Fancy
that.)

 - Dennis


    
                                                
    
					
                                                    

        
                                                
				
    
                                                
    
                                                    
                                                        
                                                        Original Message


  • 4.  RE: [office] RE: Directories in Zip packages

    Posted 09-26-2010 23:47 
    Dennis,

>2. There is no reason that ODF 1.2 ever needs such things in a package,
>since the manifest is independent of that stuff.

Well, perhaps no "need", but if it's allowed...it can be in there...

>3. It is not possible to sign a directory/folder entry in the Zip because
>there is nothing to sign. 

Actually, the same goes for the 0-size file, even an empty file can be signed :-)


> There is no data in such an entry and there should be no way to insert data
> in such an entry without it being a file instead of a folder

It is possible to craft a zip like that, although trying to unzip this could fail.
But who knows, perhaps some strange bug could be exploited because it is
not expected behavior.


>Oddly enough, the 0-length file Configurations2\accelerator\current.xml
>*has* data, and it is compressed, but the uncompressed size is 0.  Fancy
>that.

Since the storage method is deflate, not stored, it could perhaps be some
deflate header ?

Bart


  • 5.  RE: [office] RE: Directories in Zip packages

    Posted 09-27-2010 00:30
      |   view attached



  • 6.  RE: [office] RE: Directories in Zip packages

    Posted 09-27-2010 01:40 
    Ah, WTF for sure now.

Having restored my file associations I looked more closely at what the Windows Shell does with all those apparent directories that have no files in them.

 1. It turns out *every *one* that has no content ends up being an unrecognized form of 0-length file.  That leads to treatment of the following as empty files: 

  Configurations2/images/Bitmaps (no final / of course)
  Configurations2/floater
  Configurations2/menubar
  Configurations2/popupmenu
  Configurations2/progressbar
  Configurations2/statusbar
  Configurations2/toobar

 2. For bonus credit, I *opened* Configurations2/accelerator/current.xml (which auto-launched IE8).  Guess what?

   The XML page cannot be displayed 

   Cannot view XML input using style sheet. Please correct the error and 
   then click the Refresh button, or try again later. 

   ------------------------------------------------------------------------

   XML document must have a top level element. Error 
   processing resource 'file:///C:/Documents and Settings/orcmid/Local Setti...

Makes sense to me, or as we once found useful to say, "Garbage in, Garbage out."

 - Dennis


    
                                                
    
					
                                                    

        
                                                
				
    
                                                
    
                                                    
                                                        
                                                        Original Message


  • 7.  RE: [office] RE: Directories in Zip packages

    Posted 09-27-2010 12:56 
    I'm not sure I've read all the posts in this thread, but I believe:

1) ZIP totally allows zip items representing zero-byte files as well as 
items representing empty directories.  The later in particular is quite 
useful in general ZIP usage.  I remember seeing some bugs in the early 
1990's with some ZIP programs not handling this correctly.  But some uses, 
like self-extracting ZIPs that contain a pre-made empty directory for user 
data, will not work correctly without support for empty directories.

2) A zero-byte XML file is never correct.  Or at least it doesn't conform 
the the XML Recommendation since it is not well-formed XML.

3) On the other hand, except for the handful of ODF-defined ZIP items, 
like contents.xml, etc., we don't have any anti-spoofing requirement, 
right?  In other words we don't have a conformance requirement that says 
that content-type in the manifest matches the zip item.  If we had that 
restriction then it would not be conforming to have an zero-byte XML with 
content type text/xml or application/xml.  This would also make 
non-conformant potentially more sinister things like an EXE pretending to 
be image/png and stuff like that. 

4)However, there would be nothing wrong with a zero-byte foo.xml with a 
content type of text/plain or something similar.

5)Digital signatures apply to the contents of a file.  So you might think 
there is nothing to sign.  But in fact the zip item does bear a name and a 
time stamp, and either of these may bear information that could be harmed 
by tampering.  We cover the name by singing the manifest.  But we don't 
appear to cover tampering with the time stamp.  Of course, this is 
independent of the zero byte issue.

6) The most straightforward way for someone to implement a generic ODF 
package consumer would be to create a hashtable of each "file" in the ZIP 
and associate it with a record that contains metata on the entry (date, 
zip, compression method, etc.) as well as access to the underlying data. 
This is very simply in most programming languages.  Then when modifying 
and saving the package, I would recreate the manifest and write out all 
the other ZIP items. 

My guess is authors of this straightforward approach will often fail to 
properly handle the empty directory case, both on reading and on writing. 
(We have no way of notating an empty directory in the manifest).  So I'd 
favor a recommendation against (should not) or a prohibition against 
(shall not) an ODF package containing empty directories.  We have no need 
of it, and it will probably not work well across implementations.

-Rob


  • 8.  Re: [office] RE: Directories in Zip packages

    Posted 09-27-2010 14:58 
    Hi

I just completed a small experiment.  A small program reading through
a zip package created by openoffice (3.1.1) copying zipentries from
this file to a new zipfile, but leaving out any entry which is either
a directory or of zero length.

As I suspected, besides being marginally (1k) smaller, this remains a
fully acceptable odf file, at least so far as OOo is concerned.  What
is maybe a bit surprizing is that the manifest now refers to entries
which are no  longer in the zip package.  I was expecting a complaint
there but none was forthcoming.  Anyway, being a bit tidier, I
modified the manifest of the newly created file to remove the
redundant entries and I still remain with what seems to be a perfectly
acceptable package with no information loss.

All of which I think reinforce my earlier point.  ZIP may well have
valid use cases for storing directory information and zero length
files, and as Rob points out, the appnote totally allows it.  But from
our perspective (which is to package odf streams into a single archive
rather than to emulate a file system), there seems to be no good
reason to package these types of entries.  And at least one leading
implementation seems not to care if they are not there.  Removing
them, removes ambiguity over what should or not be signed.

So I would say that an odf producer should only produce entries in the
zipfile for non-zero-length streams (this would by default also
excludes directories).  And that each of these shall be referenced in
a full document signature.

An odf consumer, when validating a signature, shall verify that the
signature references all non-zero-length entries in the package.  The
presence of other zipentries in the package could be either ignored or
treated as an error.  Following Postel, I am leaning towards the more
permissive approach.  The benefit of simply ignoring being that it
would allow naive general purpose zip tools to produce valid odf
files, even though they would likely be violating the recommendation
above regarding odf producers.  I think this is reasonable given the
various toolchains people might construct which might involve an
eventual packaging stage using pkzip or something similar.

Regards
Bob

PS.  test files attached

On 27 September 2010 13:58,


  • 9.  Re: [office] RE: Directories in Zip packages

    Posted 09-27-2010 16:20 
    
  
    
  
  
    On 27.09.2010 16:57, Bob Jolliffe wrote:
    
    9A9ODyMhQ-M49m7eTxSA9h5w2jx3EhcD41KnG@mail.gmail.com" type="cite">
      
    ...

So I would say that an odf producer should only produce entries in the
zipfile for non-zero-length streams (this would by default also
excludes directories).  And that each of these shall be referenced in
a full document signature.

    
    
    
    May I remind the TC, that from the perspective of the package (ODF
    1.2 part 3 - which can be seen as the base layer of ODF 1.2) an
    entity called a document is defined by a directory related with a
    media type (in the manifest.xml).

    In general I believe when a generic functionality does not hurt, it
    should not be forbidden nor dropped as other yet unknown user
    scenarios might rely on it.

    Regards,
    Svante

    --
      

        Svante
              Schubert
              | ODF Standardization
              Phone: +49 40 23646 965
                Oracle Office
              GBU
    
      
    ORACLE
              Deutschland B.V. & Co. KG | Nagelsweg 55 | 20097
              Hamburg
              Hauptverwaltung: Riesstr. 25,
              D-80992 München
              Registergericht: Amtsgericht München, HRA
              95603

              Komplementärin: ORACLE Deutschland Verwaltung
              B.V.
              Rijnzathe 6, 3454PV De Meern, Niederlande
              Handelsregister
              der Handelskammer Midden-Niederlande, Nr. 30143697
              Geschäftsführer:
              Jürgen Kunz, Marcel van de Molen, Alexander van der Ven     
      
    
      
    
        
          
    
            
              
    
                		
            
              
    Oracle is
                      committed to developing practices and products
                      that help protect the environment


  • 10.  RE: [office] RE: Directories in Zip packages

    Posted 09-27-2010 17:31 
    Svante, a sub-document doesn't have to be an "actual" directory in any
literal sense.  This was clear in ODF 1.0/1.1 and I know of nothing that
requires us to change that.

The sub-document


  • 11.  RE: [office] RE: Directories in Zip packages

    Posted 09-27-2010 17:43 
    Arrgh :-) Folks, this is going back and forth, while I really would like to see some work on XAdES :-)

My suggestion:

- discourage (but not forbid) the use of empty directories and empty files in ODF packages
- sign everything in the zip (including empty directories and empty files)
- change the 1.2 draft about document signatures and mention "sign every entry" instead of 
"sign every file"

This means there's no impact on unsigned ODF docs (because every implementation can still
use empty things, although it is discouraged)

There is impact on signed implementations signing/verifying signed docs, but those products
probably have to be updated anyway (to support correct prefixes, version attributes etc)


Bart


________________________________________
From: Dennis E. Hamilton [dennis.hamilton@acm.org]
Sent: Monday, September 27, 2010 7:30 PM
To: 'Svante Schubert'
Cc: ODF TC List
Subject: RE: [office] RE: Directories in Zip packages

Svante, a sub-document doesn't have to be an "actual" directory in any
literal sense.  This was clear in ODF 1.0/1.1 and I know of nothing that
requires us to change that.

The sub-document


  • 12.  Res: RE: [office] RE: Directories in Zip packages

    Posted 09-27-2010 20:20 
    +1

    
                                                
    
					
                                                    

        
                                                
				
    
                                                
    
                                                    
                                                        
                                                        Original Message


  • 13.  RE: [office] RE: Directories in Zip packages

    Posted 09-27-2010 21:12 
    -1

Unless there is a proposal for how to accomplish the signing of
"directories" via XML Digital Signature (and also directory entries for
package files, which we don't sign either).

Please include a specific rule for knowing what is a directory entry in the
Zip, because there is no help in APPNOTE 6.2.0 that I can find.

 - Dennis


    
                                                
    
					
                                                    

        
                                                
				
    
                                                
    
                                                    
                                                        
                                                        Original Message


  • 14.  RE: [office] RE: Directories in Zip packages

    Posted 09-27-2010 21:57 
    Given a ZIP entry of:  A/B

How do we know whether:

B is an empty directory in A

versus

B is a zero byte file in A?

I thought A/B was a file and A/B/ was a directory.

That said, there is a lot of code, including libraries,  out there that 
doesn't follow the above rule and relies on figuring out implicitly what 
directories need to be made, according to the paths encoded in non-zero 
byte files.  So I still think allowing zero-byte files or empty 
directories is a portability concern.

-Rob



"Dennis E. Hamilton"


  • 15.  RE: [office] RE: Directories in Zip packages

    Posted 09-27-2010 23:13 
    +1 I agree completely


    
                                                
    
					
                                                    

        
                                                
				
    
                                                
    
                                                    
                                                        
                                                        Original Message


  • 16.  Re: [office] RE: Directories in Zip packages

    Posted 09-28-2010 07:20 
    Hi Rob,

robert_weir@us.ibm.com wrote:
> Given a ZIP entry of:  A/B
> 
> How do we know whether:
> 
> B is an empty directory in A
> 
> versus
> 
> B is a zero byte file in A?
> 
> I thought A/B was a file and A/B/ was a directory.
> 
> That said, there is a lot of code, including libraries,  out there that 
> doesn't follow the above rule and relies on figuring out implicitly what 
> directories need to be made, according to the paths encoded in non-zero 
> byte files.  So I still think allowing zero-byte files or empty 
> directories is a portability concern.

Is it? I mean, we have ODF since 2005, and did we ever have an issue 
with that?

It's not that this may not be discussed, but do we really know what 
impact a change in this area has? Are we sure we don't break existing 
applications? I personally would recommend to not make such changes for 
ODF 1.2 as late unless we are crystal clear about the impact and the 
benefits.

Michael
> 
> -Rob
> 
> 
> 
> "Dennis E. Hamilton"


  • 17.  RE: [office] RE: Directories in Zip packages

    Posted 09-27-2010 16:29 
    +1

Yes, I think we should define the confirming-package Zip to not include
0-length stuff, whether thought to be directories or files (although I have
a counter-example where a 0-length file if compressed has non-zero
compressed size (2 bytes, actually).

I also agree that consumers *should* be permissive for the use case you
describe.


    
                                                
    
					
                                                    

        
                                                
				
    
                                                
    
                                                    
                                                        
                                                        Original Message


  • 18.  RE: [office] RE: Directories in Zip packages

    Posted 09-27-2010 16:29 
    +1


    
                                                
    
					
                                                    

        
                                                
				
    
                                                
    
                                                    
                                                        
                                                        Original Message


  • 19.  RE: [office] RE: Directories in Zip packages

    Posted 09-27-2010 17:00 
    I second that, but then it must be prohibited (if it is only discouraged, one still
has to sign those empty dirs, just in case... that's probably what David meant
in his previous post)

Bart
________________________________________
From: Dennis E. Hamilton [dennis.hamilton@acm.org]
Sent: Monday, September 27, 2010 6:28 PM
To: robert_weir@us.ibm.com; Hanssens Bart
Cc: 'David LeBlanc'; Cornelis Frank; office@lists.oasis-open.org
Subject: RE: [office] RE: Directories in Zip packages

+1


    
                                                
    
					
                                                    

        
                                                
				
    
                                                
    
                                                    
                                                        
                                                        Original Message


Related Content