MantisBT - VTK
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0002379VTK(No Category)public2005-10-17 11:002015-01-09 13:49
ReporterMike.Stephens 
Assigned ToBrad King 
PriorityhighSeveritymajorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version 
Project
Type
Summary0002379: problem with XML Data I/O--unable to handle large files
Descriptionthis one needs to go to brad king.

--- original message to brad-----
brad,

I have a question about the “XML” file formats being used in vtk/paraview.

I searched through the source code but haven’t been able to satisfy myself as to the answer to my question

which is pretty simple:

for serial XML files is there a size limit on individual data arrays?

I’ve made a rectilinear grid by hand that and wrote out a Serial XML file.

when I try reading this file in paraview, CVS version around oct 1, 2005, or a VTK 4.4/dev app i wrote I get the following error:

prism% ./paraview

ErrorMessage

# Error or warning: There was a VTK Error in file: /viz/home/stephens/Kitware/Paraview/paraview-CVS/ParaView/VTK/IO/vtkXMLStructuredDataReader.cxx (309)

 vtkXMLRectilinearGridReader (190a6598): Error reading extent 0 1824 0 2080 0 373 from piece 0 (mms comment "Note the sizes of the grid" and they're floats)

ErrorMessage end (mms comment: the raw data is 4-byte floats)

ErrorMessage

# Error or warning: There was a VTK Error in file: /viz/home/stephens/Kitware/Paraview/paraview-CVS/ParaView/VTK/IO/vtkXMLDataReader.cxx (475)

 vtkXMLRectilinearGridReader (190a6598): Cannot read point data array "xEnergy_aaa" from PointData in piece 0. The data array in the element may be too short.

ErrorMessage end

-----------------------------------------------------------------

I’m working on an sgi.

 

I am linking against a 64-bit version of vtk and built 64-bit applications (paraview, etc);

64 bit Ids as well) but if vtk uses regular int and not long int for sizes of things

then there’s a 4-byte (32 bit limitation)

then there is a limit on file/array stuff.

 

that last bit was me rambling to myself but please check my thinking here.

 

let me know.

 

-m

 

ps: I’ve been told that out support contract issues have been resolved and we are back in kitware’s good graces.
Steps To Reproduce
Additional Information
TagsNo tags attached.
Relationships
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2011-06-16 13:11Zack GalbreathCategory => (No Category)
2012-05-16 16:22Utkarsh AyachitStatusexpired => backlog
2012-05-16 16:23Utkarsh AyachitStatusbacklog => todo
2012-05-16 16:23Utkarsh AyachitResolutionreopened => fixed
2012-05-16 16:23Utkarsh AyachitStatustodo => active development
2012-05-16 16:23Utkarsh AyachitStatusactive development => gatekeeper review
2012-05-16 16:23Utkarsh AyachitStatusgatekeeper review => customer review
2012-05-16 16:25Brad KingNote Added: 0028614
2012-05-16 16:26Brad KingNote Edited: 0028614bug_revision_view_page.php?bugnote_id=28614#r444
2012-05-16 16:27Brad KingNote Added: 0028615
2013-04-05 20:34Berk GeveciStatuscustomer review => closed
2015-01-09 13:48Utkarsh AyachitSource_changeset_attached => VTK master 9dcb01d6
2015-01-09 13:48Utkarsh AyachitSource_changeset_attached => VTK master 9b1a0f4f
2015-01-09 13:49Utkarsh AyachitSource_changeset_attached => VTK master 444257e5

Notes
(0003154)
Brad King   
2005-10-17 12:51   
I'm assinging this bug to myself because I asked Mike to submit it.
(0003155)
Brad King   
2005-10-17 12:55   
The problem is that the vtkXMLDataReader::ReadData method and the vtkXMLDataParser methods it calls were not dealing with file sizes using a big enough integer type.

The following changes should at least partially address this problem:

/cvsroot/VTK/VTK/IO/vtkXMLDataParser.cxx,v <-- vtkXMLDataParser.cxx
new revision: 1.30; previous revision: 1.29
/cvsroot/VTK/VTK/IO/vtkXMLDataParser.h,v <-- vtkXMLDataParser.h
new revision: 1.14; previous revision: 1.13
/cvsroot/VTK/VTK/IO/vtkXMLDataReader.cxx,v <-- vtkXMLDataReader.cxx
new revision: 1.28; previous revision: 1.27
/cvsroot/VTK/VTK/IO/vtkXMLDataReader.h,v <-- vtkXMLDataReader.h
new revision: 1.11; previous revision: 1.10

Under the assumption that this is the only problem with reading large data on a 64-bit platform with 64-bit ids I'm marking the bug as Resolved/Fixed. I'm not able to test it on such a large file myself though. Please test it and change the status to Verified if it works. Otherwise please report the new problem. Thanks.
(0003159)
Brad King   
2005-10-18 10:25   
More changes are needed to avoid compiler warnings:

/cvsroot/VTK/VTK/IO/vtkXMLDataParser.cxx,v <-- vtkXMLDataParser.cxx
new revision: 1.31; previous revision: 1.30
(0003228)
Brad King   
2005-11-11 11:10   
It looks like there are problems with writing large files too. I'm changing the bug summary accordingly. I'm also changing the bug status to Reopened/None until the writers can be fixed too. I'm also changing the VTK version number to 5.0.0 because this will not be fixed in the 4.x branches.
(0003229)
Brad King   
2005-11-11 11:16   
The bug in writing big files is similar to the reading problem. The internal vtkXMLWriter::WriteBinaryData methods that actually send the data somewhere are using "int" for the number of words to write. I'm changing these methods and the related infrastructure to use an OffsetType that can be set to a large integer type.

A bigger problem is that the encoding of the binary data blocks in the VTK XML format itself stores the length of the data using a 32-bit unsigned integer type. This fundamentally limits the size of the data that can be stored in each block (DataArray) even if the total file size is larger. The HeaderType defined by vtkXMLWriter and vtkXMLDataParser determines the integer type stored in the files. Changing it to a 64-bit type will solve the problem but then older readers will not be able to handle the files. We will need to store in a VTKFile XML attribute the size of the HeaderType and support reading/writing both. That way new files that do not need large data can be written in a way that can be read by VTK 4.x readers.
(0003230)
Brad King   
2005-11-11 11:33   
I have made the following changes to convert file positions and offsets to use OffsetType:

/cvsroot/VTK/VTK/IO/vtkOffsetsManagerArray.h,v <-- vtkOffsetsManagerArray.h
new revision: 1.6; previous revision: 1.5
/cvsroot/VTK/VTK/IO/vtkXMLWriter.cxx,v <-- vtkXMLWriter.cxx
new revision: 1.60; previous revision: 1.59
/cvsroot/VTK/VTK/IO/vtkXMLWriter.h,v <-- vtkXMLWriter.h
new revision: 1.23; previous revision: 1.22

This does not yet address the 32-bit HeaderType problem.
(0003255)
David Cole   
2005-11-17 14:14   
The first set of changes listed in the "Date: 12:55 PM 10-17-2005" comment have been merged into the VTK-5-0 branch. Here are the cvs commands executed in the VTK-5-0 branch and the resulting file revisions:

cvs -q up -j1.13 -j1.14 IO/vtkXMLDataParser.h
cvs commit -m "ENH: Merge changes from main tree into VTK-5-0 branch. (cvs -q up -j1.13 -j1.14 IO/vtkXMLDataParser.h)" IO/vtkXMLDataParser.h
  /cvsroot/VTK/VTK/IO/vtkXMLDataParser.h,v <-- vtkXMLDataParser.h
  new revision: 1.12.28.1; previous revision: 1.12

cvs -q up -j1.27 -j1.28 IO/vtkXMLDataReader.cxx
cvs commit -m "ENH: Merge changes from main tree into VTK-5-0 branch. (cvs -q up -j1.27 -j1.28 IO/vtkXMLDataReader.cxx)" IO/vtkXMLDataReader.cxx
  /cvsroot/VTK/VTK/IO/vtkXMLDataReader.cxx,v <-- vtkXMLDataReader.cxx
  new revision: 1.25.6.1; previous revision: 1.25

cvs -q up -j1.10 -j1.11 IO/vtkXMLDataReader.h
cvs commit -m "ENH: Merge changes from main tree into VTK-5-0 branch. (cvs -q up -j1.10 -j1.11 IO/vtkXMLDataReader.h)" IO/vtkXMLDataReader.h
  /cvsroot/VTK/VTK/IO/vtkXMLDataReader.h,v <-- vtkXMLDataReader.h
  new revision: 1.9.6.1; previous revision: 1.9

cvs -q up -j1.29 -j1.31 IO/vtkXMLDataParser.cxx
cvs commit -m "ENH: Merge changes from main tree into VTK-5-0 branch. (cvs -q up -j1.29 -j1.31 IO/vtkXMLDataParser.cxx)" IO/vtkXMLDataParser.cxx
  /cvsroot/VTK/VTK/IO/vtkXMLDataParser.cxx,v <-- vtkXMLDataParser.cxx
  new revision: 1.29.6.1; previous revision: 1.29
(0003359)
Brad King   
2005-12-05 16:12   
Bug 0002561 may also be related to the problems with file sizes requiring 64-bit offsets.
(0028614)
Brad King   
2012-05-16 16:25   
(edited on: 2012-05-16 16:26)
I've finished the changes needed to support 64-bit sizes. The main changes are:

 vtkXML*: Optionally support 64-bit binary data headers
 http://vtk.org/gitweb?p=VTK.git;a=commitdiff;h=b6fbf0ca [^]

 vtkXML*: Use 64-bit integer types where appropriate
 http://vtk.org/gitweb?p=VTK.git;a=commitdiff;h=7fcb2cda [^]

but depend on some code cleanup:

 Remove compatibility with non-standard C++ streams
 http://vtk.org/gitweb?p=VTK.git;a=commitdiff;h=3279f44d [^]

 vtkXMLDataElement: Add 64-bit scalar/vector attribute API
 http://vtk.org/gitweb?p=VTK.git;a=commitdiff;h=b0a1c940 [^]

 vtkXMLDataParser: Remove unused FindInlineDataPosition method
 http://vtk.org/gitweb?p=VTK.git;a=commitdiff;h=0c3a868c [^]

 Rename IO/Core/vtkOffsetsManagerArray.h to IO/XML/vtkXMLOffsetsManager.h
 http://vtk.org/gitweb?p=VTK.git;a=commitdiff;h=090a6ca2 [^]

and some lower-level API integer type cleanup:

 vtk(|ZLib)DataCompressor: Use size_t for buffer length arguments
 http://vtk.org/gitweb?p=VTK.git;a=commitdiff;h=0ae2c1ed [^]

 vtkXML(DataParser|Writer): Use size_t in PerformByteSwap
 http://vtk.org/gitweb?p=VTK.git;a=commitdiff;h=7db71e5c [^]

 vtkByteSwap: Use size_t for buffer length arguments
 http://vtk.org/gitweb?p=VTK.git;a=commitdiff;h=51335bc0 [^]

 vtk(Input|Output)Stream: Use appropriate types in API
 http://vtk.org/gitweb?p=VTK.git;a=commitdiff;h=6d2b9838 [^]

VTK's Git master branch now contains everything needed:

 http://vtk.org/gitweb?p=VTK.git;a=commit;h=9b9a769e [^]
(0028615)
Brad King   
2012-05-16 16:27   
I've updated our XML format Wiki page:

 http://www.vtk.org/Wiki/VTK_XML_Formats [^]

The new format is incompatible with old VTK readers prior to these changes because they do not know how to load the 64-bit header values. The major file format version number must be incremented for files using such sizes. In order to avoid creating incompatible files and disrupting everyone whether they care about this or not, the new format must be enabled explicitly:

 writer->SetHeaderTypeToUInt64()
 // OR
 writer->SetHeaderType(vtkXMLWriter::UInt64)
 // OR
 writer->SetHeaderType(64)