[Paraview] Error when reading large XMLUnstructuredData file

[Robert Maynard]

Hi,

This is a known issue when a VTK XML file has an appended data section
larger than 2GB(http://public.kitware.com/Bug/view.php?id=6938).

To work around the problem you can look at writing out the file as any of
the following:

   - Partitioned VTK XML
   - Ensight Gold Binary
   - Exodus II ( Will require you too link to the exodus library found at
   http://sourceforge.net/projects/exodusii/ )


On Tue, Jan 4, 2011 at 5:45 AM, Guenter Bartsch <
guenter.bartsch at googlemail.com> wrote:
>
> hi all,
>
> we have run into a problem reading a pretty large (3.6 GB) .vtu file.
> the file was produced by a fortran program that writes vtk data in the
> old ascii format which results in a 17G ascii file which takes very
> long to load in paraview (and results in an error message). so, we
> have written a converter program that reads the ascii file and outputs
> xml:
>
>
> int main(int argc, char *argp[])
> {
>  if (argc != 2) {
>    cerr << "usage: " << argp[0] << " <vtk filename>\n\n";
>    return 1;
>  }
>
>  char *filename = argp[1];
>
>  cout << "Input filename: " << filename << "\n";
>
>  vtkSmartPointer<vtkUnstructuredGridReader> reader =
> vtkSmartPointer<vtkUnstructuredGridReader>::New();
>  reader->SetFileName(filename);
>
>  char *ofilename = "foo.vtu";
>  cout << "Output filename: " << ofilename << "\n";
>  vtkSmartPointer<vtkXMLUnstructuredGridWriter> writer =
> vtkSmartPointer<vtkXMLUnstructuredGridWriter>::New();
>  writer->SetFileName(ofilename);
>  writer->SetInputConnection(reader->GetOutputPort());
>  writer->Write();
>
>  return EXIT_SUCCESS;
> }
>
> this program runs fine. now, if i open the resulting file in paraview,
> i run into this error as soon as i hit apply:
>
> ERROR: In
/opt/paraview-3.8.1/src/ParaView-3.8.1/VTK/IO/vtkXMLUnstructuredDataReader.cxx,
> line 652
> vtkXMLUnstructuredGridReader (0x143f0c40): Cannot read cell
> connectivity from Cells in piece 0 because the "connectivity" array is
> not long enough.
>
> the converter did work on a smaller test data set and paraview could
> open it - only now with the large data set paraview fails. here is the
> header of the failing file:
>
> <?xml version="1.0"?>
> <VTKFile type="UnstructuredGrid" version="0.1"
> byte_order="LittleEndian" compressor="vtkZLibDataCompressor">
>  <UnstructuredGrid>
>    <Piece NumberOfPoints="106496000"
> NumberOfCells="105122745"           >
>      <PointData Vectors="vectors">
>        <DataArray type="Float32" Name="vectors"
> NumberOfComponents="3" format="appended" RangeMin="0.0021820231521"
>  RangeMax="24.459394872"         offset="0"                   />
>      </PointData>
>      <CellData>
>      </CellData>
>      <Points>
>        <DataArray type="Float32" Name="Points" NumberOfComponents="3"
> format="appended" RangeMin="0"
> RangeMax="5007.1818403"         offset="1580185620"          />
>      </Points>
>      <Cells>
>        <DataArray type="Int64" Name="connectivity" format="appended"
> RangeMin=""                     RangeMax=""
> offset="2045521660"          />
>        <DataArray type="Int64" Name="offsets" format="appended"
> RangeMin=""                     RangeMax=""
> offset="3504937336"          />
>        <DataArray type="UInt8" Name="types" format="appended"
> RangeMin=""                     RangeMax=""
> offset="3671971228"          />
>      </Cells>
>    </Piece>
>  </UnstructuredGrid>
>  <AppendedData encoding="base64">
>   _WJgAAACAAAAAAAAAvXYAALF2AADJdgAAUHcAA...
>
> does anyone know what could be going on here? do we need to configure
> paraview to handle larger files? is this a bug? could the original
> ascii data set have been corrupt and the error reflects now in the xml
> file?
>
> thanks in advance and best regards,
>
>   guenter
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:
http://paraview.org/Wiki/ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://www.paraview.org/mailman/listinfo/paraview



--
Robert Maynard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.paraview.org/pipermail/paraview/attachments/20110104/d84b54b0/attachment.htm>