[Paraview] Parallel file formats (again)... (UNCLASSIFIED)

Renato N. Elias rnelias at nacad.ufrj.br
Fri Jun 13 16:11:55 EDT 2008


Hi Jerry,

As I described in my last email, I managed to compile and link the 
Fortran example supplied with the XDMF library (in fact, I changed the 
example a little bit writing a Fortran interface explicitly to call the 
extern "C" routine name). It gave me 10 XDMF and 10 HDF5 files. Each 
XDMF pointing to a corresponding HDF5. Ok, everything easy and 
running... now, I'm guessing, from a naive approach, that I could just 
follow the same example for a parallel running where each process, 
having its own piece of the domain, would write its data in pairs of 
files XDMF/HDF5. Thus, for 2 processes, I'd have something like:

Iteration 1
Demo_00001.00.xmf  --> Demo_00001.00.h5
Demo_00001.01.xmf --> Demo_00001.01.h5

Iteration 2
Demo_00002.00.xmf  --> Demo_00002.00.h5
Demo_00002.01.xmf --> Demo_00002.01.h5
...
and so on

Maybe, all 20 XDMF files in this example could be replaced by just one 
containing all iterations and processes information, but, I haven't 
found it explained on XDMF's wiki.

Do you have a simple example of parallel XDMF file? Moreover, Is 
ParaView able to load transient parallel files in XDMF?

thanks

Renato.


Clarke, Jerry (Civ, ARL/CISD) wrote:
> Classification:  UNCLASSIFIED 
> Caveats: NONE
>
> Renato,
>
> www.arl.hpc.mil/ice doesn't exist anymore.
>
> Let me know what you are trying to do and I can provide an example. 
>
> If you're trying to work in parallel, you probably want to use
> <Grid GridType="Collection" CollectionType="Spatial" ...
> 	<Grid Name="Grid from node 0"
> 	<Grid Name="Grid from node 1"
> 	etc.
> </Grid>
>
> For the non 0 started arrays, there is an Offset="1" option in the XML
>
> As for the row major / column major arrays, I link C++ to Fortran, but
> we have been talking about putting a transpose method in the DataItem
> object.
>
>
> Jerry Clarke
>
> -----Original Message-----
> From: paraview-bounces at paraview.org
> [mailto:paraview-bounces at paraview.org] On Behalf Of Renato N. Elias
> Sent: Friday, June 13, 2008 1:55 PM
> To: Dominik Szczerba
> Cc: paraview at paraview.org
> Subject: Re: [Paraview] Parallel file formats (again)...
>
>
> It seems that XDMF is the unique way to go if we'd like to get parallel
> data into ParaView using HDF5. No problem since XDMF can be seen as a
> HDF5 extension. In fact, I thought about doing exactly what you cited --
> use the HDF5 API and write simple XML/XDMF files using Fortran, no
> matter if my data is heavy or light.
>
> The problem with XDMF is the lack of information. The first link pointed
> by Google is always down (www.arl.hpc.mil/ice/) and the "official" wiki
> site doesn't offer so much. I already got something on doing Fortran
> talk with XDMF (we can also write interfaces from the Fortran side to
> talk with C-like routine names) but, now, I'd like to go a bit further
> and use parallelism but there's no examples covering the subject using
> Fortran. My chance is debugging C++ examples and try to make some
> correlation.
>
> Regarding row-major order and non 0 started arrays I can't say anything.
>
> I only say that, for Fortran programmers, it's getting a bit harder to
> work without having to deal with C++ and all that OOP stuffs. In this
> sense, I love Metis, so powerful, so easy, so fast, so simple and
> everything written in C ANSI. Just minor efforts to get it working with
> Fortran. As we say in Brazil, sometimes people like to kill cockroaches
> using bazookas instead of flip-flops... for writing files*my guess* is
> that straight C would do the job nicely
>
> Dominik, the problem with Fortran is that everybody associates it with
> 77 (just that old programming language). Maybe, they should change the
> name of the language from Fortran 2003 to F++ ;oP
>
> Renato.
>
> Dominik Szczerba wrote:
>   
>> And how would he handle hard-coded row-major ordering in XDMF?
>> -- Dominik
>>
>> Chris Kees wrote:
>>     
>>> You might want to reconsider XDMF or something based on it. I'm not 
>>> sure that XDMF is significantly harder to implement in fortran than 
>>> straight HDF5. It's just a matter  of doing some additional text i/o 
>>> on a relatively simple XML file. XDMF splits the data (with some
>>> redundancy) into light/meta data stored as simple XML (ascii) file 
>>> and an HDF5 archive of the "heavy" data.  You can read and write the 
>>> XML file directly from fortran without using the XDMF library and
>>> then use the HDF5 fortran API directly to write the heavy data.   You
>>>       
>
>   
>>> have the option of storing the heavy data in the XML file as text 
>>> when HDF5 isn't available (or when debugging/running on small
>>> data).   To me it looks like the posts you cite are pointing in this
>>>       
>
>   
>>> direction though they were unhappy with some aspects of XDMF.  It's 
>>> not clear to me whether it's the XDMF xml format, the documentation 
>>> of that format, or the C API that needs work in order to make it more
>>>       
>
>   
>>> useful.
>>> Also, it sounds like you've already decided against a mixed language 
>>> approach, but the the book by H. P. Langtangen "Python Scripting for 
>>> Computational Science" advocates a fortran/python pairing to deal 
>>> with some of your  general concerns.
>>> Chris
>>> *  *
>>> On Jun 13, 2008, at 7:58 AM, Renato N. Elias wrote:
>>>
>>>       
>>>> Can anyone shed some light above how is the support status for 
>>>> parallel file formats in ParaView?
>>>>
>>>> In my lab most of the students still work with Fortran. It seems 
>>>> that "the universe nowadays only speaks C++ (and Python for 
>>>> scripting)" which force us to do an extensive evaluation for a good 
>>>> and well supported parallel file format to invest before struggling 
>>>> with all that mixed languages interface/wrapping annoyances (not 
>>>> everybody working with programs are programmers, there's still some 
>>>> engineers like civil, mechanical, chemical, etc... doing
>>>>         
> science...).
>   
>>>> I could say that our my concerns about choosing a file format to 
>>>> sticky with is:
>>>>
>>>> -- Easiness for installation and use (in this sense, Ensight is 
>>>> wonderful since we don't need extra libraries. It's insane when we 
>>>> need to compile 50 MB of libraries to link with a 2 MB program that 
>>>> uses just one routine of such library);
>>>> -- Easiness for interfacing (most of the libraries nowadays is 
>>>> written in C++ for C++ programmers which discourage its use by C and
>>>>         
>
>   
>>>> Fortran programs. Ok, we can always spend some time in interfacing 
>>>> it, but, a library should offer more functionality and flexibility 
>>>> than annoyances)
>>>> -- Portability.
>>>>
>>>> Some time ago there was some interesting posts from Jean Favre and 
>>>> Dominic about this, which give us some overview about the subject.
>>>>
>>>> http://www.paraview.org/pipermail/paraview/2008-May/008070.html
>>>> http://www.paraview.org/pipermail/paraview/2008-May/008071.html
>>>>
>>>> My 2 cents for the discussion, *from a Fortran perspective*, is:
>>>>
>>>> 1). ENSIGHT:
>>>> 1.1. Quite simple to implement and use (no need for extra libraries 
>>>> and all that stuff. Just a few Fortran statements do the job); 1.2. 
>>>> Implicit support for transient data and parallelism; 1.3. Depending 
>>>> on the number of processes we might have a huge number of 
>>>> small/medium files since each point and cell data variable is stored
>>>>         
>
>   
>>>> in one file (sometimes it can be a serious problem); 1.4. Not 
>>>> compressed (too bad); 1.5. Not so well supported *as a parallel 
>>>> format* by ParaView yet.
>>>> After the change to deal (after PV 2.2.1) with multigroup datasets 
>>>> some functionalities were lost until reimplementation.
>>>> 1.6. Supported by ParaView, Visit and Ensight (of course)
>>>>
>>>> 2). XML/VTK:
>>>> 2.1. Almost impossible for a Fortran user to implement, so, we're 
>>>> forced to interface with VTK in order to write something; 2.2. Time 
>>>> series support has been introduced in some sense ;o) 2.3. It's a bit
>>>>         
>
>   
>>>> complicated to understand. Ok, it's XML and we should use it (and 
>>>> believe on it ;o) ) through some library, so, it's not supposed to 
>>>> "hand-implementation"; 2.4. Encoding/compression is supported (which
>>>>         
>
>   
>>>> is really good) 2.5. It should be the most well parallel file format
>>>>         
>
>   
>>>> supported by ParaView (after EXODUS, maybe) 2.6. Only supported by 
>>>> VTK based softwares (ParaView, Visit, MayaVi)
>>>>
>>>> 3). XDMF/HDF5:
>>>> 3.1. Same as 2.1, 2.2 and 2.3
>>>> 3.2. The website describing the library is a bit down lately...
>>>> 3.3. HDF5 seems a very promising file format. It has some 
>>>> development concern about its use by other scientific languages 
>>>> besides being flexible, compressed, cross platform, etc... .
>>>> 3.4. From my knowledge, XDMF is supported by Ensight, ParaView and 
>>>> Visit also --> not sure about how good is that support.
>>>>
>>>> 4). EXODUS II:
>>>> 4.1. Same as 2.1 --> I already tried more than once to find 
>>>> something about Exodus format. There's a good documentation in 
>>>> SANDIA/SEACAS page but the library is not open source (it's a 
>>>> license based distribution) which turns it a bit complicated to 
>>>> adopt; 4.2. Nothing to say about timea nd compression support since 
>>>> I never used it; 4.3. It must be well supported by PV since it's a 
>>>> Sandia's format;
>>>>
>>>> regards
>>>>
>>>> Renato.
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> ParaView mailing list
>>>> ParaView at paraview.org
>>>> http://www.paraview.org/mailman/listinfo/paraview
>>>>         
>>> ---------------------------------------------------------------------
>>> ---
>>>
>>> _______________________________________________
>>> ParaView mailing list
>>> ParaView at paraview.org
>>> http://www.paraview.org/mailman/listinfo/paraview
>>>       
>
> _______________________________________________
> ParaView mailing list
> ParaView at paraview.org
> http://www.paraview.org/mailman/listinfo/paraview
> Classification:  UNCLASSIFIED 
> Caveats: NONE
>
>   



More information about the ParaView mailing list