[Paraview] cluster: works with -np 1, crashes when runs in parallel :-(

Fri Nov 4 11:53:20 EDT 2011

Thanks for helping me and so quickly!

I applied your patch and here is what  I get:

Waiting for client
Connection URL: cs://localhost:11111
Client connected.
0x1585fa0: 1/2
0x1585fa0: Controller: 0 (None)
0x1d98770: 0/2
0x1d98770: Controller: 0x1cb5d40 vtkSocketController
pvserver: /home/ucl/tfl/sbackaer/build/ParaView_dev/ParaView/ParaViewCore/ClientServerCore/vtkPVClientServerSynchronizedRenderers.cxx:76: virtual void vtkPVClientServerSynchronizedRenderers::SlaveEndRender(): Assertion `this->ParallelController->IsA("vtkSocketController")' failed.
[hmem07:32325] *** Process received signal ***
[hmem07:32325] Signal: Aborted (6)
[hmem07:32325] Signal code:  (-6)
pvserver: /home/ucl/tfl/sbackaer/build/ParaView_dev/ParaView/ParaViewCore/ClientServerCore/vtkPVClientServerSynchronizedRenderers.cxx:48: virtual void vtkPVClientServerSynchronizedRenderers::MasterEndRender(): Assertion `this->ParallelController->IsA("vtkSocketController")' failed.
[hmem07:32324] *** Process received signal ***
[hmem07:32324] Signal: Aborted (6)
[hmem07:32324] Signal code:  (-6)

I also tried with another version: 3.12.0-RC3 (sources come from the download page). 
Compiled with Release flag, the server launched with -np 1 works. With -np 2, it does not crash when the client connects to it but the client does not respond at all (client version 3.12.0-RC3 on OSX Lion)... I am recompiling this version with Debug to see if I get the same error message on the server side...

Stephane

On 4 nov. 2011, at 15:12, Utkarsh Ayachit wrote:

> Stephane,
> 
> Sebastien was referring to a different issue and that problem is not
> in 3.12-RC3, so we can ignore that response for the time being.
> 
> I tried to reproduce the problem with the same version as you're using
> (thanks for reporting the full version number, btw), and things seems
> to work fine. I've attached a patch that should print out additional
> debug information that will help us diagnose the problem. Do you mind
> applying the patch to the server-side and posting the output printed
> on pvserver?
> 
> You should get something like follows:
> 
> mpirun -np 2 ./bin/pvserver --use-offscreen-rendering
> Waiting for client
> Connection URL: cs://localhost:11111
> Client connected.
> 0x29be7c0: 0/2
> 0x29be7c0: Controller: 0x28ec0f0 vtkSocketController
> 0xf3a950: 1/2
> 0xf3a950: Controller: 0 (None)
> Exiting...
> Exiting...
> 
> Thanks
> Utkarsh
> 
> On Fri, Nov 4, 2011 at 9:01 AM, Sebastien Jourdain
> <sebastien.jourdain at kitware.com> wrote:
>> Hi Stephane,
>> 
>> Thanks for reporting it. In fact, that bug has been fixed in
>> ParaView/next and you can simply avoid it if you compile ParaView in
>> Release mode not in Debug.The problem come from an invalid test in an
>> assert().
>> I will make sure that the fix will be in 3.12.
>> 
>> Seb
>> 
>> On Fri, Nov 4, 2011 at 5:43 AM, Stéphane Backaert
>> <stephanebackaert at gmail.com> wrote:
>>> Hello,
>>> 
>>> I have installed paraview version 3.12.0-RC3-23-g712c45e on a cluster: compiled with intel compiler, openmpi-1.5.3, no hardware acc. so Mesa-7.9 and --use-offscreen-rendering flag at startup, MPI set to ON in ccmake.
>>> 
>>> I launch the server with mpirun -np x ./pvserver  --use-offscreen-rendering and, through an ssh connection, the client on my laptop (latest version too).  That works fine :-) ... but slow :-(.
>>> So I try to use more procs, for example, mpirun -np 2 ./pvserver  --use-offscreen-rendering. There are two processes running on the cluster (I see them with 'ps x' command).
>>> 
>>> My problem: when I connect my client to this server, the connection is etablished but the server crashed immediately with the message:
>>> 
>>> Waiting for client
>>> Connection URL: cs://localhost:11111
>>> Client connected.
>>> pvserver: /home/ucl/tfl/sbackaer/build/ParaView/ParaViewCore/ClientServerCore/vtkPVClientServerSynchronizedRenderers.cxx:76: virtual void vtkPVClientServerSynchronizedRenderers::SlaveEndRender(): Assertion `this->ParallelController->IsA("vtkSocketController")' failed.
>>> [hmem00:08278] *** Process received signal ***
>>> [hmem00:08278] Signal: Aborted (6)
>>> [hmem00:08278] Signal code:  (-6)
>>> pvserver: /home/ucl/tfl/sbackaer/build/ParaView/ParaViewCore/ClientServerCore/vtkPVClientServerSynchronizedRenderers.cxx:48: virtual void vtkPVClientServerSynchronizedRenderers::MasterEndRender(): Assertion `this->ParallelController->IsA("vtkSocketController")' failed.
>>> [hmem00:08277] *** Process received signal ***
>>> [hmem00:08277] Signal: Aborted (6)
>>> [hmem00:08277] Signal code:  (-6)
>>> hmem00:08278] [ 0] /lib64/libpthread.so.0() [0x3f8a40f4c0]
>>> [hmem00:08278] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x3f89c329a5]
>>> [hmem00:08278] [ 2] /lib64/libc.so.6(abort+0x175) [0x3f89c34185]
>>> [hmem00:08278] [ 3] /lib64/libc.so.6(__assert_fail+0xf5) [0x3f89c2b935]
>>> [hmem00:08278] [ 4] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkPVClientServerCore.so(_ZN38vtkPVClientServerSynchronizedRenderers14SlaveEndRenderEv+0x56) [0x7fa499c32d6e]
>>> [hmem00:08278] [ 5] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkParallel.so.pv3.12(_ZN24vtkSynchronizedRenderers15HandleEndRenderEv+0xfe) [0x7fa493cc1e4c]
>>> [hmem00:08278] [ 6] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkParallel.so.pv3.12(_ZN24vtkSynchronizedRenderers15HandleEndRenderEv+0x72) [0x7fa493cc1dc0]
>>> [hmem00:08278] [ 7] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkParallel.so.pv3.12(+0x26c470) [0x7fa493cc4470]
>>> [hmem00:08278] [ 8] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkCommon.so.pv3.12(+0x26ae71) [0x7fa48f102e71]
>>> [hmem00:08278] [ 9] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkCommon.so.pv3.12(_ZN9vtkObject11InvokeEventEmPv+0x41) [0x7fa48f103381]
>>> [hmem00:08278] [10] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkRendering.so.pv3.12(_ZN11vtkRenderer6RenderEv+0xdcb) [0x7fa492c18267]
>>> [hmem00:08278] [11] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkRendering.so.pv3.12(_ZN21vtkRendererCollection6RenderEv+0xca) [0x7fa492c15fb4]
>>> [hmem00:08278] [12] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkRendering.so.pv3.12(_ZN15vtkRenderWindow14DoStereoRenderEv+0xee) [0x7fa492c2c84c]
>>> [hmem00:08278] [13] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkRendering.so.pv3.12(_ZN15vtkRenderWindow10DoFDRenderEv+0x54a) [0x7fa492c2c754]
>>> [hmem00:08278] [14] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkRendering.so.pv3.12(_ZN15vtkRenderWindow10DoAARenderEv+0x7c3) [0x7fa492c2c1ff]
>>> [hmem00:08278] [15] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkRendering.so.pv3.12(_ZN15vtkRenderWindow6RenderEv+0x868) [0x7fa492c2b7ca]
>>> [hmem00:08278] [16] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkPVClientServerCore.so(_ZN30vtkPVSynchronizedRenderWindows6RenderEj+0x95) [0x7fa499c9caed]
>>> [hmem00:08278] [17] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkPVClientServerCore.so(+0x1dab8e) [0x7fa499c9ab8e]
>>> [hmem00:08278] [18] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkParallel.so.pv3.12(_ZN25vtkMultiProcessController10ProcessRMIEiPvii+0x3cb) [0x7fa493bc99bd]
>>> [hmem00:08278] [19] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkParallel.so.pv3.12(_ZN25vtkMultiProcessController11ProcessRMIsEii+0x6a8) [0x7fa493bc957a]
>>> [hmem00:08278] [20] /home/ucl/tfl/sbackaer/build/ParaView-bin/bin/libvtkParallel.so.pv3.12(_ZN25vtkMultiProcessController11ProcessRMIsEv+0x22) [0x7fa493bc8ed0]
>>> [hmem00:08278] [21] ./bin/pvserver() [0x401a70]
>>> [hmem00:08278] [22] ./bin/pvserver(main+0x25) [0x401aef]
>>> [hmem00:08278] [23] /lib64/libc.so.6(__libc_start_main+0xfd) [0x3f89c1ec5d]
>>> [hmem00:08278] [24] ./bin/pvserver() [0x4017c9]
>>> [hmem00:08278] *** End of error message ***
>>> 
>>> 
>>> 
>>> I tried with different configuration options in ccmake related to MPI, no changes. The cluster works for other MPI programs...
>>> 
>>> Any idea?
>>> 
>>> Thanks!
>>> 
>>> Stephane
>>> 
>>> 
>>> _______________________________________________
>>> Powered by www.kitware.com
>>> 
>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>> 
>>> Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView
>>> 
>>> Follow this link to subscribe/unsubscribe:
>>> http://www.paraview.org/mailman/listinfo/paraview
>>> 
>> _______________________________________________
>> Powered by www.kitware.com
>> 
>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>> 
>> Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView
>> 
>> Follow this link to subscribe/unsubscribe:
>> http://www.paraview.org/mailman/listinfo/paraview
>> 
> <PrintDebugTxt.patch>