[Paraview] Problems with Paraview 3.6.1 on cluster (client - server mode)
myaccountmail9 myaccountmail9
myaccountmail9 at gmail.com
Thu Dec 10 04:13:48 EST 2009
Hi Ken,
can you tell us the distribution and Linux version that you use?
Thank you
2009/12/7 Moreland, Kenneth <kmorel at sandia.gov>
> Luis,
>
> I saw your email, but I have been on travel and don’t have a quick answer
> for your problem, so I have not had a chance to respond. This is why it is
> always a good idea to cc the ParaView mailing list.
>
> Offhand I do not know what the problem might be. The first thing I would
> check would be to make sure that none of the processes has crashed. You are
> loading a lot of data, so if it is not partitioned in a way that allows it
> to be load balanced well you could be running out of memory. If you find
> that all the processes are doing something, then it would be helpful to know
> exactly where they are spending a lot of time.
>
> -Ken
>
>
>
> On 12/7/09 12:47 AM, "myaccountmail9 myaccountmail9" <
> myaccountmail9 at gmail.com> wrote:
>
> Hi Ken, did you received my last e-mail (on 1st december)? I write you
> again...
>
>
> It seems that we have finally resolved the problem regarding OSMesa.
>
> Now can you help us to configure paraview options to improve the
> performances on our cluster (8 nodes dual quad core -> 8 core per node = 64
> cores; no GPU; 10 GB ram per node)?
>
> If we run pvserver vith 64 processes (8 processes per node with: mpirun -np
> 64 -machinefile list pvserver --use-offscreen-rendering)
> we have problems to load and/or execute operations with big models (20*10^6
> cells) exported from CFD++ in Ensight binary mode (it seems that Paraview
> loop endless).
>
> Our client configuration is (version 3.6.1):
>
> edit -> settings -> render view:
> - use immediate mode rendering: checked
> - use triangle strips: unchecked
> - LOD threshold: unchecked
> - outline threshold: 500 MCells
> - allow rendering interrups: unchecked
> - enable deept pelling: checked
> - number of peels: 4
> - use offscreen rendering for screenshot: checked
>
> edit -> settings -> server:
> - remote rendering threshold: checked, 0 MBytes
> - suppress ordered compositing: checked
> - subsample rate: checked, 4 pixels
> - squirt compression: checked, 10Bits
> - still subsample rate: checked, 4 pixels
> - client collect: checked, 97 MBytes
> - compositing threshold: unchecked
>
>
> Can you help us?
> Thanks again
> Luis
>
>
> 2009/11/10 Moreland, Kenneth <kmorel at sandia.gov>
>
> If you are using OSMesa, you should not need X running on any of the nodes.
> It should be sufficient to have the OSMesa libraries and associated Mesa GL
> libraries installed on a shared folder. However, make sure that folder is
> actually mounted everywhere (a common failure in my experience) and that
> they are the ones actually loaded for each node.
>
> -Ken
>
>
>
> On 11/10/09 2:31 AM, "myaccountmail9 myaccountmail9" <
> myaccountmail9 at gmail.com <http://myaccountmail9@gmail.com> > wrote:
>
> I'm sorry, I explained the problem very bad; we are using OSMesa library.
>
> If we run pvserver only on the master node:
>
> mpirun -np 2 pvserver --use-offscreen-rendering
>
> It's all ok (even if DISPLAY variable is not set: we don't know if it's
> necessary).
>
>
> If we try to use 2 nodes (master and node1), running this command on the
> master node:
>
> mpirun -x
> LD_LIBRARY_PATH=/home/user/ParaView/ParaView-bin/bin/:/usr/lib/:/home/user/ParaView/ComponentiAggiuntivi/lib
> -np 2 -machinefile /home/user/ParaView/NodesList
> /home/user/ParaView/ParaView-bin/bin/pvserver --use-offscreen-rendering
>
> the error is:
>
> Listen on port: 11111
> Waiting for client...
> Client connected.
> [node1:06211] *** Process received signal ***
> [node1:06211] Signal: Segmentation fault (11)
> [node1:06211] Signal code: Address not mapped (1)
> [node1:06211] Failing at address: 0x21
> mpirun noticed that job rank 0 with PID 22632 on node master exited on
> signal 15 (Terminated).
> 1 additional process aborted (not shown)
>
>
> X is running on the second node (called node1); DISPLAY is not set neither
> on node master nor in the second node.
> Must X server run on each node and must DISPLAY variable to be set also if
> we are using OSMesa library?
>
> Must we install OSMesa on each node or it's enough to have installed that
> library on a shared folder (/home/user/ParaView/Components/lib) in the
> master node?
>
> Have you any other idea?
>
> Thank you very much for your patience
> Luis
>
>
> 2009/11/9 Moreland, Kenneth <kmorel at sandia.gov <http://kmorel@sandia.gov>
> >
>
> By “usual error” do you mean the crash in the GL or pthreads library? It
> could be that the environment in the non-master nodes is incorrect.
>
> -Ken
>
>
>
> On 11/6/09 9:26 AM, "myaccountmail9 myaccountmail9" <
> myaccountmail9 at gmail.com <http://myaccountmail9@gmail.com> <
> http://myaccountmail9@gmail.com> > wrote:
>
> Hi ken, thanks again.
>
> we recompiled again OSMesa library (only on the master node of our cluster;
> the libraries file are generated on a shared folder -
> /home/user/ParaView/Components/lib - visible on each node), then we
> recompiled Paraview with these libraries (and setting shared libraries=off
> in ccmake menu); now we are able to connect the client with pvserver
> provided that pvserver runs only on the master node.
>
> If we run pvserver on two or more nodes with the command:
>
> mpirun -x LD_LIBRARY_PATH=/home/user/ParaView/Components/lib -np 2
> -machinefile NodesList pvserver --use-offscreen-rendering
>
> we always get the usual error when we try to connect the client with the
> server
>
> Must X server run on each node? Currently X server is not running on any
> node (if we execute "pidof X" we obtain nothing) and DISPLAY variable is not
> set, but we can connect the client with the master node...
>
> Have you any idea? Can you help us?
>
> Thank you very much
> Luis
>
>
>
> 2009/10/29 Moreland, Kenneth <kmorel at sandia.gov <http://kmorel@sandia.gov>
> <http://kmorel@sandia.gov> >
>
> (Responding back to the ParaView list.)
>
> These symbols should be defined in the GL library that is part of Mesa.
> You should have a library like
> /home/myuser/TempOSMesa/Mesa-7.6/lib/libGL.so. That is where the symbols
> should be defined. libOSMesa.so simply references those libraries. It is
> because the symbols cannot be found in the GL library that you are getting
> the error. You can use the ‘nm’ command to query what symbols are defined
> are what are required.
>
> -Ken
>
>
>
> On 10/28/09 2:16 AM, "myaccountmail9 myaccountmail9" <
> myaccountmail9 at gmail.com <http://myaccountmail9@gmail.com> <
> http://myaccountmail9@gmail.com> <http://myaccountmail9@gmail.com> >
> wrote:
>
> It sounds like you are still linking to the system OpenGL libraries
> somehow. Those symbols should be defined in the libGL.so that is built with
> Mesa (whereas the /usr/lib/libGL.so on your system probably does not have
> them).
>
>
>
> Hi Ken,
> thanks again.
>
> Looking the errors such as:
>
> "home/myuser/TempOSMesa/Mesa-7.6/lib/libOSMesa.so: undefined reference to
> `_mesa_free_context_data'"
>
> I believe those symbols must be defined in libOSMesa.so library and not in
> /usr/lib/libGL.so library.
>
> In fact, if we install OSMessa using Debian packages we can compile
> ParaView, but we can't connect client and server; the error always is
> similar to:
>
> ParaView3/Servers/Common/vtkServerConnection.cxx, line 67
> vtkServerConnection (0x3182220): Server Connection Closed!
>
> Executing with: 0
> *** Process received signal ***
> Signal: Segmentation fault (11)
> Signal code: (128)
> Failing at address: (nil)
> [ 0] /lib/libpthread.so.0 [0x7f4fc6536a80]
> [ 1] /usr/lib/libGL.so.1(glBlendFuncSeparateEXT+0x16) [0x7f4fc7bc12e6]
> *** End of error message ***
> Segmentation fault
>
>
> If instead we compile OSMesa ourselves we can't compile ParaView: I believe
> that something went wrong compiling OSMesa (some simbols aren't present in
> libOSMesa.so created), but I don't know if compiling OSMesa ourselves can
> help us to resolve our problem.
>
> We also tried to install on our cluster ParaView 3.6.1already compiled
> (downloaded from http://www.paraview.org/paraview/resources/software.html)
> and it works: we can establish client-server connection.
>
> But if we try to run more 2 or more pvserver instances on the same node
> (mpirun -np 2 pvserver), we have this error:
>
> "vtkProcessModuleConnectionManager (0x23d6da0): Failed to set up server
> socket"
>
>
>
> Did you do a clean build of ParaView (that is, delete the binary directory
> and start over from scratch)? If you just tried to change some CMake
> variables in an existing build, you may have some stale libraries built.
>
>
> We did "make clean".
>
>
> You should also check to make sure that the executables being built are
> linking to your compiled Mesa libraries instead of the system libraries.
> You can check this with the ldd command. You may have to set your
> LD_LIBRARY_PATH environment variable to point to
> /home/myuser/TempOSMesa/Mesa-7.6.
>
>
> We also tried to set LD_LIBRARY_PATH.
>
> Can you help us?
> Luis
>
>
>
>
> -Ken
>
>
>
>
>
>
>
> On 10/23/09 3:27 AM, "myaccountmail9 myaccountmail9" <
> myaccountmail9 at gmail.com <http://myaccountmail9@gmail.com> <
> http://myaccountmail9@gmail.com> <http://myaccountmail9@gmail.com> <
> http://myaccountmail9@gmail.com> > wrote:
>
> Hi Ken,
>
> thank you.
> We compiled Mesa with:
>
> ./configure --enable-gl-osmesa --disable-gallium
> make
> make install
>
> and we configured Paraview with:
>
> OPENGL_INCLUDE_DIR=/home/myuser/TempOSMesa/Mesa-.6/include
> OPENGL_gl_LIBRARY=/home/myuser/TempOSMesa/Mesa-7.6/lib/libGL.so
> OPENGL_glu_LIBRARY=/home/myuser/TempOSMesa/Mesa-7.6/lib/libGLU.so
> OPENGL_xmesa_INCLUDE_DIR=/home/myuser/TempOSMesa/Mesa-7.6/include
> OSMESA_INCLUDE_DIR=/home/myuser/TempOSMesa/Mesa-7.6/include
>
> OSMESA_LIBRARY=/home/myuser/TempOSMesa/Mesa-7.6/lib/libOSMesa.so
>
> but we have many error compiling Paraview; for example:
>
> /home/myuser/TempOSMesa/Mesa-7.6/lib/libOSMesa.so: undefined reference to
> `_mesa_free_context_data'
> /home/myuser/TempOSMesa/Mesa-7.6/lib/libOSMesa.so: undefined reference to
> `_mesa_get_current_context'
> /home/myuser/TempOSMesa/Mesa-7.6/lib/libOSMesa.so: undefined reference to
> `_mesa_enable_2_1_extensions'
>
> Do you think there is something went wrong installing Mesa?
> Can you help us?
>
> Thank you again.
> Luis
>
>
>
>
> 2009/10/21 Moreland, Kenneth <kmorel at sandia.gov <http://kmorel@sandia.gov>
> <http://kmorel@sandia.gov> <http://kmorel@sandia.gov> <
> http://kmorel@sandia.gov> >
>
> Not really, but I would consider trying downloading the Mesa source and
> compiling it myself. Since the error is happening somewhere in the OpenGL
> libraries (and is not really being replicated on other user’s platforms),
> the only cause I can come up with is a problem with the libraries (and
> mismatched libraries is the most common problem). Compiling Mesa is
> actually pretty easy, so it is worth taking the effort to check.
>
> -Ken
>
>
>
> On 10/21/09 12:54 AM, "myaccountmail9 myaccountmail9" <
> myaccountmail9 at gmail.com <http://myaccountmail9@gmail.com> <
> http://myaccountmail9@gmail.com> <http://myaccountmail9@gmail.com> <
> http://myaccountmail9@gmail.com> <http://myaccountmail9@gmail.com> >
> wrote:
>
> Hi Ken,
> thank you for your reply.
> No, we didn't compile OSMesa ourselves, but we installed OSMesa library on
> each node downloading the .deb packages with:
>
> apt-get install libosmesa6
> apt-get install libosmesa6-dev
>
> Have you got any other idea?
> Thank you again.
> Luis
>
>
> 2009/10/20 Moreland, Kenneth <kmorel at sandia.gov <http://kmorel@sandia.gov>
> <http://kmorel@sandia.gov> <http://kmorel@sandia.gov> <
> http://kmorel@sandia.gov> <http://kmorel@sandia.gov> >
>
> Could it be a problem with mismatched libraries? I notice in the error
> below that it occurs in /usr/lib/libGL.so.1. This is a system library. Did
> you compile OSMesa yourself? If so, then you should not be using the system
> GL libraries; you should be using the libraries that you compiled when you
> also generated the OSMesa library.
>
> -Ken
>
>
>
> On 10/20/09 1:56 AM, "myaccountmail9 myaccountmail9" <
> myaccountmail9 at gmail.com <http://myaccountmail9@gmail.com> <
> http://myaccountmail9@gmail.com> <http://myaccountmail9@gmail.com> <
> http://myaccountmail9@gmail.com> <http://myaccountmail9@gmail.com> <
> http://myaccountmail9@gmail.com> > wrote:
>
> Hi,
> we have a Linux (Debian) cluster without gpu and we have a big problem
> using Paraview 3.6.1in client - server mode.
> The server has been compiled with OSMesa library and following the
> instructions reported here (in particular the section "OSMesa support"):
>
> http://www.itk.org/Wiki/Setting_up_a_ParaView_Server
>
> If we try to connect the client started with the command:
>
> paraview
>
> and the server, started with the command:
>
> pvserver --use-offscreen-rendering
>
> we must disable the checkbox "Remote rendering threshold" (but this causes
> bad performances) in the client settings (Edit -> Settings -> Render view ->
> Server), othervise Paraview crashes after loading a big input or performing
> some operation such us changing "outline" to "wireframe"...
>
> The error always is (similar to):
>
> ParaView3/Servers/Common/vtkServerConnection.cxx, line 67
> vtkServerConnection (0x3182220): Server Connection Closed!
>
> Executing with: 0
> *** Process received signal ***
> Signal: Segmentation fault (11)
> Signal code: (128)
> Failing at address: (nil)
> [ 0] /lib/libpthread.so.0 [0x7f4fc6536a80]
> [ 1] /usr/lib/libGL.so.1(glBlendFuncSeparateEXT+0x16) [0x7f4fc7bc12e6]
> *** End of error message ***
> Segmentation fault
>
>
> This error also occurs if we start client and server logging directly
> (without any ssh connection) on the master node; on master node X is running
> and we can execute the test:
>
> mpirun -np 4 /usr/X11R6/bin/glxgears
>
> So, it doen't seem a problem regarding X connection and permission...
>
> Moreover, executing the test:
>
> /ParaView/ParaView-bin$ ctest -I ,,10
>
> we obtain some segmentation fault, but the log file can't give any useful
> information.
>
> 111/475 Testing TestGradientBackground ***Exception: SegFault
> 121/475 Testing TestOSConeCxx ***Exception: SegFault
> 131/475 Testing TestTranslucentLUTDepthPeeling***Exception: SegFault
> 141/475 Testing BoxWidget ***Exception: SegFault
> 151/475 Testing TestParallelopipedWidget ***Exception: SegFault
>
> ..
>
> On our local machine with gpu 100% test passed, and we haven't any problem
> to connect client and server on that machine.
>
>
> MPI version is: mpirun (Open MPI) 1.2.7rc2
> OSMesa version is: 7.0.3-7
>
>
> We searched on google, on Paraview wiki... without any succes: can you help
> us please?
>
> If you need more information let us now
>
> Thank you very much
>
> Luis
>
>
>
>
>
>
>
>
> **** Kenneth Moreland
> *** Sandia National Laboratories
> ***********
> *** *** *** email: kmorel at sandia.gov <http://kmorel@sandia.gov> <
> http://kmorel@sandia.gov> <http://kmorel@sandia.gov> <
> http://kmorel@sandia.gov> <http://kmorel@sandia.gov> <
> http://kmorel@sandia.gov>
> ** *** ** phone: (505) 844-8919
> *** web: http://www.cs.unm.edu/~kmorel<http://www.cs.unm.edu/%7Ekmorel><
> http://www.cs.unm.edu/%7Ekmorel> <http://www.cs.unm.edu/%7Ekmorel> <
> http://www.cs.unm.edu/%7Ekmorel> <http://www.cs.unm.edu/%7Ekmorel> <
> http://www.cs.unm.edu/%7Ekmorel> <http://www.cs.unm.edu/%7Ekmorel>
>
>
>
>
>
> **** Kenneth Moreland
> *** Sandia National Laboratories
> ***********
> *** *** *** email: kmorel at sandia.gov <http://kmorel@sandia.gov> <
> http://kmorel@sandia.gov> <http://kmorel@sandia.gov> <
> http://kmorel@sandia.gov> <http://kmorel@sandia.gov>
> ** *** ** phone: (505) 844-8919
> *** web: http://www.cs.unm.edu/~kmorel<http://www.cs.unm.edu/%7Ekmorel><
> http://www.cs.unm.edu/%7Ekmorel> <http://www.cs.unm.edu/%7Ekmorel> <
> http://www.cs.unm.edu/%7Ekmorel> <http://www.cs.unm.edu/%7Ekmorel> <
> http://www.cs.unm.edu/%7Ekmorel>
>
>
>
>
>
> **** Kenneth Moreland
> *** Sandia National Laboratories
> ***********
> *** *** *** email: kmorel at sandia.gov <http://kmorel@sandia.gov> <
> http://kmorel@sandia.gov> <http://kmorel@sandia.gov> <
> http://kmorel@sandia.gov>
> ** *** ** phone: (505) 844-8919
> *** web: http://www.cs.unm.edu/~kmorel<http://www.cs.unm.edu/%7Ekmorel><
> http://www.cs.unm.edu/%7Ekmorel> <http://www.cs.unm.edu/%7Ekmorel> <
> http://www.cs.unm.edu/%7Ekmorel> <http://www.cs.unm.edu/%7Ekmorel>
>
>
>
>
>
> **** Kenneth Moreland
> *** Sandia National Laboratories
> ***********
> *** *** *** email: kmorel at sandia.gov <http://kmorel@sandia.gov> <
> http://kmorel@sandia.gov> <http://kmorel@sandia.gov>
> ** *** ** phone: (505) 844-8919
> *** web: http://www.cs.unm.edu/~kmorel<http://www.cs.unm.edu/%7Ekmorel><
> http://www.cs.unm.edu/%7Ekmorel> <http://www.cs.unm.edu/%7Ekmorel> <
> http://www.cs.unm.edu/%7Ekmorel>
>
>
>
>
>
> **** Kenneth Moreland
> *** Sandia National Laboratories
> ***********
> *** *** *** email: kmorel at sandia.gov <http://kmorel@sandia.gov> <
> http://kmorel@sandia.gov>
> ** *** ** phone: (505) 844-8919
> *** web: http://www.cs.unm.edu/~kmorel<http://www.cs.unm.edu/%7Ekmorel><
> http://www.cs.unm.edu/%7Ekmorel> <http://www.cs.unm.edu/%7Ekmorel>
>
>
>
>
>
> **** Kenneth Moreland
> *** Sandia National Laboratories
> ***********
> *** *** *** email: kmorel at sandia.gov <http://kmorel@sandia.gov>
> ** *** ** phone: (505) 844-8919
> *** web: http://www.cs.unm.edu/~kmorel<http://www.cs.unm.edu/%7Ekmorel><
> http://www.cs.unm.edu/%7Ekmorel>
>
>
>
>
>
> **** Kenneth Moreland
> *** Sandia National Laboratories
> ***********
> *** *** *** email: kmorel at sandia.gov
> ** *** ** phone: (505) 844-8919
> *** web: http://www.cs.unm.edu/~kmorel<http://www.cs.unm.edu/%7Ekmorel>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.paraview.org/pipermail/paraview/attachments/20091210/5172c0ca/attachment-0001.htm>
More information about the ParaView
mailing list