Reverse connection and port forwarding: Difference between revisions

From KitwarePublic
Jump to navigationJump to search
No edit summary
 
(24 intermediate revisions by the same user not shown)
Line 1: Line 1:
This page describes how to configure a ParaView client/server connection between a compute node cluster and your desktop when it is not possible to open a direct connection between compute node and desktop. There are a number of options to accomplish this task. The choice of which one is right given your situation depends on the configuration of both your local network and the cluster.  
This page describes how to use ssh to create a tunnel between hosts on two networks so that a ParaView client running on a host one of the networks can connect to a ParaView server running on a host on the other network, and vise-versa. These two scenarios are called forward and reverse connections respectively. Often each of the involved networks has a firewall blocking incoming traffic on all but a few ports. If one or both of the two networks are not guarded by a firewall, or specific ports on one of the firewalls are left open for ParaView's use, then a tunnel is not necessary. Even when one has the option to open ports on a firewall, it is advantageous to use an ssh tunnel for security purposes. When running the ParaView client and server processes on the same host or multiple hosts on the same network the tunnel is not necessary. The networking tool called ''portfwd'' is also discussed.


== Reverse connection over an ssh tunnel ==
= Reverse Connection =
[[Image:rc.png|frame|The ParaView server makes a reverse connection (blue arrow) through a server side firewall to a waiting ParaView client.]]
When only the server side firewall is preventing the connection no tunnel is required if ParaView's reverse connection mechanism is employed. In the reverse connection scenario the ParaView client blocks, listening on a port for an incoming connection request from the ParaView server. The ParaView client can automatically launch a ParaVeiw server and establish a reverse connection. This can be configured as follows:
# In the ParaView client from either the File menu or main toolbar open the Connect dialog.
# Add a server
## Configure the server
### Name: REMOTEHOST-XXXXX-rc
### Type: Client-Server (reverse connection)
### Port: XXXXX
## Configure the command
### Startup type: command
### Command: ssh -i /LOCAL/PATH/TO/SSH/PRIVATE/KEY REMOTEHOST /REMOTE/PATH/TO/pvserver --reverse-connection --server-port=XXXXX --client-host=LOCALHOST
You will replace the values of REMOTEHOST with the hostname of the host where the ParaView server will run, LOCALHOST with the hostname of the host where the ParaView client will run, XXXXX with the port number you wish to use, and the /LOCAL/PATH... and /REMOTE/PATH... entries with paths from the respective systems. This command assumes that you have configured ssh for password-less login. In the last step where we specified the start up command we started a single pvserver running on the login node. However, usually one will need to submit a batch job that will start a team of pvserver's via MPI. This is accomplished by modifying the command, one may substitute any other other executable command. For example the command would change to : ''ssh -i /LOCAL/PATH/TO/SSH/PRIVATE/KEY REMOTEHOST EXECUTABLE_COMMAND'' where EXECUTABLE_COMMAND is replaced by a batch submission command.
 
= Reverse Connection Over an ssh Tunnel=
[[Image:rc-tunnel.png|frame|ParaView reverse connection (blue arrow) over a reverse ssh tunnel (black arrow). ]]
[[Image:rc-tunnel.png|frame|ParaView reverse connection (blue arrow) over a reverse ssh tunnel (black arrow). ]]
When there are firewalls on both the client and server side, and the user has no control over either or it is undesirable to open ports on the client side firewall, ssh can be used to set up a tunnel between the two networks. The tunnel can be created such that ParaView can then establish either a reverse or a forward connection. We will only describe the configuration used for the reverse connection because creating the reverse connection can be automated by the ParaView client.


This method can be automated in ParaView or executed manually. In the automated approach the .pvsc calls ssh with both the tunnel options and the commands to submit the batch job. For example:
This can be configured as follows:
<pre>
# In the ParaView client from either the File menu or main toolbar open the Connect dialog.
ssh -R XXXX:localhost:YYYY remote_machine submit_my_job.sh
# Add a server
</pre>
## Configure the server
This means that port XXXX on remote_machine will be the port to which the server must connect. Port YYYY (e.g., 11111) on your client machine is the one on which PV listens. You'd have to tell the server (in the batch submission script, for example) the name of the node and port XXXX to which to connect. For example the ParaView server might be started like this:
### Name: REMOTEHOST-XXXXX-rc
<pre>
### Type: Client-Server (reverse connection)
mpiexec pvserver --reverse-connection --client-host=WWWW --server-port=XXXX
### Port: XXXXX
</pre>
## Configure the command
For more information on the .pvsc format see [[ParaView:Server Configuration|this]].
### Startup type: command
### Command: ssh -R YYYYY:localhost:XXXXX -i /LOCAL/PATH/TO/SSH/PRIVATE/KEY REMOTEHOST /REMOTE/PATH/TO/pvserver --reverse-connection --server-port=YYYYY --client-host=LOCALHOST


=== Caveats ===
You will replace the values of REMOTEHOST with the hostname of the host where the ParaView server will run, LOCALHOST with the hostname of the host where the ParaView client will run, XXXXX with the port number you wish to use on the client side and YYYYY with the port number you wish to use on the server side, and the /LOCAL/PATH... and /REMOTE/PATH... entries with paths from the respective systems. This command assumes that you have configured ssh for password-less login. In the last step where we specified the start up command we started a single pvserver running on the login node. However, usually one will need to submit a batch job that will start a team of pvserver's via MPI. This is accomplished by modifying the command, one may substitute any other other executable command. For example the command would change to : ''ssh -i /LOCAL/PATH/TO/SSH/PRIVATE/KEY REMOTEHOST EXECUTABLE_COMMAND'' where EXECUTABLE_COMMAND is replaced by a batch submission command.
# Two important options in the server side sshd configuration are ''GatewayPorts'' and ''AllowTcpForwarding''.
# A firewall between the client and server may also interfere.


== Forward or Reverse Connection Over A Two Hop ssh Tunnel ==
== Caveats ==
[[Image:Two-hop-tunnel.png|frame|ParaView forward connection over a two hop ssh tunnel (black arrow).]]
When a batch system is used to schedule resources and start the ParaView server, the ssh daemon must be configured for port forwarding. The relevant ssh daemon configuration options are ''AllowTcpForwarding'' and ''GatewayPorts''. The former defaults to ''yes'' while the latter defaults to ''no'', in which case the port forwarding only works for connections coming into the loop back device. This will prevent ParaView server running on compute nodes from connecting to the tunnel. One generally has a couple of options, the simplest of which is to set ''GatewayPorts=yes'' in the server side ''sshd_config'' files. If this is impossible a couple of work arounds exist. One is to use the portfwd utility. A second is to create a tunnel directly to the compute node. These options are discussed in detail in the following sections.


In the case there is an additional firewall in between the cluster's front end and compute nodes or the cluster sshd configuration prevents the above method from working one can make a two hop tunnel. This is slightly more complicated and requires two ssh sessions, and may not be automated. (If you know of a way please let us know!).
= A Forward Connection Over an ssh Tunnel =
[[Image:Two-hop-tunnel.png|frame|ParaView forward connection over a two hop ssh tunnel (black arrows).]]
In order for the previous method to work the server side ssh configuration must include ''GatewayPorts=yes''. If it does not then the reverse connection from the compute node will be prevented from connecting to the ssh tunnel. One work around is to create an ssh tunnel directly to the compute node on which the ParaView server will run. This is slightly more complicated and requires two ssh sessions, and is difficult to automate.


This will require two terminals, the first terminal is used to submit the batch job, once the job is scheduled, one manually gets the root node's hostname and uses the second terminal to establish the tunnel. Once the tunnel is established one launches ParaView server and proceeds as usual. A normal or reverse ssh tunnel may be used, however since this is a manual method the forward tunnel may be easier.
This will require two terminals(or putty instances), the first terminal is used to submit the batch job, once the job is scheduled, one manually gets the root node's hostname and uses the second terminal to establish the tunnel. Once the tunnel is established one launches ParaView server and proceeds as usual. A normal or reverse ssh tunnel may be used, however since this is a manual method the forward tunnel may be easier.


In the following denoted by t1$ and t2$, say fe is the front end on your cluster. In the first terminal:
In the following the two terminals are denoted by ''t1$'' and ''t2$'', and ''fe'' denotes the hostname of the cluster's front end (login node). In the first terminal you'll submit your batch job, for example:
<pre>
<pre>
t1$ ssh fe
t1$ ssh fe
t1$ qsub -I -V -l select=XX -l walltime=XX:XX:XX
t1$ qsub -I -V -l select=XX -l walltime=XX:XX:XX
qsub: waiting for job 498525.pbspl1 to start       
Job 498525.pbspl1 started on Mon Feb 08 12:51:25 PST 2010
On NODE:
NODE$
</pre>
</pre>
XX is replaced by your values. After the job is scheduled you're automatically ssh'd into some compute node, which we'll say has hostname NODE. In the second terminal:
where XX is replaced by your values. After the job is scheduled you're automatically ssh'd into some compute node, which has the hostname NODE. Now in the second terminal, you will create the ssh tunnel:
<pre>
<pre>
t2$ ssh -L ZZZZ:NODE:YYYY fe
t2$ ssh -L ZZZZ:NODE:YYYY fe
</pre>
</pre>
ZZZZZ is a port number on your workstation. YYYYY is a port number on the server that is not blocked by the clusters internal firewall (see your sys admin). Now back to terminal one, and your waiting compute node:
where ZZZZ is a port number on your workstation. YYYY is a port number on the server. Now switch back to the first terminal, and your waiting compute node and start the ParaView server. For example:
<pre>
<pre>
t1$ mpiexec pvserver --server-port=YYYYY
NODE$ mpiexec pvserver --server-port=YYYY
 
Listen on port: YYYY
Waiting for client...
</pre>
</pre>
At this point you may start a ParaView client on you workstation and proceed as normal by connecting to localhost port ZZZZ.
== Caveats ==
# The tunnel must established before the ParaView server is started.


==Reverse Connection with Portfwd==
= Reverse Connection with portfwd =
Using a reverse connection the paraview client will bind to a port and wait for an incoming connection from the server.  When the mpi enabled paraview server (referred to as '''pvserver''') is launched on a set of compute nodes, the 0th pvserver process opens a connection to the host machine where the paraview client is waiting.
Using a reverse connection the paraview client will bind to a port and wait for an incoming connection from the server.  When the mpi enabled paraview server (referred to as '''pvserver''') is launched on a set of compute nodes, the 0th pvserver process opens a connection to the host machine where the paraview client is waiting.


Line 72: Line 102:


Now when the pvserver connects to login_node:11111 the traffic will be forwarded to client_host:11111.
Now when the pvserver connects to login_node:11111 the traffic will be forwarded to client_host:11111.
= Reverse Connection over a Reverse ssh Tunnel with portfwd =
This is described [https://visualization.hpc.mil/wiki/ParaView_Client-Server_on_Crays here]
= Resources =
# [[ParaView:Server Configuration|Server Configuration]]
# [[Starting the server]]

Latest revision as of 17:54, 20 February 2012

This page describes how to use ssh to create a tunnel between hosts on two networks so that a ParaView client running on a host one of the networks can connect to a ParaView server running on a host on the other network, and vise-versa. These two scenarios are called forward and reverse connections respectively. Often each of the involved networks has a firewall blocking incoming traffic on all but a few ports. If one or both of the two networks are not guarded by a firewall, or specific ports on one of the firewalls are left open for ParaView's use, then a tunnel is not necessary. Even when one has the option to open ports on a firewall, it is advantageous to use an ssh tunnel for security purposes. When running the ParaView client and server processes on the same host or multiple hosts on the same network the tunnel is not necessary. The networking tool called portfwd is also discussed.

Reverse Connection

The ParaView server makes a reverse connection (blue arrow) through a server side firewall to a waiting ParaView client.

When only the server side firewall is preventing the connection no tunnel is required if ParaView's reverse connection mechanism is employed. In the reverse connection scenario the ParaView client blocks, listening on a port for an incoming connection request from the ParaView server. The ParaView client can automatically launch a ParaVeiw server and establish a reverse connection. This can be configured as follows:

  1. In the ParaView client from either the File menu or main toolbar open the Connect dialog.
  2. Add a server
    1. Configure the server
      1. Name: REMOTEHOST-XXXXX-rc
      2. Type: Client-Server (reverse connection)
      3. Port: XXXXX
    2. Configure the command
      1. Startup type: command
      2. Command: ssh -i /LOCAL/PATH/TO/SSH/PRIVATE/KEY REMOTEHOST /REMOTE/PATH/TO/pvserver --reverse-connection --server-port=XXXXX --client-host=LOCALHOST

You will replace the values of REMOTEHOST with the hostname of the host where the ParaView server will run, LOCALHOST with the hostname of the host where the ParaView client will run, XXXXX with the port number you wish to use, and the /LOCAL/PATH... and /REMOTE/PATH... entries with paths from the respective systems. This command assumes that you have configured ssh for password-less login. In the last step where we specified the start up command we started a single pvserver running on the login node. However, usually one will need to submit a batch job that will start a team of pvserver's via MPI. This is accomplished by modifying the command, one may substitute any other other executable command. For example the command would change to : ssh -i /LOCAL/PATH/TO/SSH/PRIVATE/KEY REMOTEHOST EXECUTABLE_COMMAND where EXECUTABLE_COMMAND is replaced by a batch submission command.

Reverse Connection Over an ssh Tunnel

ParaView reverse connection (blue arrow) over a reverse ssh tunnel (black arrow).

When there are firewalls on both the client and server side, and the user has no control over either or it is undesirable to open ports on the client side firewall, ssh can be used to set up a tunnel between the two networks. The tunnel can be created such that ParaView can then establish either a reverse or a forward connection. We will only describe the configuration used for the reverse connection because creating the reverse connection can be automated by the ParaView client.

This can be configured as follows:

  1. In the ParaView client from either the File menu or main toolbar open the Connect dialog.
  2. Add a server
    1. Configure the server
      1. Name: REMOTEHOST-XXXXX-rc
      2. Type: Client-Server (reverse connection)
      3. Port: XXXXX
    2. Configure the command
      1. Startup type: command
      2. Command: ssh -R YYYYY:localhost:XXXXX -i /LOCAL/PATH/TO/SSH/PRIVATE/KEY REMOTEHOST /REMOTE/PATH/TO/pvserver --reverse-connection --server-port=YYYYY --client-host=LOCALHOST

You will replace the values of REMOTEHOST with the hostname of the host where the ParaView server will run, LOCALHOST with the hostname of the host where the ParaView client will run, XXXXX with the port number you wish to use on the client side and YYYYY with the port number you wish to use on the server side, and the /LOCAL/PATH... and /REMOTE/PATH... entries with paths from the respective systems. This command assumes that you have configured ssh for password-less login. In the last step where we specified the start up command we started a single pvserver running on the login node. However, usually one will need to submit a batch job that will start a team of pvserver's via MPI. This is accomplished by modifying the command, one may substitute any other other executable command. For example the command would change to : ssh -i /LOCAL/PATH/TO/SSH/PRIVATE/KEY REMOTEHOST EXECUTABLE_COMMAND where EXECUTABLE_COMMAND is replaced by a batch submission command.

Caveats

When a batch system is used to schedule resources and start the ParaView server, the ssh daemon must be configured for port forwarding. The relevant ssh daemon configuration options are AllowTcpForwarding and GatewayPorts. The former defaults to yes while the latter defaults to no, in which case the port forwarding only works for connections coming into the loop back device. This will prevent ParaView server running on compute nodes from connecting to the tunnel. One generally has a couple of options, the simplest of which is to set GatewayPorts=yes in the server side sshd_config files. If this is impossible a couple of work arounds exist. One is to use the portfwd utility. A second is to create a tunnel directly to the compute node. These options are discussed in detail in the following sections.

A Forward Connection Over an ssh Tunnel

ParaView forward connection over a two hop ssh tunnel (black arrows).

In order for the previous method to work the server side ssh configuration must include GatewayPorts=yes. If it does not then the reverse connection from the compute node will be prevented from connecting to the ssh tunnel. One work around is to create an ssh tunnel directly to the compute node on which the ParaView server will run. This is slightly more complicated and requires two ssh sessions, and is difficult to automate.

This will require two terminals(or putty instances), the first terminal is used to submit the batch job, once the job is scheduled, one manually gets the root node's hostname and uses the second terminal to establish the tunnel. Once the tunnel is established one launches ParaView server and proceeds as usual. A normal or reverse ssh tunnel may be used, however since this is a manual method the forward tunnel may be easier.

In the following the two terminals are denoted by t1$ and t2$, and fe denotes the hostname of the cluster's front end (login node). In the first terminal you'll submit your batch job, for example:

t1$ ssh fe
t1$ qsub -I -V -l select=XX -l walltime=XX:XX:XX

qsub: waiting for job 498525.pbspl1 to start         

Job 498525.pbspl1 started on Mon Feb 08 12:51:25 PST 2010

On NODE:
NODE$

where XX is replaced by your values. After the job is scheduled you're automatically ssh'd into some compute node, which has the hostname NODE. Now in the second terminal, you will create the ssh tunnel:

t2$ ssh -L ZZZZ:NODE:YYYY fe

where ZZZZ is a port number on your workstation. YYYY is a port number on the server. Now switch back to the first terminal, and your waiting compute node and start the ParaView server. For example:

NODE$ mpiexec pvserver --server-port=YYYY

Listen on port: YYYY
Waiting for client...

At this point you may start a ParaView client on you workstation and proceed as normal by connecting to localhost port ZZZZ.

Caveats

  1. The tunnel must established before the ParaView server is started.

Reverse Connection with portfwd

Using a reverse connection the paraview client will bind to a port and wait for an incoming connection from the server. When the mpi enabled paraview server (referred to as pvserver) is launched on a set of compute nodes, the 0th pvserver process opens a connection to the host machine where the paraview client is waiting.


The problem arises when it is not possible for the compute node to connect to the host machine- the compute node can only connect to a login node. The problem is solved by forwarding traffic from a port on the login node to a port on the host machine. This can be accomplished using a simple utility called portfwd found at http://portfwd.sourceforge.net/.


For this example, let's say the paraview client is waiting for a reverse connection on port 11111 of host client_host (client_host:11111), and the compute node can only connect to port 11111 of host login_node (login_node:11111). In this example pvserver is launched on the compute nodes using the command:

mpirun -np 512 pvserver --reverse-connection --client-host=login_node --server-port=11111

Next, we use portfwd to forward login_node:11111 to client_host:11111. In some situtations it is possible to accomplish this kind of port fowarding using ssh tunnels, but portfwd is much simpler. First write a configuration file, fwd.cfg:

  /*
  fwd.cfg
  forwards localhost:11111 to client_host:11111
  */
  
  tcp { 11111 { => client_host:11111 } }


Then launch portfwd on the login node (this example uses the flag --foreground to keep portfwd running in the foreground so it can be killed with control-c):

portfwd --config fwd.cfg --foreground


Now when the pvserver connects to login_node:11111 the traffic will be forwarded to client_host:11111.

Reverse Connection over a Reverse ssh Tunnel with portfwd

This is described here

Resources

  1. Server Configuration
  2. Starting the server