Using Phi at LPNHE

To know what is a phi, see by example the Intel review on this subject.

In practise, each Xeon Phi boards owns 4 processor, each with 60 computing cores. At the end, each board is like a NUMA machine with 240 cores, available through a MPI or OpenMP programming. See the OpenMP page at LPNHE.

Basic usage

Accessing the host machine

The access should be ask to [Victor Mendoza](mailto:mendoza@lpnhe.in2p3.fr?subject=Accessing LPNHE Phi machine).

Thus, the access is done via :

ssh lpnxphi.in2p3.fr

Caution, in that case we are logged on the host machine but not on a phi machine.

In practise, to avoid typing its password each time, one should a more full ssh command, with connection agent forward :

ssh -Y -2 -A lpnxphi.in2p3.fr

Accessing the phi

From the host machine. one have access to two phi boards with ssh : phi boards are just like other Linux machines. they are named mic0 and mic1 (mic is the other name of the phi architecture).

ssh mic0  # ou mic1

Environment

The home of the lab is directly available on the host machine.

But on phi, the seen home is /home/$USER, which is /local/home/$USER on the host machine. Needed file copies must be done. Thus, it would be usefull to create a /local/home/$USER/.ssh with an authorized_keys file.

Compiling for phi

Intel compiler

The use of the Intel compiler icc is mandatory. To do so, we are using tokens from computing center ccin2p3, binaries coming from CERN via AFS.

export INTEL_LICENSE_FILE=/home/beau/intel/licenses
. /afs/cern.ch/sw/IntelSoftware/linux/x86_64/xe2015/bin/iccvars.sh intel64
# example with tests on phi, OpenMP mode
cd ~/HPC/gpu/pi-test
icc -mmic -O3 -qopenmp --static pi_omp.c

Executing

cp a.out /local/home/$USER  
ssh mic1   # to log onto the Phi machine
export OMP_NUM_THREADS=239 # there are 240 cores by phi device
./a.out

GNU compiler

In principle, it is possible to use gcc... but the full procedure is not yet running.

A priori, we should use either the one from MPSS :

/opt/mpss/3.6.1/sysroots/x86_64-mpsssdk-linux/usr/bin/k1om-mpss-linux/k1om-mpss-linux-gcc

or the onde from redhat/SL :

scl enable devtoolset-4 'bash'
gcc -dumpversion

To go further

OpenMP

environment

export INTEL_LICENSE_FILE=$HOME/intel/licenses . /afs/cern.ch/sw/IntelSoftware/linux/x86_64/xe2015/bin/iccvars.sh intel64

2. compiling

    ```bash
icc -openmp -mmic -O2 pi_omp.c -o /local/home/$USER/bin/pi_omp.mic

session ( /local/home/$USER/.profile )

PS1='[\u@\h \W]$ ' export I_MPI_ROOT=/opt/intel/xe2015/impi/5.0.3.048 export PATH=/usr/bin:/usr/sbin:/bin:/sbin export PATH=PATH:I_MPI_ROOT/mic/bin:HOME/bin export LD_LIBRARY_PATH=LD_LIBRARY_PATH:$I_MPI_ROOT/mic/lib:/opt/intel/xe2015/lib/mic

4. execution (240 cores) :

    ```bash
ssh mic0
export OMP_NUM_THREADS=240
~/bin/pi_omp.mic 501000000000

MPI

MPI is installed on phi boards, which allows to use our two Phi boards and the host machine for the same computation task.

Methodology : 0. ssh configuration

Create the file `/local/home/$USER/.ssh/known_hosts`, with :

mic0,192.168.111.1 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ5M4tOENUmsAZ3I6DWaMLBIA55Ph5vWeNMTeIYmeIkLYS+63lAnAWD6RQygb06jw/CS232+LA/ec+fK8qktDKI=
mic1,192.168.111.2 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOUYIRxFQA+HNDuGFegHKHMljuZpdj/t0hENwyT97yFwusZR1cnKC9z7qGaV/yN7ww9I7R18tnQzjzin1O0mnMY=
lpnxphi.in2p3.fr,lpnxphi,host,192.168.111.254 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqq9zTsTOPoOvJru01sJjiLtq4GJyN7SgrnRXF3diuHZbcTifDej4i6PegzQWj05bYKUEnvoTRZJICdOsdU2eXonaQ5ZxYp4nbW1y1/ESLgj9uA8RfgZSvRX35uxdzcbYeZDyEnMtW6o0STramWc3cGEnmP36uDrnMKbN77raGmVhdKvLhhAelK8W445bkq27pNnaJAwaECfVwBGU6Ba/lbAJThcEy1q9NbVM90PEvDpdpn8dIYBNQPDQEpk92/itgt0kaBmbytrhMQUeYMSQTqjO5gqAs+A48Z4NVStbvLFO4xH7q1NlO0KY+QtVgFqjsfIGZiRnPtQM5gDdnqctCw==

environment

export INTEL_LICENSE_FILE=$HOME/intel/licenses . /afs/cern.ch/sw/IntelSoftware/linux/x86_64/xe2015/bin/iccvars.sh intel64 . /afs/cern.ch/sw/IntelSoftware/linux/x86_64/xe2015/impi/5.0.3.048/bin64/mpivars.sh

2. compiling

    ```bash
mpiicc -O3 -o ~/bin/pi pi_mpi.c
mpiicc -O3 -mmic -o /local/home/$USER/bin/pi.mic pi_mpi.c

execution : (512 cores)

export I_MPI_MIC=enable export I_MPI_MIC_POSTFIX=.mic export I_MPI_MIC_PROXY_PATH=/opt/intel/xe2015/impi/5.0.3.048/mic/bin export I_MPI_FABRICS=tcp cat > targets lpnxphi:32 mic0:240 mic1:240

mpirun -env LD_LIBRARY_PATH /opt/intel/xe2015/impi/5.0.3.048/mic/lib:/opt/intel/xe2015/lib/mic-machinefile targets /home/$USER/bin/pi


### Documentation ###
http://lpnhe.in2p3.fr/uploads/uploads/ceb7d7ec56eacfd7c4ba4/mpss_users_guide.pdf

Comments

Please register or sign in to add a comment.

PhiAtLpnhe