|
# Utilisation des phi au LPNHE #
|
|
# Using Phi at LPNHE
|
|
Pour savoir qu'est-ce qu'un phi, voir par exemple [la revue de Intel](http://www.intel.com/content/www/us/en/processors/xeon/xeon-phi-coprocessor-overview.html) à ce sujet.
|
|
To know what is a phi, see by example [the Intel review](http://www.intel.com/content/www/us/en/processors/xeon/xeon-phi-coprocessor-overview.html) on this subject.
|
|
|
|
|
|
Dans la pratique, chauqe carte Xeon Phi comporte 4 processeur chacun comportant 60 cœurs de calcul. Finalement, chaque carte se comporte comme une machine NUMA à 240 cœurs, machine que l'on utilisera via OpenMP ou MPI aisément. Voir utilement [la page consacrée à OpenMP au labo](OpenmpLpnhe).
|
|
In practise, each Xeon Phi boards owns 4 processor, each with 60 computing cores. At the end, each board is like a NUMA machine with 240 cores, available through a MPI or OpenMP programming. See [the OpenMP page at LPNHE](OpenmpLpnhe).
|
|
|
|
|
|
## Utisation de base ##
|
|
## Basic usage
|
|
### Accès à la machine hôte ###
|
|
### Accessing the host machine
|
|
L'accès doit être demandé auprès de [Victor Mendoza](mailto:mendoza@lpnhe.in2p3.fr?subject=Demande accès machine Phi au LPNHE).
|
|
The access should be ask to [Victor Mendoza](mailto:mendoza@lpnhe.in2p3.fr?subject=Accessing LPNHE Phi machine).
|
|
|
|
|
|
Ensuite, l'accès peut passer par une commande
|
|
Thus, the access is done via :
|
|
```bash
|
|
```bash
|
|
ssh lpnxphi.in2p3.fr
|
|
ssh lpnxphi.in2p3.fr
|
|
```
|
|
```
|
|
Attention, on arrive dans ce cas sur la machine hôte mais pas sur une machine phi.
|
|
Caution, in that case we are logged on the host machine but not on a phi machine.
|
|
|
|
|
|
Dans la pratique, pour éviter de taper son mot de passe systématiquement, on utilisera utilement une commande plus complète de `ssh`, avec le *forward* de l'agent de connexion :
|
|
In practise, to avoid typing its password each time, one should a more full `ssh` command, with connection agent *forward* :
|
|
```bash
|
|
```bash
|
|
ssh -Y -2 -A lpnxphi.in2p3.fr
|
|
ssh -Y -2 -A lpnxphi.in2p3.fr
|
|
```
|
|
```
|
|
### Accès aux phi ###
|
|
### Accessing the phi ###
|
|
À partir de la machine hôte, on a accès aux deux cartes phi par `ssh`, les cartes phi sont vues comme d'autres machines Linux. Elles se nomment `mic0` et `mic1` (**mic** est l'autre nom de l'architecture phi).
|
|
From the host machine. one have access to two phi boards with `ssh` : phi boards are just like other Linux machines. they are named `mic0` and `mic1` (**mic** is the other name of the phi architecture).
|
|
```bash
|
|
```bash
|
|
ssh mic0 # ou mic1
|
|
ssh mic0 # ou mic1
|
|
```
|
|
```
|
|
### Environnement ###
|
|
### Environment ###
|
|
Le `home` du labo est disponible sur la machine hôte directement.
|
|
The `home` of the lab is directly available on the host machine.
|
|
|
|
|
|
En revanche sur les phi, c'est le `home` vu du phi `/home/$USER` qui est vu. Il s'agit aussi de `/local/home/$USER` de la machine hôte. Il faut effectuer les copies qui s'imposent. Ainsi on créera utilement le répertoire `/local/home/$USER/.ssh` avec le fichier `authorized_keys`.
|
|
But on phi, the seen home is `/home/$USER`, which is `/local/home/$USER` on the host machine. Needed file copies must be done. Thus, it would be usefull to create a `/local/home/$USER/.ssh` with an `authorized_keys` file.
|
|
|
|
|
|
## Compiler pour les phi ##
|
|
## Compiling for phi
|
|
### Compilateur Intel ###
|
|
### Intel compiler
|
|
L'utilisation du [compilateur Intel `icc`](https://software.intel.com/en-us/c-compilers) est nécessaire. On utilise pour cela les jetons du centre de calcul, les binaires venant du CERN via AFS.
|
|
The use of the [Intel compiler `icc`](https://software.intel.com/en-us/c-compilers) is mandatory. To do so, we are using tokens from computing center [ccin2p3](http://cc.in2p3.fr), binaries coming from CERN via AFS.
|
|
```bash
|
|
```bash
|
|
export INTEL_LICENSE_FILE=/home/beau/intel/licenses
|
|
export INTEL_LICENSE_FILE=/home/beau/intel/licenses
|
|
. /afs/cern.ch/sw/IntelSoftware/linux/x86_64/xe2015/bin/iccvars.sh intel64
|
|
. /afs/cern.ch/sw/IntelSoftware/linux/x86_64/xe2015/bin/iccvars.sh intel64
|
|
# exemple avec les tests sur pi en OpenMP
|
|
# example with tests on phi, OpenMP mode
|
|
cd ~/HPC/gpu/pi-test
|
|
cd ~/HPC/gpu/pi-test
|
|
icc -mmic -O3 -qopenmp --static pi_omp.c
|
|
icc -mmic -O3 -qopenmp --static pi_omp.c
|
|
```
|
|
```
|
|
### Execution ###
|
|
### Executing
|
|
```bash
|
|
```bash
|
|
cp a.out /local/home/$USER
|
|
cp a.out /local/home/$USER
|
|
ssh mic1 # pour se logger sur la machine Phi
|
|
ssh mic1 # to log onto the Phi machine
|
|
export OMP_NUM_THREADS=239 # il y a 240 coeurs par device phi
|
|
export OMP_NUM_THREADS=239 # there are 240 cores by phi device
|
|
./a.out
|
|
./a.out
|
|
```
|
|
```
|
|
### Compilateur GNU ###
|
|
### GNU compiler
|
|
Il est possible en principe d'utiliser `gcc`... mais la procédure complète ne fonctionne pas encore.
|
|
In principle, it is possible to use `gcc`... but the full procedure is not yet running.
|
|
|
|
|
|
Il faudrait a priori utiliser soit celui fourni par MPSS :
|
|
A priori, we should use either the one from MPSS :
|
|
```bash
|
|
```bash
|
|
/opt/mpss/3.6.1/sysroots/x86_64-mpsssdk-linux/usr/bin/k1om-mpss-linux/k1om-mpss-linux-gcc
|
|
/opt/mpss/3.6.1/sysroots/x86_64-mpsssdk-linux/usr/bin/k1om-mpss-linux/k1om-mpss-linux-gcc
|
|
```
|
|
```
|
|
soit celui fourni par redhat/SL :
|
|
or the onde from redhat/SL :
|
|
```bash
|
|
```bash
|
|
scl enable devtoolset-4 'bash'
|
|
scl enable devtoolset-4 'bash'
|
|
gcc -dumpversion
|
|
gcc -dumpversion
|
|
```
|
|
```
|
|
|
|
|
|
## Pour aller plus loin ##
|
|
## To go further
|
|
### OpenMP ###
|
|
### OpenMP ###
|
|
1. environnement
|
|
1. environment
|
|
|
|
|
|
```bash
|
|
```bash
|
|
export INTEL_LICENSE_FILE=$HOME/intel/licenses
|
|
export INTEL_LICENSE_FILE=$HOME/intel/licenses
|
|
. /afs/cern.ch/sw/IntelSoftware/linux/x86_64/xe2015/bin/iccvars.sh intel64
|
|
. /afs/cern.ch/sw/IntelSoftware/linux/x86_64/xe2015/bin/iccvars.sh intel64
|
|
```
|
|
```
|
|
2. compilation
|
|
2. compiling
|
|
|
|
|
|
```bash
|
|
```bash
|
|
icc -openmp -mmic -O2 pi_omp.c -o /local/home/$USER/bin/pi_omp.mic
|
|
icc -openmp -mmic -O2 pi_omp.c -o /local/home/$USER/bin/pi_omp.mic
|
... | @@ -79,7 +79,7 @@ export PATH=/usr/bin:/usr/sbin:/bin:/sbin |
... | @@ -79,7 +79,7 @@ export PATH=/usr/bin:/usr/sbin:/bin:/sbin |
|
export PATH=$PATH:$I_MPI_ROOT/mic/bin:$HOME/bin
|
|
export PATH=$PATH:$I_MPI_ROOT/mic/bin:$HOME/bin
|
|
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$I_MPI_ROOT/mic/lib:/opt/intel/xe2015/lib/mic
|
|
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$I_MPI_ROOT/mic/lib:/opt/intel/xe2015/lib/mic
|
|
```
|
|
```
|
|
4. exécution (240 cores) :
|
|
4. execution (240 cores) :
|
|
|
|
|
|
```bash
|
|
```bash
|
|
ssh mic0
|
|
ssh mic0
|
... | @@ -88,30 +88,31 @@ export OMP_NUM_THREADS=240 |
... | @@ -88,30 +88,31 @@ export OMP_NUM_THREADS=240 |
|
```
|
|
```
|
|
|
|
|
|
### MPI ###
|
|
### MPI ###
|
|
MPI est installé sur les phi, ce qui peut notamment permettre d'utiliser les 2 cartes Phi et la machine hôte en pour un même calcul.
|
|
MPI is installed on phi boards, which allows to use our two Phi boards *and* the host machine for the same computation task.
|
|
Méthode :
|
|
|
|
0. configuration ssh
|
|
|
|
|
|
|
|
Créer le fichier `/local/home/$USER/.ssh/known_hosts` contenant :
|
|
Methodology :
|
|
|
|
0. ssh configuration
|
|
|
|
|
|
|
|
Create the file `/local/home/$USER/.ssh/known_hosts`, with :
|
|
```
|
|
```
|
|
mic0,192.168.111.1 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ5M4tOENUmsAZ3I6DWaMLBIA55Ph5vWeNMTeIYmeIkLYS+63lAnAWD6RQygb06jw/CS232+LA/ec+fK8qktDKI=
|
|
mic0,192.168.111.1 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ5M4tOENUmsAZ3I6DWaMLBIA55Ph5vWeNMTeIYmeIkLYS+63lAnAWD6RQygb06jw/CS232+LA/ec+fK8qktDKI=
|
|
mic1,192.168.111.2 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOUYIRxFQA+HNDuGFegHKHMljuZpdj/t0hENwyT97yFwusZR1cnKC9z7qGaV/yN7ww9I7R18tnQzjzin1O0mnMY=
|
|
mic1,192.168.111.2 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOUYIRxFQA+HNDuGFegHKHMljuZpdj/t0hENwyT97yFwusZR1cnKC9z7qGaV/yN7ww9I7R18tnQzjzin1O0mnMY=
|
|
lpnxphi.in2p3.fr,lpnxphi,host,192.168.111.254 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqq9zTsTOPoOvJru01sJjiLtq4GJyN7SgrnRXF3diuHZbcTifDej4i6PegzQWj05bYKUEnvoTRZJICdOsdU2eXonaQ5ZxYp4nbW1y1/ESLgj9uA8RfgZSvRX35uxdzcbYeZDyEnMtW6o0STramWc3cGEnmP36uDrnMKbN77raGmVhdKvLhhAelK8W445bkq27pNnaJAwaECfVwBGU6Ba/lbAJThcEy1q9NbVM90PEvDpdpn8dIYBNQPDQEpk92/itgt0kaBmbytrhMQUeYMSQTqjO5gqAs+A48Z4NVStbvLFO4xH7q1NlO0KY+QtVgFqjsfIGZiRnPtQM5gDdnqctCw==
|
|
lpnxphi.in2p3.fr,lpnxphi,host,192.168.111.254 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqq9zTsTOPoOvJru01sJjiLtq4GJyN7SgrnRXF3diuHZbcTifDej4i6PegzQWj05bYKUEnvoTRZJICdOsdU2eXonaQ5ZxYp4nbW1y1/ESLgj9uA8RfgZSvRX35uxdzcbYeZDyEnMtW6o0STramWc3cGEnmP36uDrnMKbN77raGmVhdKvLhhAelK8W445bkq27pNnaJAwaECfVwBGU6Ba/lbAJThcEy1q9NbVM90PEvDpdpn8dIYBNQPDQEpk92/itgt0kaBmbytrhMQUeYMSQTqjO5gqAs+A48Z4NVStbvLFO4xH7q1NlO0KY+QtVgFqjsfIGZiRnPtQM5gDdnqctCw==
|
|
```
|
|
```
|
|
1. environnement
|
|
1. environment
|
|
|
|
|
|
```bash
|
|
```bash
|
|
export INTEL_LICENSE_FILE=$HOME/intel/licenses
|
|
export INTEL_LICENSE_FILE=$HOME/intel/licenses
|
|
. /afs/cern.ch/sw/IntelSoftware/linux/x86_64/xe2015/bin/iccvars.sh intel64
|
|
. /afs/cern.ch/sw/IntelSoftware/linux/x86_64/xe2015/bin/iccvars.sh intel64
|
|
. /afs/cern.ch/sw/IntelSoftware/linux/x86_64/xe2015/impi/5.0.3.048/bin64/mpivars.sh
|
|
. /afs/cern.ch/sw/IntelSoftware/linux/x86_64/xe2015/impi/5.0.3.048/bin64/mpivars.sh
|
|
```
|
|
```
|
|
2. compilation
|
|
2. compiling
|
|
|
|
|
|
```bash
|
|
```bash
|
|
mpiicc -O3 -o ~/bin/pi pi_mpi.c
|
|
mpiicc -O3 -o ~/bin/pi pi_mpi.c
|
|
mpiicc -O3 -mmic -o /local/home/$USER/bin/pi.mic pi_mpi.c
|
|
mpiicc -O3 -mmic -o /local/home/$USER/bin/pi.mic pi_mpi.c
|
|
```
|
|
```
|
|
3. exécution : (512 cores)
|
|
3. execution : (512 cores)
|
|
|
|
|
|
```bash
|
|
```bash
|
|
export I_MPI_MIC=enable
|
|
export I_MPI_MIC=enable
|
... | @@ -122,7 +123,6 @@ cat > targets |
... | @@ -122,7 +123,6 @@ cat > targets |
|
lpnxphi:32
|
|
lpnxphi:32
|
|
mic0:240
|
|
mic0:240
|
|
mic1:240
|
|
mic1:240
|
|
^D
|
|
|
|
|
|
|
|
mpirun -env LD_LIBRARY_PATH
|
|
mpirun -env LD_LIBRARY_PATH
|
|
/opt/intel/xe2015/impi/5.0.3.048/mic/lib:/opt/intel/xe2015/lib/mic-machinefile targets /home/$USER/bin/pi
|
|
/opt/intel/xe2015/impi/5.0.3.048/mic/lib:/opt/intel/xe2015/lib/mic-machinefile targets /home/$USER/bin/pi
|
... | | ... | |