Commit f3a23f71 authored by Fabio Hernandez's avatar Fabio Hernandez
Browse files

Add lsst-v12.1 on both Linux and Darwin

parent 72c22508
...@@ -18,15 +18,15 @@ First, you need to instruct EUPS to use a database on your `$HOME` directory, in ...@@ -18,15 +18,15 @@ First, you need to instruct EUPS to use a database on your `$HOME` directory, in
export EUPS_PATH=$HOME/eups export EUPS_PATH=$HOME/eups
``` ```
* Each time you want to work with a particular version of the stack source its bootstrap script. For instance, for working with version `12.0` on a Linux machine do: * Each time you want to work with a particular version of the stack source its bootstrap script. For instance, for working with version `12.1` on a Linux machine do:
```bash ```bash
$ source /cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.0/loadLSST.bash $ source /cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1/loadLSST.bash
``` ```
Please note that there are versions of the `loadLSST.*` bootstrap script to be used with other shell flavors (such as CSH, ZSH and KSH). Look in the same directory to see what is available for your version of interest. You may want to add this line to your shell profile so your environment is set for every session. Please note that there are versions of the `loadLSST.*` bootstrap script to be used with other shell flavors (such as CSH, ZSH and KSH). Look in the same directory to see what is available for your version of interest. You may want to add this line to your shell profile so your environment is set for every session.
After bootstraping your LSST environment, the value of the `$EUPS_PATH` variable will be modified to contain both your own path (i.e. `$HOME/eups`) and the path of the binary distribution (i.e. `/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.0`). After bootstraping your LSST environment, the value of the `$EUPS_PATH` variable will be modified to contain both your own path (i.e. `$HOME/eups`) and the path of the binary distribution (i.e. `/cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1`).
Once your environment is ready, you can start developing against the stack. Let's say you want to improve one of the packages of the stack, namely, `obs_cfht`, and your work is kept on a particular branch. You need first to checkout the relevant branch and instruct EUPS to use that package: Once your environment is ready, you can start developing against the stack. Let's say you want to improve one of the packages of the stack, namely, `obs_cfht`, and your work is kept on a particular branch. You need first to checkout the relevant branch and instruct EUPS to use that package:
......
# Distribution of LSST software via CernVM FS # Distribution of LSST software via CernVM-FS
## Table of contents ## Table of contents
...@@ -15,26 +15,26 @@ ...@@ -15,26 +15,26 @@
## Introduction ## Introduction
This document provides the steb-by-step instructions to configure your computer to use the binary distribution of the [LSST science pipelines](https://pipelines.lsst.io) using [CernVM FS](http://cernvm.cern.ch/portal/filesystem). This document provides the steb-by-step instructions to configure your computer to use the binary distribution of the [LSST science pipelines](https://pipelines.lsst.io) using [CernVM-FS](http://cernvm.cern.ch/portal/filesystem).
CERN's CernVM FS is a software component which allow you to mount a remote software repository in **read-only mode**, which will appear to your computer as if the software was locally installed. At [CC-IN2P3](http://cc.in2p3.fr) we prepared a binary distribution of LSST stack to be used through CernVM FS. You will find below the procedure for installing CernVM FS and configuring it to use the binary LSST software repository. CERN's CernVM-FS is a software component which allows you to mount a remote software repository in **read-only mode**. It will appear to your computer as if the software were locally installed. At [CC-IN2P3](http://cc.in2p3.fr) we prepared a binary distribution of the LSST stack to be used through CernVM-FS. You will find below the procedure for installing CernVM-FS and configuring it to use the binary LSST software repository.
Context and perspectives about this work can be found in [this presentation](https://speakerdeck.com/airnandez/experimenting-with-cernvm-fs-for-distributing-lsst-software). Context and perspectives about this work can be found in [this presentation](https://speakerdeck.com/airnandez/experimenting-with-cernvm-fs-for-distributing-lsst-software).
**WARNING** : *please bear in mind that this work is experimental. Your feedback on how to improve it is very welcome. Scroll to the end of this document to know how you can provide feedback.* **WARNING** : *please bear in mind that this work is experimental. Your feedback on how to improve it is very welcome. Scroll to the end of this document to know how you can provide feedback.*
## Benefits ## Benefits
With this method, you need to install and configure CernVM FS only once. Once this is done, when your computer is connected to the network, you will find the available versions of the LSST software stack under the local directory: With this method, you need to install and configure CernVM-FS only once. Once this is done, when your computer is connected to the network, you will find the available versions of the LSST software stack under the local directory:
/cvmfs/lsst.in2p3.fr /cvmfs/lsst.in2p3.fr
This method of distributing the software is particularly useful for individuals: you don't need to install each official LSST software release from sources on your personal computer, but rather to automatically mount and use the binary distributions prepared for your convenience. This method of distributing the software is particularly useful for individuals: you don't need to build and install each official LSST software release from sources on your personal computer. You rather mount and use the binary distribution prepared for your convenience.
The figure below shows the namespace and the releases currently available: The figure below shows the namespace and the releases currently available:
![](namespace.png) ![](namespace.png)
Please note that you don't need special privileges to use the LSST software stack distributed this way: any user on a pre-configured computer can use the software. However, in order to install and configure CernVM FS, a one-time process, you need super-user privileges on the target machine. Please note that you don't need special privileges to use the LSST software stack distributed this way: any user on a pre-configured computer can use the software. However, in order to install and configure CernVM-FS, a one-time process, you need super-user privileges on the target machine.
## Installation ## Installation
We have succesfully tested several versions of the LSST software using this installation on MacOS X 10.10 Yosemite, Scientific Linux 6, Scientific Linux 7, CentOS 7 and Ubuntu 14.04. It may work on other Linux distributions as well. We have succesfully tested several versions of the LSST software using this installation on MacOS X 10.10 Yosemite, Scientific Linux 6, Scientific Linux 7, CentOS 7 and Ubuntu 14.04. It may work on other Linux distributions as well.
...@@ -52,7 +52,7 @@ Some packages are either necessary or recommended on Ubuntu. To install them do: ...@@ -52,7 +52,7 @@ Some packages are either necessary or recommended on Ubuntu. To install them do:
$ sudo apt-get install autofs attr gdb git sysv-rc-conf uuid libossp-uuid16 $ sudo apt-get install autofs attr gdb git sysv-rc-conf uuid libossp-uuid16
Download and install CernVM FS: Download and install CernVM-FS:
$ cd /tmp $ cd /tmp
$ curl -O https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.1.20/cvmfs_2.1.20_amd64.deb $ curl -O https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.1.20/cvmfs_2.1.20_amd64.deb
...@@ -62,9 +62,9 @@ Download and install CernVM FS: ...@@ -62,9 +62,9 @@ Download and install CernVM FS:
*NOTE: this method has been reported to work also on Linux Mint v17.3* *NOTE: this method has been reported to work also on Linux Mint v17.3*
### Installing on OS X 10.9 Mavericks, 10.10 Yosemite, 10.11 El Capitan ### Installing on OS X 10.9 Mavericks, 10.10 Yosemite, 10.11 El Capitan
* Download and install the **latest stable** release of [FUSE for OS X](https://osxfuse.github.io/). This is a dependency of the CernVM FS client. * Download and install the **latest stable** release of [FUSE for OS X](https://osxfuse.github.io/). This is a dependency of the CernVM-FS client.
* Download and install [the CernVM FS client package](https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.1.20/cvmfs-2.1.20.pkg). Alternatively, you can do it manually if you prefer: * Download and install [the CernVM-FS client package](https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.1.20/cvmfs-2.1.20.pkg). Alternatively, you can do it manually if you prefer:
``` ```
$ cd /tmp $ cd /tmp
...@@ -72,11 +72,11 @@ Download and install CernVM FS: ...@@ -72,11 +72,11 @@ Download and install CernVM FS:
$ open cvmfs-2.1.20.pkg $ open cvmfs-2.1.20.pkg
``` ```
At this point, you have installed the CernVM FS client software on your computer. For configuring it please see the next section. At this point, you have installed the CernVM-FS client software on your computer. For configuring it please see the next section.
## Configuration ## Configuration
Configuring the CernVM FS client to use the binary distribution of LSST software served by CC-IN2P3 is a *one-time operation*: Configuring the CernVM-FS client to use the binary distribution of LSST software served by CC-IN2P3 is a *one-time operation*:
* Clone this repository and run the provided configuration script. The configuration script needs super-user privileges: * Clone this repository and run the provided configuration script. The configuration script needs super-user privileges:
...@@ -97,7 +97,7 @@ Configuring the CernVM FS client to use the binary distribution of LSST software ...@@ -97,7 +97,7 @@ Configuring the CernVM FS client to use the binary distribution of LSST software
$ sudo umount /cvmfs/lsst.in2p3.fr $ sudo umount /cvmfs/lsst.in2p3.fr
* **[Linux - recommended]** On Linux, the CernVM FS client uses `autofs` for automatically mounting and unmounting the file system. We recommend to configure the `autofs` service to start at boot time. On Scientific Linux and CentOS do: * **[Linux - recommended]** On Linux, the CernVM-FS client uses `autofs` for automatically mounting and unmounting the file system. We recommend to configure the `autofs` service to start at boot time. On Scientific Linux and CentOS do:
$ sudo chkconfig autofs on $ sudo chkconfig autofs on
...@@ -111,9 +111,9 @@ Now you are ready to use the stack. See next section. ...@@ -111,9 +111,9 @@ Now you are ready to use the stack. See next section.
## Usage ## Usage
Once the software is installed and configured in your computer, a one-time process, you don't need super-user privileges to use the LSST software. Once the software is installed and configured in your computer, a one-time process, you don't need super-user privileges to use the LSST software.
In order to use the LSST science pipelines, you need first to bootstrap your environment for a specific version of the LSST software for which a binary distribution is available. For instance, to use LSST `v12.0` on a Linux machine do: In order to use the LSST science pipelines, you need first to bootstrap your environment for a specific version of the LSST software for which a binary distribution is available. For instance, to use LSST `v12.1` on a Linux computer do:
$ cd /cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.0 $ cd /cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v12.1
$ source loadLSST.bash $ source loadLSST.bash
Now you are ready to test your installation by [downloading and running the demo project](https://pipelines.lsst.io/install/demo.html#download-the-demo-project). Now you are ready to test your installation by [downloading and running the demo project](https://pipelines.lsst.io/install/demo.html#download-the-demo-project).
...@@ -133,8 +133,8 @@ Currently you will find the releases presented in the table below: ...@@ -133,8 +133,8 @@ Currently you will find the releases presented in the table below:
| Platform | Available versions of LSST software | | Platform | Available versions of LSST software |
| --------------------- | ----------------------------------- | | --------------------- | ----------------------------------- |
| Linux, x86_64, 64bits | `v10.1`, `v11.0`, `v12.0` | | Linux, x86_64, 64bits | `v10.1`, `v11.0`, `v12.0`, `v12.1` |
| Darwin, x86_64, 64bits | `v10.1`, `v11.0`, `v12.0` | | Darwin, x86_64, 64bits | `v10.1`, `v11.0`, `v12.0`, `v12.1` |
For details on the platform each binary release was built on please refer to the `README` file in the corresponding directory. For details on the platform each binary release was built on please refer to the `README` file in the corresponding directory.
...@@ -157,17 +157,17 @@ Please also refer to the [known issues](https://github.com/airnandez/lsst-cvmfs/ ...@@ -157,17 +157,17 @@ Please also refer to the [known issues](https://github.com/airnandez/lsst-cvmfs/
Your feedback is very welcome. Please feel free to [open an issue](https://github.com/airnandez/lsst-cvmfs/issues). Your feedback is very welcome. Please feel free to [open an issue](https://github.com/airnandez/lsst-cvmfs/issues).
* **Where can I get more detailed information on CernVM FS?** * **Where can I get more detailed information on CernVM-FS?**
The [CernVM FS downloads page](http://cernvm.cern.ch/portal/filesystem/downloads) contains additional information. In addition, you may want to read the [CernVM FS Technical Information](http://cernvm.cern.ch/portal/filesystem/techinformation) for more in-depth information on how CernVM FS works. The [CernVM-FS downloads page](http://cernvm.cern.ch/portal/filesystem/downloads) contains additional information. In addition, you may want to read the [CernVM-FS Technical Information](http://cernvm.cern.ch/portal/filesystem/techinformation) for more in-depth information on how CernVM-FS works.
* **Can I use my remote LSST software distribution while disconnected from the network?** * **Can I use my remote LSST software distribution while disconnected from the network?**
The CernVM FS client caches all the file metadata and the contents of the accessed remote files in the local disk of your computer. If you have previously used the stack it is likely that the relevant files are locally available in your local disk, in which case, you may work while disconnected. However, we have not tested this thoroughly, so let us know how it works for you. The CernVM-FS client caches all the file metadata and the contents of the accessed remote files in the local disk of your computer. If you have previously used the stack it is likely that the relevant files are locally available in your local disk, in which case, you may work while disconnected. However, we have not tested this thoroughly, so let us know how it works for you.
* **Can I use this for my Docker containers?** * **Can I use this for my Docker containers?**
Yes, you can configure your container for automatically mounting a read-only file system with LSST software stack ready to use. Sébastien Binet did exactly this, so you can just use as is or as a baseline for your own containers. You will find all the details [here](https://github.com/hepsw/docks/tree/master/cvmfs-lsst). Yes, you can configure your container for automatically mounting a read-only file system with LSST software stack ready to use. Sébastien Binet did exactly this, so you can just use as is or as a baseline for your own containers. You will find all the details [here](https://github.com/hepsw/docks/tree/master/cvmfs-lsst).
## Credits ## Credits
This work was done by Fabio Hernandez from [IN2P3/CNRS computing center](http://cc.inp3.fr) (Lyon, France) with very valuable help from Vanessa Hamar who set up the CernVM FS server and proxy infrastructure. This work was done by Fabio Hernandez from [IN2P3/CNRS computing center](http://cc.inp3.fr) (Lyon, France) with very valuable help from Vanessa Hamar who set up the CernVM-FS server and proxy infrastructure.
namespace.png

137 KB | W: | H:

namespace.png

174 KB | W: | H:

namespace.png
namespace.png
namespace.png
namespace.png
  • 2-up
  • Swipe
  • Onion skin
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment