README.md 9.4 KB
Newer Older
Fabio Hernandez's avatar
Fabio Hernandez committed
1 2
<br>
<h3>
3
Please refer to <a href="https://sw.lsst.eu">https://sw.lsst.eu</a> for an improved version of this service. This repository won't be updated anymore.
Fabio Hernandez's avatar
Fabio Hernandez committed
4 5 6
</h3>
<br>

7 8
# Distribution of LSST software via CernVM-FS

Fabio Hernandez's avatar
Fabio Hernandez committed
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
## Table of contents

* [Introduction](#introduction)
* [Benefits](#benefits)
* [Installation](#installation)
* [Configuration](#configuration)
* [Usage](#usage)
* [Available releases](#available-releases)
* [Advanced Usage](#advanced-usage)
* [Troubleshooting](#troubleshooting)
* [Frequently Asked Questions](#frequently-asked-questions)
* [Credits](#credits)


## Introduction
24
This document provides the steb-by-step instructions to configure your computer to use the binary distribution of the [LSST science pipelines](https://pipelines.lsst.io) using [CernVM-FS](http://cernvm.cern.ch/portal/filesystem).
Fabio Hernandez's avatar
Fabio Hernandez committed
25

26
CERN's CernVM-FS is a software component which allows you to mount a remote software repository in **read-only mode**. It will appear to your computer as if the software were locally installed. At [CC-IN2P3](http://cc.in2p3.fr) we prepared a binary distribution of the LSST stack to be used through CernVM-FS. You will find below the procedure for installing CernVM-FS and configuring it to use the binary LSST software repository.
Fabio Hernandez's avatar
Fabio Hernandez committed
27

28
**WARNING** : *please bear in mind that this work is experimental. Your feedback on how to improve it is very welcome. Scroll to the end of this document to know how you can provide feedback.*
Fabio Hernandez's avatar
Fabio Hernandez committed
29

Fabio Hernandez's avatar
Fabio Hernandez committed
30
## Benefits
31
With this method, you need to install and configure CernVM-FS only once. Once this is done, when your computer is connected to the network, you will find the available versions of the LSST software stack under the local directory:
Fabio Hernandez's avatar
Fabio Hernandez committed
32 33

	/cvmfs/lsst.in2p3.fr
34

35
This method of distributing the software is particularly useful for individuals: you don't need to build and install each official LSST software release from sources on your personal computer. You rather mount and use the binary distribution prepared for your convenience.
Fabio Hernandez's avatar
Fabio Hernandez committed
36

Fabio Hernandez's avatar
Fabio Hernandez committed
37 38 39 40
The figure below shows the namespace and the releases currently available:

![](namespace.png)

41
Please note that you don't need special privileges to use the LSST software stack distributed this way: any user on a pre-configured computer can use the software. However, in order to install and configure CernVM-FS, a one-time process, you need super-user privileges on the target machine.
42

Fabio Hernandez's avatar
Fabio Hernandez committed
43
## Installation
44
We have succesfully tested several versions of the LSST software using this installation on MacOS X 10.10 Yosemite, CentOS 7 and Ubuntu 14.04. It may work on other Linux distributions as well.
Fabio Hernandez's avatar
Fabio Hernandez committed
45

46
### Installing on CentOS 7 (64 bits)
47
To download the software from CERN's repository and install it do:
Fabio Hernandez's avatar
Fabio Hernandez committed
48

49
   	$ cd /tmp
50 51
	$ curl -O https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.3.5/cvmfs-2.3.5-1.el7.centos.x86_64.rpm
	$ curl -O https://ecsft.cern.ch/dist/cvmfs/cvmfs-config/cvmfs-config-none-1.0-2.noarch.rpm
52
	$ sudo yum install --assumeyes ./cvmfs-*.rpm
Fabio Hernandez's avatar
Fabio Hernandez committed
53 54

### Installing on Ubuntu 14.04 (64 bits)
55
Some packages are either necessary or recommended on Ubuntu. To install them do:
Fabio Hernandez's avatar
Fabio Hernandez committed
56

Fabio Hernandez's avatar
Fabio Hernandez committed
57
    $ sudo apt-get install autofs attr gdb git sysv-rc-conf uuid libossp-uuid16
58

59
Download and install CernVM-FS:
60

61
    $ cd /tmp
62 63 64
    $ curl -O https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.3.5/cvmfs_2.3.5_amd64.deb
    $ curl -O https://ecsft.cern.ch/dist/cvmfs/cvmfs-config/cvmfs-config-none_1.0-2_all.deb
    $ sudo dpkg -i ./cvmfs_2.3.5_amd64.deb ./cvmfs-config-none_1.0-2_all.deb
Fabio Hernandez's avatar
Fabio Hernandez committed
65 66
    
*NOTE: this method has been reported to work also on Linux Mint v17.3*
67

Fabio Hernandez's avatar
Fabio Hernandez committed
68
### Installing on OS X 10.9 Mavericks, 10.10 Yosemite, 10.11 El Capitan
69
* Download and install the **latest stable** release of [FUSE for OS X](https://osxfuse.github.io/). This is a dependency of the CernVM-FS client.
70

71
* Download and install [the CernVM-FS client package](https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.1.20/cvmfs-2.1.20.pkg). Alternatively, you can do it manually if you prefer:
72

Fabio Hernandez's avatar
Fabio Hernandez committed
73 74 75 76 77
```bash
$ cd /tmp
$ curl -O https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.1.20/cvmfs-2.1.20.pkg
$ open cvmfs-2.1.20.pkg
```
78

79
At this point, you have installed the CernVM-FS client software on your computer. For configuring it please see the next section.
80

Fabio Hernandez's avatar
Fabio Hernandez committed
81

Fabio Hernandez's avatar
Fabio Hernandez committed
82
## Configuration
83
Configuring the CernVM-FS client to use the binary distribution of LSST software served by CC-IN2P3 is a *one-time operation*:
Fabio Hernandez's avatar
Fabio Hernandez committed
84

85
* Clone this repository and run the provided configuration script. The configuration script needs super-user privileges:
Fabio Hernandez's avatar
Fabio Hernandez committed
86

87 88 89
		$ cd /tmp
		$ git clone https://github.com/airnandez/lsst-cvmfs.git
		$ cd lsst-cvmfs
90
		$ sudo bash ./configure.sh
Fabio Hernandez's avatar
Fabio Hernandez committed
91 92

	After this step, among other things, an unprivileged user `cvmfs` is created in your computer and several configuration files with sensible default values are located under `/etc/cvmfs`.
93

94
  You can tell the configuration process was successful if you don't see any error message.
95

Fabio Hernandez's avatar
Fabio Hernandez committed
96
* **[OS X only]** On OS X, you need to manually mount the file system:
97

98
		$ sudo mount -t cvmfs lsst.in2p3.fr /cvmfs/lsst.in2p3.fr
99

100
	For unmounting do:
101

102
		$ sudo umount /cvmfs/lsst.in2p3.fr
103

104
* **[Linux - recommended]** On Linux, the CernVM-FS client uses `autofs` for automatically mounting and unmounting the file system. We recommend to configure the `autofs` service to start at boot time. On CentOS do:
105

106
		$ sudo chkconfig autofs on
107

108
	and on Ubuntu, do:
109

110
		$ sudo sysv-rc-conf autofs on
111

112

113
Now you are ready to use the stack. See next section.
114

Fabio Hernandez's avatar
Fabio Hernandez committed
115
## Usage
116 117
Once the software is installed and configured in your computer, a one-time process, you don't need super-user privileges to use the LSST software.

Fabio Hernandez's avatar
Fabio Hernandez committed
118
In order to use the LSST science pipelines, you need first to bootstrap your environment for a specific version of the LSST software for which a binary distribution is available. For instance, to use LSST `v15.0` on a Linux computer do:
Fabio Hernandez's avatar
Fabio Hernandez committed
119

Fabio Hernandez's avatar
Fabio Hernandez committed
120
		$ cd /cvmfs/lsst.in2p3.fr/software/linux-x86_64/lsst-v15.0
121
		$ source loadLSST.bash
122

123
Now you are ready to test your installation by [downloading and running the demo project](https://pipelines.lsst.io/install/demo.html#download-the-demo-project).  
124

Fabio Hernandez's avatar
Fabio Hernandez committed
125 126
**NOTE**: *there are some known issues with some releases of the LSST software distributed via this channel, which may not work on all Linux distributions. See the [known issues](https://github.com/airnandez/lsst-cvmfs/issues) if you experience some problems and don't hesitate to provide feedback.*

Fabio Hernandez's avatar
Fabio Hernandez committed
127 128
## Available releases
At any moment, you can see what releases are available for Linux-based machines by inspecting the directory:
129

Fabio Hernandez's avatar
Fabio Hernandez committed
130
	ls -l /cvmfs/lsst.in2p3.fr/software/linux-x86_64
131

Fabio Hernandez's avatar
Fabio Hernandez committed
132
You can list the releases available for OS X with the command:
133

Fabio Hernandez's avatar
Fabio Hernandez committed
134
	ls -l /cvmfs/lsst.in2p3.fr/software/darwin-x86_64
135

136 137 138
Currently you will find the releases presented in the table below:

| Platform                | Available versions of LSST software |
Fabio Hernandez's avatar
Fabio Hernandez committed
139
| ---------------------   | ----------------------------------- |
Fabio Hernandez's avatar
Fabio Hernandez committed
140 141
| Linux, x86_64, 64bits   |   `v10.1`, `v11.0`, `v12.0`, `v12.1`, `v13.0`, `v14.0`, `v15.0` |
| Darwin, x86_64, 64bits  |   `v10.1`, `v11.0`, `v12.0`, `v12.1`, `v14.0`, `v15.0` |
142

143
For details on the platform each binary release was built on please refer to the `README` file in the corresponding directory.
144

Fabio Hernandez's avatar
Fabio Hernandez committed
145
## Advanced usage
146 147
Details on how to use this distribution mechanism for more advanced use cases are provided in the [Advanced Usage](AdvancedUsage.md) document. There you will find details on how you can develop your own software package which depends on other packages already present in the binary distribution.

Fabio Hernandez's avatar
Fabio Hernandez committed
148

Fabio Hernandez's avatar
Fabio Hernandez committed
149
## Troubleshooting
Fabio Hernandez's avatar
Fabio Hernandez committed
150 151
Please note that in order for this distribution mechanism to work for you, you need your machine to be connected to the network and able to contact CC-IN2P3 server. To check this is the case please do:

152
	$ curl --proxy http://cclsstsqd.in2p3.fr:3128 --head http://cclssts1.in2p3.fr/cvmfs/lsst.in2p3.fr/.cvmfspublished
153

Fabio Hernandez's avatar
Fabio Hernandez committed
154 155
You should see a line containning `HTTP/1.0 200 OK` which indicates that your machine can talk to the relevant server.

Fabio Hernandez's avatar
Fabio Hernandez committed
156 157
Please also refer to the [known issues](https://github.com/airnandez/lsst-cvmfs/issues).

Fabio Hernandez's avatar
Fabio Hernandez committed
158
## Frequently Asked Questions
159

Fabio Hernandez's avatar
Fabio Hernandez committed
160 161 162 163
* **How can I provide feedback?**

  Your feedback is very welcome. Please feel free to [open an issue](https://github.com/airnandez/lsst-cvmfs/issues).

164
*  **Where can I get more detailed information on CernVM-FS?**
Fabio Hernandez's avatar
Fabio Hernandez committed
165

166
	The [CernVM-FS downloads page](http://cernvm.cern.ch/portal/filesystem/downloads) contains additional information. In addition, you may want to read the [CernVM-FS Technical Information](http://cernvm.cern.ch/portal/filesystem/techinformation) for more in-depth information on how CernVM-FS works.
Fabio Hernandez's avatar
Fabio Hernandez committed
167 168 169

* **Can I use my remote LSST software distribution while disconnected from the network?**

170
  The CernVM-FS client caches all the file metadata and the contents of the accessed remote files in the local disk of your computer. If you have previously used the stack it is likely that the relevant files are locally available in your local disk, in which case, you may work while disconnected. However, we have not tested this thoroughly, so let us know how it works for you.
171

172 173 174
* **Can I use this for my Docker containers?**

  Yes, you can configure your container for automatically mounting a read-only file system with LSST software stack ready to use. Sébastien Binet did exactly this, so you can just use as is or as a baseline for your own containers. You will find all the details [here](https://github.com/hepsw/docks/tree/master/cvmfs-lsst).
Fabio Hernandez's avatar
Fabio Hernandez committed
175

Fabio Hernandez's avatar
Fabio Hernandez committed
176
## Credits
177
This work was done by Fabio Hernandez from [IN2P3/CNRS computing center](http://cc.inp3.fr) (Lyon, France) with very valuable help from Vanessa Hamar who set up the CernVM-FS server and proxy infrastructure.