Commit bd6d3953 authored by Vuillaume's avatar Vuillaume Committed by Enrique Garcia
Browse files

Deleted .singularityci

parent 8f89b62e
BSD 3-Clause License
Copyright (c) 2018, Vanessa Sochat
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
# README _.singularityci_ module
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3834833.svg)](https://doi.org/10.5281/zenodo.3834833)
**PLEASE HAVE A LOOK TO THE LICENSING SECTION BELOW BEFORE IMPLEMENTING ANY PART OF THIS CODE INTO YOURS !!**
## Building Singularity Containers using the continuous integration in GitLab
The source code contained in this folder is based on the following
[GitLab-CI project](https://gitlab.com/singularityhub/gitlab-ci). You can find a very didactic **tutorial** of how
to implement this code [here](https://vsoch.github.io/2018/gitlab-singularity-ci/).
This software (composed of the files within this directory **AND** part of the the `.gitlab-ci.yml` file in the
root directory - the `build_singularity_image` stage) provide the necessary tools to:
- Use a container image stored in the DockerHub site to create a Singularity container of your repository
(thus no need of having installed the Singularity source code), and upload it directly to your GitLab repository.
- The Singularity recipe must be provided, of course.
- In case the container is too large (10 Gb total storage limit for a single GitLab project), you must pass
through a cloud service (various examples of different services are shown in the tutorial and the original repository).
## License of the `template_project_repository`:
The `template_project_repository` contains code from different projects. This 'mixing' can be done because:
- Both 'parent' projects are Open Source.
- None of the original licenses are copy-left.
- Both `BSD 3-Clause` and `MIT`, are permissive licenses. All of these points mean that source code distributed with
a BSD 3-Clause license can be included in a project with a MIT license - and of course be re-distributed under `MIT`:
- Note however, that the **original licenses** must be included in the resulting project and mentioned in the
LICENSE/documentation files.
**PLEASE TAKE THE TIME TO CHECK AND VERIFY LICENSES AND THEIR COMPATIBILITIES**
\ No newline at end of file
#!/bin/bash
# This source code is based on https://gitlab.com/singularityhub/gitlab-ci,
# with DOI: https://doi.org/10.5281/zenodo.3834833
# provided under the following license:
### BSD 3-Clause License
###
### Copyright (c) 2018, Vanessa Sochat
### All rights reserved.
###
### Redistribution and use in source and binary forms, with or without
### modification, are permitted provided that the following conditions are met:
###
### * Redistributions of source code must retain the above copyright notice, this
### list of conditions and the following disclaimer.
###
### * Redistributions in binary form must reproduce the above copyright notice,
### this list of conditions and the following disclaimer in the documentation
### and/or other materials provided with the distribution.
###
### * Neither the name of the copyright holder nor the names of its
### contributors may be used to endorse or promote products derived from
### this software without specific prior written permission.
###
### THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
### AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
### IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
### DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
### FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
### DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
### SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
### CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
### OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
### OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
###
# build.sh will build a Singularity container. It's not overly complicated.
#
# USAGE: build.sh --uri collection-name/container-name --cli registry Singularity
# build.sh --uri collection-name/container-name --cli registry
# build.sh Singularity
# Copyright (C) 2017-2018 Vanessa Sochat.
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or (at your
# option) any later version.
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public
# License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
set -o errexit
set -o nounset
function usage() {
echo "USAGE: build [recipe] [options]"
echo ""
echo "OPTIONS:
Image Format
--uri -u if uploading, a uri to give to sregistry
--cli -c the sregistry client to use (if uploading)
--help -h show this help and exit
"
}
# --- Option processing --------------------------------------------------------
uri=""
cli=""
tag=""
while true; do
case ${1:-} in
-h|--help|help)
usage
exit 0
;;
-u|--uri)
shift
uri="${1:-}"
shift
;;
-t|--tag)
shift
tag="${1:-}"
shift
;;
-c|--cli)
shift
cli="${1:-}"
shift
;;
\?) printf "illegal option: -%s\n" "${1:-}" >&2
usage
exit 1
;;
-*)
printf "illegal option: -%s\n" "${1:-}" >&2
usage
exit 1
;;
*)
break;
;;
esac
done
################################################################################
### Recipe File ################################################################
################################################################################
if [ $# == 0 ] ; then
recipe="Singularity"
else
recipe=$1
fi
echo ""
echo "Image Recipe: ${recipe}"
################################################################################
### Storage Client #############################################################
################################################################################
is_valid_client () {
local e match="$1"
shift
for e; do [[ "$e" == "$match" ]] && return 0; done
return 1
}
# Test if client is valid
clients=("google-storage" "registry" "globus" "dropbox" "google-drive")
if [ "${cli}" != "" ]; then
is_valid_client "${cli}" "${clients[@]}"
if [ $? -ne 0 ]; then
echo "${cli} is not a valid choice! Choose from ${clients[@]}";
exit 1
fi
echo "Storage Client: ${cli}"
else
echo "Storage Client: none"
fi
################################################################################
### Build! #####################################################################
################################################################################
# Continue if the image recipe is found
if [ -f "$recipe" ]; then
imagefile="${recipe}.simg"
echo "Creating $imagefile using $recipe..."
singularity build $imagefile $recipe
# If the image is successfully built, test it and upload (examples)
if [ -f "${imagefile}" ]; then
# Example testing using run (you could also use test command)
echo "Testing the image... Marco!"
singularity exec $imagefile echo "Polo!"
# Example sregistry commands to push to endpoints
if [ "${cli}" != "" ]; then
# If the uri isn't provided, he gets a robot name
if [ "${uri}" == "" ]; then
uri=$(python -c "from sregistry.logger.namer import RobotNamer; bot=RobotNamer(); print(bot.generate())")
fi
# If a tag is provided, add to uri
if [ "${tag}" != "" ]; then
uri="${uri}:${tag}"
fi
echo "Pushing ${uri} to ${cli}://"
echo "SREGISTRY_CLIENT=${cli} sregistry push --name ${uri} ${imagefile}"
SREGISTRY_CLIENT="${cli}" sregistry push --name "${uri}" "${imagefile}"
else
echo "Skipping upload. Image $imagefile is finished!"
fi
fi
else
echo "Singularity recipe ${recipe} not found!"
exit 1
fi
#!/bin/bash
# This source code is based on https://gitlab.com/singularityhub/gitlab-ci,
# with DOI: https://doi.org/10.5281/zenodo.3834833
# provided under the following license:
### BSD 3-Clause License
###
### Copyright (c) 2018, Vanessa Sochat
### All rights reserved.
###
### Redistribution and use in source and binary forms, with or without
### modification, are permitted provided that the following conditions are met:
###
### * Redistributions of source code must retain the above copyright notice, this
### list of conditions and the following disclaimer.
###
### * Redistributions in binary form must reproduce the above copyright notice,
### this list of conditions and the following disclaimer in the documentation
### and/or other materials provided with the distribution.
###
### * Neither the name of the copyright holder nor the names of its
### contributors may be used to endorse or promote products derived from
### this software without specific prior written permission.
###
### THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
### AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
### IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
### DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
### FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
### DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
### SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
### CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
### OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
### OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
###
#
# Installation Singularity and sregistry
apt-get update && apt-get install -y wget git \
build-essential \
squashfs-tools \
libtool \
autotools-dev \
libarchive-dev \
automake \
autoconf \
uuid-dev \
libssl-dev
sed -i -e 's/^Defaults\tsecure_path.*$//' /etc/sudoers
# Check Python
echo "Python Version:"
python --version
pip install sregistry[all]
sregistry version
echo "sregistry Version:"
# Install Singularity
cd /tmp && \
git clone -b vault/release-2.5 https://www.github.com/sylabs/singularity.git
cd singularity && \
./autogen.sh && \
./configure --prefix=/usr/local && \
make && make install
......@@ -12,6 +12,22 @@ https://mybinder.org/v2/gh/https%3A%2F%2Fgitlab.in2p3.fr%2Fescape2020%2Fwp3%2Fte
<img src="https://cdn.eso.org/images/large/ann18084a.jpg" width="640" height="453"/>
</p>
# Content
- [About this project](#about-this-project)
- [Licensing](#license)
- [Contribute to the ESCAPE OSSR](#contribute-to-the-escape-ossr)
- [Install](#install)
- [Report an issue or ask a question](#report-an-issue--ask-a-question)
- [Contact](#contact)
- [Citing this project](#citing)
- [Examples contained in this project](#examples-contained-in-the-project)
- [Automatise building Singularity images using CI/CD](#1-how-to-automatise-the-building-of-a-singularity-image-and-upload-it-to-zenodo-using-the-gitlab-ci)
- [Automatise building Docker containers using CI/CD](#2-how-to-automatise-the-building-of-a-docker-container-and-upload-it-to-the-gitlab-container-registry)
# About this project
A simple template project to provide software to ESCAPE.
This repository shows the **basic content** that should be included in a project (following the
......@@ -42,6 +58,26 @@ It would be highly suitable to include too:
RENATER), you will need to [add a SSH key](https://gitlab.in2p3.fr/help/ssh/README#generating-a-new-ssh-key-pair) to
your GitLab profile if you want to 'push' your changes to the server.
# License
Portions of this code are based on the `GitLab-CI` project ([repository link](https://gitlab.com/singularityhub/gitlab-ci),
[Zenodo link and DOI](https://zenodo.org/record/3834833)) held by a
[`BSD 3-Clause License`](https://opensource.org/licenses/BSD-3-Clause). You can check the complete license text
[here](https://gitlab.com/singularityhub/gitlab-ci/-/blob/master/LICENSE).
## License of the `template_project_repository`:
The `template_project_repository` contains code from a third party project. This 'mixing' can be done because:
- Both 'parent' projects are Open Source.
- None of the original licenses are copy-left.
- Both `BSD 3-Clause` and `MIT`, are permissive licenses. This means that source code distributed with
a BSD 3-Clause license can be included in a project with a MIT license and be re-distributed under `MIT`:
- Note however, that the **original licenses** must be included in the resulting project and mentioned in the
LICENSE/documentation files.
**PLEASE TAKE THE TIME TO CHECK AND VERIFY LICENSES AND THEIR COMPATIBILITIES**
# Contribute to the ESCAPE OSSR
If you want to provide software to the ESCAPE repository:
......@@ -55,25 +91,6 @@ If you want to provide software to the ESCAPE repository:
- Check the following [tutorial on how to publish content in Zenodo](https://escape2020.pages.in2p3.fr/wp3/ossr-pages/page/contribute/publish_tutorial/),
and how to automatise the upload of each new release of your project.
# This project also includes
## 1. How to automatise the building of a Singularity image and upload it to Zenodo using the GitLab-CI
A working example of how to automatise the GitLab-CI to;
1. create a Singularity image / container of your code,
2. make it available as a downloadable artifact within your project and
3. upload it to the [ESCAPE OSSR](https://zenodo.org/communities/escape2020),
can be found in the `.singularityci`, and `Singularity` directories and in the `.gitlab-ci.yml` file - the
`build_singularity_image` stage. Please read carefully all the README files.
For an easy example of how to create a Singularity receipt from scratch (and its corresponding container when executed),
please have a look to the `singularity_utils` directory.
## 2. How to automatise the building of a Docker container and upload it to the GitLab Container Registry
An example can be found in the `Docker` directory and in the `.gitlab-ci.yml` file - the
`build_docker_image` stage.
# Install
Example of how to show installing instructions (and indeed the way to install this project).
......@@ -84,6 +101,12 @@ Example of how to show installing instructions (and indeed the way to install th
$ pip install .
```
# Report an issue / Ask a question
Use the [GitLab repository Issues](https://gitlab.in2p3.fr/escape2020/wp3/template_project_escape/-/issues).
# Contact
Email to vuillaume [at] lapp.in2p3.fr / garcia [at] lapp.in2p3.fr.
# Citing
Example of citing (as well as the DOI to cite this project),
......@@ -97,13 +120,23 @@ In case of citing this repository, use the following DOI:
Do not forget to include your code / container into the [Zenodo ESCAPE community](https://zenodo.org/communities/escape2020/).
- ***Note that*** a DOI will be assigned in the moment create a new record/entry in Zenodo.
# License
Please check the licenses of the code within the `.singularityci` directory before adding this template
to your project.
# Examples contained in this project
# Report an issue / Ask a question
Use the [GitLab repository Issues](https://gitlab.in2p3.fr/escape2020/wp3/template_project_escape/-/issues).
## 1. How to automatise the building of a Singularity image and upload it to Zenodo using the GitLab-CI
# Contact
Email to vuillaume [at] lapp.in2p3.fr / garcia [at] lapp.in2p3.fr.
A working example of how to automatise the GitLab-CI to;
1. create a Singularity image / container of your code,
2. make it available as a downloadable artifact within your project and
3. upload it to the [ESCAPE OSSR](https://zenodo.org/communities/escape2020),
can be found in the `.singularityci`, and `Singularity` directories and in the `.gitlab-ci.yml` file - the
`build_singularity_image` stage. Please read carefully all the README files.
For an easy example of how to create a Singularity receipt from scratch (and its corresponding container when executed),
please have a look to the `singularity_utils` directory.
## 2. How to automatise the building of a Docker container and upload it to the GitLab Container Registry
An example can be found in the `Docker` directory and in the `.gitlab-ci.yml` file - the
`build_docker_image` stage.
......@@ -14,7 +14,7 @@
"installUrl": "https://gitlab.in2p3.fr/escape2020/wp3/template_project_escape/-/blob/master/setup.py",
"releaseNotes": "Update Jupyter notebook with an example related with the repository.",
"dateCreated": "2019-11-05",
"datePublished": "2022-01-25",
"datePublished": "2022-01-27",
"dateModified": "2021-05-25",
"isAccessibleForFree": true,
"isPartOf": [
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment