README.md 7.5 KB
Newer Older
Enrique Garcia's avatar
Enrique Garcia committed
1
# ZenodoCI
2
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.4786641.svg)](https://doi.org/10.5281/zenodo.4786641)
3 4
[![pipeline status](https://gitlab.in2p3.fr/escape2020/wp3/zenodoci/badges/master/pipeline.svg)](
https://gitlab.in2p3.fr/escape2020/wp3/zenodoci/-/commits/master)
Enrique Garcia's avatar
Enrique Garcia committed
5
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
6 7
[![coverage report](https://gitlab.in2p3.fr/escape2020/wp3/zenodoci/badges/master/coverage.svg)](
https://gitlab.in2p3.fr/escape2020/wp3/zenodoci/-/commits/master)
Enrique Garcia's avatar
Enrique Garcia committed
8 9 10 11


Library to manage an upload to Zenodo through its REST API.

Enrique Garcia's avatar
Enrique Garcia committed
12 13 14 15 16 17 18 19
## Install

```sh
  $ git clone https://gitlab.in2p3.fr/escape2020/wp3/zenodoci.git
  $ cd zenodoci
  $ pip install .
``` 

Enrique Garcia's avatar
Enrique Garcia committed
20 21 22 23 24 25 26 27 28 29
Then you would be able to run any of the following;
```sh
  $ upload_new_deposit --token $ZENODO_TOKEN --sandbox False --input_dir <./path_proj_dir>
```
```sh
  $ upload_new_version_deposit -t $ZENODO_TOKEN -s False -i <./your_proj_dir> -id <ZENODO_PROJ_ID>
```
```sh
  $ test_connection_zenodo -t $ZENODO_TOKEN -s False --repo_dir <PROJ_DIR>
```
Enrique Garcia's avatar
Enrique Garcia committed
30

Enrique Garcia's avatar
Enrique Garcia committed
31 32
-----------

Enrique Garcia's avatar
Enrique Garcia committed
33 34 35
Please note that all the code developed in this project, together with the [codemeta2zenodo crosswalk](
https://gitlab.in2p3.fr/escape2020/wp3/codemeta2zenodo) are included and installed into a Docker container that can be 
found at [the container registry](https://gitlab.in2p3.fr/escape2020/wp3/zenodoci/container_registry) of this project.
36

Enrique Garcia's avatar
Enrique Garcia committed
37 38 39 40 41
**Fell free to `pull` it and use at any moment**.

## _Quickstart_ - Deploy your project to Zenodo (`one-click-build-and-publish`). 

1. Include a `codementa.json` metadata file to your project.
Enrique Garcia's avatar
Enrique Garcia committed
42
    - [CodeMeta generator](https://codemeta.github.io/codemeta-generator/).
Enrique Garcia's avatar
Enrique Garcia committed
43 44 45 46 47 48 49 50 51 52 53 54
    - The ZenodoCI pipeline will not work without one.
2. Create a token at [zenodo](https://zenodo.org/) and add it as an environment variable to your project (see 
[below](#zenodo-token-gitlab-ci-environment-variable)).
3. Add in your `.gitlab-ci.yml` file the following stage to deploy your project into Zenodo.
4. Create a release.

The code to be added to the `.gitlab-ci.yml` file is the following (**COMMON FOR EVERY PROJECT**).

### To create a new entry or deposit
```
stages:
    - `(...) all your CI stages (...)`
Enrique Garcia's avatar
Enrique Garcia committed
55
    - deploy
Enrique Garcia's avatar
Enrique Garcia committed
56 57 58

`(...) All the code to be run in your CI pipeline (...)`

Enrique Garcia's avatar
Enrique Garcia committed
59 60
deploy_to_zenodo:
    stage: deploy
Enrique Garcia's avatar
Enrique Garcia committed
61
    image: gitlab-registry.in2p3.fr/escape2020/wp3/zenodoci:v1.2
Enrique Garcia's avatar
Enrique Garcia committed
62 63 64 65
    before_script:
      - test_connection_zenodo --token $ZENODO_TOKEN --sandbox False -p $CI_PROJECT_DIR
    script:
      - mkdir -p build
Enrique Garcia's avatar
Enrique Garcia committed
66
      - parse_last_release_git.sh $CI_PROJECT_NAME $CI_PROJECT_URL
Enrique Garcia's avatar
Enrique Garcia committed
67
      - if [[ -f ./codemeta.json ]]; then cp ./codemeta.json ./build; fi
Enrique Garcia's avatar
Enrique Garcia committed
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111
      - ls ./build

      - upload_new_deposit --token $SANDBOX_ZENODO_TOKEN --sandbox True --input-dir ./build
    only:
      - tags
```

### To create a new version of an existing entry 
Substitute
```
      - upload_new_version_deposit -t $ZENODO_TOKEN -s True -i ./build -id $ZENODO_PROJECT_ID
```
for 
```
      - upload_new_version_deposit -t $ZENODO_TOKEN -s True -i ./build -id $ZENODO_PROJECT_ID
```
and please check the correct indentation on the file.

## Zenodo token & GitLab CI environment variable

To allow GitLab communicate with Zenodo through their APIs, a personal access token must be created and included into 
the GitLab project (as a **masked !** environment variable).

Please note that (to date) the token can be assigned **only to a single** Zenodo account. To create the token:
 - Go to [zenodo.org](https://zenodo.org/).
 - Click on `Account` --> `Applications` --> `Personal access token` --> `New token` and Select the desired `Scopes`. 
 
This token will be passed later in the deployment stage of the ZenodoCI pipeline (the same that you should also include 
into your project to automatise the upload). 

For not sharing publicly your personal token, you should create an environment variable in your GitLab repository. 
This way, the token could be used as a variable without revealing its value. To create an an environment variable:
  - Go to your GitLab project.
  - Click on `Settings` --> `CI/CD` --> `Variables` --> `Add variable` --> Fill the fields --> **Mask your variable(s) !!**

Please name your environment variables as follows (so that no changes should be done to the `.gitlab-ci.yml` file);
  - `ZENODO_TOKEN` or `SANDBOX_ZENODO_TOKEN`
  - `ZENODO_PROJECT_ID`
  
so that environment variable(s) should look like:

```sh
    $ upload_new_deposit --token $ZENODO_TOKEN --sandbox False --input-directory <./build> 
    $ upload_new_version_deposit -t $ZENODO_TOKEN -s False -i <./your_proj_dir> -id $ZENODO_PROJECT_ID
112
```
Enrique Garcia's avatar
Enrique Garcia committed
113

Enrique Garcia's avatar
Enrique Garcia committed
114 115 116 117 118 119 120
# Contact

[Contact us](mailto:garcia@lapp.in2p3.fr) for any question or doubt related with the current library, the ESCAPE project, 
the metadata schema context or the integration of any project to Zenodo, Gitlab or the ESCAPE OSSR.


<!-- COMMENT
Enrique Garcia's avatar
Enrique Garcia committed
121
## Continuous Integration to Zenodo
Enrique Garcia's avatar
Enrique Garcia committed
122

123
The library was developed specifically to perform a deploy stage (to the Zenodo repository) within a GitLab CI 
Enrique Garcia's avatar
Enrique Garcia committed
124
pipeline that **could be implemented in any external project**. 
125 126

It provides all the needed Scripts to handle the upload of the desired 
127
file(s) to the [ESCAPE2020 Zenodo community](https://zenodo.org/communities/escape2020/). 
Enrique Garcia's avatar
Enrique Garcia committed
128 129
Please **only upload stable versions/releases of source code and/or image containers!**

130 131 132 133 134
A `codemeta.json` metadata file (see below) **MUST BE ADDED** before uploading an entry to Zenodo or triggering the
 GitLabCI pipeline.

The `deploy` stage in the CI pipeline (see the `.gitlab-ci.yml` file) will make use of the `zenodoapi` library ( as well
as the built Singularity container created in the previous CI stage check [the ESCAPE project template](
135
https://gitlab.in2p3.fr/escape2020/wp3/template_project_escape) 
Enrique Garcia's avatar
Enrique Garcia committed
136 137
 ) to:
 - Either upload the desired file(s) to the ESCAPE community in Zenodo.
138
 - Either upload a new version of an existing entry to Zenodo. 
139 140 141 142
 
Also, depending on the case, the corresponding python script (`upload_new_deposit.py` or `upload_new_version_of_deposit.py`) 
must be adapted and included into the `.gitlab-ci.yml` file with its corresponding arguments (examples are shown in the yml file). 

Enrique Garcia's avatar
Enrique Garcia committed
143
 - **You need a more detailed tutorial ?** Please check our more detailed tutorials at the [**OSSR entry point site**](
144 145
https://escape2020.pages.in2p3.fr/wp3/ossr-pages/page/repository/publish_in_repository/).

Enrique Garcia's avatar
Enrique Garcia committed
146
 - **Problems creating a `.gitlab-ci.yml` file ?** Please check the `gitlab_generator` module implemented in [**this ESCAPE project**](
147 148 149 150
https://gitlab.in2p3.fr/escape2020/wp3/template_project_escape/-/tree/master/gitlabci_generator).
 
#### **Use of new metadata context ! Please check the news !**

Enrique Garcia's avatar
Enrique Garcia committed
151
    We are no longer supporting the use of a `repository_information.json` file to provide metadata to Zenodo.
152 153 154 155 156 157 158 159

We are currently moving to a [CodeMeta metadata context](https://codemeta.github.io/).
 This metadata standard provides a complete metadata schema context supported by many other services and search engines.   

Adding a single `codemeta.json` file to the root directory of your project will be enough ! Please check out the
[ESCAPE metadata template](https://gitlab.in2p3.fr/escape2020/wp3/escape_metadata_template) project for a _quickstart_ on
how to easily create a `codemeta.json` file. 

Enrique Garcia's avatar
Enrique Garcia committed
160
Last but not least ! Please note that during the CI pipeline the `zenodoci` module will search for a `codemeta.json` file
161 162
and will automatically create the equivalent file to provide metadata to the Zenodo repository. The `.zenodo.json` file
will contain the exactly same information that in the `codemeta.json` file but using the Zenodo syntax. 
Enrique Garcia's avatar
Enrique Garcia committed
163
-->