README.md 7.27 KB
Newer Older
Enrique Garcia's avatar
Enrique Garcia committed
1
# ZenodoCI
2 3
[![pipeline status](https://gitlab.in2p3.fr/escape2020/wp3/zenodoci/badges/master/pipeline.svg)](
https://gitlab.in2p3.fr/escape2020/wp3/zenodoci/-/commits/master)
Enrique Garcia's avatar
Enrique Garcia committed
4
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
Enrique Garcia's avatar
Enrique Garcia committed
5

Enrique Garcia's avatar
Enrique Garcia committed
6 7 8 9


Library to manage an upload to Zenodo through its REST API.

Enrique Garcia's avatar
Enrique Garcia committed
10 11 12 13 14 15 16 17
## Install

```sh
  $ git clone https://gitlab.in2p3.fr/escape2020/wp3/zenodoci.git
  $ cd zenodoci
  $ pip install .
``` 

18 19 20 21 22 23 24 25 26 27
Then you would be able to run any of the following;
```sh
  $ upload_new_deposit --token $ZENODO_TOKEN --sandbox False --input_dir <./path_proj_dir>
```
```sh
  $ upload_new_version_deposit -t $ZENODO_TOKEN -s False -i <./your_proj_dir> -id <ZENODO_PROJ_ID>
```
```sh
  $ test_connection_zenodo -t $ZENODO_TOKEN -s False --repo_dir <PROJ_DIR>
```
Enrique Garcia's avatar
Enrique Garcia committed
28

Enrique Garcia's avatar
Enrique Garcia committed
29 30 31 32
Please note that all the code developed in this project, together with the [codemeta2zenodo crosswalk](
https://gitlab.in2p3.fr/escape2020/wp3/codemeta2zenodo) are included and installed into a Docker container that can be 
found at [the container registry](https://gitlab.in2p3.fr/escape2020/wp3/zenodoci/container_registry) of this project.

Enrique Garcia's avatar
Enrique Garcia committed
33
**Fell free to `pull` it and use at any moment**.
Enrique Garcia's avatar
Enrique Garcia committed
34

Enrique Garcia's avatar
Enrique Garcia committed
35
## _Quickstart_ - Deploy your project to Zenodo (`one-click-build-and-publish`). 
36

37
1. Include a `codementa.json` metadata file to your project.
Enrique Garcia's avatar
Enrique Garcia committed
38
    - [CodeMeta generator](https://codemeta.github.io/codemeta-generator/).
39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107
    - The ZenodoCI pipeline will not work without one.
2. Create a token at [zenodo](https://zenodo.org/) and add it as an environment variable to your project (see 
[below](#zenodo-token-gitlab-ci-environment-variable)).
3. Add in your `.gitlab-ci.yml` file the following stage to deploy your project into Zenodo.
4. Create a release.

The code to be added to the `.gitlab-ci.yml` file is the following (**COMMON FOR EVERY PROJECT**).

### To create a new entry or deposit
```
stages:
    - `(...) all your CI stages (...)`
    - deploy_to_zenodo

`(...) All the code to be run in your CI pipeline (...)`

deploy:
    stage: deploy_to_zenodo
    image: gitlab-registry.in2p3.fr/escape2020/wp3/zenodoci
    before_script:
      - wget -q https://gitlab.in2p3.fr/escape2020/wp3/zenodoci/-/raw/master/zenodoci/parse_last_release.sh
      - test_connection_zenodo --token $ZENODO_TOKEN --sandbox False -p $CI_PROJECT_DIR
    script:
      - mkdir -p build
      - /bin/bash parse_last_release.sh $CI_PROJECT_NAME $CI_PROJECT_URL
      - ls ./build

      - upload_new_deposit --token $SANDBOX_ZENODO_TOKEN --sandbox True --input-dir ./build
    only:
      - tags
```

### To create a new version of an existing entry 
Substitute
```
      - upload_new_version_deposit -t $ZENODO_TOKEN -s True -i ./build -id $ZENODO_PROJECT_ID
```
for 
```
      - upload_new_version_deposit -t $ZENODO_TOKEN -s True -i ./build -id $ZENODO_PROJECT_ID
```
and please check the correct indentation on the file.

## Zenodo token & GitLab CI environment variable

To allow GitLab communicate with Zenodo through their APIs, a personal access token must be created and included into 
the GitLab project (as a **masked !** environment variable).

Please note that (to date) the token can be assigned **only to a single** Zenodo account. To create the token:
 - Go to [zenodo.org](https://zenodo.org/).
 - Click on `Account` --> `Applications` --> `Personal access token` --> `New token` and Select the desired `Scopes`. 
 
This token will be passed later in the deployment stage of the ZenodoCI pipeline (the same that you should also include 
into your project to automatise the upload). 

For not sharing publicly your personal token, you should create an environment variable in your GitLab repository. 
This way, the token could be used as a variable without revealing its value. To create an an environment variable:
  - Go to your GitLab project.
  - Click on `Settings` --> `CI/CD` --> `Variables` --> `Add variable` --> Fill the fields --> **Mask your variable(s) !!**

Please name your environment variables as follows (so that no changes should be done to the `.gitlab-ci.yml` file);
  - `ZENODO_TOKEN` or `SANDBOX_ZENODO_TOKEN`
  - `ZENODO_PROJECT_ID`
  
so that environment variable(s) should look like:

```sh
    $ upload_new_deposit --token $ZENODO_TOKEN --sandbox False --input-directory <./build> 
    $ upload_new_version_deposit -t $ZENODO_TOKEN -s False -i <./your_proj_dir> -id $ZENODO_PROJECT_ID
108
```
Enrique Garcia's avatar
Enrique Garcia committed
109

110 111 112 113 114 115 116
# Contact

[Contact us](mailto:garcia@lapp.in2p3.fr) for any question or doubt related with the current library, the ESCAPE project, 
the metadata schema context or the integration of any project to Zenodo, Gitlab or the ESCAPE OSSR.


<!-- COMMENT
Enrique Garcia's avatar
Enrique Garcia committed
117
## Continuous Integration to Zenodo
Enrique Garcia's avatar
Enrique Garcia committed
118

119
The library was developed specifically to perform a deploy stage (to the Zenodo repository) within a GitLab CI 
Enrique Garcia's avatar
Enrique Garcia committed
120
pipeline that **could be implemented in any external project**. 
121 122

It provides all the needed Scripts to handle the upload of the desired 
123
file(s) to the [ESCAPE2020 Zenodo community](https://zenodo.org/communities/escape2020/). 
Enrique Garcia's avatar
Enrique Garcia committed
124 125
Please **only upload stable versions/releases of source code and/or image containers!**

126 127 128 129 130
A `codemeta.json` metadata file (see below) **MUST BE ADDED** before uploading an entry to Zenodo or triggering the
 GitLabCI pipeline.

The `deploy` stage in the CI pipeline (see the `.gitlab-ci.yml` file) will make use of the `zenodoapi` library ( as well
as the built Singularity container created in the previous CI stage check [the ESCAPE project template](
131
https://gitlab.in2p3.fr/escape2020/wp3/template_project_escape) 
Enrique Garcia's avatar
Enrique Garcia committed
132 133
 ) to:
 - Either upload the desired file(s) to the ESCAPE community in Zenodo.
134
 - Either upload a new version of an existing entry to Zenodo. 
135 136 137 138
 
Also, depending on the case, the corresponding python script (`upload_new_deposit.py` or `upload_new_version_of_deposit.py`) 
must be adapted and included into the `.gitlab-ci.yml` file with its corresponding arguments (examples are shown in the yml file). 

Enrique Garcia's avatar
Enrique Garcia committed
139
 - **You need a more detailed tutorial ?** Please check our more detailed tutorials at the [**OSSR entry point site**](
140 141
https://escape2020.pages.in2p3.fr/wp3/ossr-pages/page/repository/publish_in_repository/).

Enrique Garcia's avatar
Enrique Garcia committed
142
 - **Problems creating a `.gitlab-ci.yml` file ?** Please check the `gitlab_generator` module implemented in [**this ESCAPE project**](
143 144 145 146
https://gitlab.in2p3.fr/escape2020/wp3/template_project_escape/-/tree/master/gitlabci_generator).
 
#### **Use of new metadata context ! Please check the news !**

Enrique Garcia's avatar
Enrique Garcia committed
147
    We are no longer supporting the use of a `repository_information.json` file to provide metadata to Zenodo.
148 149 150 151 152 153 154 155

We are currently moving to a [CodeMeta metadata context](https://codemeta.github.io/).
 This metadata standard provides a complete metadata schema context supported by many other services and search engines.   

Adding a single `codemeta.json` file to the root directory of your project will be enough ! Please check out the
[ESCAPE metadata template](https://gitlab.in2p3.fr/escape2020/wp3/escape_metadata_template) project for a _quickstart_ on
how to easily create a `codemeta.json` file. 

Enrique Garcia's avatar
Enrique Garcia committed
156
Last but not least ! Please note that during the CI pipeline the `zenodoci` module will search for a `codemeta.json` file
157 158
and will automatically create the equivalent file to provide metadata to the Zenodo repository. The `.zenodo.json` file
will contain the exactly same information that in the `codemeta.json` file but using the Zenodo syntax. 
159
-->