diff --git a/codemeta.json b/codemeta.json
index 478f5e9a8fb117629764f888721badf574629f37..bc62fcf02a6f95879b3417699719e2073044f15a 100644
--- a/codemeta.json
+++ b/codemeta.json
@@ -38,7 +38,7 @@
"softwareVersion": ">=5.4.2"
}
],
- "keywords": [],
+ "keywords": ["jupyter-notebook"],
"runtimePlatform": "",
"downloadUrl": "",
"installUrl": "",
diff --git a/eossr/examples/notebooks/Harvest Zenodo - OAI-PMH and REST API.ipynb b/eossr/examples/notebooks/Harvest Zenodo - OAI-PMH and REST API.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..1bf491532777534c48911691a28e22f177bfa0a6
--- /dev/null
+++ b/eossr/examples/notebooks/Harvest Zenodo - OAI-PMH and REST API.ipynb
@@ -0,0 +1,919 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "b0fcf1bf",
+ "metadata": {},
+ "source": [
+ "<h1><center> <font size=\"36\"> How to harvest metadata from Zenodo </font> </center></h1>\n",
+ "\n",
+ "---------------------\n",
+ "#### Notebook outline \n",
+ " - Zenodo OAI-PMH protocol\n",
+ " - Zenodo REST API\n",
+ " - Explore the REST API answer (payload) with the `request` library\n",
+ " - Using `eossr` library\n",
+ " - Using `PyZenodo3` library\n",
+ " - Pros and cons of both methods\n",
+ " \n",
+ "---------------------"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2529eacc",
+ "metadata": {},
+ "source": [
+ "## TL;DR: Pros and cons of each method\n",
+ " - Using AOI-PMH for harvesting;\n",
+ " + $+$ More efficient harvest:\n",
+ " - faster,\n",
+ " - thought for large and continues queries of a repository.\n",
+ " - No token needed to harvest and fetch entries.\n",
+ " + $-$ Metadata representation of files is provided by the data provider.\n",
+ " - Using the REST API;\n",
+ " + $+$ Access to the full entry/record/community information.\n",
+ " + $-$ An [access token](https://zenodo.org/account/settings/applications/) is needed to communicate with the REST API.\n",
+ " + $-$ Harvest not optimised for large searches."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2193adc5",
+ "metadata": {},
+ "source": [
+ "## OAI-PMH protocol"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9bb7e516",
+ "metadata": {},
+ "source": [
+ "#### - First have a lookg to a nice [tutorial to the protocol](https://indico.cern.ch/event/5710/sessions/108048/attachments/988151/1405129/Simeon_tutorial.pdf)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1bcf7733",
+ "metadata": {},
+ "source": [
+ "The [OAI-PMH protocol](https://www.openarchives.org/pmh/) uses a base URL + special syntax ('verbs') to query and find metadata representation(s) of a data provider.\n",
+ "\n",
+ "In the case of zenodo the base URL is: https://zenodo.org/oai2d.\n",
+ "\n",
+ "For example; \n",
+ " - to retrieve all the entries (`verb=ListRecords`)\n",
+ " - belonging to escape2020 community (`set=user-escape2020`)\n",
+ " - in the OAI DataCite metadata representation (`metadataPrefix=oai_datacite`)\n",
+ " \n",
+ "https://zenodo.org/oai2d?verb=ListRecords&set=user-escape2020&metadataPrefix=oai_datacite\n",
+ "\n",
+ "\n",
+ "Ex2:\n",
+ " - To obtain a single entry (`verb=GetRecord`)\n",
+ " - of a certain zenodo record - identified by the entry_id (`identifier=oai:zenodo.org:4105896`)\n",
+ " - in the Dublin Core metadata representation (`metadataPrefix=oai_dc`)\n",
+ " \n",
+ "https://zenodo.org/oai2d?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai:zenodo.org:4105896"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "26eba497",
+ "metadata": {},
+ "source": [
+ "## Example with the OAI-PMH protocol: A python OAI-Harvester"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "145835bd",
+ "metadata": {},
+ "source": [
+ "```\n",
+ "pip install oaiharvest\n",
+ "oai-harvest -h\n",
+ "\n",
+ "# Examples of usage\n",
+ "oai-harvest https://zenodo.org/oai2d -s \"user-escape2020\" -d oai_dc\n",
+ "oai-harvest https://zenodo.org/oai2d -s \"user-escape2020\" -d oai_datacite4\n",
+ "oai-harvest https://zenodo.org/oai2d -s \"user-escape2020\" -d datacite3\n",
+ "\n",
+ "# Example of output\n",
+ "$ oai-harvest https://zenodo.org/oai2d -s \"user-escape2020\" -d datacite3\n",
+ "$ cd datacite3\n",
+ "$ ls\n",
+ "oai:zenodo.org:1689986.oai_dc.xml oai:zenodo.org:3884963.oai_dc.xml\n",
+ "oai:zenodo.org:2533132.oai_dc.xml oai:zenodo.org:3967386.oai_dc.xml\n",
+ "oai:zenodo.org:2542652.oai_dc.xml oai:zenodo.org:4012169.oai_dc.xml\n",
+ "oai:zenodo.org:2542664.oai_dc.xml oai:zenodo.org:4028908.oai_dc.xml\n",
+ "oai:zenodo.org:3356656.oai_dc.xml oai:zenodo.org:4044010.oai_dc.xml\n",
+ "oai:zenodo.org:3362435.oai_dc.xml oai:zenodo.org:4055176.oai_dc.xml\n",
+ "oai:zenodo.org:3572655.oai_dc.xml oai:zenodo.org:4105896.oai_dc.xml\n",
+ "oai:zenodo.org:3614662.oai_dc.xml oai:zenodo.org:4311271.oai_dc.xml\n",
+ "oai:zenodo.org:3659184.oai_dc.xml oai:zenodo.org:4419866.oai_dc.xml\n",
+ "oai:zenodo.org:3675081.oai_dc.xml oai:zenodo.org:4601451.oai_dc.xml\n",
+ "oai:zenodo.org:3734091.oai_dc.xml oai:zenodo.org:4687123.oai_dc.xml\n",
+ "oai:zenodo.org:3743489.oai_dc.xml oai:zenodo.org:4786641.oai_dc.xml\n",
+ "oai:zenodo.org:3743490.oai_dc.xml oai:zenodo.org:4790629.oai_dc.xml\n",
+ "oai:zenodo.org:3854976.oai_dc.xml\n",
+ "$ cat <FILE>\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f9ad3584",
+ "metadata": {},
+ "source": [
+ " No token is needed to fetch metadata files provided by Zenodo (the provider). However please note that the **metadata schema representation of the records is chosen by the provider !** \n",
+ " \n",
+ "Zenodo supports the following schema representations:\n",
+ " - `DataCite` (various version),\n",
+ " - `Dublin Core`,\n",
+ " - `MARC21`,\n",
+ " - However it **does not provide** metadata under the `codemeta.json` schema."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c6a47567",
+ "metadata": {},
+ "source": [
+ "# Query Zenodo's records through its REST API"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "26424a79",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# pip install request"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "e7a84906",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import requests"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 24,
+ "id": "076bfee8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "token = ''"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "87f186f9",
+ "metadata": {},
+ "source": [
+ "We would need to specify some arguments to reduce the search"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "5ee3a192",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "parameters = {'access_token': token,\n",
+ " 'communities': 'escape2020',\n",
+ " 'size':100}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4cd8011b",
+ "metadata": {},
+ "source": [
+ "## Example with the `requests` lib - How to recover all ESCAPE2020 community records ?"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "fde5ee19",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "dict_keys(['aggregations', 'hits', 'links'])"
+ ]
+ },
+ "execution_count": 5,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "escape2020 = requests.get('https://zenodo.org/api/records', params=parameters).json()\n",
+ "escape2020.keys()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9e564b25",
+ "metadata": {},
+ "source": [
+ "Let's explore the REST API payload to find the desired information."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "2506740f",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'access_right': {'buckets': [{'doc_count': 15, 'key': 'open'}],\n",
+ " 'doc_count_error_upper_bound': 0,\n",
+ " 'sum_other_doc_count': 0},\n",
+ " 'file_type': {'buckets': [{'doc_count': 7, 'key': 'zip'},\n",
+ " {'doc_count': 4, 'key': 'pdf'},\n",
+ " {'doc_count': 3, 'key': 'gz'},\n",
+ " {'doc_count': 2, 'key': 'json'},\n",
+ " {'doc_count': 1, 'key': ''},\n",
+ " {'doc_count': 1, 'key': 'md'},\n",
+ " {'doc_count': 1, 'key': 'simg'},\n",
+ " {'doc_count': 1, 'key': 'tar'}],\n",
+ " 'doc_count_error_upper_bound': 0,\n",
+ " 'sum_other_doc_count': 0},\n",
+ " 'keywords': {'buckets': [{'doc_count': 3, 'key': 'ESCAPE'},\n",
+ " {'doc_count': 2, 'key': 'CTA'},\n",
+ " {'doc_count': 2, 'key': 'python'},\n",
+ " {'doc_count': 1, 'key': 'AGN'},\n",
+ " {'doc_count': 1, 'key': 'EOSC'},\n",
+ " {'doc_count': 1,\n",
+ " 'key': 'European Open Science Cloud, ESFRI, e-Infrastructures'},\n",
+ " {'doc_count': 1, 'key': 'Horizon Europe'},\n",
+ " {'doc_count': 1, 'key': 'Interoperability'},\n",
+ " {'doc_count': 1, 'key': 'MWL'},\n",
+ " {'doc_count': 1,\n",
+ " 'key': 'Machine Learning, Big Data, Aapche Kafka, Gravitational Wave'}],\n",
+ " 'doc_count_error_upper_bound': 0,\n",
+ " 'sum_other_doc_count': 17},\n",
+ " 'type': {'buckets': [{'doc_count': 10,\n",
+ " 'key': 'software',\n",
+ " 'subtype': {'buckets': [],\n",
+ " 'doc_count_error_upper_bound': 0,\n",
+ " 'sum_other_doc_count': 0}},\n",
+ " {'doc_count': 3,\n",
+ " 'key': 'publication',\n",
+ " 'subtype': {'buckets': [{'doc_count': 1, 'key': 'other'},\n",
+ " {'doc_count': 1, 'key': 'report'},\n",
+ " {'doc_count': 1, 'key': 'workingpaper'}],\n",
+ " 'doc_count_error_upper_bound': 0,\n",
+ " 'sum_other_doc_count': 0}},\n",
+ " {'doc_count': 1,\n",
+ " 'key': 'lesson',\n",
+ " 'subtype': {'buckets': [],\n",
+ " 'doc_count_error_upper_bound': 0,\n",
+ " 'sum_other_doc_count': 0}},\n",
+ " {'doc_count': 1,\n",
+ " 'key': 'poster',\n",
+ " 'subtype': {'buckets': [],\n",
+ " 'doc_count_error_upper_bound': 0,\n",
+ " 'sum_other_doc_count': 0}}],\n",
+ " 'doc_count_error_upper_bound': 0,\n",
+ " 'sum_other_doc_count': 0}}"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Nice summary of the request we just made\n",
+ "escape2020['aggregations']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "dd36b4be",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "dict_keys(['hits', 'total'])\n",
+ "15\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Total number of entries in the payload\n",
+ "print(escape2020['hits'].keys())\n",
+ "print(escape2020['hits']['total'])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "0550cffb",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "all_entries = escape2020['hits']['hits']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "94a4e8c9",
+ "metadata": {
+ "scrolled": false
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'conceptdoi': '10.5281/zenodo.5176088',\n",
+ " 'conceptrecid': '5176088',\n",
+ " 'created': '2021-08-16T07:21:15.005975+00:00',\n",
+ " 'doi': '10.5281/zenodo.5176089',\n",
+ " 'files': [{'bucket': '409f3f8e-cb73-4a10-b718-3b8fc238a616',\n",
+ " 'checksum': 'md5:eab8fbaa4c318cbd75629eb6a7719ecb',\n",
+ " 'key': 'EOSC_SYMPOSIUM_2021_Report.pdf',\n",
+ " 'links': {'self': 'https://zenodo.org/api/files/409f3f8e-cb73-4a10-b718-3b8fc238a616/EOSC_SYMPOSIUM_2021_Report.pdf'},\n",
+ " 'size': 1654918,\n",
+ " 'type': 'pdf'}],\n",
+ " 'id': 5176089,\n",
+ " 'links': {'badge': 'https://zenodo.org/badge/doi/10.5281/zenodo.5176089.svg',\n",
+ " 'bucket': 'https://zenodo.org/api/files/409f3f8e-cb73-4a10-b718-3b8fc238a616',\n",
+ " 'conceptbadge': 'https://zenodo.org/badge/doi/10.5281/zenodo.5176088.svg',\n",
+ " 'conceptdoi': 'https://doi.org/10.5281/zenodo.5176088',\n",
+ " 'doi': 'https://doi.org/10.5281/zenodo.5176089',\n",
+ " 'html': 'https://zenodo.org/record/5176089',\n",
+ " 'latest': 'https://zenodo.org/api/records/5176089',\n",
+ " 'latest_html': 'https://zenodo.org/record/5176089',\n",
+ " 'self': 'https://zenodo.org/api/records/5176089'},\n",
+ " 'metadata': {'access_right': 'open',\n",
+ " 'access_right_category': 'success',\n",
+ " 'communities': [{'id': 'envri'},\n",
+ " {'id': 'eosc_synergy'},\n",
+ " {'id': 'eoscsecretariat'},\n",
+ " {'id': 'escape2020'},\n",
+ " {'id': 'expands'},\n",
+ " {'id': 'ni4os-europe'},\n",
+ " {'id': 'sshoc'}],\n",
+ " 'contributors': [{'affiliation': 'Trust-IT Services',\n",
+ " 'name': 'Ferguson, Nicholas',\n",
+ " 'orcid': '0000-0001-5523-6430',\n",
+ " 'type': 'WorkPackageLeader'}],\n",
+ " 'creators': [{'affiliation': 'Technopolis Group Belgium',\n",
+ " 'name': 'Bertacchini, Veronica'},\n",
+ " {'affiliation': 'Trust-IT Services',\n",
+ " 'name': 'Drago, Federico',\n",
+ " 'orcid': '0000-0002-1333-4478'},\n",
+ " {'affiliation': 'TU Wien',\n",
+ " 'name': 'Flicker, Katharina',\n",
+ " 'orcid': '0000-0001-6040-2798'},\n",
+ " {'affiliation': 'KIT', 'name': 'Gebreyesus, Netsanet'},\n",
+ " {'affiliation': 'GÉANT', 'name': 'Grant, Annabel'},\n",
+ " {'affiliation': 'CERN',\n",
+ " 'name': 'Jones, Bob',\n",
+ " 'orcid': '0000-0001-9092-4589'},\n",
+ " {'affiliation': 'CSC-IT Center for Science', 'name': 'Liinamaa, Iiris'},\n",
+ " {'affiliation': 'CSC-IT Center for Science', 'name': 'Märkälä, Anu'},\n",
+ " {'affiliation': 'Athena Research Center',\n",
+ " 'name': 'Marinos-Kouris, Christos'},\n",
+ " {'affiliation': 'GO FAIR Foundation',\n",
+ " 'name': 'Meerman, Bert',\n",
+ " 'orcid': '0000-0002-0071-2660'},\n",
+ " {'affiliation': 'TU Wien',\n",
+ " 'name': 'Saurugger, Bernd',\n",
+ " 'orcid': '0000-0001-5730-3983'},\n",
+ " {'affiliation': 'Trust-IT Services',\n",
+ " 'name': 'Smith, Zachary',\n",
+ " 'orcid': '0000-0002-9984-008X'}],\n",
+ " 'description': '<p>The EOSC Symposium 2021 provided a key engagement opportunity for the EOSC community after the European Open Science Cloud finally entered its highly-anticipated implementation phase in 2021. Delivered online to just under 1,000 EOSC stakeholders from over 63 different countries, this was not only the largest EOSC Symposium yet, but it was also an essential opportunity for convergence and alignment on principles and priorities.</p>\\n\\n<p>The EOSC Association will play an important role in this phase. With already over 210 member and observer organisations from across Europe, the Association represents a single voice for the advocacy and representation of the broader EOSC Stakeholder community in Europe, promoting alignment of EU research policy and priorities.</p>\\n\\n<p>The Association will continuously develop the EOSC Strategic Research and Innovation Agenda (SRIA) which will influence future EOSC activities at institutional, national and EU level (including the EOSC-related work programmes in Horizon Europe). This living document will adapt to the changing EOSC ecosystem and the needs of EOSC stakeholders. The Association is setting up a series of Advisory Groups (AG) with Task Forces (TF) to engage with the EOSC community around priority areas, namely:</p>\\n\\n<ul>\\n\\t<li>Implementation of EOSC</li>\\n\\t<li>Metadata and Data Quality</li>\\n\\t<li>Research Careers and Curricula</li>\\n\\t<li>Sustaining EOSC</li>\\n\\t<li>Technical Challenges on EOSC</li>\\n</ul>\\n\\n<p>The Symposium was the first opportunity for the Association to present the draft charters of the Task Forces. A key objective of the event was also for the Association to understand what work has been carried out, is in progress, or is planned on the topics of the AGs and TFs. A call for contributions ran throughout May 2021, with a total of 137 applications received. Through presentations, lightning talks, and panels, over 70 community members were able to highlight key findings and recommendations for the AGs and TFs to take into consideration for their work.</p>',\n",
+ " 'doi': '10.5281/zenodo.5176089',\n",
+ " 'grants': [{'acronym': 'EOSCsecretariat.eu',\n",
+ " 'code': '831644',\n",
+ " 'funder': {'acronyms': [],\n",
+ " 'doi': '10.13039/501100000780',\n",
+ " 'links': {'self': 'https://zenodo.org/api/funders/10.13039/501100000780'},\n",
+ " 'name': 'European Commission'},\n",
+ " 'links': {'self': 'https://zenodo.org/api/grants/10.13039/501100000780::831644'},\n",
+ " 'program': 'H2020',\n",
+ " 'title': 'EOSCsecretariat.eu'}],\n",
+ " 'keywords': ['EOSC', 'Open Science', 'Horizon Europe', 'Interoperability'],\n",
+ " 'language': 'eng',\n",
+ " 'license': {'id': 'CC-BY-4.0'},\n",
+ " 'publication_date': '2021-08-10',\n",
+ " 'related_identifiers': [{'identifier': '10.5281/zenodo.5176088',\n",
+ " 'relation': 'isVersionOf',\n",
+ " 'scheme': 'doi'}],\n",
+ " 'relations': {'version': [{'count': 1,\n",
+ " 'index': 0,\n",
+ " 'is_last': True,\n",
+ " 'last_child': {'pid_type': 'recid', 'pid_value': '5176089'},\n",
+ " 'parent': {'pid_type': 'recid', 'pid_value': '5176088'}}]},\n",
+ " 'resource_type': {'subtype': 'report',\n",
+ " 'title': 'Report',\n",
+ " 'type': 'publication'},\n",
+ " 'title': 'EOSC Symposium 2021 Report'},\n",
+ " 'owners': [91736],\n",
+ " 'revision': 8,\n",
+ " 'stats': {'downloads': 438.0,\n",
+ " 'unique_downloads': 374.0,\n",
+ " 'unique_views': 475.0,\n",
+ " 'version_downloads': 438.0,\n",
+ " 'version_unique_downloads': 374.0,\n",
+ " 'version_unique_views': 475.0,\n",
+ " 'version_views': 494.0,\n",
+ " 'version_volume': 724854084.0,\n",
+ " 'views': 494.0,\n",
+ " 'volume': 724854084.0},\n",
+ " 'updated': '2021-08-24T14:27:14.603504+00:00'}"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# The content of the first entry of the payload - It contain all the info that we can also find in Zenodo\n",
+ "all_entries[0]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "0b48bcff",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "5176089 \t EOSC Symposium 2021 Report\n",
+ "5153369 \t agnpy: Modelling Active Galactic Nuclei radiative processes with python.\n",
+ "5093909 \t ESCAPE Data Science Summer School 2021\n",
+ "4923992 \t ESCAPE template project\n",
+ "4786641 \t ZenodoCI\n",
+ "4601451 \t gLike: numerical maximization of heterogeneous joint likelihood functions of a common free parameter plus nuisance parameters\n",
+ "4419866 \t IndexedConv/IndexedConv: v1.3\n",
+ "4044010 \t EOSC - a tool for enabling Open Science in Europe\n",
+ "3854976 \t FairRootGroup/DDS\n",
+ "3743489 \t ESCAPE the maze\n",
+ "3675081 \t ESFRI cluster projects - Position papers on expectations and planned contributions to the EOSC\n",
+ "3659184 \t ctapipe_io_mchdf5\n",
+ "3614662 \t FairRoot\n",
+ "3362435 \t FairMQ\n",
+ "3356656 \t A prototype for a real time pipeline for the detection of transient signals and their automatic classification\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Example to retrieve entries_ids and titles\n",
+ "for entry in all_entries:\n",
+ " print(f\"{entry['id']} \\t {entry['metadata']['title']}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "id": "e2afd195",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "5176089 \t ['EOSC', 'Open Science', 'Horizon Europe', 'Interoperability']\n",
+ "5153369 \t ['radiative processes', 'blazars', 'radio galaxies', 'AGN', 'jets', 'MWL', 'astropy', 'numpy', 'python']\n",
+ "5093909 \t ['python', 'lesson']\n",
+ "4923992 \t ['ESCAPE', 'jupyter-notebook']\n",
+ "4786641 \t ['ESCAPE']\n",
+ "4419866 \t ['CTA']\n",
+ "4044010 \t ['European Open Science Cloud, ESFRI, e-Infrastructures']\n",
+ "3743489 \t ['ESCAPE']\n",
+ "3659184 \t ['CTA']\n",
+ "3614662 \t ['geant4', 'c-plus-plus', 'cmake', 'reconstruction', 'vmc', 'modular', 'analysis', 'simulation']\n",
+ "3356656 \t ['Machine Learning, Big Data, Aapche Kafka, Gravitational Wave']\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Example of all the keywords within each entry\n",
+ "for entry in all_entries:\n",
+ " try:\n",
+ " print(f\"{entry['id']} \\t {entry['metadata']['keywords']}\")\n",
+ " except KeyError:\n",
+ " pass"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e9007eef",
+ "metadata": {},
+ "source": [
+ "#### Let's explore a specific ESCAPE2020 entry, for example `agnpy`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "id": "75b4de93",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "dict_keys(['conceptdoi', 'conceptrecid', 'created', 'doi', 'files', 'id', 'links', 'metadata', 'owners', 'revision', 'stats', 'updated'])"
+ ]
+ },
+ "execution_count": 12,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "agnpy = requests.get('https://zenodo.org/api/records/4687123', params=parameters).json()\n",
+ "agnpy.keys()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "id": "39be15f1",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'access_right': 'open',\n",
+ " 'access_right_category': 'success',\n",
+ " 'communities': [{'id': 'escape2020'}],\n",
+ " 'creators': [{'affiliation': \"Institut de FÃsica d'Altes Energies (IFAE)\",\n",
+ " 'name': 'Cosimo Nigro'},\n",
+ " {'name': 'Julian Sitarek'},\n",
+ " {'affiliation': 'Minnesota State University Moorhead', 'name': 'Matt Craig'},\n",
+ " {'name': 'Paweł Gliwny'},\n",
+ " {'affiliation': '@sourcery-ai', 'name': 'Sourcery AI'}],\n",
+ " 'description': '<p>In this release the major features added are:</p>\\n<ul>\\n<li><p>an exponential cutoff power-law for the electron spectra;</p>\\n</li>\\n<li><p>the possibility to compute the gamma-gamma opacity for misaligned sources (<code>viewing angle != 0</code>) for the following targets: point source behind the jet, BLR and the DT.</p>\\n</li>\\n</ul>',\n",
+ " 'doi': '10.5281/zenodo.4687123',\n",
+ " 'license': {'id': 'other-open'},\n",
+ " 'publication_date': '2021-04-14',\n",
+ " 'related_identifiers': [{'identifier': 'https://github.com/cosimoNigro/agnpy/tree/v0.0.10',\n",
+ " 'relation': 'isSupplementTo',\n",
+ " 'scheme': 'url'},\n",
+ " {'identifier': '10.5281/zenodo.4055175',\n",
+ " 'relation': 'isVersionOf',\n",
+ " 'scheme': 'doi'}],\n",
+ " 'relations': {'version': [{'count': 7,\n",
+ " 'index': 3,\n",
+ " 'is_last': False,\n",
+ " 'last_child': {'pid_type': 'recid', 'pid_value': '5153369'},\n",
+ " 'parent': {'pid_type': 'recid', 'pid_value': '4055175'}}]},\n",
+ " 'resource_type': {'title': 'Software', 'type': 'software'},\n",
+ " 'title': 'cosimoNigro/agnpy: v0.0.10: added EPWL for electrons and off-axis absorption calculation',\n",
+ " 'version': 'v0.0.10'}"
+ ]
+ },
+ "execution_count": 13,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "agnpy['metadata']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "id": "1ee7197f",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "https://zenodo.org/api/files/a806b549-922e-4025-9453-a5f4c0913fdd/cosimoNigro/agnpy-v0.0.10.zip\n"
+ ]
+ }
+ ],
+ "source": [
+ "for file in agnpy['files']:\n",
+ " print(file['links']['self'])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bb63887b",
+ "metadata": {},
+ "source": [
+ "We could do a simple `wget` of the previous URL and recover the file updoaded to Zenodo.\n",
+ "\n",
+ "Let's see and example with various files uploaded."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "id": "16db6ee0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "ESCAPE_template = requests.get('https://zenodo.org/api/records/4790629', params=parameters).json()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "id": "9feca5e6",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "https://zenodo.org/api/files/923a2614-a0fa-4927-bb3b-704168f3c768/codemeta.json\n",
+ "https://zenodo.org/api/files/923a2614-a0fa-4927-bb3b-704168f3c768/Singularity\n",
+ "https://zenodo.org/api/files/923a2614-a0fa-4927-bb3b-704168f3c768/Singularity.simg\n",
+ "https://zenodo.org/api/files/923a2614-a0fa-4927-bb3b-704168f3c768/template_project_escape-v2.1.zip\n"
+ ]
+ }
+ ],
+ "source": [
+ "for file in ESCAPE_template['files']:\n",
+ " print(file['links']['self'])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4070d988",
+ "metadata": {},
+ "source": [
+ "## eossr\n",
+ "\n",
+ "All these methods are implemented in the [Zenodo client](https://gitlab.in2p3.fr/escape2020/wp3/eossr/-/blob/master/eossr/api/zenodo.py) (a REST API handler) of the [eossr library](https://gitlab.in2p3.fr/escape2020/wp3/eossr). \n",
+ "\n",
+ "The library is also in charge of automatise the project's uploads from GitLab to Zenodo (by the use of the GitLab-CI and the REST API handler)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "id": "6cd7f714",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# pip install https://gitlab.in2p3.fr/escape2020/wp3/eossr/-/archive/master/eossr-master.zip"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "id": "0dbbbd64",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from eossr.api.zenodo import ZenodoAPI\n",
+ "z = ZenodoAPI(access_token=token, sandbox=False)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "id": "2037338d",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "15"
+ ]
+ },
+ "execution_count": 19,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "entries = z.fetch_community_entries(community_name='escape2020', \n",
+ " results_per_query=100)\n",
+ "entries.json()['hits']['total']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "id": "3985383f",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "5176089 EOSC Symposium 2021 Report\n",
+ "5153369 agnpy: Modelling Active Galactic Nuclei radiative processes with python.\n",
+ "5093909 ESCAPE Data Science Summer School 2021\n",
+ "4923992 ESCAPE template project\n",
+ "4786641 ZenodoCI\n",
+ "4601451 gLike: numerical maximization of heterogeneous joint likelihood functions of a common free parameter plus nuisance parameters\n",
+ "4419866 IndexedConv/IndexedConv: v1.3\n",
+ "4044010 EOSC - a tool for enabling Open Science in Europe\n",
+ "3854976 FairRootGroup/DDS\n",
+ "3743489 ESCAPE the maze\n",
+ "3675081 ESFRI cluster projects - Position papers on expectations and planned contributions to the EOSC\n",
+ "3659184 ctapipe_io_mchdf5\n",
+ "3614662 FairRoot\n",
+ "3362435 FairMQ\n",
+ "3356656 A prototype for a real time pipeline for the detection of transient signals and their automatic classification\n"
+ ]
+ }
+ ],
+ "source": [
+ "ids = z.fetch_community_entries_per_id(community_name='escape2020', \n",
+ " results_per_query=100)\n",
+ "\n",
+ "titles = z.fetch_community_entries_per_title(community_name='escape2020', \n",
+ " results_per_query=100)\n",
+ "\n",
+ "for id, title in zip(ids, titles):\n",
+ " print(id, title)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "595ba083",
+ "metadata": {},
+ "source": [
+ "## PyZenodo3\n",
+ "\n",
+ "Another equivalent example with the pyzenodo3 library"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "id": "91985172",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# pip install pyzenodo3"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "id": "7cd937b9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import pyzenodo3\n",
+ "\n",
+ "zen = pyzenodo3.Zenodo()\n",
+ "records = zen.search('agnpy')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 23,
+ "id": "815578d9",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'conceptdoi': '10.5281/zenodo.5174757',\n",
+ " 'conceptrecid': '5174757',\n",
+ " 'created': '2021-08-10T07:43:16.545873+00:00',\n",
+ " 'doi': '10.5281/zenodo.5174758',\n",
+ " 'files': [{'bucket': '4a6752c9-f922-45f5-a6e5-82f9a016ba87',\n",
+ " 'checksum': 'md5:516ae662f4d8a251a7d8b9fb41007e56',\n",
+ " 'key': 'cosimoNigro/agnpy_paper-v0.1.0.zip',\n",
+ " 'links': {'self': 'https://zenodo.org/api/files/4a6752c9-f922-45f5-a6e5-82f9a016ba87/cosimoNigro/agnpy_paper-v0.1.0.zip'},\n",
+ " 'size': 45740,\n",
+ " 'type': 'zip'}],\n",
+ " 'id': 5174758,\n",
+ " 'links': {'badge': 'https://zenodo.org/badge/doi/10.5281/zenodo.5174758.svg',\n",
+ " 'bucket': 'https://zenodo.org/api/files/4a6752c9-f922-45f5-a6e5-82f9a016ba87',\n",
+ " 'conceptbadge': 'https://zenodo.org/badge/doi/10.5281/zenodo.5174757.svg',\n",
+ " 'conceptdoi': 'https://doi.org/10.5281/zenodo.5174757',\n",
+ " 'doi': 'https://doi.org/10.5281/zenodo.5174758',\n",
+ " 'html': 'https://zenodo.org/record/5174758',\n",
+ " 'latest': 'https://zenodo.org/api/records/5174758',\n",
+ " 'latest_html': 'https://zenodo.org/record/5174758',\n",
+ " 'self': 'https://zenodo.org/api/records/5174758'},\n",
+ " 'metadata': {'access_right': 'open',\n",
+ " 'access_right_category': 'success',\n",
+ " 'creators': [{'affiliation': \"Institut de FÃsica d'Altes Energies (IFAE), The Barcelona Institute of Science and Technology, Campus UAB, 08193 Bellaterra (Barcelona), Spain\",\n",
+ " 'name': 'Cosimo Nigro',\n",
+ " 'orcid': '0000-0001-8375-1907'},\n",
+ " {'affiliation': 'University of Lodz, Faculty of Physics and Applied Informatics, Department of Astrophysics, 90-236 Lodz, Poland',\n",
+ " 'name': 'Julian Sitarek',\n",
+ " 'orcid': '0000-0002-1659-5374'},\n",
+ " {'affiliation': 'University of Lodz, Faculty of Physics and Applied Informatics, Department of Astrophysics, 90-236 Lodz, Poland',\n",
+ " 'name': 'Paweł Gliwny',\n",
+ " 'orcid': '0000-0002-4183-391X'},\n",
+ " {'affiliation': \"Laboratoire d'Annecy de Physique des Particules, Univ. Grenoble Alpes, Univ. Savoie Mont Blanc, CNRS, LAPP, 74000 Annecy, France\",\n",
+ " 'name': 'David Sanchez'},\n",
+ " {'affiliation': 'Minnesota State University Moorhead, Moorhead, Minnesota, US',\n",
+ " 'name': 'Matthew Craig',\n",
+ " 'orcid': '0000-0002-4183-391X'}],\n",
+ " 'description': \"This repository contains the scripts to generate the figures included in the paper 'agnpy: an open-source python package modelling the radiative processes of jetted active galactic nuclei'.\",\n",
+ " 'doi': '10.5281/zenodo.5174758',\n",
+ " 'keywords': ['radiative processes',\n",
+ " 'blazars',\n",
+ " 'radio galaxies',\n",
+ " 'AGN',\n",
+ " 'jets',\n",
+ " 'MWL',\n",
+ " 'astropy',\n",
+ " 'numpy',\n",
+ " 'python'],\n",
+ " 'license': {'id': 'other-open'},\n",
+ " 'publication_date': '2021-08-10',\n",
+ " 'related_identifiers': [{'identifier': 'https://github.com/cosimoNigro/agnpy_paper/tree/v0.1.0',\n",
+ " 'relation': 'isSupplementTo',\n",
+ " 'scheme': 'url'},\n",
+ " {'identifier': '10.5281/zenodo.5174757',\n",
+ " 'relation': 'isVersionOf',\n",
+ " 'scheme': 'doi'}],\n",
+ " 'relations': {'version': [{'count': 1,\n",
+ " 'index': 0,\n",
+ " 'is_last': True,\n",
+ " 'last_child': {'pid_type': 'recid', 'pid_value': '5174758'},\n",
+ " 'parent': {'pid_type': 'recid', 'pid_value': '5174757'}}]},\n",
+ " 'resource_type': {'title': 'Software', 'type': 'software'},\n",
+ " 'title': 'agnpy: an open-source python package modelling the radiative processes of jetted active galactic nuclei',\n",
+ " 'version': '0.1.0'},\n",
+ " 'owners': [99841],\n",
+ " 'revision': 3,\n",
+ " 'stats': {'downloads': 0.0,\n",
+ " 'unique_downloads': 0.0,\n",
+ " 'unique_views': 12.0,\n",
+ " 'version_downloads': 0.0,\n",
+ " 'version_unique_downloads': 0.0,\n",
+ " 'version_unique_views': 12.0,\n",
+ " 'version_views': 13.0,\n",
+ " 'version_volume': 0.0,\n",
+ " 'views': 13.0,\n",
+ " 'volume': 0.0},\n",
+ " 'updated': '2021-08-10T13:48:43.185119+00:00'}"
+ ]
+ },
+ "execution_count": 23,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "records[0].data"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a304d374",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.10"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}