diff --git a/CHANGELOG b/CHANGELOG index 617d64242ddd0b67cc6ae6ccdf34c39cd47c8dc4..45231f2e88375d9980c2f63a545e87a9cda85d5c 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -2,7 +2,18 @@ HEAD -2.14.6-ext6 +2.18.3 (Jan 2020) + - Full stack is running with python 3 + - Base image is alpine 3.10 instead of Centos7 + - Migrate to web2py 2.18.3 + - Tools jsduck, pdflatex and senchacmd are moved to separated images. + - Adapt configure.py and build.py + - Add image gunicorn in which web2py applications are running behind + the gunicorn front-end instead of the build-in rocker. + This is the base image to build a server. + - Remove image_server + +2.14.6-ext6 (Oct 2016) - Similar to 2.14.6 but running with ExtJS 6.0.1 2.14.6 (June 2016) diff --git a/README.md b/README.md index e21599652a224a760df37a12444c5e1887b814b5..163499e5d2e612fae9ec4fb8aa58c583867bada3 100644 --- a/README.md +++ b/README.md @@ -1,30 +1,48 @@ # w2pext/docker Tool to build [docker](https://www.docker.com/) images -running the [web2py](http://web2py.com/) buildin HTTP rocket server and -user applications. +running the [web2py](http://web2py.com/) server. -## build +Two images can be constructed: + + * the *base image* `w2pegp` which runs the wep2py server behind its + buildin HTTP rocker server. This image is useful to develop web2py + applications. + + * the *server image* `w2pegp-nginx` which runs the web2py rocket server + behing the front-end [nginx](http://nginx.org/). + This image is useful to run web2py application in production environment. + It is built on top of the base image. + +## build the base image The build is performed in two steps: -1. launch the script `configure.py` to select release numbers - for the framework `web2py` and for third party libraries. -2. launch the script `build.py` to buid the web2py image with third - party libraries. On top of the valilla web2py framework, - several layers can be added: +1. Edit the configuration file `w2pext_base.json` with the choosen release + for `web2py` and third party libraries: + - the script `configure.py` can help to select release numbers. + - for `web2py` it is possible to select an archive version like `2.18.3` + or the `latest` one. + +2. launch the script `build.py` to build the base image. + Several layers can be added on top of the valilla web2py framework: - `export`: LibreOffice converter - `graph`: [pandas](http://pandas.pydata.org), [matplotlib](http://matplotlib.org) and their friends - - `doc`: [jsduck](https://github.com/senchalabs/jsduck) and - [Sphinx](http://www.sphinx-doc.org) - - `senchacmd`: [SenchaCmd](https://www.sencha.com/products/sencha-cmd) - is an utilities to minified javascript library + - `pytest`: [pytest](http://pytest.org/en/latest/) - `all` + By default, only the web2py layers is built by the script. + Additional layers can be selected via arguments of the `build.py` script. + +## buid the server image +The build is performed in two steps: + +1. edit the file `w2pext_server.json` to select the release of the base image + and to define the tag for the server image. -The process is configured via the [JSON](http://json.org/) file `web2py.json`. +2. launch the script `build.py` to build the server image. ## License The code is released under [CeCILL License](http://www.cecill.info/licences/Licence_CeCILL_V2.1-en.htm) -Copyright © 2016 Renaud Le Gac +Copyright © 2016-2019 Renaud Le Gac diff --git a/VERSION b/VERSION new file mode 100644 index 0000000000000000000000000000000000000000..26452813e00684c19e64aa1c29a6ad1120576639 --- /dev/null +++ b/VERSION @@ -0,0 +1 @@ +2.18.3 diff --git a/build.py b/build.py deleted file mode 100755 index ecb586b48ea54d235a8d0a6c89c1e527855372ab..0000000000000000000000000000000000000000 --- a/build.py +++ /dev/null @@ -1,283 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- -""" NAME - build.py -- build docker image w2pext running the web2py service - - SYNOPSIS - build.py [Options] command [layers] - - DESCRIPTION - Build the docker images w2pext. It contains the web2py framework, - the building HTTP front-end, rocket, and third party libraries. - - The building is in several steps: - - 1. launch the script configure.py to select the third party - libraries use by the web2py applications - 2. build the web2py image(s). - - Third party libraries are group in layers which can be merged at - the building time: - - * base - the base service running on centos. - this layer is always present. - The web2py source file is download from the github repository. - - * export - add the export functionalities. - Latex can be used to generated PDF file and LibreOffice - to translate HTML into doc document. They are installed - via the rpm of the centos distribution. - - * graph - add the graph functionalities. - The python modules pandas, as well as matplotlib are - available. They are installed using pip. - - * doc - add tools to generate the documentation (sphinx and jsduck). - - * senchacmd - add the tool to minified javascript libraries (SenchaCMD). - - To build web2py with layers added on top of it. - - > ./build.py - > ./build.py export - > ./build.py export graph doc js - > ./build.py all - - the realse of the third party libraries are defined in the - configuration file. The latter can be generated using the script - configure.py. - - The name image depends on the layers selection and on - the third party dependencies: - - * w2pext:2.9.11 - base image, web2py release 2.9.11. - - * w2pext-degs:2.9.11 - base image with all layers. - - The release numbers for third party libraries are stored - in the image via docker LABEL. - - OPTIONS - -h, --help - - EXAMPLES - > ./build.py - - AUTHOR - R. Le Gac, legac@cppm.in2p3.fr - -""" -import argparse -import json -import os -import string -import sys - - -from subprocess import call, CalledProcessError, check_output - - -DOCKERFILE = "Dockerfile" -IMG_BASE = "w2pext" -LAYERS = ["base", "doc", "export", "graph", "senchacmd"] -PATH_LAYER = "layers/%s_layer" -PLUGINS_GIT = os.path.expanduser("~/mywap/w2pext/plugins") -TAR_PLG = "plugins_%s.tar" - - -def check_docker_daemon(): - """Exit if the docker daemon is not running. - - """ - try: - check_output(["docker", "info"]) - - except CalledProcessError: - print "\n\tDocker daemon is not running§\n" - sys.exit(1) - - -def build(body, name): - """build the docker image - - Args: - body (str): the content of the docker file. - name (str): name of the image. - - """ - with open(DOCKERFILE, "w") as fi: - fi.write(body) - - print "\n", "."*80, "\n" - print "construct the image", name, "..." - call(["docker", "build", "-t", name, "."]) - - print "\n", "."*80, "\n" - call(["docker", "history", name]) - - print "\n", "."*80, "\n" - call(["docker", "images"]) - - # clean - os.remove(DOCKERFILE) - - -def build_w2pext(args): - """Build the w2pext image. - - Args: - args (argparse.Namespace): command arguments - - """ - check_docker_daemon() - - print "Start building w2pext image.\n" - - # protection - if len(args.layers) == 1 and args.layers[0] == "all": - args.layers = LAYERS - - elif len(args.layers) > 0: - delta = set(args.layers).difference(set(LAYERS)) - if delta: - print "Invalid layer(s):", " ".join(list(delta)) - sys.exit(1) - - # meta data and image - print "Prepare the dockerfile..." - - meta = json.load(args.configuration) - img_name = image_name(args, meta) - - # the base layer - print "\tAdd the base layer" - body = read_layer("base") - - # web2py layer - print "\tadd the web2py layer" - - meta["waps_build"] = \ - check_output(["git", "describe", "--always"]).strip("\n") - - template = string.Template(read_layer("web2py")) - layer = template.substitute(meta) - - if meta["pydal"] == "": - layer = layer.replace("LABEL pyDAL =", "") - layer = layer.replace("RUN pip install PYDAL==", "") - - body += layer - - # export layer - if "export" in args.layers: - print "\tadd the export layer" - body += read_layer("export") - body += "\n" - - # layers with substitution - # order matter since pip install python modules - for layer in ("graph", "doc", "senchacmd"): - if layer in args.layers: - print "\tadd the %s layer" % layer - template = string.Template(read_layer(layer)) - layer = template.substitute(meta) - body += layer - - # docker file - build(body, img_name) - - -def image_name(args, meta): - """Construct the name of the image from meta-data. - - Args: - args (argparse.Namespace): command arguments - meta (dict): - - returns: - str: w2pext-degs:2.9.11 - - """ - name = IMG_BASE - - # list of layers - layers = [el[0:1] for el in args.layers if el != "base"] - layers.sort() - if len(layers) > 0: - name = "%s-%s" % (name, "".join(layers)) - - # web2py release - name = "%s:%s" % (name, meta["tag"]) - - return name - - -def read_layer(layer): - """Read docker instruction for the layer. - - Args: - layer (str): name of the layer - - Returns: - str: - - """ - with open(PATH_LAYER % layer) as fi: - data = fi.read() - - return data - - -def tar_plugins(release, git_repository): - """Helper function to tar the web2py plugins: ace, extjs, mathjax - and plugin_dbui. - - Args: - release (str): the plugins release - git_repository (str): git repository containing the compressed versions - for the plugins - - """ - cwd = os.getcwd() - tar_file = os.path.join(cwd, TAR_PLG % release) - - if not os.path.exists(tar_file): - print "Generating tar file for plugins", tar_file - os.chdir(git_repository) - tag = ("master" if release == "latest" else release) - call(["git", "checkout", "-b", "build", tag]) - call("tar -cf %s web2py.plugin.*.w2p" % tar_file, shell=True) - call(["git", "checkout", "master"]) - os.chdir(cwd) - -if __name__ == '__main__': - - AGP = argparse.ArgumentParser() - - AGP.add_argument( - "-c", "--configuration", - default="w2pext.json", - help="JSON configuration file [%(default)s].", - metavar="<file>", - type=argparse.FileType('r')) - - txt = "possible values are %s and all" % ", ".join(LAYERS) - AGP.add_argument( - "layers", - default=["all"], - help=txt + " [%(default)s].", - nargs="*") - - AGP.set_defaults(func=build_w2pext) - - ARGS = AGP.parse_args() - ARGS.func(ARGS) - - sys.exit(0) diff --git a/image_gunicorn/Dockerfile b/image_gunicorn/Dockerfile new file mode 100644 index 0000000000000000000000000000000000000000..082ad6a26d8fd33abc8341c03fcb975c052549da --- /dev/null +++ b/image_gunicorn/Dockerfile @@ -0,0 +1,70 @@ +# +# NAME +# w2pext_gunicorn +# +# SYNOPSIS +# Run web2py with HTTP gunicorn front-end isntead of the build-in rocket +# +# DESCRIPTION +# Base image contain web2py running on alpine and third party libraries. +# The HTTP front end is gunicorn instead of the build-in rocker one +# +# Set the password for the admin interface to "f" +# +# Environment variables: +# +# - BIND [127.0.0.1:9090] +# - UID [0] +# - GID [0] +# - WORKERS [2] +# +# By default, guincorn and web2py servers are run as root. +# They can run as a given user by defning environment variable UID and GID. +# +# The following volumes can be mount for persistency: +# +# * /opt/web2py/parameters_443.py : admin password for the web2py server +# * /opt/web2py/applications : web2py applications +# +# EXAMPLE +# as root user +# $ dev --data-container +# $ docker run --rm -d --network host --volumes-from devvol w2pegp-gunicorn +# $ firefox http://localhost:9090/test_limbra +# +# as user 1001:1200 +# $ dev --data-container +# $ docker run --rm -d -e UID=1001 -e GID=1200 --network host --volumes-from devvol w2pegp-gunicorn +# $ firefox http://localhost:9090/test_limbra +# +# $ docker run --rm -d -e UID=1001 -e GID=1200 -e BIND 127.0.0.1:8000 --network host --volumes-from devvol w2pegp-gunicorn +# $ firefox http://localhost:8000/test_limbra +# +# +ARG image_base + +FROM ${image_base} + +MAINTAINER Renaud Le Gac <legac@cppm.in2p3.fr> + +RUN apk add --no-cache su-exec \ + && /opt/miniconda/bin/pip install gunicorn==20.0.4 \ + && mv /opt/web2py/handlers/wsgihandler.py /opt/web2py/ + +COPY bin/ / + +#............................................................................. +# +# FIX 191212 gunicorn (20.0.4) in order to run with web2py and python3 (line 326) +# +COPY wsgi.py /opt/miniconda/lib/python3.7/site-packages/gunicorn/http/ + + +#............................................................................. +# +# run +# +ENV BIND=127.0.0.1:9090 GID=0 UID=0 WORKERS=2 + +WORKDIR /opt/web2py +CMD /usr/local/sbin/runw2p.sh diff --git a/image_gunicorn/bin/usr/local/sbin/runw2p.sh b/image_gunicorn/bin/usr/local/sbin/runw2p.sh new file mode 100755 index 0000000000000000000000000000000000000000..d7dce16c8d707f6b8daced008a48675b1ae11a95 --- /dev/null +++ b/image_gunicorn/bin/usr/local/sbin/runw2p.sh @@ -0,0 +1,38 @@ +#! /bin/sh +# +# NAME +# runw2p +# +# SYNOPSIS +# runw2p +# +# DESCRIPTION +# Run web2py using gunicorn HTTP front-end. +# By default, guincorn and web2py servers are run as root. +# The script allows to run the wep2py part as a selected user. +# It is enough to define the enviroment variables 'UID' and 'GID'. +# +# EXAMPLE +# +# $ docker run --rm -it -e UID=1001 -e GID=1200 w2pegp-gunicorn:latest +# + +if [ "$GID" = "" ]; then + GID=0 +fi + +case $UID in + 0) + cd /opt/web2py + /opt/miniconda/bin/gunicorn -b $BIND -w $WORKERS wsgihandler:application + ;; + "") + cd /opt/web2py + /opt/miniconda/bin/gunicorn -b $BIND -w $WORKERS wsgihandler:application + ;; + *) + chown -R ${UID}:${GID} /opt/web2py + cd /opt/web2py + /sbin/su-exec ${UID}:${GID} /opt/miniconda/bin/gunicorn -b $BIND -w $WORKERS wsgihandler:application + ;; +esac diff --git a/image_gunicorn/build.py b/image_gunicorn/build.py new file mode 100755 index 0000000000000000000000000000000000000000..c1066f90f50df6d20cc44e82d8849be7269e4708 --- /dev/null +++ b/image_gunicorn/build.py @@ -0,0 +1,150 @@ +#!/usr/bin/python3 +""" NAME + build.py -- build docker images + + SYNOPSIS + build.py [Options] + + DESCRIPTION + Build the docker images for the w2pext gunicorn edition. + + It runs the web2py framework, third party libraries + when the HTTP front-end is gunicorn instead of the build in rocket. + + OPTIONS + -h, --help + + EXAMPLES + > ./build.py + > ./build.py -c w2pext_gunicorn.json + + AUTHOR + R. Le Gac, legac@cppm.in2p3.fr + +""" +import argparse +import docker +import json +import logging +import sys + + +DOCKERFILE = "Dockerfile" +DOCKER_API_VERSION = "auto" +IMG_BASE = "w2p" +JSON = "w2pext_gunicorn.json" +TAG = "gunicorn" +VERSION = "VERSION" + + +def cli(): + + logging.basicConfig( + format="%(asctime)s %(levelname)s %(message)s", + level=logging.INFO, + datefmt="%H:%M:%S") + + parser = argparse.ArgumentParser() + + parser.add_argument( + "-c", "--configuration", + default=JSON, + help="JSON configuration file [%(default)s].", + metavar="<path>", + type=argparse.FileType('r')) + + parser.set_defaults(func=build_w2pext_server) + + args = parser.parse_args() + args.func(args) + + +def build_w2pext_server(args): + """Build the w2pext image for the gunicorn edition. + + Args: + args (argparse.Namespace): command arguments + + """ + logging.info("start building w2pext server image...") + + # check that the docker service is running + try: + client = docker.from_env(version=DOCKER_API_VERSION) + client.ping() + + except BaseException: + logging.error("docker daemon is not running!!!\n") + sys.exit(1) + + # load meta data + meta = json.load(args.configuration) + img_base = meta["image_base"] + img_name = "%s-%s:%s" % (img_base[:img_base.index(":")], TAG, meta["tag"]) + + # ........................................................................ + # + # build the image removing intermediate container + # use the low level API in order to display the build messages + # + logging.info(f"\tconstruct image {img_name}...\n") + + out = client.api.build(path=".", + buildargs=meta, + rm=True, + tag=img_name) + + for line in out: + di = json.loads(line) + if "stream" in di: + print(di["stream"].strip("\n").replace(" \n", "", 1)) + + # ........................................................................ + # + # display the history of the image + # + print() + print("\t%-8s %-45s %-8s %-s" % ("IMAGE", "CREATED BY", "SIZE", "COMMENT")) + + try: + client.images.get(img_name) + + fmt = "\t%-8s %-45s %05.1f MB %-s" + for elt in client.api.history(img_name): + print(fmt % (elt["Id"][7:15], + elt["CreatedBy"][:45], + round(elt["Size"]/1E6, 1), + elt["Comment"])) + + except docker.errors.ImageNotFound: + logging.error(f"\timage {img_name} not found!!!") + logging.error("\ttry: docker images...\n") + return + + # ........................................................................ + # + # Display the list of existing images + # + print() + print("\t%-25s %-15s %-15s %s" % ("REPOSITORY", "TAG", "IMAGE ID", "SIZE")) + + fmt = "\t%-25s %-15s %-15s %6.3f GB" + for img in client.images.list(): + attrs = img.attrs + + lst = attrs["RepoTags"] + value = (lst[0] if len(lst) > 0 else "<none>:<none>") + name, tag = value.split(":") + + if name.startswith(IMG_BASE): + print(fmt % (name, + tag, + attrs["Id"][7:15], + round(attrs["Size"]/1E9, 3))) + print() + + +if __name__ == '__main__': + + cli() + sys.exit(0) diff --git a/image_gunicorn/w2pext_gunicorn.json b/image_gunicorn/w2pext_gunicorn.json new file mode 100644 index 0000000000000000000000000000000000000000..5945e085e823f9a086304145b491bccfa2336d26 --- /dev/null +++ b/image_gunicorn/w2pext_gunicorn.json @@ -0,0 +1,4 @@ +{ + "image_base": "w2pegp:2.18.3", + "tag": "2.18.3" +} diff --git a/image_gunicorn/wsgi.py b/image_gunicorn/wsgi.py new file mode 100644 index 0000000000000000000000000000000000000000..6a618cf2c182cb66fe51128d1f800c59c24fc933 --- /dev/null +++ b/image_gunicorn/wsgi.py @@ -0,0 +1,408 @@ +# -*- coding: utf-8 - +# +# This file is part of gunicorn released under the MIT license. +# See the NOTICE for more information. + +import io +import logging +import os +import re +import sys + +from gunicorn.http.message import HEADER_RE +from gunicorn.http.errors import InvalidHeader, InvalidHeaderName +from gunicorn import SERVER_SOFTWARE +import gunicorn.util as util + +# Send files in at most 1GB blocks as some operating systems can have problems +# with sending files in blocks over 2GB. +BLKSIZE = 0x3FFFFFFF + +HEADER_VALUE_RE = re.compile(r'[\x00-\x1F\x7F]') + +log = logging.getLogger(__name__) + + +class FileWrapper(object): + + def __init__(self, filelike, blksize=8192): + self.filelike = filelike + self.blksize = blksize + if hasattr(filelike, 'close'): + self.close = filelike.close + + def __getitem__(self, key): + data = self.filelike.read(self.blksize) + if data: + return data + raise IndexError + + +class WSGIErrorsWrapper(io.RawIOBase): + + def __init__(self, cfg): + # There is no public __init__ method for RawIOBase so + # we don't need to call super() in the __init__ method. + # pylint: disable=super-init-not-called + errorlog = logging.getLogger("gunicorn.error") + handlers = errorlog.handlers + self.streams = [] + + if cfg.errorlog == "-": + self.streams.append(sys.stderr) + handlers = handlers[1:] + + for h in handlers: + if hasattr(h, "stream"): + self.streams.append(h.stream) + + def write(self, data): + for stream in self.streams: + try: + stream.write(data) + except UnicodeError: + stream.write(data.encode("UTF-8")) + stream.flush() + + +def base_environ(cfg): + return { + "wsgi.errors": WSGIErrorsWrapper(cfg), + "wsgi.version": (1, 0), + "wsgi.multithread": False, + "wsgi.multiprocess": (cfg.workers > 1), + "wsgi.run_once": False, + "wsgi.file_wrapper": FileWrapper, + "wsgi.input_terminated": True, + "SERVER_SOFTWARE": SERVER_SOFTWARE, + } + + +def default_environ(req, sock, cfg): + env = base_environ(cfg) + env.update({ + "wsgi.input": req.body, + "gunicorn.socket": sock, + "REQUEST_METHOD": req.method, + "QUERY_STRING": req.query, + "RAW_URI": req.uri, + "SERVER_PROTOCOL": "HTTP/%s" % ".".join([str(v) for v in req.version]) + }) + return env + + +def proxy_environ(req): + info = req.proxy_protocol_info + + if not info: + return {} + + return { + "PROXY_PROTOCOL": info["proxy_protocol"], + "REMOTE_ADDR": info["client_addr"], + "REMOTE_PORT": str(info["client_port"]), + "PROXY_ADDR": info["proxy_addr"], + "PROXY_PORT": str(info["proxy_port"]), + } + + +def create(req, sock, client, server, cfg): + resp = Response(req, sock, cfg) + + # set initial environ + environ = default_environ(req, sock, cfg) + + # default variables + host = None + script_name = os.environ.get("SCRIPT_NAME", "") + + # add the headers to the environ + for hdr_name, hdr_value in req.headers: + if hdr_name == "EXPECT": + # handle expect + if hdr_value.lower() == "100-continue": + sock.send(b"HTTP/1.1 100 Continue\r\n\r\n") + elif hdr_name == 'HOST': + host = hdr_value + elif hdr_name == "SCRIPT_NAME": + script_name = hdr_value + elif hdr_name == "CONTENT-TYPE": + environ['CONTENT_TYPE'] = hdr_value + continue + elif hdr_name == "CONTENT-LENGTH": + environ['CONTENT_LENGTH'] = hdr_value + continue + + key = 'HTTP_' + hdr_name.replace('-', '_') + if key in environ: + hdr_value = "%s,%s" % (environ[key], hdr_value) + environ[key] = hdr_value + + # set the url scheme + environ['wsgi.url_scheme'] = req.scheme + + # set the REMOTE_* keys in environ + # authors should be aware that REMOTE_HOST and REMOTE_ADDR + # may not qualify the remote addr: + # http://www.ietf.org/rfc/rfc3875 + if isinstance(client, str): + environ['REMOTE_ADDR'] = client + elif isinstance(client, bytes): + environ['REMOTE_ADDR'] = client.decode() + else: + environ['REMOTE_ADDR'] = client[0] + environ['REMOTE_PORT'] = str(client[1]) + + # handle the SERVER_* + # Normally only the application should use the Host header but since the + # WSGI spec doesn't support unix sockets, we are using it to create + # viable SERVER_* if possible. + if isinstance(server, str): + server = server.split(":") + if len(server) == 1: + # unix socket + if host: + server = host.split(':') + if len(server) == 1: + if req.scheme == "http": + server.append(80) + elif req.scheme == "https": + server.append(443) + else: + server.append('') + else: + # no host header given which means that we are not behind a + # proxy, so append an empty port. + server.append('') + environ['SERVER_NAME'] = server[0] + environ['SERVER_PORT'] = str(server[1]) + + # set the path and script name + path_info = req.path + if script_name: + path_info = path_info.split(script_name, 1)[1] + environ['PATH_INFO'] = util.unquote_to_wsgi_str(path_info) + environ['SCRIPT_NAME'] = script_name + + # override the environ with the correct remote and server address if + # we are behind a proxy using the proxy protocol. + environ.update(proxy_environ(req)) + return resp, environ + + +class Response(object): + + def __init__(self, req, sock, cfg): + self.req = req + self.sock = sock + self.version = SERVER_SOFTWARE + self.status = None + self.chunked = False + self.must_close = False + self.headers = [] + self.headers_sent = False + self.response_length = None + self.sent = 0 + self.upgrade = False + self.cfg = cfg + + def force_close(self): + self.must_close = True + + def should_close(self): + if self.must_close or self.req.should_close(): + return True + if self.response_length is not None or self.chunked: + return False + if self.req.method == 'HEAD': + return False + if self.status_code < 200 or self.status_code in (204, 304): + return False + return True + + def start_response(self, status, headers, exc_info=None): + if exc_info: + try: + if self.status and self.headers_sent: + util.reraise(exc_info[0], exc_info[1], exc_info[2]) + finally: + exc_info = None + elif self.status is not None: + raise AssertionError("Response headers already set!") + + self.status = status + + # get the status code from the response here so we can use it to check + # the need for the connection header later without parsing the string + # each time. + try: + self.status_code = int(self.status.split()[0]) + except ValueError: + self.status_code = None + + self.process_headers(headers) + self.chunked = self.is_chunked() + return self.write + + def process_headers(self, headers): + for name, value in headers: + if not isinstance(name, str): + raise TypeError('%r is not a string' % name) + + if HEADER_RE.search(name): + raise InvalidHeaderName('%r' % name) + + if not isinstance(value, str): + raise TypeError('%r is not a string' % value) + + if HEADER_VALUE_RE.search(value): + raise InvalidHeader('%r' % value) + + value = value.strip() + lname = name.lower().strip() + if lname == "content-length": + self.response_length = int(value) + elif util.is_hoppish(name): + if lname == "connection": + # handle websocket + if value.lower().strip() == "upgrade": + self.upgrade = True + elif lname == "upgrade": + if value.lower().strip() == "websocket": + self.headers.append((name.strip(), value)) + + # ignore hopbyhop headers + continue + self.headers.append((name.strip(), value)) + + def is_chunked(self): + # Only use chunked responses when the client is + # speaking HTTP/1.1 or newer and there was + # no Content-Length header set. + if self.response_length is not None: + return False + elif self.req.version <= (1, 0): + return False + elif self.req.method == 'HEAD': + # Responses to a HEAD request MUST NOT contain a response body. + return False + elif self.status_code in (204, 304): + # Do not use chunked responses when the response is guaranteed to + # not have a response body. + return False + return True + + def default_headers(self): + # set the connection header + if self.upgrade: + connection = "upgrade" + elif self.should_close(): + connection = "close" + else: + connection = "keep-alive" + + headers = [ + "HTTP/%s.%s %s\r\n" % (self.req.version[0], + self.req.version[1], self.status), + "Server: %s\r\n" % self.version, + "Date: %s\r\n" % util.http_date(), + "Connection: %s\r\n" % connection + ] + if self.chunked: + headers.append("Transfer-Encoding: chunked\r\n") + return headers + + def send_headers(self): + if self.headers_sent: + return + tosend = self.default_headers() + tosend.extend(["%s: %s\r\n" % (k, v) for k, v in self.headers]) + + header_str = "%s\r\n" % "".join(tosend) + util.write(self.sock, util.to_bytestring(header_str, "latin-1")) + self.headers_sent = True + + def write(self, arg): + self.send_headers() + # RLG FX 191212 + if isinstance(arg, str): + arg = arg.encode("utf-8") + # END FIX + elif not isinstance(arg, bytes): + raise TypeError('%r is not a byte' % arg) + arglen = len(arg) + tosend = arglen + if self.response_length is not None: + if self.sent >= self.response_length: + # Never write more than self.response_length bytes + return + + tosend = min(self.response_length - self.sent, tosend) + if tosend < arglen: + arg = arg[:tosend] + + # Sending an empty chunk signals the end of the + # response and prematurely closes the response + if self.chunked and tosend == 0: + return + + self.sent += tosend + util.write(self.sock, arg, self.chunked) + + def can_sendfile(self): + return self.cfg.sendfile is not False + + def sendfile(self, respiter): + if self.cfg.is_ssl or not self.can_sendfile(): + return False + + if not util.has_fileno(respiter.filelike): + return False + + fileno = respiter.filelike.fileno() + try: + offset = os.lseek(fileno, 0, os.SEEK_CUR) + if self.response_length is None: + filesize = os.fstat(fileno).st_size + + # The file may be special and sendfile will fail. + # It may also be zero-length, but that is okay. + if filesize == 0: + return False + + nbytes = filesize - offset + else: + nbytes = self.response_length + except (OSError, io.UnsupportedOperation): + return False + + self.send_headers() + + if self.is_chunked(): + chunk_size = "%X\r\n" % nbytes + self.sock.sendall(chunk_size.encode('utf-8')) + + sockno = self.sock.fileno() + sent = 0 + + while sent != nbytes: + count = min(nbytes - sent, BLKSIZE) + sent += os.sendfile(sockno, fileno, offset + sent, count) + + if self.is_chunked(): + self.sock.sendall(b"\r\n") + + os.lseek(fileno, offset, os.SEEK_SET) + + return True + + def write_file(self, respiter): + if not self.sendfile(respiter): + for item in respiter: + self.write(item) + + def close(self): + if not self.headers_sent: + self.send_headers() + if self.chunked: + util.write_chunk(self.sock, b"") diff --git a/image_jsduck/Dockerfile b/image_jsduck/Dockerfile new file mode 100644 index 0000000000000000000000000000000000000000..56c7d800871bb530c57e672c3661623468dd1221 --- /dev/null +++ b/image_jsduck/Dockerfile @@ -0,0 +1,14 @@ +FROM alpine:latest +MAINTAINER Philippe Poumaroux <poum@cpan.org> + + +RUN apk add --update curl ruby ruby-dev make gcc libc-dev ruby-rdoc ruby-irb && \ + curl -o /rubygems.gem https://rubygems.org/downloads/rubygems-update-2.6.7.gem && \ + gem install --local /rubygems.gem && \ + update_rubygems --no-ri --no-rdoc && \ + gem uninstall rubygems-update -x && \ + gem install jsduck && \ + apk del ruby-dev make gcc libc-dev curl && \ + rm -rf /var/cache/apk/* + +ENTRYPOINT ["/usr/bin/jsduck"] diff --git a/image_pdflatex/Dockerfile b/image_pdflatex/Dockerfile new file mode 100644 index 0000000000000000000000000000000000000000..2426b0a04de7ba6499969f0823ad0301f519043b --- /dev/null +++ b/image_pdflatex/Dockerfile @@ -0,0 +1,28 @@ +# +# NAME +# pdflatex +# +# SYNOPSIS +# Tool to run pdflatex +# +# DESCRIPTION +# The base operating system is alpine. +# +# To run the pdflatex on the file foo.tex located in the directory latex +# and own by the user 1001: +# +# $ cd latex +# $ docker run --rm -v `pwd`:/wd -w /wd -u 1001 pdflatex:2019 foo.tex +# +# +FROM alpine:3.10.3 + +MAINTAINER Renaud Le Gac <legac@cppm.in2p3.fr> + +LABEL alpine=3.10.3 + +RUN apk add --no-cache texlive \ + texmf-dist-latexextra + + +ENTRYPOINT ["pdflatex"] \ No newline at end of file diff --git a/image_senchacmd/Dockerfile b/image_senchacmd/Dockerfile new file mode 100644 index 0000000000000000000000000000000000000000..6b8e3a5f2f983c2ba84de104a5e369074d2de26d --- /dev/null +++ b/image_senchacmd/Dockerfile @@ -0,0 +1,25 @@ +FROM openjdk:8-jre +LABEL maintainer "Philippe Poumaroux <poum@cpan.org>" + +ENV VERSION=6.6.0.13 + +RUN curl -o /cmd.run.zip http://cdn.sencha.com/cmd/$VERSION/no-jre/SenchaCmd-$VERSION-linux-amd64.sh.zip && \ + unzip -p /cmd.run.zip > /cmd-install.run && \ + chmod +x /cmd-install.run && \ + /cmd-install.run -q -Dall=true -dir /opt/Sencha/Cmd/$VERSION && \ + install -dm777 -o root -g root /opt/Sencha/Cmd/repo && \ + rm /cmd-install.run /cmd.run.zip && \ + ln -s /opt/Sencha/Cmd/$VERSION/sencha /opt/Sencha/sencha && \ + apt-get update && apt-get install -y --no-install-recommends \ + ruby \ + libffi6 \ + build-essential \ + ruby-dev \ + libffi-dev && \ + gem update --system && \ + gem install compass && \ + apt-get remove -y ruby-dev build-essential libffi-dev && \ + apt-get autoremove -y && \ + rm -rf /var/lib/apt/lists/* + +ENTRYPOINT ["/opt/Sencha/sencha"] diff --git a/image_sphinx/Dockerfile b/image_sphinx/Dockerfile new file mode 100644 index 0000000000000000000000000000000000000000..d07ac0a132bbba167f54a5a54da4205da58a411b --- /dev/null +++ b/image_sphinx/Dockerfile @@ -0,0 +1,46 @@ +# +# NAME +# sphinx-build +# +# SYNOPSIS +# Tools to generate documentation for python code +# It include python source code for web2py +# +# DESCRIPTION +# The base operating system is alpine with python3. +# +# To run the sphinx-build on files located in the directory docs/api +# and own by the user 1001. The output will go to directory docs/out +# +# $ cd docs +# $ docker run --rm -v `pwd`:/wd -w /wd -u 1001 sphinx:2.2.1 api out +# +# +FROM python:3.7.5-alpine3.10 + +MAINTAINER Renaud Le Gac <legac@cppm.in2p3.fr> + +ENV sphinx 2.2.1 +ENV web2py R-2.18.3 +ENV pydal 19.04 + +LABEL alpine=3.10 python=3.7.5 pydal=$pydal sphinx=$sphinx web2py=$web2py + +RUN pip install -U sphinx==$sphinx \ +# +# Install web2py python source including pydal and yatl +# + && cd /opt \ + && wget -q https://github.com/web2py/web2py/archive/$web2py.tar.gz \ + && tar xf $web2py.tar.gz \ + && mv web2py-$web2py web2py \ + && rm $web2py.tar.gz \ + && cd web2py \ + && rm -rf applications docker docs examples extras handlers \ + && rm -rf scripts site-packages \ + && rm -rf ABOUT CHANGELOG MANIFEST.in Makefile README.markdown \ + && rm -rf anyserver.py appveyor.yml fabfile.py setup.py tox.ini web2py.py \ + && pip install PYDAL==$pydal \ + && pip install yatl + +ENTRYPOINT ["/usr/local/bin/sphinx-build"] diff --git a/image_web2py/build.py b/image_web2py/build.py new file mode 100755 index 0000000000000000000000000000000000000000..ecba94919d085961efa377fa9a89e3b28e900f08 --- /dev/null +++ b/image_web2py/build.py @@ -0,0 +1,313 @@ +#!/usr/bin/python3 +""" NAME + build.py -- build docker image w2pext running the web2py service + + SYNOPSIS + build.py [Options] command [layers] + + DESCRIPTION + Build the docker images w2pext. It contains the web2py framework, + the building HTTP front-end, rocket, and third party libraries. + + The building is done in several steps: + + 1. Edit the file w2pext_base.json or launch the script + configure.py to select version of third party libraries used + by the web2py applications + 2. build the web2py image(s). + + Third party libraries are grouped in layers which can be merged at + the building time: + + * export + add the export functionalities. + Latex can be used to generated PDF file and LibreOffice + to translate HTML into odt document. + + * graph + add the graph functionalities. + The python modules pandas, as well as matplotlib are + available. + + * pytest + add pytest functionalities. + + To build web2py with layers added on top of it. + + > ./build.py + > ./build.py export + > ./build.py export graph pytest + > ./build.py all + + the release of the third party libraries are defined in the + w2pext_base.jon file. The latter can be edited or generated by using + the script configure.py. + + The name image depends on the layers selection and on + the third party dependencies: + + * w2pext:2.9.11 + web2py server release 2.9.11. + + * w2pext-egp:2.9.11 + web2py server with all layers. + + Release numbers for third party libraries are stored + in the image via docker LABEL. + + OPTIONS + -h, --help + + EXAMPLES + > ./build.py + + AUTHOR + R. Le Gac, legac@cppm.in2p3.fr + +""" +import argparse +import docker +import json +import logging +import sys + + +from pathlib import Path +from subprocess import PIPE, run + + +DOCKER_API_VERSION = "auto" +DOCKERFILE = "Dockerfile" +IMG_BASE = "w2p" +LAYERS = ["export", "graph", "pytest"] + + +def cli(): + + logging.basicConfig( + format="%(asctime)s %(levelname)s %(message)s", + level=logging.INFO, + datefmt="%H:%M:%S") + + parser = argparse.ArgumentParser() + + parser.add_argument( + "-c", "--configuration", + default="w2pext_base.json", + help="JSON configuration file [%(default)s].", + metavar="<file>", + type=argparse.FileType('r')) + + parser.add_argument( + "-s", "--save-dockerfile", + action="store_true", + help="Save the dockerFile and exit [%(default)s].") + + txt = "possible values are %s and all" % ", ".join(LAYERS) + parser.add_argument( + "layers", + help=txt + " [%(default)s].", + nargs="*") + + parser.set_defaults(func=build_w2pext_base) + + args = parser.parse_args() + args.func(args) + + +def build(name, meta): + """build the docker image + + Args: + name (str): name of the image. + meta (dict): + the keys are: + * jsduck + * matplotlib + * numpy + * pandas + * pydal + * sphinx + * tag + * waps_build + * web2py + + """ + client = docker.from_env(version=DOCKER_API_VERSION) + + # ........................................................................ + # + # build the image removing intermediate container + # use the latest centos image + # use the low level API in order to display the build messages + # + print() + logging.info(f"Construct the image {name}...") + + out = client.api.build(path=".", + buildargs=meta, + pull=True, + rm=True, + tag=name) + + for line in out: + di = json.loads(line) + if "stream" in di: + print(di["stream"].strip("\n").replace(" \n", "", 1)) + + # ........................................................................ + # + # display the history of the image + # + # TODO: + # img = client.images.get(name) + # for elt in img.history(): + # blabla ... + # be careful the history is in the reserver order + print() + print("%-8s %-45s %-8s %-s" % ("IMAGE", "CREATED BY", "SIZE", "COMMENT")) + + try: + client.images.get(name) + + fmt = "%-8s %-45s %05.1f MB %-s" + for elt in client.api.history(name): + print(fmt % (elt["Id"][7:15], + elt["CreatedBy"][:45], + round(elt["Size"]/1E6, 1), + elt["Comment"])) + + except docker.errors.ImageNotFound: + logging.error(f"image {name} not found!!!") + logging.error("\ttry: docker images...\n") + return + + # ........................................................................ + # + # Display the list of existing images + # + print() + print("%-22s %-15s %-15s %s" % ("REPOSITORY", "TAG", "IMAGE ID", "SIZE")) + + fmt = "%-22s %-15s %-15s %5.3f GB" + for img in client.images.list(): + attrs = img.attrs + + lst = attrs["RepoTags"] + value = (lst[0] if len(lst) > 0 else "<none>:<none>") + name, tag = value.split(":") + + if name.startswith(IMG_BASE): + print(fmt % (name, + tag, + attrs["Id"][7:15], + round(attrs["Size"]/1E9, 3))) + + +def build_w2pext_base(args): + """Build the w2pext image for the base edition. + + Args: + args (argparse.Namespace): command arguments + + """ + print() + + # check that the docker service is running + try: + client = docker.from_env(version=DOCKER_API_VERSION) + client.ping() + + except BaseException: + logging.error("docker daemon is not running!!!") + print() + sys.exit(1) + + logging.info("start building w2pext image...") + + # list of additional layers + if args.layers is None: + args.layers = list() + + elif len(args.layers) == 1 and args.layers[0] == "all": + args.layers = LAYERS + + else: + delta = set(args.layers).difference(set(LAYERS)) + if delta: + logging.error("Invalid layer(s): {}".format(" ".join(list(delta)))) + print() + sys.exit(1) + + # meta data and image name + logging.info("prepare the dockerfile...") + + meta = json.load(args.configuration) + + proc = run(["git", "describe", "--always"], stdout=PIPE, check=True) + meta["waps_build"] = proc.stdout.decode("utf-8").strip("\n") + + img_name = image_name(args, meta) + + # build the docker file + pdocker = Path(DOCKERFILE) + if pdocker.exists(): + pdocker.unlink() + + logging.info("\tadd the web2py layer") + + layer = "web2py_latest" if meta["web2py"] == "latest" else "web2py_archive" + player = Path("layers", layer) + + with pdocker.open("w") as fdocker: + fdocker.write(player.read_text()) + + # add optional layers + for layer in ("export", "graph", "pytest"): + if layer in args.layers: + logging.info(f"\tadd the {layer} layer") + player = Path("layers", layer) + fdocker.write(player.read_text()) + + logging.info(f"\tdockerfile is {pdocker.absolute()}") + if args.save_dockerfile: + sys.exit(0) + + # build the image from the Dockerfile + build(img_name, meta) + + # clean the Dockerfile + logging.info("\tclean Dockerfile\n") + pdocker.unlink() + + +def image_name(args, meta): + """Construct the name of the image from meta-data. + + Args: + args (argparse.Namespace): command arguments + meta (dict): + + returns: + str: + w2pegp:2.18.5 + + """ + name = IMG_BASE + + # list of layers + layers = [el[0:1] for el in args.layers if el != "web2py"] + layers.sort() + if len(layers) > 0: + name = "{}{}".format(name, "".join(layers)) + + # web2py release + name = "{}:{}".format(name, meta["tag"]) + + return name + + +if __name__ == '__main__': + + cli() + sys.exit(0) diff --git a/configure.py b/image_web2py/configure.py similarity index 59% rename from configure.py rename to image_web2py/configure.py index 56ea0c7685450bf26243a52ad4202587f01bb054..2bc306708153a405f29af38f14ea70eee5212c6c 100755 --- a/configure.py +++ b/image_web2py/configure.py @@ -1,5 +1,4 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- +#!/usr/bin/python3 """ NAME configure.py -- create the configuration file for web2py image @@ -31,74 +30,85 @@ import sys from subprocess import check_output -CFG_FN = "web2py.json" +CFG_FN = "w2pext_base.json" JSDUCK_TAGS = "https://api.github.com/repos/senchalabs/jsduck/tags" MATPLOTLIB_TAGS = "https://api.github.com/repos/matplotlib/matplotlib/tags" -NUMPY_TAGS = "https://api.github.com/repos/numpy/numpy/tags" -PANDAS_TAGS = "https://api.github.com/repos/pydata/pandas/tags" +PANDAS_TAGS = "https://api.github.com/repos/pandas-dev/pandas/tags" PYDAL_TAGS = "https://api.github.com/repos/web2py/pydal/tags" +PYTEST_TAGS = "https://api.github.com/repos/pytest-dev/pytest/tags" SPHINX_TAGS = "https://api.github.com/repos/sphinx-doc/sphinx/tags" WEB2PY_TAGS = "https://api.github.com/repos/web2py/web2py/tags" -def do_configuration(args): - """ - Args: - args (argparse.Namespace): command arguments +def cli_options(): - """ - meta = dict(web2py="", - pydal="", - numpy="", - matplotlib="", + parser = argparse.ArgumentParser(usage="%(prog)s <command> [option]") + + parser.add_argument( + "-d", "--directory", + default=os.getcwd(), + help="directory to store configuration files [%(default)s]", + metavar="<dir>") + + return parser.parse_args() + + +def cli(): + + args = cli_options() + + meta = dict(matplotlib="", pandas="", - jsduck="", - sphinx="", - tag="") + pydal="", + pytest="", + tag="", + web2py="") # web2py and pydal-- require by the base layer # pydal is needed when web2py version > 2.9.12 meta["web2py"] = select_tag(WEB2PY_TAGS) - release_map = [int(el) for el in meta["web2py"].split(".")] - if release_map > [2, 9, 12]: + require_pydal = True + if meta["web2py"] != "latest": + ver = meta["web2py"].replace("R-", "") + release_map = [int(el) for el in ver.split(".")] + if release_map <= [2, 9, 12]: + require_pydal = False + + if require_pydal: meta["pydal"] = select_tag(PYDAL_TAGS) - # numpy, mathplotlib and pandas -- require by the graph layer - meta["numpy"] = select_tag(NUMPY_TAGS) + # matplotlib, pandas and pytest meta["matplotlib"] = select_tag(MATPLOTLIB_TAGS) meta["pandas"] = select_tag(PANDAS_TAGS) - - # jsduck and sphinx -- require by the doc layer - meta["jsduck"] = select_tag(JSDUCK_TAGS) - meta["sphinx"] = select_tag(SPHINX_TAGS) + meta["pytest"] = select_tag(PYTEST_TAGS) # image tag: tag = meta["web2py"] - rep = raw_input("\nImage tag [%s]: " % tag) + rep = input(f"\nImage tag [{tag}]: ") meta["tag"] = (tag if len(rep) == 0 else rep) # write the configuration files fn = CFG_FN - rep = raw_input("Name of the configuration file [%s]:" % fn) + rep = input(f"Name of the configuration file [{fn}]:" ) if len(rep) > 0: fn = rep path = os.path.join(args.directory, fn) - fi = open(path, "w") - json.dump(meta, fi) - fi.close() + with open(path, "w") as fi: + json.dump(meta, fi, indent=4, separators=(",", ": "), sort_keys=True) - print "configuration file saved in", path, "\n" + print(f"configuration file saved in {path} \n") def github_tags(url): """Retrieve the list of valid tags from github. - It uses the github API. T + It uses the github API. Args: - url (str): the github url containing the list of tag, i.e + url (str): + the github url containing the list of tag, i.e "https://api.github.com/repos/numpy/numpy/tags" Returns: @@ -106,15 +116,15 @@ def github_tags(url): """ out = check_output(["curl", "-s", url]) - return [di[u"name"] for di in json.loads(out)] + return [di["name"] for di in json.loads(out)] def select_tag(url): """Select a release for a software located in github. Args: - software (str): name of the software - url (str): the github url containing the list of tag for the + url (str): + the github url containing the list of tag for the software, i.e "https://api.github.com/repos/numpy/numpy/tags" Returns: @@ -147,19 +157,19 @@ def select_tag(url): li = list(tags[:20]) li[19] = "..." - print - print "\nLatest releases for", software, "are:" - for i in xrange(5): - print "\t%15s %15s %15s %15s" % (li[i], li[i+5], li[i+10], li[i+15]) + print() + print(f"\nLatest releases for {software} are:") + for i in range(5): + print(f"\t{li[i]:15} {li[i+5]:15} {li[i+10]:15} {li[i+15]:15}") # select the release - rep = raw_input("\nSelect the release [%s]: " % tags[0]) + rep = input(f"\nSelect the release [{tags[0]}]: ") if len(rep) == 0: return tags[0] elif rep not in tags: - print "\n\tThe release", rep, "does not exist." - print "\tCheck on %s\n" % url + print(f"\n\tThe release {rep} does not exist.") + print(f"\tCheck on {url}\n") sys.exit(1) return rep @@ -167,15 +177,5 @@ def select_tag(url): if __name__ == '__main__': - APS = argparse.ArgumentParser(usage="%(prog)s <command> [option]") - - APS.add_argument("-d", "--directory", - default=os.getcwd(), - help="directory to store configuration files " - "[%(default)s]", - metavar="<dir>") - - ARGS = APS.parse_args() - - do_configuration(ARGS) + cli() sys.exit(0) diff --git a/image_web2py/layers/export b/image_web2py/layers/export new file mode 100644 index 0000000000000000000000000000000000000000..ba8d53f566fe97a0f1801a560491e5686ca5a7e7 --- /dev/null +++ b/image_web2py/layers/export @@ -0,0 +1,14 @@ +# +# NAME +# export layer +# +# SYNOPSIS +# Add export functionalities to the base web2py service +# +# DESCRIPTION +# Add pdflatex and LibreOffice converter +# + +RUN apk add --no-cache libreoffice-writer \ + texlive \ + texmf-dist-latexextra diff --git a/image_web2py/layers/graph b/image_web2py/layers/graph new file mode 100644 index 0000000000000000000000000000000000000000..fb93da8f9f408fc8f9b9087c018cc8448a8e03f8 --- /dev/null +++ b/image_web2py/layers/graph @@ -0,0 +1,17 @@ +# +# NAME +# graph layer +# +# SYNOPSIS +# Add graph funtionalities to the web2py server. +# +# DESCRIPTION +# Add the python modules pandas, matplotlib and their friends +# +ARG matplotlib +ARG pandas + +LABEL matplotlib=$matplotlib pandas=$pandas + +RUN /opt/miniconda/bin/conda install -q matplotlib==$matplotlib \ + && /opt/miniconda/bin/conda install -q pandas==$pandas diff --git a/image_web2py/layers/pytest b/image_web2py/layers/pytest new file mode 100644 index 0000000000000000000000000000000000000000..7c02768d98eddee3c52f8ca16b99a7e07b3d2bec --- /dev/null +++ b/image_web2py/layers/pytest @@ -0,0 +1,16 @@ +# +# NAME +# pytest layer +# +# SYNOPSIS +# Add pytest funtionalities to the web2py server. +# +# DESCRIPTION +# Add the python modules pytest and pytest-profiling +# +ARG pytest + +LABEL pytest=$pytest + +RUN /opt/miniconda/bin/conda install -q pytest==$pytest \ + && /opt/miniconda/bin/pip install -q pytest-profiling diff --git a/image_web2py/layers/web2py_archive b/image_web2py/layers/web2py_archive new file mode 100644 index 0000000000000000000000000000000000000000..e6eeff4a37603a99ed751d3b1874ab83cada5903 --- /dev/null +++ b/image_web2py/layers/web2py_archive @@ -0,0 +1,62 @@ +# +# NAME +# web2py layer +# +# SYNOPSIS +# The base image for a web2py server +# +# DESCRIPTION +# The base image is alpine. +# The basic command "conda", "pip" are added. +# Conda is installed in /opt/miniconda +# The web2py server is installed in /opt/web2py. +# The "welcome" and "examples" applications are removed +# The "admin" application is kept (it can alo be removed if need) +# Add a dummy file to avoid the creation of the welcome application +# +FROM alpine:3.10.3 + +MAINTAINER Renaud Le Gac <legac@cppm.in2p3.fr> + +ARG web2py +ARG pydal +ARG waps_build + +LABEL alpine=3.10.3 web2py=$web2py pyDAL=$pydal waps_build=$waps_build + +ENV GLIBC 2.30-r0 +ENV MINICONDA Miniconda3-latest-Linux-x86_64.sh + +RUN \ +# +# Install a version of glibc required by the miniconda insaller +# From https://hub.docker.com/r/petronetto/miniconda-alpine +# + wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub \ + && wget -q https://github.com/sgerrand/alpine-pkg-glibc/releases/download/$GLIBC/glibc-$GLIBC.apk \ + && wget -q https://github.com/sgerrand/alpine-pkg-glibc/releases/download/$GLIBC/glibc-bin-$GLIBC.apk \ + && apk add glibc-$GLIBC.apk \ + && apk add glibc-bin-$GLIBC.apk \ + && rm glibc-$GLIBC.apk glibc-bin-$GLIBC.apk \ +# +# Install miniconda +# From https://hub.docker.com/r/petronetto/miniconda-alpine +# + && wget -q https://repo.anaconda.com/miniconda/$MINICONDA \ + && /bin/sh $MINICONDA -f -b -p /opt/miniconda \ + && rm $MINICONDA \ + && /opt/miniconda/bin/conda init \ +# +# Install web2py +# + && cd /opt \ + && wget -q https://github.com/web2py/web2py/archive/R-$web2py.tar.gz \ + && tar xzf R-$web2py.tar.gz \ + && mv web2py-R-$web2py web2py \ + && rm R-$web2py.tar.gz \ + && rm -rf /opt/web2py/applications/examples \ + && rm -rf /opt/web2py/examples \ + && rm -rf /opt/web2py/applications/welcome \ + && touch /opt/web2py/welcome.w2p \ + && /opt/miniconda/bin/pip install -q PYDAL==$pydal \ + && /opt/miniconda/bin/pip install -q yatl diff --git a/image_web2py/layers/web2py_latest b/image_web2py/layers/web2py_latest new file mode 100644 index 0000000000000000000000000000000000000000..83759c0bb47652a8c6875cad7e545802fbaf29b0 --- /dev/null +++ b/image_web2py/layers/web2py_latest @@ -0,0 +1,37 @@ +# +# NAME +# web2py layer +# +# SYNOPSIS +# The base image for a web2py server +# +# DESCRIPTION +# The base image is alpine. +# Take the latest release for web2py. It contains pyDAL and yatl +# The web2py server is installed in /opt/web2py. +# The "welcome" and "examples" applications are removed +# The "admin" application is kept (it can alo be removed if need) +# Add a dummy file to avoid the creation of the welcome application +# +FROM alpine:3.10.3 + +MAINTAINER Renaud Le Gac <legac@cppm.in2p3.fr> + +ARG web2py +ARG waps_build + +LABEL alpine=3.10.3 web2py=$web2py waps_build=$waps_build + + +RUN apk add --no-cache python3 \ +# +# Install web2py +# + && cd /opt \ + && wget -q http://web2py.com/examples/static/web2py_src.zip \ + && unzip web2py_src.zip \ + && rm web2py_src.zip \ + && rm -rf /opt/web2py/applications/examples \ + && rm -rf /opt/web2py/examples \ + && rm -rf /opt/web2py/applications/welcome \ + && touch /opt/web2py/welcome.w2p diff --git a/image_web2py/w2pext_base.json b/image_web2py/w2pext_base.json new file mode 100644 index 0000000000000000000000000000000000000000..75bc60c09643f001746122df59aaa171c7d173e5 --- /dev/null +++ b/image_web2py/w2pext_base.json @@ -0,0 +1,8 @@ +{ + "matplotlib": "3.1.1", + "pandas": "0.25.2", + "pydal": "19.04", + "pytest": "5.2.2", + "tag": "2.18.3", + "web2py": "2.18.3" +} diff --git a/layers/base_layer b/layers/base_layer deleted file mode 100644 index 56d2ad7566c2463f8886d7c17f6092114f17c87b..0000000000000000000000000000000000000000 --- a/layers/base_layer +++ /dev/null @@ -1,22 +0,0 @@ -# -# NAME -# base -# -# SYNOPSIS -# The base image for a web2py server -# -# DESCRIPTION -# The base operating system is centos with EPEL repository. -# The basic command "pip" is added. -# -FROM centos:7 - -MAINTAINER Renaud Le Gac <legac@cppm.in2p3.fr> - -RUN yum -y install deltarpm \ - && yum -y update \ - && yum -y install epel-release \ - && yum -y update \ - && yum -y install python-pip which \ - && yum clean all \ - && pip install --upgrade pip diff --git a/layers/doc_layer b/layers/doc_layer deleted file mode 100644 index f2a90276c30abc33bcc53e8c73651310a52b011e..0000000000000000000000000000000000000000 --- a/layers/doc_layer +++ /dev/null @@ -1,54 +0,0 @@ -# -# NAME -# doc layer -# -# SYNOPSIS -# Add tools to generated documentation. -# -# DESCRIPTION -# Add the tool jsduck and sphinx -# -#............................................................................. -LABEL jsduck = $jsduck - -RUN yum -y install gcc \ - make \ - ruby \ - ruby-devel \ - rubygem-rake \ - rubygem-rake-compiler \ - && yum clean all \ - && gem install rdiscount \ - && gem install jsduck -v $jsduck - -#............................................................................. -LABEL Sphinx = $sphinx - -RUN yum -y update \ - && yum -y install texlive-cmap \ - texlive-cm-super \ - texlive-ec \ - texlive-fancybox \ - texlive-collection-fontsrecommended \ - texlive-framed \ - texlive-mdwtools \ - texlive-multirow \ - texlive-parskip \ - texlive-threeparttable \ - texlive-titlesec \ - texlive-wrapfig \ - && yum clean all - -COPY rpms/ /opt/rpms/ - -RUN cd /opt/rpms \ - && yum -y install texlive-capt-of-svn29803.0-19.fc23.noarch.rpm \ - texlive-trimspaces-svn15878.1.1-18.fc23.noarch.rpm \ - texlive-environ-svn29600.0.3-19.fc23.noarch.rpm \ - texlive-eqparbox-svn29419.4.0-19.fc23.noarch.rpm \ - texlive-needspace-svn29601.1.3d-19.fc23.noarch.rpm \ - texlive-upquote-svn26059.v1.3-18.fc23.noarch.rpm \ - && yum clean all \ - && rm -rf /opt/rpms - -RUN pip install Sphinx==$sphinx diff --git a/layers/export_layer b/layers/export_layer deleted file mode 100644 index a829f33ae21c5bd3214af9b2b2aa607b77597594..0000000000000000000000000000000000000000 --- a/layers/export_layer +++ /dev/null @@ -1,22 +0,0 @@ -# -# NAME -# export layer -# -# SYNOPSIS -# Add export functionalities to the base web2py service -# -# DESCRIPTION -# Add latex, pdflatex and LibreOffice writer -# -RUN yum -y install libreoffice-headless \ - libreoffice-writer \ - texlive-collection-latex \ - texlive-ucs \ - && yum clean all - -COPY rpms/ /opt/rpms/ - -RUN cd /opt/rpms \ - && yum -y install texlive-paper-svn25802.1.0l-19.fc23.noarch.rpm \ - && yum clean all \ - && rm -rf /opt/rpms diff --git a/layers/graph_layer b/layers/graph_layer deleted file mode 100644 index 290781900059b3bc76045c483c4c10a77b679fb2..0000000000000000000000000000000000000000 --- a/layers/graph_layer +++ /dev/null @@ -1,35 +0,0 @@ -# -# NAME -# graph layer -# -# SYNOPSIS -# Add graph funtionalities to the base web2py service. -# -# DESCRIPTION -# Add the python modules pandas, matplotlib and their friends -# The installation is performed using pip. -# - -LABEL matplotlib = $matplotlib -LABEL numpy = $numpy -LABEL pandas = $pandas - -RUN yum -y install Cython \ - agg-devel \ - atlas-devel \ - cairo-devel \ - freetype-devel \ - gcc \ - gcc-c++ \ - gcc-fortan \ - lapack-devel \ - libpng-devel \ - python-devel \ - python-pycxx-devel \ - && yum clean all - -RUN pip install numpy==$numpy \ - matplotlib==$matplotlib \ - pandas==$pandas \ - statsmodels \ - numexpr diff --git a/layers/senchacmd_layer b/layers/senchacmd_layer deleted file mode 100644 index 4f3ec6a310e06601009669d00dbf974b8daed8a9..0000000000000000000000000000000000000000 --- a/layers/senchacmd_layer +++ /dev/null @@ -1,31 +0,0 @@ -# -# NAME -# senchacmd layer -# -# SYNOPSIS -# Add tool to generate javascript libraries. -# -# DESCRIPTION -# Add the command sencha -# -# NOTE -# list of SenchaCmd releases can be found at: -# https://www.sencha.com/products/extjs/cmd-download/ -# -# A release can be upgraded: -# > sencha upgrade -check -# > sencha upgrade - -LABEL SenchaCmd = 6.1.3.42 - -RUN yum -y install java-1.8.0-openjdk unzip \ - && yum clean all \ - && cd /opt \ - && mkdir Sencha Sencha/Cmd Sencha/Cmd/6.1.3.42 \ - && curl -sSL -o tmp.sh.zip https://cdn.sencha.com/cmd/6.1.3/no-jre/SenchaCmd-6.1.3-linux-amd64.sh.zip \ - && unzip tmp.sh.zip \ - && rm tmp.sh.zip \ - && ./SenchaCmd-6.1.3.42-linux-amd64.sh -q -dir /opt/Sencha/Cmd/6.1.3.42 \ - && rm SenchaCmd-6.1.3.42-linux-amd64.sh - -COPY sencha /usr/local/bin/ \ No newline at end of file diff --git a/layers/web2py_layer b/layers/web2py_layer deleted file mode 100644 index b5b1659ae8adac8d1232a067c87f5f42124ed12e..0000000000000000000000000000000000000000 --- a/layers/web2py_layer +++ /dev/null @@ -1,27 +0,0 @@ -# -# NAME -# web2py layer -# -# SYNOPSIS -# The web2py server -# -# DESCRIPTION -# The web2py server is installed in /opt/web2py. -# The "welcome" and "examples" applications are removed -# The "admin" application is kept (it can alo be removed if need) -# Add a dummy file to avoid the creation of the welcome application -# Install PyDAL and pytest -# -LABEL web2py = $web2py -LABEL pyDAL = $pydal -LABEL waps_build = $waps_build - -RUN cd /opt \ - && curl -sSL https://github.com/web2py/web2py/archive/R-$web2py.tar.gz | tar xz \ - && mv web2py-R-$web2py web2py \ - && rm -rf web2py/applications/examples web2py/applications/welcome \ - && touch /opt/web2py/welcome.w2p - -RUN pip install --upgrade pip \ - && pip install PYDAL==$pydal \ - && pip install pytest diff --git a/rpms/texlive-capt-of-svn29803.0-19.fc23.noarch.rpm b/rpms/texlive-capt-of-svn29803.0-19.fc23.noarch.rpm deleted file mode 100644 index dcbbd4c67f48d1552a93e63f4396989688a696f7..0000000000000000000000000000000000000000 Binary files a/rpms/texlive-capt-of-svn29803.0-19.fc23.noarch.rpm and /dev/null differ diff --git a/rpms/texlive-environ-svn29600.0.3-19.fc23.noarch.rpm b/rpms/texlive-environ-svn29600.0.3-19.fc23.noarch.rpm deleted file mode 100644 index 4c6137019f2c98739be25fc5a5dd7aed4cdb4241..0000000000000000000000000000000000000000 Binary files a/rpms/texlive-environ-svn29600.0.3-19.fc23.noarch.rpm and /dev/null differ diff --git a/rpms/texlive-eqparbox-svn29419.4.0-19.fc23.noarch.rpm b/rpms/texlive-eqparbox-svn29419.4.0-19.fc23.noarch.rpm deleted file mode 100644 index 439781997c7a6599ee200e44db62256a8ded8aa1..0000000000000000000000000000000000000000 Binary files a/rpms/texlive-eqparbox-svn29419.4.0-19.fc23.noarch.rpm and /dev/null differ diff --git a/rpms/texlive-needspace-svn29601.1.3d-19.fc23.noarch.rpm b/rpms/texlive-needspace-svn29601.1.3d-19.fc23.noarch.rpm deleted file mode 100644 index 3ebb7c08b5b4ec57930db92c4fb3490d3f21e80d..0000000000000000000000000000000000000000 Binary files a/rpms/texlive-needspace-svn29601.1.3d-19.fc23.noarch.rpm and /dev/null differ diff --git a/rpms/texlive-paper-svn25802.1.0l-19.fc23.noarch.rpm b/rpms/texlive-paper-svn25802.1.0l-19.fc23.noarch.rpm deleted file mode 100644 index a5c6ad9563fb84a9365e75351447b433dd28c30d..0000000000000000000000000000000000000000 Binary files a/rpms/texlive-paper-svn25802.1.0l-19.fc23.noarch.rpm and /dev/null differ diff --git a/rpms/texlive-trimspaces-svn15878.1.1-18.fc23.noarch.rpm b/rpms/texlive-trimspaces-svn15878.1.1-18.fc23.noarch.rpm deleted file mode 100644 index 0514a0839873b375cc40d86c31f3388e28b10fb9..0000000000000000000000000000000000000000 Binary files a/rpms/texlive-trimspaces-svn15878.1.1-18.fc23.noarch.rpm and /dev/null differ diff --git a/rpms/texlive-upquote-svn26059.v1.3-18.fc23.noarch.rpm b/rpms/texlive-upquote-svn26059.v1.3-18.fc23.noarch.rpm deleted file mode 100644 index bb8b895195c8e1d718555c1b05bb6b00342f0bae..0000000000000000000000000000000000000000 Binary files a/rpms/texlive-upquote-svn26059.v1.3-18.fc23.noarch.rpm and /dev/null differ diff --git a/sencha b/sencha deleted file mode 100755 index 865b88d317f75b45062b49efc840c499c03e266c..0000000000000000000000000000000000000000 --- a/sencha +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/bash -# -# Run the Sencha Cmd -# http://www.sencha.com/products/sencha-cmd/download -# -# Complete documentation can be found in the Ext JS documentation: -# http://docs.sencha.com/extjs/4.2.2/#!/guide/command -# -cd /opt/Sencha/Cmd -./sencha $* diff --git a/w2pext.json b/w2pext.json deleted file mode 100644 index f7e85a0106684dad8b32f321000f7cfd1204e7bc..0000000000000000000000000000000000000000 --- a/w2pext.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "sphinx": "1.4.1", - "jsduck": "5.3.4", - "matplotlib": "1.5.1", - "pandas": "0.18.1", - "web2py": "2.14.6", - "numpy": "1.11.0", - "pydal": "16.03", - "tag": "2.14.6-ext6" -} \ No newline at end of file