VIFIB DESCENTRALIZED CLOUD COMPUTING

SlapOS is a decentralized Cloud Computing technology that can automate the deployment and configuration of applications in a heterogeneous environment.

How to customize SlapOS

You can view this tutorial in presentation mode.

Agenda

  • SlapOS Internals
  • How to customize the "helloworld" software release step by step:
    1. How to customize each instance of a unique software
    2. How to add a simple feature
    3. How to add new components

Requirements

  • Good understanding of Buildout (via tutorials on buildout.org)
  • SlapOS Tutorials (day 1)
  • SlapOS PHPMyAdmin Tutorial
  • Git knowledge

SlapOS internals

Same as Buildout

  • Profile = Software Release (software.cfg)
  • Recipe = slapos.cookbook

Profiles

SlapOS uses Buildout to install software and deploy instances.

In SlapOS, a Software Release is defined by its URL. It is in fact a Buildout Profile. When SlapOS installs a Software Release, it launches Buildout with this URL.

A Software Release file, named software.cfg, can reuse other Buildout profiles using the extends mechanism. It allows you to create a software.cfg in a few lines.

Many different profiles exist in SlapOS. You can use components that are utilities and/or binaries like cURL, PHP, Apache, Java. You can also use profiles installing libraries like libxml, to fulfil requirements.

If you need, you can also use what we call "stacks". A stack is a group of components working together, in a reusable way. For example, when creating a PHP Sofware Release, you will extend the "LAMP" stack, containing all what is needed to run PHP applications (Apache, MySQL, etc) so that you can define only what is specific to your software in a few lines.

Recipes

In order to deploy Software Instances, SlapOS uses custom recipes. If you have read the Buildout tutorial, you know that recipes are Python Eggs. You can see the source code of these recipes by looking in slapos/recipe. Recipes achieving high quality level are included in a single Egg : slapos.cookbook. You can of course use your own egg, usually mycompanyname.recipe.myinstancename. For more information about defining your own custom recipes please visit www.buildout.org/en/latest/docs/recipe.html

Git repository architecture

  • /component
       Software buiding profiles
  • /stack
      Software release wit multiple instances profiles
  • /software
      Software releases
  • /slapos/recipe
      Recipes to build software releases

The official git repository is separated in several directories. Each category defined in the previous section can be found here.

How to customize the "helloworld" software release step by step

Use cases

  • You want to give to some instances a special behaviour, depending on the use case.
  • You want to add a new functionnality to your software, using a software which already exists.

Presentation of the example

We will modify the recipe of the "helloworld" software. Currently, the "helloworld"
software will create a simple web server that publishes the text "hello <configuration.name>" to the web.
Where configuration.name is defined as one of the instance parameters. 

How to customize each instance of a unique software

The Software Release of "hello world" is composed of two files:

  • "software.cfg", which will install the software release on the computer, and
  • "instance.cfg.in" which will be copied and filled by "software.cfg".
  • It will be used to deploy (configure and run) instances of your software. After buildout has filled
    the instance.cfg.in file you should see a hidden file called .installed.cfg, according to their environment.

    Let's have a look at the "software.cfg" file:

    [buildout]

                    
    extends = 
    # "slapos" stack describes basic things needed for 99.9% of SlapOS Software
    # Releases
      ../../stack/slapos.cfg
    # Extend here component profiles, like openssl, apache, mariadb, curl...
    # Or/and extend a stack (lamp, tomcat) that does most of the work for you
    # In this example we will extend the helloweb component.
      ../../component/helloweb/buildout.cfg
    
    parts =
    # Call installation of slapos.cookbook egg defined in stack/slapos.cfg (needed
    # in 99,9% of Slapos Software Releases)
      slapos-cookbook
    # Call creation of instance.cfg file that will be called for deployment of
    # instance
      template
    
    # Download instance.cfg.in (buildout profile used to deployment of instance),
    # replace all ${foo:bar} parameters by real values, and change $${foo:bar} to
    # ${foo:bar}
    [template]
    recipe = slapos.recipe.template
    url = ${:_profile_base_location_}/instance.cfg.in
    output = ${buildout:directory}/instance.cfg
    # MD5 checksum can be skipped for development (easier to develop).
    # While developping, comment this line to avoid a crash while deploying the software. 
    # For production, you must give the md5 hash of instance.cfg.in
    md5sum = ab51bfb473e030514997c7691951601f
    mode = 0644
                  

    The profile extends slapos.cfg (contain basic slapos deployment tools) and "component/helloweb/buildout.cfg"
    (a buildout profile to create some programs that say hello to the web in three different languages [python, ruby, and golang]).
    It also describes how to parse the "instance.cfg.in" instance profile
    templates into "instance.cfg" which will be used to configure and run the instance of our software release.  Note: read the comment about the md5 sum and then proceed to comment it out, or replace it with the md5 hash of _your_ instance.cfg.in
    Otherwise the build will fail upon checking the integrity of the file.

    A way to offer flexibility to an instance is to provide instance parameters in your requests instance on "slapos.org" using the "instance XML" field.

    Let's add a new parameter that we can be manually defined later. We'll call it "title" for the title of our web pages.

    This is the new "instance.cfg.in" file (in bold are the new and interesting lines to look at :

                
    #############################
    #
    # Deploy hello-world instance
    #
    #############################
    #
    #  In this file is explained how to add a new value that will be handled by the software 
    #
    #############################
    
    [buildout]
    parts =
    directory
    publish-connection-parameter
    
    # Define egg directories to be the one from Software Release
    # (/opt/slapgrid/...)
    # Always the same.
    eggs-directory = ${buildout:eggs-directory}
    develop-eggs-directory = ${buildout:develop-eggs-directory}
    offline = true
    
    
    [instance-parameter]
    # Fetch arbitrary parameters defined by the user in SlapOS Master for his instance.
    # We use the slapconfiguration recipe with a few parameters (partition id,
    # computer id, certificate, etc).
    # It will then authenticate to SlapOS Master and fetch the instance parameters.
    # The parameters are accessible from $${instance-parameter:configuration.parameter-name}
    # Always the same. Just copy/paste.
    # See docstring of slapos.cookbook:slapconfiguration for more informations.
    recipe = slapos.cookbook:slapconfiguration
    computer = $${slap_connection:computer_id}
    partition = $${slap_connection:partition_id}
    url = $${slap_connection:server_url}
    key = $${slap_connection:key_file}
    cert = $${slap_connection:cert_file}
    # Define default parameter(s) that will be used later, in case user didn't
    # specify it.
    # All used parameters should have a default in order to avoid an error if the parameter is missing form the request.
    # In our use case, we are expecting from the user to specify one (optional) parameter: "name". We put the default value here if he doesn't specify it, so that it doesn't crash.
    configuration.name = John Doe
    # Here is the declaration of the default value for the parameter "title" (None here). If commented, the recipe will crash.
    # The "configuration." part of the parameter name means that the value will be the one given (if present) in instance parameters
    configuration.title = 
                
    
    # Create all needed directories, depending on your needs
    
    [directory]
    recipe = slapos.cookbook:mkdirectory  # Note use of recipe to make directories.
    home = $${buildout:directory}
    etc = $${:home}/etc
    var = $${:home}/var
    # Executables put here will be started but not monitored (for start-up/reload scripts)
    script = $${:etc}/run/
    # Executables put here will be started and monitored (for daemons)
    service = $${:etc}/service
    # Executables put here will be launched after buildout has completed to see
    # if instance is running
    promise = $${:etc}/promise/
    # Path of the log directory used by our service (see [hello-world])
    log = $${:var}/log
    
    # Create a simple web server that says "hello <configuration.name>" to the web.
    [helloweb]
    # helloworld service is listening on:
    # - global IPv6 address, and
    # - fixed port
    #
    # NOTE because every computer partition is allocated its own global IPv6
    # address, it is ok to fix the port - different hello-world instances will have
    # different IPv6 addresses and they all will be accessible at the same time.
    ipv6 = ${instance-parameter:ipv6-random}
    # full URL - for convenience
    url = http://[${:ipv6}]:${:port}
    
    # the service will log here
    logfile = ${directory:log}/helloweb-${:kind}.log
    
    # Actual script that starts the service:
    # This recipe will try to "exec" the command-line after separating parameters.
    recipe = slapos.cookbook:wrapper
    command-line =
    {{ buildout['bin-directory'] }}/helloweb-${:kind} --logfile ${:logfile}
      ${:ipv6} ${:port} ${instance-parameter:configuration.name}
    # Put this shell script in the "etc/service" directory. Each executable of this
    # repository will be started and monitored by supervisord. If a service
    # exits/crashes, it will trigger a "bang" and cause a re-run of the instance.
    wrapper-path = ${directory:service}/helloweb-${:kind}
    
    

    # promise, that checks that helloweb service is alive [helloweb-promise] recipe = slapos.cookbook:check_port_listening path = ${directory:promise}/helloweb-${:kind} {# macro to instantiate service of `kind` to listen on `port` #} {% set service_list = [] %} {% macro hellowebsrv(kind, port) %} {% do service_list.append(kind) %} [helloweb-{{ kind }}] <= helloweb kind = {{ kind }} port = {{ port }} [helloweb-{{ kind }}-promise] <= helloweb-promise kind = {{ kind }} hostname= ${helloweb-{{ kind }}:ipv6} port = {{ port }} {% endmacro %} # services instantiation {{ hellowebsrv('python', 7777) }} {{ hellowebsrv('ruby', 7778) }} {{ hellowebsrv('go', 7779) }} # Publish all the parameters needed for the user to connect to the instance. # It can be anything: URL(s), password(s), or arbitrary parameters. # Here we'll just echo back the entered name as instance parameter [publish-connection-parameter] recipe = slapos.cookbook:publish name = Hello $${instance-parameter:configuration.name}! # Adds the published parameter "title". By default, its value is the value declared in the section [instance-parameter] above. # The "publish" recipe lets you give information to the user title = $${instance-parameter:configuration.title} {%- for kind in service_list %} url.{{ kind }} = ${helloweb-{{ kind }}:url} {%- endfor %}

    While there is a lot going on here, you only need to add some text next to the configuration.title parameter.
    You can now check the result in the webrunner, by looking at "Inspect instance">"SLAP Response". It is also one of the info which will be given on slapos.org, in the "My Services" section.

    How to add a simple feature

    This part explains how to add a new feature to your software release. After having manipulated existing sections, we will add new ones with different recipes to add it.

    In this example, we will add a file in the instance tree using a template and instance parameters (here rendered Jinja2). It will show the method used to write configuration files for services. 

    Prepare it in the software profile

    Here is the Jinja2 template that we will be using. Please copy it in a new folder "template". (For more information, find and read the Jinja doc).

    {% if title %}

    {{ title }}
    -
    {% endif %}
    Hello {{ name }} 

     We now have to modify the software profile ("software.cfg") to import the template in the software release folder. Add this to the end of "software.cfg" :


    # To make a render thanks to Jinja2, first we need to get the template
    # The url is where to find it. Location is just there to share information through buildout, and corresponds to a default location
    [template-get]
    recipe = hexagonit.recipe.download
    url = ${:_profile_base_location_}/template/${:filename}
    location = ${buildout:parts-directory}/${:_buildout_section_name_}
    filename = rendered.in
    download-only = true
    mode = 0644

    Do not forget to include the name of the new part in the section [buildout]. 

    Deploy it during instance set up

    It is now time to use it in our instance profile by adding a new section with the needed arguments. Here is the diff doing so.

    @@ -4,7 +4,7 @@
     #
     #############################
     #
    -# In this file is explained how to add a new value that the software will handle
    +# Now we are going how to add new parts, using new recipes
     #
     #############################
     [buildout]
    @@ -12,6 +12,8 @@ parts =
       directory
       publish-connection-parameter
    +# Don't forget to include the new part ! It will also call the dependency [template-get] in software.cfg
    +  template-creation
     # Define egg directories to be the one from Software Release
     # (/opt/slapgrid/...)
    @@ -55,6 +57,8 @@ recipe = slapos.cookbook:mkdirectory
     home = $${buildout:directory}
     etc = $${:home}/etc
     var = $${:home}/var
    +# Creation of the folder wich will contain the jinja template
    +data-folder = $${:home}/srv/data
     # Executables put here will be started but not monitored (for startup scripts)
     script = $${:etc}/run/
     # Executables put here will be started and monitored (for daemons)
    @@ -65,6 +69,28 @@ promise = $${:etc}/promise/
     # Path of the log directory used by our service (see [hello-world])
     log = $${:var}/log
    +
    +# It is now time for Jinja !
    +[template-creation]
    +recipe = slapos.recipe.template:jinja2
    +# the value of template will be replaced thanks to the information given in software.cfg, during software building.
    +# This is why there is only one $.
    +# The $$ symbol is processed during the buildout of software.cfg and is then changed in an only $.
    +template = ${template-get:location}/${template-get:filename}
    +# It is also possible define the template inline.
    +#template = inline:
    +#  ----------------------------------
    +#  Title : {{title}}
    +#  ----------------------------------
    +#  Hello {{name}}
    +# Here we can declare where we want to get the new file rendered by Jinja
    +rendered = $${directory:data-folder}/data1.txt
    +mode = 700
    +# In the context we can define the specific values wich will be substituted in the template
    +# "key" means that the value is defined by buildout. Go to http://git.erp5.org/gitweb/slapos.recipe.template.git/blob/HEAD:/slapos/recipe/template/README.jinja2.txt?js=1 for more information
    +context =
    +  key name instance-parameter:configuration.name +  key title instance-parameter:configuration.title

    You can now find at the location ${template-creation:rendered} a file containing the rendered template. 

    You can find a lot of recipes on the internet (see the buildout documentation), or in the slapos git repository. It is easy to find documentation on each of them in order to know mandatory and optional parameters than can be provided.

    How to add new components

    You may also need to add a new component to your software release in order to add a new feature. A lot of component are already available in slapos repository .

    In this tutorial we will implement log rotation feature using logrotate, cron and gzip. 

    Add them in the software profile

    Let's just install them, in this way ("software.cfg") :

    @@ -9,6 +9,10 @@ extends =
     # In this example we extend from the helloweb component
     # shell script.
       ../../component/helloweb/buildout.cfg
    +# Add needed components
    +  ../../component/dcron/buildout.cfg
    +  ../../component/logrotate/buildout.cfg
    +  ../../component/gzip/buildout.cfg
     parts =
     # Call installation of slapos.cookbook egg defined in stack/slapos.cfg (needed
    @@ -17,6 +21,10 @@ parts =
     # Call creation of instance.cfg file that will be called for deployment of
     # instance
       template
    +# Deployment of the new components. The section can be found
    +# in files used to extends the profile
    +  dcron
    +  logrotate

    Logrotate offer rotation of log files, cron will execute it regularly, and gzip is just here to have a more concrete example.

    Configure the new components in the instance profile

    We first need to create new directories that will contain the configs and the logs ("instance.cfg") :

    @@ -56,9 +60,11 @@ configuration.title =
     recipe = slapos.cookbook:mkdirectory
     home = $${buildout:directory}
     etc = $${:home}/etc
    +bin = $${:home}/bin
     var = $${:home}/var
    +srv = $${:home}/srv
    -data-fold = $${:home}/srv/data
    +data-fold = $${:srv}/data
     # Path of the log directory used by our service (see [hello-world])
     log = $${:var}/log
    +# Cron directories
    +cron-entries = $${:etc}/cron.d
    +crontabs = $${:etc}/crontabs
    +cronstamps = $${:etc}/cronstamps
    +# Logrotate directories
    +backup = $${:srv}/backup
    +logrotate-backup = $${:backup}/logrotate
    +logrotate-entries = $${:etc}/logrotate.d
    +

    We can now define the section to deploy the feature. Let's first install and configure cron : 

    # Deploy cron service

    [cron]
    recipe = slapos.cookbook:cron
    dcrond-binary = ${dcron:location}/sbin/crond
    cron-entries = $${directory:cron-entries}
    crontabs = $${directory:crontabs}
    cronstamps = $${directory:cronstamps}
    catcher = $${cron-simplelogger:wrapper}
    binary = $${directory:service}/crond

    # Add log to cron
    [cron-simplelogger]
    recipe = slapos.cookbook:simplelogger
    wrapper = $${directory:bin}/cron_simplelogger
    log = $${directory:log}/cron.log

    # Add a cron entry
    [cron-entry-logrotate]
    <= cron
    recipe = slapos.cookbook:cron.d
    name = logrotate
    frequency = 0 * * * *
    # Now that the cron is set up, we can define
    # the command to launch at the defined frequency
    command = $${logrotate:wrapper}

    First, we deploy cron using cron's recipe. In this recipe we use as a catcher for logs with a wrapper defined in the "cron-simplelogger" section.

    These buildout sections where inspired from the "apache-frontend" software profile. It means that if you don't want to spend time to understand every recipe, you can just copy the section from another profile having the wanted feature and change a few lines to match your needs (here it was just the command to launch and the frequency in [cron-entry-logrotate]). 

    Let's now add the logrogate section as followed:

    # Deploy Logrotate
    # To know more about necessary arguments
    # and why they are used for, you need to read the recipe
    # in ../../slapos/recipe/logrogate.py
    [logrotate]
    recipe = slapos.cookbook:logrotate
    # Binaries
    logrotate-binary = ${logrotate:location}/usr/sbin/logrotate
    gzip-binary = ${gzip:location}/bin/gzip
    gunzip-binary = ${gzip:location}/bin/gunzip
    # Directories
    wrapper = $${directory:bin}/logrotate
    conf = $${directory:etc}/logrotate.conf
    # in this folder will be defined tho files to save, and how to save them
    logrotate-entries = $${directory:logrotate-entries}
    # in this folder the logs will be compressed then saved
    backup = $${directory:logrotate-backup}
    state-file = $${directory:srv}/logrotate.status

    [logrotate-entry]
    <= logrotate
    recipe = slapos.cookbook:logrotate.d
    name = helloworld-logrotate
    log = $${directory:log}/log.log
    frequency = daily
    rotatep-num = 30
    sharedscripts = true
    notifempty = true
    create = true