VIFIB DESCENTRALIZED CLOUD COMPUTING

SlapOS is a decentralized Cloud Computing technology that can automate the deployment and configuration of applications in a heterogeneous environment.

SlapOS Conventions

Global concepts naming conventions

There are several concepts named differently depending on the people talking about it. There should be only the following concepts, and those concepts are not swapable each others:

  • SlapOS Node (not "Server", not "computer" except in the SlapOS Master Backend)
  • Software Release
  • Software Instance
  • Service (!= Software Instance, a service can be combined of several instances)
  • Computer Partition (!= Software Instance, an instances is deployed inside a Partition). Should become Node Partition

Buildout profile conventions

About Buildout Profiles, we follow Buildout conventions, especially:

  • Section / parameter name: Use dash, not underscore: "a-b". No "a_b". Exceptions allowed for sections named from file names: [install-foo_bar.conf] if filename is foo_bar.conf.

Exceptions:

  • Section name: Should reflect what the part installs, i.e for a part deploying kvm, name it "kvm". For a part creating directories, name it "directory".
  • To ease integration with Buildout, all falsy (empty string, None, [], ...) parameters from profiles and recipes should be equivalent to the non-existence of this parameter.

Software Releases conventions

As a reminder, here is how SlapOS works:

  • Software Releases, once correctly installed, are frozen, and won't be processed anymore: slapgrid will IGNORE them.
  • All instances are supposed to be processed (i.e Buildout is run) at least once per day. There should be no exception.

In consequence, here are a few conventions while writing profiles and recipes:

  • Stable (i.e used in production) Software Release profiles themselves should be frozen into a git tag. Acceptable:

    http://git.erp5.org/gitweb/slapos.git/blob_plain/refs/tags/slapos-0.137:/software/kvm/software.cfg
    

    Not acceptable:

    http://git.erp5.org/gitweb/slapos.git/blob_plain/refs/heads/kvm:/software/kvm/software.cfg
    
  • Instances profiles and recipes should be promise-based and shouldn't break/alter existing data of instance if any. I.e, all promises should check for existing content in instance before injecting/altering any content. Not acceptable:

    # Recipe code that blindly sets application password without checking if it has been changed
    def install(self):
       set_password()
    

    Acceptable:

    def install(self):
      if not password_already_set:
        set_password()
    
  • Developping a Software Release should be done inside of a webrunner. Developping with a bare slapproxy can be accepted but not supported.

Instance Parameters
  • For maintenance and security reasons, it should NOT be possible to give as an instance parameter a configuration file. Only atomic parameters (such as "password", "cache-size") should be allowed.
  • The same parameters are used in 90% of the cases. One should try to use the following names if using one of them, so that SlapOS end-users see the same kind of parameters:
    • host
    • ip
    • port
    • url
    • password
Exception:

Defining all needed and possible parameters in a Software Release can be tricky, resulting in a non-flexible Software Release. For practical reasons, giving a configuration file in such a case is accepted for the first versions of the Software Release. This should tend to disappear as new atomic possible parameters are added to the Software Release.

Component conventions

  • A component "foo v1.2.3" should be in a file named component/foo/buildout.cfg.

  • It should have a "main" section named "[foo]".

  • Several incompatible versions ("foo v1.2", "foo v1.3") can live in the same buildout profile (component/foo/buildout.cfg). A main section ([foo]) must extend the latest one. The different versions should be in different sections ("[foo v1.2.3]", "[foo v1.2.4]"). Example:

    [foo]
    <= foo-1.2
    
    [foo-1.2]
    recipe = slapos.recipe.cmmi
    url = http://www.foo.com/foo-1.2.6.tar.gz
    md5sum = abcdef012345679
    
    [foo-1.3]
    recipe = slapos.recipe.cmmi
    url = http://www.foo.com/foo-1.3.2.tar.gz
    md5sum = 987654321fedcba
    
  • No "parts" should be defined in the [buildout] section.

  • Patches should go to component/foo/my-patch.patch and be downloaded by a section like this:

    [my-patch.patch]
    recipe = hexagonit.recipe.download
    filename = ${:_buildout_section_name_}
    url = ${:_profile_base_location_}/${:filename}
    md5sum = ac06cbaa298ac686d0b0c04bc03e6ad8
    download-only = true
    
  • Minor dependencies only used by this component can live in the same buildout profile.

Software Instance conventions

  • Each processes of all instances composing the service (a.k.a "instance tree") must use a different port so that it can be developped in the webrunner (containing only one IPv6 and one IPv4 for itself + all the instances deployed inside of it).
  • All instances should deploy at least one "promise" script testing for fullfillment of the promise: "Can I bind to XXX?", "Does YYY answers HTTP 200?". See developer documentation to know how to do.
Example:

A Wordpress service composed by Apache instance and mariadb instance.

Apache instance contains httpd process listening to port W of IPv4 and stunnel listening to port X of IPv6.

Mariadb instance contains mysqld process listening to port X of IPv4 and stunnel listening to port Z of IPv6.

W, X should be different. X, Z should be different.

Moreover, service should not use webrunner IPv6 ports (30000, 50000, 2222).

Recipes conventions

Design conventions
  • Golden rule: re-use existing "generic" recipes like "create directory" (slapos.cookbook:makedirectory) or "create a wrapper" (slapos.cookbook:wrapper). Only highly customized Software Releases should write custom recipes.
  • Obvious duplicates should be avoided. Example: ERP5 Software Release and Wordpress Software Release should use the same recipe to deploy MySQL.
Be nice at failing
  • Errors should be obvious. If someone encounters an error, it should be easy for him to know the origin.
  • One consequence of this is "fail early". Don't silently continue then let some another recipe raise with a completely different error message.
  • Expected error (like raising because of a not-yet ready child instance) should be easy to understand, and should not be mistaken with an unexpected error.
Code conventions
  • Recipe name should be "Recipe".
  • librecipe / SLAP parameters:
    • Recipes should extend slapos.recipe.librecipe.GenericBaseRecipe by default.
    • Recipes should take their SLAP instance parameters from the Buildout Instance Profile, as parameters. They should NOT (except in case of special need) retrieve parameters by calling the slap library.
    • Publishing SLAP connection parameters should be done using slapos.cookbook:publish in the Buildout Instance Profile, not directly from the code in a recipe.
    • If they need to get/set SLAP parameters for cases not covered (example: slave handling), they must extend slapos.recipe.librecipe.GenericSlapRecipe.

Code style in recipes must follow ERP5 conventions.

slapos.git conventions

Files
  • Everything related to a Software Release should be in a software/nameofsoftware directory.
  • Every installable, atomic component that can be factored should go to component directory (Example: openssl, mariadb).
  • Every high-level section that can be factored should go to stack directory (Example: LAMP service, Tomcat service).
  • Software Release Buildout profile should be named software.cfg.
  • Instance Buildout profile(s) should be named like instance*.cfg.in and be downloaded as instance*.cfg
  • Other Buildout Profiles should be named buildout.cfg
  • (Optional) development profile of Software Release should be located at software/nameofsoftware/development.cfg
Branches

Golden rule: don't work in master branch if using git.erp5.org. Create your own branch.

When possible, create a branch for a specific, atomic, feature, then when tests are passing, ask review then merge to master.

Rule: NO branch is allowed to diverge too much. All work done in branch should be integrated into master sooner or later (the sooner the better).

Branches shouldn't be named with name of developer. Branches should be named by their functionnality or their software release:

  • Acceptable: wordpress, wordpress-upgrade-3.5, erp5-multihomed, resiliency-cleanup
  • Not acceptable: cedric
slapos.git erp5 workflow

erp5 is a bit special in slapos.git. In order to the ERP5-related branches don't diverge too much from the other branches based on master branch, the following workflow should be followed:

  • 1/ merge master into erp5-component. Use a temporary branch to test that tests pass
  • 2/ merge erp5-component into erp5 when it is stable. Use a temporary branch to test that tests pass
  • 3/ merge erp5 into master when it is stable. Use a temporary branch to test that tests pass

Of course, the same should be done for all branches (testnode, wordpress, etc).

SlapOS Coding Crimes

Never copy / paste more than 30 lines of code
Rule

Never copy / paste more than lines of code. This does increase cost of maintaining code in multiple places, discourages code re-use and efforts of making generic software components that can be used in different recipes.

Always pin versions of eggs / products
Rule

SlapOS / buildout will try to install the most recent versions of eggs available by default. If one doesn't pin versions there's no guarantee that you will get the version you expect as you might get the newest one which might break the system.

Do not release a tagged software release for public usage unless it's already pre compiled and "frozen" in shacache.org
Rule

If you want to release a software release for public it's required that you pre compile it and sign it in shacache.org for a minimum set of operating system that you plan to support. The explanation is trivial: without shacache.org everything must be compiled from scratch which takes 4+ hours for a big ERP5 release and can sometimes depend on environmental variables in OS which have a great chance to break compilation process. With shacache.org the time to install is almost equal to time to download the already pre compiled package which is much more user friendly.

Do use tagged software release for production system
Rule

If you deploy a software release to a production system it's completely forbidden to use untagged one like git's HEAD. The reason is that in this case your software release will likely change in time and any software build can have unexpected results.

Bad example:

https://lab.nexedi.cn/nexedi/slapos/raw/master/software/erp5/software.cfg

Good example:

https://lab.nexedi.cn/nexedi/slapos/blob/1.0.13/software/erp5/software.cfg