Community Build Framework 2

Document created by Pedro Goncalves on Aug 10, 2017Last modified by Diogo Belem on Nov 9, 2017
Version 13Show Document
  • View in full screen mode


 

Focused on a multi-project / multi-environment scenario, CBF2 is the way to setup and deploy Pentaho-based applications.
CBF2 is extremely useful when working in several projects at the same time, allowing you to switch among several environments (production, development, etc.), even when customizing the Pentaho server (using different databases, changing the look and feel or applying different security types).

 

Credits: Webdetails Team, Lead - Pedro Alves

 


 

Overview

The goal of this project is to quickly spin a working Pentaho server on Docker containers. This will also provide script utilities to get the client tools.

This is a very useful project, that we used and maintained actively throughout the last decade. It still works, but now we have different paradigms to take into account.

CBF2 is first and foremost, a developer's tool and methodology so we won't make any recommendations on using it in a production environment, Pentaho's stance on this is clearly explained in this article.

 

Main Features

  • Works from binaries;
  • Works in CE and EE;
  • For EE, processes patches;
  • Setups not just Pentaho, but all the environment;
  • Ability to switch from totally different projects on the fly;
  • Supports multiple environments (eg: development, production, worker1, worker2...);
  • No changes to the original files that could get overwritten in a upgrade;
  • Supports multiple platform versions;
  • Simplifies version upgrades;
  • Debug-friendly;
  • VCS (Version Control System)-friendly;
  • Supports deploy to local/remote machines.
  • Supports all kind of different customization in different projects, from using different databases to different security types;

 


 

Requirements

  • A shell (if you're using Windows, Cygwin or equivalent is required);
  • A system with Docker installed;
  • Your VCS client;
  • lftp.

 


 

How to use

There are a few utilities here:

  • getBinariesFromBox.sh - Connects to box and builds the main images for the servers (Requires access to box. Later there will be a workaround which won't need that);
  • cbf2.sh - What you need to use to build the images;
  • getClients.sh - A utility to get the clients tools;
  • startClient.sh - A utility to start the client tools.

 


 

The software directory

This is the main starting point. If you're a Pentaho employee you will have access to using the getBinariesFromBox.sh script, but all the rest of the world can still use this by manually putting the files here.

You should put the official software files under the software/v.v.v directory. It's very important that you follow this 3 number representation.

This works for both CE and EE it's actually works better for EE, since you can also put the patches there and they will be processed. For EE, you should use the official -dist.zip artifacts. For CE, use the normal .zip file.

 


 

The licences directory

For EE, just place the *.lic license files on the licenses subdirectory. They will be installed on the images for EE builds.

 

Released versions

/X.X.X and inside drop the server, plugins and patches.

 

Nightly Builds

Drop the build artifacts directly in that directory:

 

software/
    ├── 5.2.1
    │   ├── SP201502-5.2.zip
    │   ├── biserver-ee-5.2.1.0-148-dist.zip
    │   ├── paz-plugin-ee-5.2.1.0-148-dist.zip
    │   ├── pdd-plugin-ee-5.2.1.0-148-dist.zip
    │   └── pir-plugin-ee-5.2.1.0-148-dist.zip
    ├── 5.4.0
    │   └── biserver-ce-5.4.0.0-128.zip
    ├── 5.4.1
    │   ├── SP201603-5.4.zip
    │   └── biserver-ee-5.4.1.0-169-dist.zip
    ├── 6.0.1
    │   ├── SP201601-6.0.zip
    │   ├── SP201602-6.0.zip
    │   ├── SP201603-6.0.zip
    │   ├── biserver-ce-6.0.1.0-386.zip
    │   ├── biserver-ee-6.0.1.0-386-dist.zip
    │   ├── paz-plugin-ee-6.0.1.0-386-dist.zip
    │   ├── pdd-plugin-ee-6.0.1.0-386-dist.zip
    │   └── pir-plugin-ee-6.0.1.0-386-dist.zip
    ├── 6.1-QAT-153
    │   ├── biserver-ee-6.1-qat-153-dist.zip
    │   ├── biserver-merged-ce-6.1-qat-153.zip
    │   ├── paz-plugin-ee-6.1-qat-153-dist.zip
    │   ├── pdd-plugin-ee-6.1-qat-153-dist.zip
    │   └── pir-plugin-ee-6.1-qat-153-dist.zip
    ├── 7.0-QAT-76
    │   ├── biserver-merged-ee-7.0-QAT-76-dist.zip
    │   ├── pdd-plugin-ee-7.0-QAT-76-dist.zip
    │   └── pir-plugin-ee-7.0-QAT-76-dist.zip
    └── README.txt

 


 

CBF2: The main thing

CBF1 was an ant script but CBF2 is a bash script. So, you want cbf2.sh. If you are on windows you should be able to just use cygwin.

 

Here's what you'll see when you run ./cbf2.sh :

--------------------------------------------------------------
--------------------------------------------------------------
------       CBF2 - Community Build Framework 2        -------
------                 Version: 0.9                    ------- 
------ Author: Pedro Alves (pedro.alves@webdetails.pt) ------- 
-------------------------------------------------------------- 
-------------------------------------------------------------- 

Core Images available:
----------------------

 [0] baserver-ee-5.4.1.0-169
 [1] baserver-ee-6.0.1.0-386
 [2] baserver-merged-ce-6.1-qat-153
 [3] baserver-merged-ee-6.1.0.0-192 

Core containers available:
--------------------------

 [4] (Stopped): baserver-ee-5.4.1.0-169-debug

Project images available:
-------------------------

 [5] pdu-project-nasa-samples-baserver-ee-5.4.1.0-169
 [6] pdu-project-nasa-samples-baserver-merged-ee-6.1.0.0-192

Project containers available:
-----------------------------

 [7] (Running): pdu-project-nasa-samples-baserver-ee-5.4.1.0-169-debug
 [8] (Stopped): pdu-project-nasa-samples-baserver-merged-ee-6.1.0.0-192-debug

> Select an entry number, [A] to add new image or [C] to create new project:

 

There are 4 main concepts here:

  • Core images;
  • Core containers;
  • Project images;
  • Project containers.

These should be straightforward to understand if you're familiar with Docker, but in a nutshell there are two fundamental concepts: images and containers. An image is an inert, immutable file.

The container is an instance of an image, and it's a container that will run and allow us to access the Pentaho platform.

 


 

Accessing the platform

When we run the container, it exposes a few ports, most importantly 8080.

So in order to see Pentaho running all we need to do is to access the machine where docker is running.This part may vary depending on the operating system so be sure to check your container's ip address, you should be able to access the platform by using the following URL: http://your-container-ip:8080/pentaho/Home

 

Core images

These are the core images - a clean install out of one of the available artifacts that are provided on the software directory. So the first thing we should do is add a core image. The option [A] allows us to select which image to add from an official distribution archive.

 

When we select this option, we are prompted to choose the version we want to build:

> Select an entry number, [A] to add new image or [C] to create new project: A

Servers found on the software dir:
 [0]: biserver-ee-5.2.1.0-148-dist.zip
 [1]: biserver-ce-5.4.0.0-128.zip 
 [2]: biserver-ee-5.4.1.0-169-dist.zip 
 [3]: biserver-ce-6.0.1.0-386.zip 
 [4]: biserver-ee-6.0.1.0-386-dist.zip 
 [5]: biserver-ee-6.1-qat-153-dist.zip 
 [6]: biserver-merged-ce-6.1-qat-153.zip 
 [7]: biserver-merged-ee-7.0-QAT-76-dist.zip

CBF2 will correctly know how to handle EE dist files, you'll be presented with the EULA, patches will be automatically processed and licenses will be installed.

Once an image is built, if we select that core image number you'll have the option to launch a new container or delete the image:

> Select an entry number, [A] to add new image or [C] to create new project: 0 

You selected the image baserver-ee-6.0.1.0-386 

> What do you want to do? (L)aunch a new container or (D)elete the image? [L]:

 

Core containers

You can launch a container from a core image. This will allow us to explore a completely clean version of the image you selected. This is useful for some tests, but I'd say the big value would come out of the project images. Here are the options available over containers:

> Select an entry number, [A] to add new image or [C] to create new project: 3  

You selected the container baserver-merged-ce-6.1-qat-153-debug
The container is running; Possible operations:   

 S: Stop it 
 R: Restart it 
 A: Attach to it 
 L: See the Logs 

What do you want to do? [A]:

Briefly, here are the options mean - even though they should be relatively straightforward:

  • Stop it: Stops the container. When the container is stopped you'll be able to delete the container or start it again;
  • Restart it: Simply restarts the container;
  • Attach to it: Attaches to the docker container. You'll then have a bash shell and you'll be able to play with the server;
  • See the Logs: Gets the logs from the server.

 


 

Projects

 

Definition and structure

A project is built on top of a core image. Instead of being a clean install it's meant to replicate a real project's environment. As a best practice, it should also have a well defined structure that can be stored on a VCS repository.

Projects should be cloned / checked out in to the projects directory. It's recommend for every project to be versioned in a different git or svn repository. Here's a project structure example:

pedro@orion:~/tex/pentaho/cbf2 (master *) $ tree  -l ./projects/
./projects/
└── project-nasa-samples -> ../../project-nasa-samples/
        ├── _dockerfiles 
        └── solution 
                └── public 
                        ├── Mars_Photo_Project 
                        │   ├── Mars_Photo_Project.cda 
                        │   ├── Mars_Photo_Project.cdfde 
                        │   ├── Mars_Photo_Project.wcdf 
                        │   ├── css 
                        │   │   └── styles.css 
                        │   ├── img 
                        │   │   └── nasaicon.png 
                        │   └── js 
                        │       └── functions.js 
                        └── ktr 
                                ├── NASA\ API\ KEY.txt 
                                ├── curiosity.ktr 
                                ├── getPages.ktr 
                                └── mars.ktr

 

All the solution files are going to be automatically imported, including metadata for datasources creation.

The directory _dockerfiles is a special one, you can override the default Dockerfile that's used to build a project image (dockerfiles/buildProject/Dockerfile) and just drop a project specific Dockerfile in that directory.

Note that you should not change the FROM line, as it will be dynamically replaced. This is what you want for project level configurations, like installing / restoring a specific database, an apache server on front or any fine tuned configurations.

 

Project images

The first thing that we need to do is to create a project.To do that is very simple: select one of the projects on our projects directory and a core image to install it against. This separations aims at really simplifying upgrades / tests / etc.

> Select an entry number, [A] to add new image or [C] to create new project: C 

Choose a project to build an image for: 

  [0] project-nasa-samples  

> Choose project: 0  

Select the image to use for the project  

 [0] baserver-ee-6.0.1.0-386 
 [1] baserver-merged-ce-6.1-qat-153 
 [2] baserver-merged-ee-6.1.0.0-192  

> Choose image: 2

 

Once the project image created, you have access to the same options you had for the core images, which is basically launching a container or deleting the image.

 

Project containers

Like the images, project containers work very similarly to core containers, but also have two extra options available:

  • Export the solution: Exports the solution to your project folder;
  • Import the solution: Imports the solution from your project folder into the running containers. This would be equivalent to rebuilding the image.

 

Note that by design CBF2 only exports the folders in public that are already part of the project. You'll need to manually create the directory if you add a top level one.

 


 

The client tools

This also provides two utilities to handle the client tools; One of them, the getClients.sh, is probably something you can't use since it's for internal pentaho people only.

The other one, startClients.sh, may be more useful; It requires the client tools to be downloaded into a dir called /clients with a certain structure:

pedro@orion:~/tex/pentaho/cbf2 (master *) $ tree -L 4 clients/ 
clients/ 
├── pad-ce 
│   └── 6.1.0.0 
├── pdi-ce 
│   ├── 6.1-QAT 
│   │   └── 156 
│   │       └── data-integration 
│   ├── 6.1.0.0 
│   │   └── 192 
│   │       └── data-integration 
│   └── 7.0-QAT 
│       └── 57 
│           └── data-integration 
├── pdi-ee-client 
│   └── 6.1.0.0 
│       └── 192 
│           ├── data-integration 
│           ├── jdbc-distribution 
│           └── license-installer 
├── pme-ce 
│   └── 6.1.0.0 
│       └── 182 
│           └── metadata-editor 
├── prd-ce 
│   └── 6.1.0.0 
│       └── 182 
│           └── report-designer 
└── psw-ce         └── 6.1.0.0

If you use this, then the startClients.sh simplifies launching them; Note that, unlike the platform, this will run on the local machine, not on a docker VM:

pedro@orion:~/tex/pentaho/cbf2 (master *) $ ./startClients.sh  

Clients found: 
--------------  

 [0] pdi-ce: 6.1-QAT-156 
 [1] pdi-ce: 6.1.0.0-192 
 [2] pdi-ce: 7.0-QAT-57 
 [3] pdi-ee-client: 6.1.0.0-192 
 [4] pme-ce: 6.1.0.0-182 
 [5] prd-ce: 6.1.0.0-182  

Select a client:

 

Attachments

    Outcomes