Notes on Jenkins integration refactoring

Contents:

basic installation

Tim's customizations

Features/Changes
problem installing toolchain
can't create image due to debian server network problem
Tim's notes from March 7 comparison

differences
things to fix
issues
questions

Here are some miscelaneous notes on the Jenkins integration refactoring done by Daniel Sangorrin:

(These notes are obsolete with version 1.1 of Fuego - this should all be integrated into that version.)

basic installation [edit section]

To use the new system do: Quickstart:

$ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
$ git clone --branch next https://bitbucket.org/nirrognas/fuego-core.git
$ cd fuego/
$ ./install.sh
$ ./fuego-host-scripts/docker-create-container.sh
$ ./fuego-host-scripts/docker-start-container.sh
docker# fuego-create-node --board docker
docker# fuego-create-jobs --board docker --testplan testplan_docker --distrib nosyslogd.dist

Then, start the "docker.testplan_docker.batch" job which will trigger the rest of jobs.

Tim's customizations [edit section]

Install my test board:

$ cp ~/work/fuego/fuego/userdata/conf/boards/bbb-poky-sdk.board ~/work/fuego-daniel/fuego/fuego-ro/conf/boards
docker# fuego-create-node --board bbb-poky-sdk
docker# fuego-create-jobs --board bbb-poky-sdk --testplan testplan_docker --distrib nosyslogd.dist

Install my toolchain:

$ sudo su
$ mkdir -p /fuego-ro/toolchains
$ cd /fuego-ro/toolchains
$ cp ~/work/yocto/build-quemuarm/yocto/poky/build-qemuarm/tmp/deploy/sdk/*.sh .
$ ./*snapshot.sh
- enter target directory for SDK: /fuego-ro/toolchains/poky
$ cd ~/work/fuego-daniel/fuego/fuego-ro/toolchains
$ cp -a /fuego-ro/toolchains/poky .
$ cp ~/work/fuego/fuego/userdata/toolchains/poky-qemuarm-tools.sh .
$ edit poky-qemuarm-tools.sh to use correct path

this did not work

$ ln -s /home/1000072457/work/fuego-daniel/fuego/fuego-ro /fuego-ro
cd /fuego-ro/toolchains
./*snapshot.sh
remove all occurences of '/home/1000072457/work/fuego-daniel/fuego' from environment-setup-armv53-poky-linux-gnueabi

this did not work

docker# cd /fuego-ro/toolchains
docker# ./*snapshot.sh
- enter target directory for SDK: /fuego-ro/toolchains/poky
  - I get the message cannot mkdir: Read-only file system

Features/Changes [edit section]

Here is a list of the most fundamental modifications that were added:

automatized the version upgrade process. So far I have updated Jenkins 3 times quite smoothly.
- The current version is the latest LTS version (2.31.1 LTS)
reduced plugin dependencies to the minimum (only 2, and they can be made optional).
removed dependencies on all groovy scripts.
Userdata is gone. Instead we now have fuego-ro and fuego-rw
- These and fuego-core are configured as external docker bind mounts.
- Only fuego-rw, which contains logs and build folders, is mounted in RW mode.
- also fixed the uid/gid of jenkins so that it matches the one in the host.
- This means you can develop comfortably on your host, with GUI tools, and don't fear a buggy script deleting your folders.
added a tool for generating jobs from testplans.

Creating jobs from testplans is powerful. It allows you, for example, to easily specify the timeout for each test on your board. The tool also generates the corresponding batch job. For example, you can easily trigger all tests for your board and a specific testplan periodically.

[Note] This will also enable the creation of scripts for comparing test results across different boards and test parameters (on my TODO list).

Other less fundamental changes include:

I added support for using the docker container itself as a target board (e.g.: for a quickstart or unit tests)
I fixed the flot plugin (java and javascript fixes)
Now you can click on a node and see which jobs (and testplans) are assigned to your board. This was one of my old feature requests and makes Fuego's GUI easier to use.
I added quite a few fixes and improvements to the core engine scripts and tests (too detailed to describe here).
I added Excel output support for IOzone
Faster docker build time (ARMhf compiler installation is now optional, no latex..)
I put a fixed name to the fuego container, instead of the "last_container_id" file.
I removed the inotify script
I added scripts for easily removing the docker container/image.

problem installing toolchain [edit section]

I couldn't install the toolchain into /fuego-ro/toolchains, because /fuego-ro is mounted read-only. I had to install the toolchain to /fuego-ro/toolchains on my host, then as root copy the directory to

/home/<user>/work/fuego/fuego/fuego-ro/toolchains - where it then appeared in the docker container in /fuego-ro/toolchains.

You have to use the same installation path on the host as it will be in the container, because the environment setup script hardcodes the path when the SDK is installed.

can't create image due to debian server network problem [edit section]

I had a problem completing the build of the docker image, due to an intermittent problem accessing debian servers during the build.

Here is an error I saw:

Get:95 http://httpredir.debian.org/debian/ jessie/main libatk-wrapper-java all 0.30.5-1 [30.3 kB]
Err http://httpredir.debian.org/debian/ jessie/main libavahi-glib1 amd64 0.6.31-5
  Error reading from server. Remote end closed connection [IP: 5.153.231.35 80]

Daniel says:

Yes, I had the same problem yesterday when I installed it on a different machine.
It seems that httpredir.debian.org is not as reliable as using a local mirror directly.
I'm not sure if I should add retries or put a default debian server as it used to be.
Could you change httpredir by the US mirror (http://ftp.us.debian.org/debian) and try again?

Tim's notes from March 7 comparison [edit section]

differences [edit section]

frontend-install only has:
- flot
- fuego-install-plugins.sh, which installs:
  - description-setter
  - pegdown-formatter
userdata moved to:
- fuego-ro, which has:
  - conf/boards/docker.board, lager.board, qemu-arm.board, etc.
  - scripts/nodes/fuego-create-node
  - scripts/nodes/fuego-delete-node
  - scripts/jobs/fuego-create-jobs
  - scripts/jobs/fuego-delete-jobs
  - toolchains/tools.sh, poky-qemuarm-tools.sh
- fuego-rw, which has:
  - buildzone
  - logs
  - work

things to fix [edit section]

fuego-core/engine/scripts/ftc is missing commit to add debug option
- I forgot to push this after the Portland trip
- It's small and disposable as a patch (that is, it's not important to integrate it)
fuego/Dockerfile is missing ttc installation
- but fuego-core/engine/overlays/base/base-board.fuegoclass has ttc support in the ov_* functions
post_test is not run
- no test directories were removed on target
LTP did not have enough space to run on the device
- my bbb only has about 300 meg. free

issues [edit section]

there's no documentation on providing your own image or container names
- install.sh [<my_image_name>]
- fuego-host-scripts/docker-create-container.sh [<my_image_name> [<my_container_name>]]
- fuego-host-scripts/docker-start-container.sh [<my_container_name>]
it would be nice to put fuego-core, fuego-ro and fuego-rw under /opt/fuego.
- for toolchain installation, it would be handy to have toolchains with the same path on the host as in the container:
  - /opt/fuego/fuego-ro/toolchains in both the container and on the host
The build shell command for jobs could (and should) be simplified:
- TESTNAME should always be fuego_test.sh
  - this requires renaming all base scripts in all tests
- DISTRIB should be in the board file
- TESTPLAN should be able to be parsed from JOB_NAME
- is NODE_NAME used for the board?
- just call "ftc run-test"
  - have it detect TESTNAME, DISTRIB, TESTPLAN, NODE_NAME
  - have it detect it is being run from a Jenkins job (check for JENKINS_URL?)
tests directories are placed at the root on the target:
- /fuego.Benchmark.dbench (instead of /home/a/fuego.Benchmark.dbench)
- new system uses BOARD_TESTDIR instead of FUEGO_HOME

questions [edit section]

question: does Dockerfile use ../fuego-core, or clone it's own?
- answer: docker-create-container.sh uses ../../fuego-core