Fuego wiki

Fuego Jamboree 2

Contents:

Introduction
Pictures from the Event
Event Details

Previous events

Topics
Schedule

Other presentations

Session descriptions

Status of Fuego (Tim)
Recent Toshiba work (Daniel)
Vision and industry initiatives (Tim)

Notes for discussion of Roadmap and Priorities

ftc run-test issues
fuego run scalability
stand-alone layers
release timing
program binary cache
Priorities

Attendees

Introduction [edit section]

The Fuego Jamboree is a technical meeting of Fuego Test System development community. We plan to hold these meetings on a regular basis, to share ideas and plan new features and improvements for the Fuego Test System. We are having our second Fuego Jamboree, coming up in June, 2018!

Pictures from the Event [edit section]

Click on a picture for the gallery.

Event Details [edit section]

Fuego Jamboree #2 was held:

Date: 23 June 2018
Time: 9:00 am to 12:00 pm
Location: Panasonic Laboratory Tokyo
- 5th Floor, TFT Building West Wing,
- 3-4-10 Ariake, Koto Ward, Tokyo
- Postal code: 135-8072
See the map
- Map(English)
- Map(Japanese)

On the 5th Floor, go to the right side (Atelier side)

Please join the Fuego mailing list to get the latest announcements.

Previous events [edit section]

Information about the previous Fuego Jamboree is here: https://elinux.org/Japan_Technical_Jamboree_58-1

You can also see Presentations from previous meetings and other events.

Topics [edit section]

This list is tentative...

Status and roadmap
Demos
Hardware testing
Performance
Customers (AGL, LTSI, LTS, CIP)
New tests
Outreach

Schedule [edit section]

Time	Topic	Presenter	Slides
9:00-9:10	Welcome and Introduction	Tim	no slides
9:10-9:40	Status of Fuego (overview)	Tim	PDF
9:40-10:20	Recent Toshiba work on Fuego	Daniel Sangorrin	PDF
10:20-10:40	Demos	?	(There was no time for this session)
10:40-10:50	BREAK	BREAK	BREAK
10:50-11:10	Vision and Industry initiatives	Tim	PDF
11:10-11:30	User feedback	?	(There was no time for this session)
11:30-12:00	Discussion of Roadmap and Priorities	Tim (discussion leader)	PDF

Session descriptions [edit section]

Status of Fuego (Tim) [edit section]

feature overview
feature status
recent work
- busybox tests
feature "grades"

Recent Toshiba work (Daniel) [edit section]

dynamic variables
using ftc from jenkins
publishing run.json and artifacts to Squad
argparse
trinity (fuzzing test)
useful scripts

Vision and industry initiatives (Tim) [edit section]

customization of expected values
- ways to customize a test (spec, expected value, criteria)
- Fuego compare reports (saving a baseline)
- ftc set-criteria (under development, see: Update Criteria )
Linux Foundation kernelci project
Automated Testing Summit
integration/use of other systems
- Standards for a Linux Test Stack
  - layers
  - board control layer
Fuego test network - status
- put-board, run-request
- adding the tbwiki code (for Fuego server)

Notes for discussion of Roadmap and Priorities [edit section]

Place notes here during or after the meeting Thanks to everyone who came to the event. It was valuable having you there!

Here are a few notes that Tim took from the meeting:

ftc run-test issues [edit section]

Daniel proposed isolating Jenkins code in ftc, so that commands that don't need to access Jenkins don't have dependencies on it (e.g. don't do 'import jenkins' unless needed)
Khiem noted that there may be concurrency issues using the command line tool only
- Jenkins handles job concurrency if the job goes through it. It won't schedule another job for the board, if one is currently running. ftc run-test doesn't have a run-time concurrency check, only a build-time concurrency check.
We definitely need to add concurrency control for a board, for ftc run-test.

fuego run scalability [edit section]

Khiem also noted that he sees a lot of buildup of run data that clogs the Jenkins interface
- Tim noted that there are mechanisms for the tables to only show the last 40 entries in a chart, but nothing in Fuego to remove old run data
- Khiem says he runs hundreds of tests a week, and the number of runs ends up clogging up the system.
- Tim says that certain things in Fuego, like the report generation, can get really slow if there is too much run data. This is a fundamental scalability issue that needs to be addressed.
  - some --where clauses require reading every run's run.json file and some don't, so you may only see slowness on some queries.
one potential solutions would be to make a script which deletes old run data
maybe add an ftc command to remove runs.
- could be something that uses --where clauses:
  - 'ftc rm-runs --where start_time=<"1 week ago"'

stand-alone layers [edit section]

Based on discussions with Hiramatsu-san (who was representing the LKFT project at this event), it seems like improving our layering would be good. Daniel has already (or will shortly) layer the code between Fuego and the aggregation/presentation layer - supporting kernelci, fuego and squad servers. Daniel mentioned isolating the parsing layer, as that seems to be something other systems don't have, that would be good to make available to other projects.

Tim wants to layer the software as well - including separating the parsing layer (he agrees with Daniel on this), as well as formalizing the board control and provisioning layer. Maybe we could have some preliminary stuff in the 1.4 release

release timing [edit section]

Khiem asked about the timing of releases. Tim said that 6 months is too long, which is what each of the last 2 releases have been. We should shoot for 3-month releases, even if it reduces the number of features in each release. The group discussed trying to achieve the 1.4 release in early October, prior to Embedded Linux Conference Europe, and the Automated Testing Summit. Tim will be busy most of July, and won't make much progress, but we should still be able to do something in that time period.

One thing that has stalled releases is release testing. We continue to increase the amount of Fuego self-test and release-test code, so the overhead of making a release should decrease over time.

program binary cache [edit section]

Tim mentioned that he has 2 reasons to want to do a test program binary cache:

to support the LAVA integration work by providing a way for LAVA worker nodes to download needed test program code independently from having a Fuego builder node
to support users who don't have an SDK for their board. This would include people using Fuego to self-test their host machine Due to this, Tim thinks it would be good to work on the test program binary cache. (Note: I have a prototype written that I worked on on the plane home, that might be part of this)

Priorities [edit section]

The following seem to be important priorities, based on discussions at the Jamboree:

ftc run-test concurrency (some kind of board lock)
ftc rm-run command, with where-clause support
isolate the parser system and make it standalone
create a test program binary cache

As well as all the other stuff that was mentioned in the slides. :-)

Attendees [edit section]

Please add your name to this list if you attended the meeting.

Name Company e-mail Present

Tim Bird Sony tim.bird (at) sony.com yes

Shinsuke Kato Panasonic kato.shinsuke (at) jp.panasonic.com yes

Nobuyuki Tanaka Sony No.Tanaka (at) sony.com yes

Hirotaka Motai Mitsubishi Electric Motai.Hirotaka (at) aj.MitsubishiElectric.co.jp yes

Hirokazu Tokuda Panasonic
yes

Zheng Ruoqin Fujitsu
yes

Wang Mingyu Fujitsu
yes

Teppei Asaba Fujitsu teppeiasaba (at) jp.fujitsu.com yes

Nguyen Phuc Loc Toshiba VN
Yes

Khiem Nguyen Renesas VN Khiem.Nguyen (at) renesas.com yes

Nguyen Nguyen Renesas VN
yes

Dien Pham Renesas VN
yes

Takuo Koguchi Hitachi takuo.koguchi.sw (at) hitachi.com yes

Hiro Fukuchi Sony
yes

Masami Hiramatsu Linaro
yes

Daniel Sangorrin Toshiba daniel.sangorrin (at) toshiba.co.jp yes

Frank Rowand Sony frank.rowand (at) sony.com yes