Fuego Jamboree 2
Introduction [edit section]
The Fuego Jamboree is a technical meeting of Fuego Test System development community. We plan to hold these meetings on a regular basis, to share ideas and plan new features and improvements for the Fuego Test System. We are having our second Fuego Jamboree, coming up in June, 2018!
Pictures from the Event [edit section]
Click on a picture for the gallery.
Event Details [edit section]
Fuego Jamboree #2 was held:- Date: 23 June 2018
- Time: 9:00 am to 12:00 pm
- Location: Panasonic Laboratory Tokyo
- 5th Floor, TFT Building West Wing,
- 3-4-10 Ariake, Koto Ward, Tokyo
- Postal code: 135-8072
- See the map
On the 5th Floor, go to the right side (Atelier side)
Please join the Fuego mailing list to get the latest announcements.
Previous events [edit section]
Information about the previous Fuego Jamboree is here: https://elinux.org/Japan_Technical_Jamboree_58-1You can also see Presentations from previous meetings and other events.
Topics [edit section]
This list is tentative...
- Status and roadmap
- Demos
- Hardware testing
- Performance
- Customers (AGL, LTSI, LTS, CIP)
- New tests
- Outreach
Schedule [edit section]
| Time | Topic | Presenter | Slides |
| 9:00-9:10 | Welcome and Introduction | Tim | no slides |
| 9:10-9:40 | Status of Fuego (overview) | Tim | |
| 9:40-10:20 | Recent Toshiba work on Fuego | Daniel Sangorrin | |
| 10:20-10:40 | Demos | ? | (There was no time for this session) |
| 10:40-10:50 | BREAK |
BREAK |
BREAK |
| 10:50-11:10 | Vision and Industry initiatives | Tim | |
| 11:10-11:30 | User feedback | ? | (There was no time for this session) |
| 11:30-12:00 | Discussion of Roadmap and Priorities | Tim (discussion leader) |
Other presentations [edit section]
Tim gave the following presentations at the Automotive Linux Summit, and the Long Term Stable Initiative Workshop, which also have information about Fuego status, and it's relationship to other industry initiatives:- Fuego Test System Status Update - Tim Bird (Sony) June 2018 at ALS
- description: Describes features of 1.3 release, some plans for future work, and steps for using Fuego
- slides: PDF
- Fuego LTS/LTSI Testing, Projects, and Initiatives - Tim Bird (Sony) June 2018 at LTSI Workshop, at ALS
- description: Describes Fuego features and ongoing projects related to LTS/LTSI testing. Also mentions industry initiatives around Linux Automated Test Stack standards.
- slides: PDF
Session descriptions [edit section]
Status of Fuego (Tim) [edit section]
- feature overview
- feature status
- recent work
- busybox tests
- feature "grades"
Recent Toshiba work (Daniel) [edit section]
- dynamic variables
- using ftc from jenkins
- publishing run.json and artifacts to Squad
- argparse
- trinity (fuzzing test)
- useful scripts
Vision and industry initiatives (Tim) [edit section]
- customization of expected values
- ways to customize a test (spec, expected value, criteria)
- Fuego compare reports (saving a baseline)
- ftc set-criteria (under development, see: Update Criteria )
- Linux Foundation kernelci project
- Automated Testing Summit
- integration/use of other systems
- Standards for a Linux Test Stack
- layers
- board control layer
- Standards for a Linux Test Stack
- Fuego test network - status
- put-board, run-request
- adding the tbwiki code (for Fuego server)
Notes for discussion of Roadmap and Priorities [edit section]
Place notes here during or after the meeting Thanks to everyone who came to the event. It was valuable having you there!
Here are a few notes that Tim took from the meeting:
ftc run-test issues [edit section]
- Daniel proposed isolating Jenkins code in ftc, so that commands that don't need to access Jenkins don't have dependencies on it (e.g. don't do 'import jenkins' unless needed)
- Khiem noted that there may be concurrency issues using the command line tool only
- Jenkins handles job concurrency if the job goes through it. It won't schedule another job for the board, if one is currently running. ftc run-test doesn't have a run-time concurrency check, only a build-time concurrency check.
- We definitely need to add concurrency control for a board, for ftc run-test.
fuego run scalability [edit section]
- Khiem also noted that he sees a lot of buildup of run data that clogs the Jenkins interface
- Tim noted that there are mechanisms for the tables to only show the last 40 entries in a chart, but nothing in Fuego to remove old run data
- Khiem says he runs hundreds of tests a week, and the number of runs ends up clogging up the system.
- Tim says that certain things in Fuego, like the report generation, can get really slow if there is too much run data. This is a fundamental scalability issue that needs to be addressed.
- some --where clauses require reading every run's run.json file and some don't, so you may only see slowness on some queries.
- one potential solutions would be to make a script which deletes old run data
- maybe add an ftc command to remove runs.
- could be something that uses --where clauses:
- 'ftc rm-runs --where start_time=<"1 week ago"'
- could be something that uses --where clauses:
stand-alone layers [edit section]
Based on discussions with Hiramatsu-san (who was representing the LKFT project at this event), it seems like improving our layering would be good. Daniel has already (or will shortly) layer the code between Fuego and the aggregation/presentation layer - supporting kernelci, fuego and squad servers. Daniel mentioned isolating the parsing layer, as that seems to be something other systems don't have, that would be good to make available to other projects.Tim wants to layer the software as well - including separating the parsing layer (he agrees with Daniel on this), as well as formalizing the board control and provisioning layer. Maybe we could have some preliminary stuff in the 1.4 release
release timing [edit section]
Khiem asked about the timing of releases. Tim said that 6 months is too long, which is what each of the last 2 releases have been. We should shoot for 3-month releases, even if it reduces the number of features in each release. The group discussed trying to achieve the 1.4 release in early October, prior to Embedded Linux Conference Europe, and the Automated Testing Summit. Tim will be busy most of July, and won't make much progress, but we should still be able to do something in that time period.One thing that has stalled releases is release testing. We continue to increase the amount of Fuego self-test and release-test code, so the overhead of making a release should decrease over time.
program binary cache [edit section]
Tim mentioned that he has 2 reasons to want to do a test program binary cache:- to support the LAVA integration work by providing a way for LAVA worker nodes to download needed test program code independently from having a Fuego builder node
- to support users who don't have an SDK for their board. This would include people using Fuego to self-test their host machine Due to this, Tim thinks it would be good to work on the test program binary cache. (Note: I have a prototype written that I worked on on the plane home, that might be part of this)
Priorities [edit section]
The following seem to be important priorities, based on discussions at the Jamboree:- ftc run-test concurrency (some kind of board lock)
- ftc rm-run command, with where-clause support
- isolate the parser system and make it standalone
- create a test program binary cache
As well as all the other stuff that was mentioned in the slides. :-)
Attendees [edit section]
Please add your name to this list if you attended the meeting.
| Name | Company | Present | |
| Tim Bird | Sony | tim.bird (at) sony.com | yes |
| Shinsuke Kato | Panasonic | kato.shinsuke (at) jp.panasonic.com | yes |
| Nobuyuki Tanaka | Sony | No.Tanaka (at) sony.com | yes |
| Hirotaka Motai | Mitsubishi Electric | Motai.Hirotaka (at) aj.MitsubishiElectric.co.jp | yes |
| Hirokazu Tokuda | Panasonic | yes | |
| Zheng Ruoqin | Fujitsu | yes | |
| Wang Mingyu | Fujitsu | yes | |
| Teppei Asaba | Fujitsu | teppeiasaba (at) jp.fujitsu.com | yes |
| Nguyen Phuc Loc | Toshiba VN | Yes | |
| Khiem Nguyen | Renesas VN | Khiem.Nguyen (at) renesas.com | yes |
| Nguyen Nguyen | Renesas VN | yes | |
| Dien Pham | Renesas VN | yes | |
| Takuo Koguchi | Hitachi | takuo.koguchi.sw (at) hitachi.com | yes |
| Hiro Fukuchi | Sony | yes | |
| Masami Hiramatsu | Linaro | yes | |
| Daniel Sangorrin | Toshiba | daniel.sangorrin (at) toshiba.co.jp | yes |
| Frank Rowand | Sony | frank.rowand (at) sony.com | yes |