Fuego wiki

(There was no time for this session)

[[image:HeadTitle_Fuego-Jamboree.png]]
{{TableOfContents}}

= Introduction =
The Fuego Jamboree is a technical meeting of Fuego Test System development community.  We plan to hold these meetings on a regular basis, to share
ideas and plan new features and improvements for the Fuego Test System.
We are having our second Fuego Jamboree, coming up in June, 2018!

= Pictures from the Event =
Click on a picture for the gallery.

{{{#!ImageGallery:fj2
images="""
FJ2-room-status-talk.JPG
FJ2-room-status-talk2.jpg
FJ2-back-of-room.JPG
FJ2-Tim-presenting.JPG
FJ2-attendees-watching.JPG
FJ2-group-picture.JPG
"""
height=220
}}}

= Event Details =
Fuego Jamboree #2 was held:
 * Date: 23 June 2018
 * Time: 9:00 am to 12:00 pm
 * Location: Panasonic Laboratory Tokyo
   * 5th Floor, TFT Building West Wing, 
   * 3-4-10 Ariake, Koto Ward, Tokyo
   * Postal code: 135-8072
 * See the map
   * [[https://elinux.org/images/2/20/Map_PanasonicLabTokyo_E.PDF|Map(English)]]
   * [[https://elinux.org/images/b/b9/Map_PanasonicLabTokyo.pdf|Map(Japanese)]]

On the 5th Floor, go to the right side (Atelier side)

Please join the [[https://lists.linuxfoundation.org/mailman/listinfo/fuego|Fuego mailing list]] to get the latest announcements.

== Previous events ==
Information about the previous Fuego Jamboree is here:
https://elinux.org/Japan_Technical_Jamboree_58-1

You can also see [[Presentations]] from previous meetings and other events.

= Topics =
''This list is tentative...''
 * Status and roadmap
 * Demos
 * Hardware testing
 * Performance
 * Customers (AGL, LTSI, LTS, CIP)
 * New tests
 * Outreach

= Schedule =
{{{#!Table
show_sort_links=0
show_edit_links=0
||Time||Topic||Presenter||Slides||
||9:00-9:10||Welcome and Introduction||Tim||''no slides''||
||9:10-9:40||Status of Fuego (overview)||Tim||[[file:Fuego-Jamboree-status-June-2018.pdf|PDF]]||
||9:40-10:20||Recent Toshiba work on Fuego||Daniel Sangorrin||[[file:fuego-jamboree2-daniel-sangorrin-23jun2018.pdf|PDF]]||
||10:20-10:40||Demos ||?||''(There was no time for this session)''||
||10:40-10:50||'''BREAK'''||'''BREAK'''||'''BREAK'''||
||10:50-11:10||Vision and Industry initiatives||Tim||[[file:Fuego-Jamboree-vision-June-2018.pdf|PDF]]||
||11:10-11:30||User feedback ||?||''(There was no time for this session)''||
||11:30-12:00||Discussion of Roadmap and Priorities||Tim (discussion leader)||[[file:Fuego-Jamboree-roadmap-priorities-June-2018.pdf|PDF]]||
}}}

== Other presentations ==
Tim gave the following presentations at the Automotive Linux Summit,
and the Long Term Stable Initiative Workshop, which also have information
about Fuego status, and it's relationship to other industry initiatives:
 * '''Fuego Test System Status Update''' - Tim Bird (Sony) June 2018 at ALS
   * description: Describes features of 1.3 release, some plans for future work, and steps for using Fuego
   * slides: [[file:Fuego-Test-System-Status-Update-June-2018-final.pdf|PDF]] 
 * '''Fuego LTS/LTSI Testing, Projects, and Initiatives''' - Tim Bird (Sony) June 2018 at LTSI Workshop, at ALS
   * description: Describes Fuego features and ongoing projects related to LTS/LTSI testing.  Also mentions industry initiatives around Linux Automated Test Stack standards.
   * slides: [[file:Fuego-LTS-workshop-June-2018.pdf|PDF]]

= Session descriptions =
== Status of Fuego (Tim) ==
 * feature overview
 * feature status
 * recent work
   * busybox tests
 * feature "grades"

== Recent Toshiba work (Daniel) ==
 - dynamic variables
 - using ftc from jenkins
 - publishing run.json and artifacts to Squad
 - argparse
 - trinity (fuzzing test)
 - useful scripts

== Vision and industry initiatives (Tim) ==
 * customization of expected values
   * ways to customize a test (spec, expected value, criteria)
   * Fuego compare reports (saving a baseline)
   * ftc set-criteria (under development, see: [[Update Criteria]] )
 * Linux Foundation kernelci project
 * Automated Testing Summit
 * integration/use of other systems
   * Standards for a Linux Test Stack
     * layers
     * board control layer
 * Fuego test network - status
   * put-board, run-request
   * adding the tbwiki code (for Fuego server)

= Notes for discussion of Roadmap and Priorities =
''Place notes here during or after the meeting''
Thanks to everyone who came to the event.  It was valuable having you there!

Here are a few notes that Tim took from the meeting:

== ftc run-test issues ==
 * Daniel proposed isolating Jenkins code in ftc, so that commands that don't need to access Jenkins don't have dependencies on it (e.g. don't do 'import jenkins' unless needed)
 * Khiem noted that there may be concurrency issues using the command line tool only
   * Jenkins handles job concurrency if the job goes through it.  It won't schedule another job for the board, if one is currently running.  ftc run-test doesn't have a run-time concurrency check, only a build-time concurrency check.
 * We definitely need to add concurrency control for a board, for ftc run-test.

== fuego run scalability ==
 * Khiem also noted that he sees a lot of buildup of run data that clogs the Jenkins interface
   * Tim noted that there are mechanisms for the tables to only show the last 40 entries in a chart, but nothing in Fuego to remove old run data
   * Khiem says he runs hundreds of tests a week, and the number of runs ends up clogging up the system.
   * Tim says that certain things in Fuego, like the report generation, can get really slow if there is too much run data.  This is a fundamental scalability issue that needs to be addressed.
      * some --where clauses require reading every run's run.json file and some don't, so you may only see slowness on some queries.
 * one potential solutions would be to make a script which deletes old run data
 * maybe add an ftc command to remove runs.
   * could be something that uses --where clauses:
     * 'ftc rm-runs --where start_time=<"1 week ago"'

== stand-alone layers ==
Based on discussions with Hiramatsu-san (who was representing the LKFT project
at this event), it seems like improving our layering would be good.
Daniel has already (or will shortly) layer the code between Fuego and 
the aggregation/presentation layer - supporting kernelci, fuego and squad
servers.  Daniel mentioned isolating the parsing layer, as that seems to be
something other systems don't have, that would be good to make available
to other projects. 

Tim wants to layer the software as well - including separating the parsing
layer (he agrees with Daniel on this), as well as formalizing the
board control and provisioning layer.  Maybe we could have some preliminary
stuff in the 1.4 release

== release timing ==
Khiem asked about the timing of releases.  Tim said that 6 months is too long,
which is what each of the last 2 releases have been.  We should shoot for 3-month releases, even if it reduces the number of features in each release.
The group discussed trying to achieve the 1.4 release in early October,
prior to Embedded Linux Conference Europe, and the Automated Testing Summit.
Tim will be busy most of July, and won't make much progress, but we should
still be able to do something in that time period.

One thing that has stalled releases is release testing. We continue to increase
the amount of Fuego self-test and release-test code, so the overhead of making
a release should decrease over time.

== program binary cache ==
Tim mentioned that he has 2 reasons to want to do a test program binary cache:
 * to support the LAVA integration work by providing a way for LAVA worker nodes to download needed test program code independently from having a Fuego builder node
 * to support users who don't have an SDK for their board.  This would include people using Fuego to self-test their host machine
Due to this, Tim thinks it would be good to work on the test program binary cache.  (''Note: I have a prototype written that I worked on on the plane home, that might be part of this'')

== Priorities ==
The following seem to be important priorities, based on discussions at the Jamboree:
 * ftc run-test concurrency (some kind of board lock)
 * ftc rm-run command, with where-clause support
 * isolate the parser system and make it standalone
 * create a test program binary cache

As well as all the other stuff that was mentioned in the slides.  :-)

= Attendees =
Please add your name to this list if you attended the meeting.

{{{#!Table:attendees
is_spreadsheet=1
show_edit_links=0
show_sort_links=0
||Name||Company||e-mail||Present||
||Tim Bird||Sony||tim.bird (at) sony.com||yes||
||Shinsuke Kato||Panasonic||kato.shinsuke (at) jp.panasonic.com ||yes||
||Nobuyuki Tanaka||Sony||No.Tanaka (at) sony.com||yes||
||Hirotaka Motai||Mitsubishi Electric||Motai.Hirotaka (at) aj.MitsubishiElectric.co.jp||yes||
||Hirokazu Tokuda||Panasonic|| ||yes||
||Zheng Ruoqin||Fujitsu|| ||yes||
||Wang Mingyu||Fujitsu|| ||yes||
||Teppei Asaba||Fujitsu||teppeiasaba (at) jp.fujitsu.com||yes||
||Nguyen Phuc Loc||Toshiba VN|| ||Yes ||
||Khiem Nguyen||Renesas VN||Khiem.Nguyen (at) renesas.com ||yes ||
||Nguyen Nguyen||Renesas VN|| ||yes||
||Dien Pham||Renesas VN|| ||yes||
||Takuo Koguchi||Hitachi||takuo.koguchi.sw (at) hitachi.com||yes||
||Hiro Fukuchi||Sony|| ||yes||
||Masami Hiramatsu||Linaro|| ||yes||
||Daniel Sangorrin||Toshiba||daniel.sangorrin (at) toshiba.co.jp||yes||
||Frank Rowand||Sony||frank.rowand (at) sony.com||yes||
}}}

Fuego wiki

Fuego Jamboree 2 in split format

Introduction [edit section]

Pictures from the Event [edit section]

Event Details [edit section]

Previous events [edit section]

Topics [edit section]

Schedule [edit section]

Other presentations [edit section]

Session descriptions [edit section]

Recent Toshiba work (Daniel) [edit section]

Vision and industry initiatives (Tim) [edit section]

Notes for discussion of Roadmap and Priorities [edit section]

ftc run-test issues [edit section]

fuego run scalability [edit section]

stand-alone layers [edit section]

release timing [edit section]

program binary cache [edit section]

Priorities [edit section]

Attendees [edit section]