In the end of September, I attended the “In-Vivo Analytics for Big Software Quality” Lorentz Center Workshop in beautiful Leiden, Netherlands. Organised by Andy Zaidman, Jürgen Cito, Arie van Deursen, and Mozhan Soltani, we spent five days discussing everything somehow covered by this umbrella title. The idea was to “bring junior and senior researchers together to address how Big Data analysis techniques on runtime data could be used to address software quality in DevOps”. The schedule was packed with presentations from a broad range of research areas, many of them outside of Software Engineering.
Initially, I was a bit unsure how my research actually fits into this scope (I do requirements engineering and model-based engineering), but I decided to give it a go. Indeed, the organisers managed to get hold of a wide variety of participants, especially reflected in the keynotes.
If you don’t want the whole story, feel free to scroll all the way down to my summary/take aways.
Monday
Monday started off with Benoit Baudry discussing “Automated Test Generation in DevOps”. Benoit started off introducing DevOps and the different elements involved, and went then on to the testing part. A large part of this keynote was really targeted at conveying the basics of DevOps. This broad scope was kept for the remainder of the keynotes. Given the broad nature of the workshop topic, this was an excellent way to get an overview of different areas and interdisciplinary topics.
Monday afternoon, Mariëlle Stoelinga covered the basics of her research about risk assessment of computer systems. In essence, she introduced the basics of fault trees and fault tree analysis. While one or another workshop participant probably knew these things, it was a great way to give the majority of the participants an out-of-the-box experience (and, hopefully, some inspiration).
We finished the work part of Monday with a 1-hour speed-dating session. In this session, we were paired with a (pseudo) random other participant for 2 minutes at a time, introducing our research and the reason we were attending the workshop. While this left all participants pretty exhausted, it helped me to get familiar with some faces and research areas (I took the participant list with me and crossed of everyone I had met – this actually helped a great deal later to remember people).
For dinner, we headed out to Café Oliver in downtown Leiden (all paid for by the Lorentz Center), continuing the discussions. I ended up at the table next to Benoit Baudry, Magnus Ågren, and Gilles Perrouin, discussing everything from Swedish habits to actually some research.
Tuesday
On Tuesday, Asterios Katsifodimos, from the area of data processing introduced Stream Processing of Runtime Log Data. This presentation was focused on getting the message across that we should be doing stream instead of batch processing and that batch processing is actually just a special case of stream processing.
The remainder of the day contained four smaller (20-minute) presentations and the so-called Writer’s and Collaborator’s Workshop. Arie discusses the idea in more depth here. In essence, we were asked to submit a paper or a paper draft to the workshop beforehand (similar to a regular conference/WS submission). Then, other workshop participants were assigned to review this paper (again, as in a conf/WS). Finally, at the workshop, we sat down and discussed the papers with the authors present. Naturally, this encouraged “nicer” discussions than you would find at a PC meeting or in conference reviews. However, it was appreciated a lot by (I think) all participants due to the direct feedback on the work, and – for junior researchers – to get insights into how a PC meeting might look like. Strongly recommended – adopt and repeat at your Workshop/Meeting/Research retreat!
Directly after the Workshop program, we had a Wine & Cheese party at the Lorentz Center, continuing the discussions (and side-tracking into communism and other topics) and eventually heading off to the hotel bar for more drinks.
Wednesday
Wednesday came with a headache – among other things. Then, there were of course more points on the agenda.
Claire Le Goues discussed Automated Fault Localization and Program Repair, also touching on the news of the week: Facebook adopting automated program repair with SapFix. Sadly, I had a hard time concentrating, due to my short night and the pressure to have some slides ready for a talk right before lunch.
Before lunch, we had four shorter presentations on a variety of topics, including my talk on LoCo CoCo (using existing requirements and traces to create communication networks).
The after-lunch keynote was given by our very own Eric Knauss, talking about RE at scale, in the context of agile and DevOps. He discussed how agile is difficult to scale in large organisations and how RE integrates with agile practices. While I was involved in a large part of that research, it was yet again interesting to get it presented from a different angle.
As a last program point, we formed breakout groups on specific topics we brainstormed right before. As a goal, we had to come up with a 4-minute elevator pitch on a topic we would like to propose to an imaginary funding agency (Andy tricked us and did in the end not provide any funding…). I ended up in the privacy/security group, discussing for an hour topics around privacy in analysis of runtime data. Essentially, we discussed how runtime data could be shared across organisations/actors, while still preserving a certain privacy. This is a trade-off between the amount of privacy you ensure, and the utility you will get out of the analysis. While the technical aspects of this topic are well-researched (e.g., how to encrypt data or how to do secure aggregation), the mix of people in the group made this a very interdisciplinary and interesting discussion!
After the workshop program, we went by bus to a place outside of Leiden, getting on a boat for a dinner cruise. We cruised through different canals and lakes around Leiden and had a fantastic evening. I ended up chatting away with José Miguel Rojas on tenure tracks, South America, and other topics only remotely connected to the workshop 🙂
Thursday
Thursday morning, Alexandru Iosup from the distributed systems community came in to share his vision of Massivizing Computer Systems. The talk was very well delivered, but at times hard to understand (in my opinion). While very ambitious, I felt that many of the things Alex was asking for have been discussed for many years (“We need to care more about human aspects”, “We need to be able to assure a better level of quality”, etc.). We surely need those things, but I wasn’t quite sure whether the talk did a good job explaining how to get there. In any case, I liked that Alex was actively trying to get people on board (connecting the different keynote topics and encouraging cooperation).
Before lunch, we had yet another session with short presentations.
Andy Zaidman delivered the afternoon keynote on his empirical studies on software quality. He basically encouraged to question beliefs in Software Engineering (especially if we don’t know were they come from).
As a final point, we had a 1.5-hour session in which participants did live demos of tools they had developed. Jürgen Cito demonstrated PerformanceHat, a tool that augments source code in the IDE with (runtime) performance information and predictions. José showed us CodeDefenders, a game for learning mutation testing. This got all the participants to happily hack away, trying to defeat each other in coming up with mutants and tests killing them. Eric then demoed T-Reqs, a tool that integrates requirements into revision control. I believe there were other demos, but at this point I was so exhausted that it got difficult following (sorry if I skipped anyone here).
Thursday did not have an evening program, which should have given me some time to visit Leiden. Instead, I was stuck in the hotel room doing lecture slides – a side effect of squeezing in the workshop into a busy teaching schedule.
Friday
Friday had only one keynote and a wrap-up session, allowing participants to head home at lunchtime.
The keynote was again rather interdisciplinary, as Fabian Knirsch and Andreas Unterweger talked about the smart grid, and security/privacy in this context. The talk again explained the basics in a very comprehensive way, sketching an interesting application field that (most likely) requires much more rigorous Software Engineering practices.
We then finished the workshop with a quick summary and said our goodbyes!
Summary/Take Aways
It’s now almost a month since the workshop, and maybe time to reflect on what I took away from the workshop, what worked, and what might be improved in similar workshops in the future.
First, I really thoroughly enjoyed the week in Leiden. Despite (or maybe due to) being seemingly a bit disconnected from the workshop topic, I learned a lot during the week, getting insights into a variety of new topics (lots of different updates on software testing, some basics in the problems encountered in Smart Grids, to name a few). Since we, as researchers, are (or should be) aiming to solve problems on a societal scale, I think broadening your own horizon, seeing some new challenges, and getting inspired by existing solutions in other areas can only be beneficial. This is a clear plus of having such a broad workshop.
At the same time, the broad scope made it hard to come up with clearer goals for the workshop. I do believe that the discussion sessions could have benefitted from more structure or a clearer direction. However, I understand that this is hard given the different backgrounds and interests of the participants.
Thirdly, only knowing a handful of attendees actually proved to be a good thing, since I ended up meeting lots of people, some of which I might end up collaborating with. This was again somewhat unexpected, as I am used to the typical conference setup, with sub-communities already knowing each other. In such a setup it is in my experience much harder to talk about new ideas and directions.
Finally, as a direct take-away, I really liked both the breakout sessions in which we had to discuss a research topic/agenda (even though it was short) and the Writer’s/Collaborator’s workshop. Both setups are definitely something I will adopt in future meetings/workshops, and urge you to do the same!