Session A1: Guidelines for Case Study Research and Publication
Chairs: Stefan Wagner and Per Runeson
Location: Max Bell Room 251

Background:
Per Runeson and Martin Höst have systematically collected existing work on case study research in software engineering. Although some people appreciate the clear structure, many strongly demand changes to the way the study is reported. This can have at least two reasons.

The guidelines:
1. Have weaknesses that should be corrected
2. Are not known well enough in the broader community.

Session goals:
Analyze the two possible reasons by identifying: 1. Weaknesses of and problems with the current guidelines to improve them.
2. Action steps to make them better known and used.

Development of the session:
Attendees are asked to:
1. Read a paper that follows the guidelines, and propose changes.
2. Answer a questionnaire that analyses the knowledge of the guidelines and if and why they are used.
Findings are presented to the audience.

Session B1: Reporting Replications
Chairs: Jeffrey Carver, Natalia Juristo and Teresa Baldassarre
Location: Max Bell Room 252

Background:
The value of experimental replications is evident to ISERN members. The software engineering community learns a great deal from performing replications, reading reports of replications performed by others and aggregating the results of replications to draw deeper conclusions that would otherwise be possible. For experimental replications to have scientific value comparable to that of other types of empirical studies, they must be published in the peer-reviewed literature. To facilitate the usefulness of these publications, we need guidelines to ensure that a consistent set of information is published about each replication. There are existing guidelines for reporting controlled experiments and case studies, which have been discussed during ISERN, but there are no guidelines specifically for reporting experimental replications. The type of report required for an experimental replication is similar to, but is not the same as that for a controlled experiment. In a replication it is important to publish information about the original study, the context of the replication, any changes made, and the results. It is not always clear how to balance these various types of information within a replication paper.

Session goal:
The goal of this session, similar to the sessions conducted in 2006 on controlled experiments and in 2007 regarding case studies, will be to evaluate and improve proposed guidelines for reporting replications. Two of the session chairs have independently developed their own proposed guidelines. ISERN members will evaluate the proposed guidelines through a hands-on exercise of trying to complete them for a set of published replications. The ultimate goal of the session is to produce a publishable set of guidelines for reporting experimental replications.

Development of the session:
The session would begin with a few introductory/motivational presentations. Little time will be spent discussing replications and how to conduct them other than what is necessary to motivate a discussion about reporting guidelines. The session chairs will present one or more proposed guidelines for reporting replications. ISERN members will be divided into groups. Based on the guidelines proposed, each group will try to identify the required information in one or more published replication papers. The goal of this exercise is to evaluate the usefulness of the reporting guidelines and identify information that should be included in the guidelines.

Session A2: Systematic Literature Reviews: Are They Really Reliable, Useful and Replicable?
Chair: Claes Wohlin
Location: Max Bell Room 251

Background:
The number of systematic literature reviews has increased over the last five years. The objective of a review is to synthesize the research in an area. But are the reviews reliable? Do we get similar results if conducting two independent reviews?

Session goal:
The goal of the session is to discuss and conclude how we as a community can both write for synthesis and conduct more reliable systematic literature reviews.

Development of the session:
The session will start with an introduction to systematic literature reviews and in particular its challenges. The session will then move into a general discussion on issues related to conducting reviews. The discussion will be based on a set of prepared questions to guide the session.

Session B2: System Evaluation
Chair: Andreas Jedlitschka
Location: Max Bell Room 252

Background:
In the past, ESE aimed mainly at evaluating SE methods with regard to certain aspects, e.g., efficiency etc. However, what does this mean for the resulting system. Is it more usable, reliable, afterwards? Several ideas are published, e.g., the unified theory of acceptance and use of technology by Venkatesh.

Session goal:
The objective of this session is to discuss the question of "how to evaluate SW-based products from an end-user perspective".

Development of the session:
Very brief introduction of the key points, perhaps supported by some people who have experience with those kinds of evaluations. The session will be guided by key questions collected from the audience during the introduction.

The future organization of the ISERN Workshop
Chair: Victor R. Basili
Location: Max Bell Room 251

Background:
There has been some discussion within the ISERN Steering Committee about the future organization of ISERN, as we grow in number. The goal is to continue to allow new members to enter, but still maintain the special flavor of in-depth workshops which push the state-of-the-art envelope. These in-depth workshops created a unique workshop environment in the past. Some of the committee members believe that we are in danger of losing this special flavor due to the facts that we (a) are growing too large, and (b) re-inventing the wheel by discussing issues with always changing attendees with different experiences. Vic Basili has agreed to moderate the meeting and will start with a collection of ideas proposed by various Steering Committee members, and solicit opinions and other ideas from the attendees.

Session A2: What are the Important Problems in Our Field?
Chairs: Guilherme Travassos and Tore Dybå
Location: Max Bell Room 251

Background:
What are the important problems in Software Engineering? Are we doing research that has an impact?

Session goal:
To discuss and prioritize the important research questions in Software Engineering accordingly the perspective of ISERN participants.

Development of the session:
At day one, as part of the welcome and introduction session, audience will be invited to write down their one, top burning research question and put it on a board during the first day. To motivate activities and give the discussion perspective, a short motivational material will be distributed. The important questions will be collected the next morning and in the session, these will be the questions to discuss and prioritize. Audience will be organized in groups to work out the questions. Then a summary will be produced.

Session B2: Software Assurance, Neglected or Unnecessary?
Chairs: Dan Port, Yuko Miyamoto and Haruka Nakao
Location: Max Bell Room 252

Background:
Recent work at JPL indicates that different groups of stakeholders have significantly different ideas about what constitutes SA activities as well as different expectations of their expected benefits and outcomes. Such differing perspectives on SA are both pervasive and persistent. There is a need to establish clarity on what activities constitute SA, the conditions in which these are needed or desirable, and the means for demonstrating their benefit. To this end an extensive empirical study was conducted at JPL with participation from JAXA and the NASA assurance community at large. The primary outcome of this study was a proposed a new definition and "value proposition" for SA, meant to clarify the nature of SA and its tangible expected value to software projects.

Session goal:
To stimulate interest and collaboration activities in utilizing the proposed new SA definition and "value proposition" as a unifying principle for SA operations and research going forward. The expected outcome of the session is to establish, clarify, and prioritize a list of "fundamental" research opportunities in SA.

Development of the session:
The session starts with an introduction to SA. It is followed by an attendee interactive discussion with panel of SA practitioners and researchers of proposed new SA definition and value proposition. Finally, from a brainstorm on research questions, research suggestions and opportunities to address questions are identified and prioritized.

Session A3: Great debate
Chair: Mike Barker
Location: Max Bell Room 251

Background:
Resolved: Using Cloud Computing means End Users don't need Empirical Software Engineering. Depending on who you listen to, cloud computing means never having to worry about programming, software, maintenance, backups, all of that stuff anymore! Just push all of your work into the cloud, access it anywhere and anytime you like, and everything will be wonderful! Right? So... does this mean that end-users and corporate cloud users can quit worrying about empirical software engineering?

Session Goal:
Share ideas and thinking about how empirical software engineering fits into an environment where most computing is done "in the cloud." What kind of "empirical software engineering literacy" does cloud computing require from its end users? Can they really just ignore everything, or does using cloud computing require them to pay attention to certain specific types of research and results?

Development of the session:
1. We'll start by assuming that cloud computing really is the answer to all our problems, and in teams, consider how much using cloud computing reduces the need for end users to understand empirical software engineering models and results.
2. Then we'll consider what empirical software engineering knowledge is needed by end users and cloud system developers and providers, and what research studies need to be done in the cloud environment.
3. We'll summarize this as challenges to ISERN that cloud computing poses.

Session B3: Empirical Approaches to Support Decision Making in Industry
Chairs/Panelists: Pete Rotella, Brian Robinson, Nachi Naggappan and Audris Mockus
Location: Max Bell Room 252

Background:
The role of measurement-based decision making has dramatically increased in the corporate software development environment over the last decade. Many of the measures are based on the data from corporate issue tracking and software development databases, much as the underlying data in the empirical study of software engineering. However, the goals of the measurement in industry are substantially different as are the standards of what constitutes valid evidence.

Session goal:
Share experiences of software quality and productivity measures that are based on corporate databases including software development, sales, and services. Explain how and why the measures were designed and are used to make business and development decisions at the levels of a developer, a project, and of entire corporation. The session also outlines industry needs to academic participants.

Development of the session:
Brief statements by panelists followed by general discussion. Each panelist:
1. Gives the primary objectives of such measurement programs in their context.
2. Outlines the approaches that worked in the past and present existing and future challenges.
3. Provides examples of what is accepted as valid evidence in a particular industry context.
4. Outlines challenges that remain. Translating the above into a language that participants from academia could understand (and act upon).

ISERN business
Chair: Victor R. Basili