These materials were assembled by the following participants in the workshop: Kristen Arbutiski, Julie Ballin, Matt Bietz, Jeremy Birnholtz, Tom Finholt, Darren Gergle, Erik Hofer, Gary Olson, and Diane Sonnenwald.

Measures of collaboratory success

All of the discussion groups, to varying degrees, began consideration of collaboratory success by first attempting to define a collaboratory. Synthesizing across these efforts produces the following description:

A collaboratory is a network-based facility and organizational entity that spans distance, supports rich and recurring human interaction oriented to a common research area, fosters contact between researchers who are both known and unknown to each other, and provides access to data sources, artifacts and tools required to accomplish research tasks.

Several features of this definition are helpful in thinking about how to measure collaboratory success. First, the emphasis on spanning distance suggests that when measuring collaboratory success, the correct comparison is with other forms of long-distance collaboration – and not with collocated collaboration. Second, the emphasis on contacts, recurring interaction, and artifacts suggests that a critical contribution of a collaboratory is to be a social arena for research work – but an arena that is not tied to a particular physical space. And finally, the emphasis on research suggests that the impact of collaboratories must be evaluated in terms of products critical to the research endeavor, such as creation of new knowledge and techniques.

The following examples illustrate different kinds of measures that may reflect success in collaboratory-based collaboration, both in terms of process and product, when compared to more traditional long-distance collaboration: frequency and impact of discoveries; frequency and impact of publications; diversity among collaborators (on multiple dimensions); satisfaction of collaborators; frequency of travel; number and prestige of honors accumulated by collaborators; proportion of collaborations composed of first-time collaborators; impact of the collaborations on the training of young scientists; attracting newcomers to the field, frequency and duration of delays; level of interpersonal trust; extent of shared mental models; degree of mutually consistent work practices; and level of public interest.

These proposed measures form the focus for three principal research threads: a) exploration of the feasibility and sensitivity of the success measures using retrospective and prospective data from operational collaboratories – in comparison with traditional long-distance collaborations; b) exploration of the different collaboratory types that have evolved (e.g., integrated versus aggregation of separate applications) – and how these types influence success; and c) exploration of how collaboratory stakeholders, level of analysis, and time interact to influence success. An overarching concern was the problem of finding a sufficient number of viable collaboratories to form a sample for examining the impact of collaboratories.

(Click here for detailed notes on measures)

Group processes in collaboratories

A way to view group processes in collaboratories is to see these as independent variables that influence the success measures summarized in the preceding section. This section, then, is divided into two parts. First, a list of candidate processes are proposed.Then, these candidate processes are analyzed in terms of their importance to understanding collaboratory success.

Candidate processes

The processes suggested in the discussion groups included: a) awareness, meaning the availability and interpretation of cues about what others are doing; b) task interdependence, such as Van de Ven et al.'s (1976) scheme of pooled, sequential, reciprocal, and team tasks; c) maintenance of common ground; d) group dynamics (including trust, leadership, conflict resolution, decision making, roles, and goal setting); e) creation of routines and procedures; f) communication, such as cultural sensitivity and fluency; and g) interaction of task and process characteristics and technology (e.g., scarcity of instrument time).

Important processes

To guide examination of the processes listed above, workshop participants discussed building on an effort launched by the Science of Collaboratories group at Michigan to produce a framework for identifying variables of interest. This framework can be developed both retrospectively based on data previously collected and prospectively for new and ongoing collaboratories. Examples of relevant contexts include face-to-face collaboration within and across science and engineering labs and distributed collaboration outside of a collaboratory.

An additional suggested starting point was to consider how the identified processes and variables might moderate and mediate collaboratory success. That is, from Baron and Kenny's (1986) description, a moderator variable influences the strength or direction of the relationship between an independent and a dependent variable and a mediator variable explains the mechanism behind the relationship. In the case of collaboratories, for example, there may be an interest in how a particular measure of collaboratory use relates to a particular measure of collaboratory success – where this relationship may be moderated by some factors (e.g., task interdependence) and mediated by others (e.g., awareness).

(Click here for detailed notes on processes)

Organizational and community consequences of collaboratory use

The discussions of organizational and community consequences of collaboratory use were focused along two lines: examination of potential consequences, assuming widespread collaboratory use; and specific collaboratory-based interventions that might have beneficial outcomes (e.g., expanding the pipeline for training of future scientists).

Potential consequences

Expanded access – While some workshop participants noted that claims for expanded access to science, through collaboratory use, are probably exaggerated – there was still hope that collaboratory development might create new opportunities for broader participation in research (particularly among institutions with fewer resources, both domestically and internationally). Specifically, collaboratories may provide a way for faculty at Master's Colleges and Universities I and II (i.e., from the Carnegie Classification – institutions that are primarily oriented to undergraduates and award 20 to 40 masters degrees per year) to maintain active research involvement with colleagues at Doctoral/Research Universities – Extensive (i.e., from the Carnegie Classification – institutions with undergraduate and graduate programs, typically awarding 50 or more doctorates per year across 15 or more disciplines). Also, collaboratories may provide a compelling mechanism for outreach, particularly to K-12 audiences (i.e., via science museums, classroom use and etc.).

Structural transformation – Again, while some participants expressed skepticism – many expected that widespread collaboratory use would produce changes in the structure of scientific communities. Specifically, collaboratories were seen as accelerating trends toward increased multi-disciplinarity (although there were concerns about the scientific value of multi-disciplinarity) and toward cross-institutional mentoring.Similarly, there was an expectation that feedback might occur, such that wider adoption of previously unorthodox practices (e.g., electronic publication) and career tracks (e.g., outside the tenure system) may also produce incentives that reward collaboratory use – which in turn, may accelerate transformation of the status quo (e.g., by increasing visibility of junior scientists or of researchers outside the scientific mainstream).

New kinds of science – A principal envisioned impact was the capability to bring previously unrelated communities of researchers together through the capacity of collaboratories to combine data in new ways. For example, in the Space Physics and Aeronomy Collaboratory, observational and simulation data can be viewed side by side, which has led to new interactions between theoretical space physicists and experimental space physicists.

Increased production – In the discussion of success measures, increased rates of publication and discovery were identified as important outcomes of collaboratory use. Discussions focused on the sensitivity of such measures with particular attention to the lag between adoption of innovative practices and impact in terms of publication or citation patterns. Comparisons to earlier innovations, like email and electronic journals, suggested that even at mature points in the penetration of these innovations – it was still hard to see impact in terms of citation data. However, broader access to the Internet does seem to have played a role in the dramatic increase of papers within one very special class (authors from ten or more different countries). Collaboratory impact might be most visible in the case of similar “mega-collaborations.” One proposal was to track the number of different congressional districts represented among authors on papers from collaboratory-based collaborations versus conventional long-distance collaborations.

Collaboratory-based interventions

In terms of next steps for research there was a reluctance to pursue the connection between collaboratory use and broad changes – both because of the immaturity of collaboratory technology and because of the multi-causal nature of large transformations (e.g., the trend toward multi-institution authorship has been increasing for several decades – pre-dating the Internet, collaboratories and other more recent inventions). However, there was enthusiasm for tightly scoped interventions with high impact, both in terms of outcomes and as demonstrations. For example, in terms of Ph.D. production in the United States, a small set of selective liberal arts colleges have performed at a disproportionately high level (Carrier & Davis-Van Atta, 1987) – with corresponding disproportionate representation among authors of highly-cited publications and among members of the National Academy of Sciences. The keys to this success are partially related to the admission criteria of these institutions – but other important factors include exposure to hands on research activity and mentorship by faculty. With the right mixture of support, collaboratories might be used to produce the conditions currently found only at these selective liberal arts colleges, but across a wider array of institutions – with corresponding increases in the number of students choosing to pursue graduate degrees and research careers.

(Click here for detailed notes on organizational and community processes)

Bibliography

Baron, R.M., & Kenny, D.A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173-1182.

Carrier, S., & Davis-Van Atta, D. (1987). Maintaining America's scientific productivity Oberlin, OH: Oberlin College.

Van de Ven, A.H., Delbecq, A. L., & Koenig, R., Jr. (1976). Determinants of coordination made within organizations. American Sociological Review, 41(3), 322-338.