The JISC Virtual Research Environment Phase Two Programme: Attitudes to and Awareness of Open Development

by Sander van der Waal on 28 August 2008 , last updated 9 September 2013

Introduction

Executive Summary

[This document describes the situation as at October 2008. Ed]

Background

OSS Watch is a national advisory service that provides open source software advice to the further and higher education communities. The remit encompasses support for JISC’s 2004 policy on open source, which established the principle that JISC-funded projects should not only release their software as open source but also follow an open development approach, including taking steps to foster communities of users and developers around their software.

In early 2008, OSS Watch won approval to significantly ramp-up their level of assistance to UK higher and further education projects, to help them implement this policy and put themselves in a much better position to fully realise the benefits of open source as a result. This study was commissioned late 2008 to help inform the new work through investigating the current level of awareness, attitudes to and understanding of the open development concept within the JISC innovation community.

The projects selected for investigation were the four funded within the JISC Virtual Research Environment Phase Two programme, which is developing innovative software for the UK research community. The study methodology combined face-to-face interviews with a focus-group-style discussion. The analysis was primarily qualitative and took into consideration the author’s own experience of managing JISC-funded projects to strike a balance between the ideals of open source on the one side and practical realities on the other. The OSS Watch team itself were not directly involved with drawing up the list of recommendations.

Findings and recommendations

The study found the four projects to be extremely positive about open source and to be enthusiastic, repeat consumers of open source software. Two out of the four projects interviewed had publicly accessible source code repositories and even those that did not were firmly committed to releasing their software as open source.

The majority had only recently become aware of open development as an approach and of the need to encourage third parties to contribute code and documentation during the lifetime of the project. As none of the projects was explicitly inviting contributions, from an open development perspective, they can be considered to be essentially closed developments.

A minority of projects doubted the applicability of open development to their projects, seeing themselves to be more at the level of testing a concept rather than developing production-quality software. Even those that saw the need expressed significant concerns around the overhead of managing the process of accepting and moderating contributions, given that they had not budgeted for it in their project proposals. Despite this, many expressed a real desire to be further educated about it.

The list of recommendations arrived at set out to address the lack of awareness around the need for and practice of following an open development approach. Information tailored to the specific circumstances of JISC-funded projects was a recurring theme, as was the need to educate bid-writers and markers at as early a stage as possible. It was also recommended that OSS Watch look into other ways this issue could be brought to the attention of the JISC community through, for example, looking at possible incentives and policy changes and through carrying out workshops to address projects’ sometimes deep-seated concerns about ‘going open’.

Background

OSS Watch provides advice on a wide variety of topics around the adoption and development of open source software to the further and higher education communities. Around the period when the second VRE programme started, the service started taking a more active role in helping JISC development projects fully implement the JISC open source policy with a particular emphasis on community development.

This report represents the result of efforts to inform this ‘project-embedding’ work by investigating firstly how successful existing JISC-funded software development projects have been in creating communities, using the JISC Virtual Research Environment Phase Two programme as a sample, and secondly the ways in which this process could be made easier, with or without the assistance of OSS Watch.

OSS Watch and the JISC Open Source Policy

OSS Watch was formed in 2003 as a small pilot advisory service with a remit initially limited to establishing and maintaining a website, organising national conferences and workshops and providing guidance to institutions as requested. When in its sixth year, it had migrated from this purely educational objective to one of active support and guidance. For example, OSS Watch, where possible, started attending and contributing to all JISC programme start-up events, conducting numerous consultations with projects and publishing case studies of a range of active open source development projects.

In 2004, OSS Watch drafted an open source policy for JISC in response to the UK Government Open Source Software Policy ¹. Most memorably, this established the principle that, by default, JISC software development projects should release their software under an Open Source Initiative (OSI) approved open source licence, refining the existing requirement for project outputs to be made freely available to the further and higher education communities. Less memorably, the policy also established guidelines for open development, recommending that projects take steps to cultivate communities of developers (and users) around their software:

4.9 Sustainability and communities

Many open source projects are supported by community effort, and the JISC is keen to encourage broad participation and contribution to open source projects.

• Projects must state in their bid whether they foresee the project continuing beyond the time span of the funding, and if so whom they see participating in the project.
• Projects should engage with end users and other parties to encourage and build self-sustaining communities.
• Projects should facilitate the use of their software in other UK HE and FE institutions.
• Projects may facilitate the use of their software in the wider community [23].
• Projects should accept bug reports, patches, translations and feedback from contributors outside the project.

– Policy on open source software for JISC projects and services

The intention was for projects to establish, through their user and developer communities, a level of support sufficient for them to move beyond their initial phases with less reliance on continuation funding.

JISC Virtual Research Environment Phase Two Programme

The JISC Virtual Research Environment Phase Two (VRE2) programme was one of around thirty JISC programmes developing and piloting software for further and higher education. According to the JISC website, a Virtual Research Environment (or VRE), ‘helps researchers in all disciplines manage the increasingly complex range of tasks involved in carrying out research’. VRE2 followed on from VRE1, a larger and more experimental programme, and was made up of four projects developing pilot virtual research environments to support the needs of UK researchers.

The VRE for the Study of Ancient Documents (VRE-SDM) project is building facilities for geographically dispersed researchers to access and annotate image collections. The VRE for Archaeology (VERA) project is developing and piloting a VRE with the Silchester Roman Town dig to enhance the means of efficiently documenting archaeological excavation and its associated artefacts. MyExperiment is creating a social network for scientists based around, but not restricted to, the sharing and execution of workflows. The Collaborative Research Events on the Web (CREW) project is producing a VRE to capture and publish the scholarly communication that takes place at and around research events.

Even within this relatively small programme, the projects address an extraordinary gamut of needs, some focusing on a single research domain, others being more generic in scope. JISC’s project portfolio as a whole shares this diversity and, as a result, it is almost impossible to pick a representative programme or set of projects. That said, there are certain characteristics that JISC projects normally share. For one, in addition to their being projects for _UK FE and HE, they are also normally _run__by UK HE or FE institutions, frequently by those in computing services or computer science departments. Secondly, JISC projects all tend to run under similar cost and time constraints: budgets are normally well under £500,000 and run for one to two years. Thirdly, projects are often geographically distributed with individual project teams split across multiple UK institutions. In all these respects, the VRE2 programme is no different.

The benefits of open development

The benefits of open source have been well documented. However, some of the reported benefits can be argued to exist in equal, if not greater, measure in closed-source developments. For example, the existence of active user communities contributing documentation, experience and advice to other users, is certainly not unique to open source.

The real advantage open source projects have is their capacity to harness the resources of numerous individuals and organisations to develop the software further, through a form of peer-review simply not available with closed source. This advantage is, however, clearly predicated on the existence of active communities of developers as well as active communities of users.

Community formation does not happen automatically just because software is released as open source. If projects want to establish them, they need to take active steps to foster them throughout the lifetime of the project. While the ultimate goal is an active developer community, user communities are of course critically important to both because without users the project would have no reason to exist, and because user communities also represent the most likely source of future developers, either through users getting involved directly or through their employing people to do so on their behalf. Consequently, it is important that user communities are supported and continue to grow.

The process of drawing in new users may represent a challenge for projects satisfying needs shared by a relatively small number of people. And even for projects addressing more widely held needs, if this also means tackling significant technical challenges, it may seem to make more sense to keep the initial user community reasonably small, while the concept is tested, rather than risk damaging the project’s reputation by releasing software that does not work.

Users are not the only source of potential developers. Those working within specific technical communities or working on technical standards may want to become involved and use the software as a vehicle for an emerging technology. This can be a win-win situation. However, there is a risk that if these people start to outnumber the end-users, the project team as a whole loses sight of its original purpose and becomes an academic exercise: all developers and no users.

Achieving a sensible compromise

OSS Watch is not an advocacy organisation and does not promote open source as such; in contrast, it provides unbiased advice on how institutions can go about adopting or developing it. Having said that, where projects developing software choose to go down the open source route, OSS Watch does advocate embracing the concept in its entirety, including taking steps towards developing communities around the software and not just ‘code dumping’ it on an open source hosting service at the end. This is not to say that OSS Watch recommends all projects should run as consensus-based democracies. Open development is not an all or nothing endeavour: it is possible to be open while retaining elements of control.

Some steps towards open development are likely to be (or perceived to be) more achievable than others.

Methodology

The fieldwork conducted for this study took the form of a series of semi-structured interviews with the four VRE2 projects between January and April 2008, complemented by a session at the VRE2 programme meeting in May 2008.

Interviews

The interviews were designed to look at each individual project’s attitudes, understanding and progress towards implementing open development, the barriers preventing it and the ways in which open development could be made easier, both with and without the assistance of OSS Watch.

The approach used for the first (joint) interview with VERA and VRE-SDM was fairly exploratory and concentrated on the two projects’ attitudes and understanding of the open development concept. The second interview with MyExperiment took place as part of an existing project meeting and less time was spent exploring attitudes and more on actual progress with community development, particularly on the user side. In order to improve coverage, an interview pro-forma² was drafted and used to guide the final interview with CREW.

The topic of community development naturally divided itself into users on one side, and developers on the other, whilst recognising overlaps between the two. For both communities, it normally made sense to distinguish between “friends and family” – those recruited at the bid writing stage – and those not initially known or involved with the project. Activities with the former group were inevitably more about engagement than development: keeping them on board and happy. Discussions on how projects had attracted outside interest led to questions on how projects had marketed themselves to potential users and the technical community, including the other VRE2 projects and groups drafting open standards.

Although not originally intended as a major topic for discussion, the issue of software licensing and legal arrangements frequently cropped up, not least because choosing an appropriate licence and legal structure often presented the most immediate barrier to opening out development.

Session at the VRE programme meeting

The programme meeting session aimed to provoke discussion and arrive at some form of consensus on possible action for OSS Watch. It was an hour long, with the first half devoted to presenting the findings from the interviews, and the second half taking the form of a discussion around the subject of open development, how achievable it is for JISC projects, and how it might be made easier through the work of OSS Watch. A preliminary list of suggestions for OSS Watch was discussed and further added to and refined. At the end, participants were able to indicate what they felt to be their number one suggestion to OSS Watch.

Findings

Attitudes and awareness

Overall, projects demonstrated an overwhelmingly positive attitude to open source. As frequent consumers of open source software, the projects saw open source as a good thing overall and all were committed to releasing their software under an OSI-approved open source licence. On the other hand, projects’ awareness and understanding of open development was less consistent. One respondent said, ‘I’ve only heard a real buzz about open development in the JISC community in the last year or so. Post when we put our bids in.’

User community development

The VRE2 call prescribed a user-centred development approach, introducing a ‘figure of eight’ model (see Figure 1) based on agile and participative software development methods. As a result, all four VRE2 projects are strongly committed to user community engagement in the sense of understanding their users’ needs and working closely with them when developing the software. From an open development perspective, this is clearly a good thing: users will turn away if their needs are not satisfied, and without users there is no project.

As a software development model as opposed to a community development model, the ‘figure of eight’ assumes the user community is already out there, waiting to be engaged with and does not place very much emphasis on drawing in new users. The VRE2 projects all had user communities written into them at their outset and genuine efforts had been made to keep them engaged and interested.

MyExperiment was the only project to have made significant attempts to recruit new users, but this was significantly more straightforward for them because their software was already accessible as a hosted ‘web 2.0’ application. CREW and VRE-SDM were still at the stage of controlling access to the software through carefully managed ‘show and tell’ usability testing sessions; for them it was simply too early to be making the software generally available ­– users needed a certain degree of ‘hand-holding’. VERA was gearing up for the Silchester summer 2008 dig, at which their users would have their first chance to get their hands on the technology developed.

Developer community development

None of the projects had really considered opening up development beyond their core teams in their initial funding periods. A representative comment made by one technical manager was, ‘We haven’t got a plan for opening it out as an open source development. That is not to say we would rule it out.’

Putting materials out

The most important step in opening out development is making the software available so that other developers can ‘play’ with it. However, this is not just about putting the source code on a website or granting the public access to the revision control system. It is generally recognised that developers (and users) are much more likely to get involved if the software is made available as a _release _with some associated statement of quality. For example, that it will compile and run at the very least.

Two out of the four VRE projects, MyExperiment and VERA, had already made their source code avilable via version control: MyExperiment through RubyForge and VERA through a local Trac system, although the VERA code is not publicly accessible³. However, for differing reasons, the projects had not carried out any formal releases. One explanation for this was that the projects had not needed to do so in order to make the software accessible to their users. MyExperiment’s users were almost all adopting the project’s hosted service (a ‘perpetual beta’) at myexperiment.org rather than installing local copies; VERA, CREW and VRE-SDM were still controlling access to the software.

Looking beyond this, projects expressed a basic hesitancy about attracting third-party developers and putting their software out in this way. There was the feeling that their projects were simply not ‘big enough and popular enough’ for it to be worth it. One project said, ‘You have got to have enough people using the software to make it viable.’ There was also the fear that putting software out opened up developers to a certain amount of exposure and there was the feeling from one project that ‘that transparency could inhibit developers’. At the VRE programme meeting, this fear was explored further.

Another concern was that by opening up development, projects would lose an element of control and even the cachet associated with having been funded to deliver the project in the first place. There was also the worry that other people might change the direction of the project to the detriment of the end user community. All projects appeared to prefer the Linux model: where they control a core part and third-party contributors develop tools around the edges.

Accepting and managing contributions

Making releases available allows developers to ‘play’ with the software. What it does not provide for is mechanisms to allow changes to be contributed back into the project.

At one level or another, the VRE2 projects recognised that carrying out releases was a necessary part of opening up development. However, what they had not expected to do was release their software any earlier than at the end. While projects had been very open about the idea of open development, at the same time they also expressed real concerns about the overhead of managing the process of accepting contributions on top of delivering the software itself. One technical manager commented, ‘[This overhead] wasn’t factored in on this project and I haven’t factored it in on any bid I’ve written.’ Another said, ‘Managing and massaging the code when other people are contributing is a lot more work than writing the code yourself.’

As a result of this reluctance, none of the projects had concrete plans for opening up development, no dedicated developer websites and no instructions on how to contribute. Despite this, there were two projects – MyExperiment and VERA – that were agreeable to outside parties developing tools and plug-ins. MyExperiment, by releasing a “RESTful”, API, were even implicitly inviting this and examples of ‘mash-ups’ integrating MyExperiment functionality already existed in the form of three Google Gadgets. MyExperiment admitted, however, that they did need a strategy for plug-ins so that those building them would know ‘where to put them’.

Another way in which the projects had hoped to open out was by spinning out sub-components as open source projects in their own right. VERA had, as part of their efforts to integrate off-the-shelf applications such as Wordpress and MediaWiki within their portal, developed a middleware component dubbed ‘RecycleBridge’ to facilitate this. CREW stated a desire to spin out an integral RDF-based annotations component having already shared it with two other JISC-funded projects at the University of Bristol Institute for Learning and Research Technology (ILRT). In both cases, it was felt that by spinning these components out, they would be made relevant to a much wider group of users and stand a better chance of attracting developers from the portals and semantic web communities respectively.

MyExperiment had engaged with technical communities too, but were unique among the VRE2 projects in that they had decided not to embrace portals as their main presentation technology. They did, however, state an ambition to launch a portals activity in parallel to the main project. MyExperiment had engaged with both the World Wide Web Consortium and digital repositories communities through their development of Encapsulated myExperiment Objects (EMOs) designed to interoperate with the emerging OAI-ORE⁴ standard. MyExperiment had also collaborated with the JISC-funded SKUA project exploring the use of Open Social⁵ as a possible collaboration path. In doing so, the SKUA project became the first and only project to install MyExperiment locally rather than using the hosted service.

Sustainability and exit plans

The projects’ general approach to sustainability was either to seek additional funding at project end or to investigate commercial exploitation opportunities. As one technical manager commented, ‘The sustainability plan is being seen as enough of a success earlier in the project that we stand a chance of bidding for more money.’

Suggestions from the projects to OSS Watch

Many of the suggestions made by projects were related to the provision of information. However, it was not so much that more information was needed; it was more that the wealth of existing information be tailored to the particular circumstances of JISC-funded projects. For example, while OSS Watch had published numerous open source case studies, there was only one that showed how an existing JISC-funded project⁶ had successfully followed an open development approach. Publishing more similar exemplars was the most popular suggestion by far because, in the words of one technical manager, ‘It’s always easier to play follow-my-leader.’

Another example of this tailoring was the suggestion of providing template governance models and contributor licence agreements. OSS Watch already had resources on drafting these documents, but the projects felt that a lot of time could be saved by allowing projects to obtain boilerplate documents for their specific circumstances. This assisted decision-making could also simplify the process of choosing the open source licence itself. Rather than relying on generic advice, projects could be guided through a series of scenarios allowing them to explore the risks, costs and benefits of taking different licensing decisions. At the same time, there was also the suggestion that OSS Watch or JISC could provide a fully fledged legal advice service for open source.

A further aspect to the information-related suggestions was the need for dissemination to be less passive and much more active. In addition to information being made available on the website and its associated RSS feeds, it should be proactively put in front of the people who need to know it, particularly the bid writers and bid markers. OSS Watch does, where possible, attend all JISC town and programme meetings, and projects are normally required to consult with OSS Watch when they get started. However, projects reported that the message about open development had been received far too late. Unless bid-writers are made aware and educated about it, open development is very unlikely to be planned or budgeted for.

No amount of information, education or advice will help some projects get over their most deep-seated fears about ‘going open’. There was the suggestion that OSS Watch should organise workshops with projects to look at their concerns and how they might be addressed, for example, through effectively managing the expectations of users. One way of helping projects to overcome their fears could be by working with them to follow their scenarios through to their logical extremes; by helping projects to visualise potential worst-case scenarios, risks are much better understood and defined.

Some of the suggestions were less about help for individual projects and more about policy at the departmental, institutional and national level. For example, could some of the hesitancy about putting software out earlier be tackled through departments and institutions putting in place incentives, in the same way that the open access movement has encouraged researchers to deposit their publications in institutional repositories? One radical but popular idea was for OSS Watch or JISC to actually ‘manage’ an open source community of developers for JISC projects. This would provide a pool of developers and contributors and help to ensure that knowledge is not lost. It was suggested that even if this might not work in practice, it might still be a valuable ‘thought experiment’ to carry out.

Conclusions

Even though the VRE2 projects are all committed to releasing their software as open source, the lack of resources for developers, formal releases and, in some cases, read access to code repositories shows that the VRE2 projects are not conducting development in a very open way. As a consequence, they are making it much less likely for development communities to form, preserve the knowledge built up by team members and take the projects beyond their initial funding.

The reasons the VRE2 projects have not embraced open development appear to be rooted in a combination of lack of awareness, psychology, and lack of resources. Awareness relates both to which steps to take as well as the importance of actually taking them: whether they are essential to open development or simply desirable. For the steps projects are aware of, there seem to be significant psychological barriers preventing them from taking them both in terms of their perceived difficulty, effort (overhead) involved and possible fall-out.

None of those interviewed were completely against open development. However, there were some that were doubtful as to how applicable it was for their projects, especially where they saw their projects to be more about testing a concept than producing production quality software. Even so, the majority were not against the idea of putting open development into practice in future projects, as long as they have sufficient resources to do it and implementing it will not distract them too much from the task of developing the software.

The concept of community development is very much a part of the JISC open source policy and the VRE2 circular explicitly required applicants to ‘make clear the licence under which software outputs will be released, mechanisms that will be put in place for community contribution (users and developers) throughout the project, and the sustainability plan for the project.’ However, this had not resulted, in the case of VRE2, in projects budgeting realistically for the overhead of managing the process of open development. The lesson here is that OSS Watch and JISC should be much more pro-active with educating the JISC innovation community, particularly the bid-writers and bid-markers, on the importance of factoring this in.

Recommendations

The following prioritised list of recommendations for OSS Watch has been distilled from the interviews and VRE programme meeting session. The OSS Watch team itself was not directly involved with drawing them up but did, wherever possible, work these recommendations into OSS Watch’s project plan for the 2008-2010 funding period.

The status comment was added to indicate the activities that have been undertaken in each area and reflects the status in 2011.

OSS Watch is happy to provide consultation on how open development can benefit your project. Contact us at mailto:[email protected]

Acknowledgements

OSS Watch is funded by the Joint Information Systems Committee (JISC) and is located at Oxford University Computing Services.

The author would like to thank the JISC VRE programme manager and members of the VRE projects for contributing their time to participate in this study and for commenting on previous drafts of this report. In particular, the author would like to thank: Ruth Kirkham, John Pybus, Ségolène Tarte, Henriette Roued Olsen, Mark Baker, Matthew Grove, Emma O’Riordan, David de Roure, Carole Goble, Danius Michaelides, David Withers, Alan Williams, Stuart Owen, Ian Dunlop, Franck Tanoh, Katy Wolstencroft, Meik Poschen, Yuwei Lin, Antoon Goderis, Jiten Bhagat, Nikki Rogers, Michael Daw and Frederique van Till.

Appendices

Appendix A - Interview Pro-Forma

This pro-forma was developed to guide the final interview with CREW.

• Introduction

• Awareness of and attitudes to open development

  • JISC open source policy

• User community development

  • Current size
  • Marketing and PR (Website, blogging, launch, national press, demos, etc)
  • Engagement with existing users (requirements workshops, usability testing, training, etc)

• Developer community development

  • Current size
  • Developer’s website: mailing list, issue tracker, SVN, documentation, etc.
  • Hosting
  • How are you attracting developers, e.g. from semantic web community, user community, etc.?
  • Any 3rd party contributions?
  • Governance model
  • Releases

• Legal status

  • Have they sought advice?
  • Copyright ownership
  • Software licence
  • 3rd party software
  • Any contributor licence agreement
  • Any applicable content licences

• Problems with being open: how could OSS watch make it easier?

  • E.g. problems: distractions, IPR
  • E.g. help: workshops, governance framework, contributor licence agreement, advice.

• Exit plan

  • Seek additional funding?
  • Taken forward by developers?
  • Who is the driving force?

• Open standards engagement

  • Semantic web
  • Portals
  • Open social
  • etc
• Review of actions
• Close

Further reading

Related information from OSS Watch:

• How to build an open source community
• Sustainable open source
• Roles in open source projects
• Sustainability Case Studies
• Software sustainability maturity model
• Open source and research infrastructure