Reproducible Research Oxford (RROx): launch event (Part I/II)

[ENG] It is becoming more apparent that there should be greater focus on reproducible and transparent research. In light of recent* public exposure of scientific research that raised concerns of reproducibility and transparency of data, I as an early career researcher (ECR) feel that this “topic” should be at the forefront of the minds of all ECR’s. 

The concept of reproducible and transparent research has been on my mind for some time and I was keen to attend the launch event for “Reproducible Research Oxford (RROx)” in January (a bit fortuitous, given recent events). It was a really interesting event that explained not only how they came about but also bringing together people from various parts of the research environment to discuss their own involvement in reproducible research. This blog is one of two. Here, I briefly summarise the launch event and add a few thoughts of my own. The second blog will be by Sara Middleton, and she will provide her view, as an ECR, on the event and the topic of reproducibility and transparency of data. 

screenshot 2020 03 23 at 08 42 23

Figure 1. Left: One of the speakers, Brian Nosek, highlighting what steps need to be taken to initiate a change in research culture. Right: Brian showing how these steps would initiate change in the scientific community.

The first talk at the launch event was given by Dorothy Bishop. She opened with the landmark finding by the Open Science Collaboration (led by Brian Nosek; Open Science Collaboration. 2015. Science, 349, aac4716) that only 1/3 to 40% of psychology studies gave the same result when they were reproduced. This highlighted a potential “problem” in that many research studies may not be reproducible. She said that this finding was the instigator to address the issue of reproducibility which she raised with colleagues at the Academy of Medical Sciences (AMS) symposium in 2015. Following this meeting there was a push to actively address the reproducibility issue and through funding, collaborations, training and large ECR involvement RROx was established (a timeline of seed stage to launch of RROx is shown in the text box below). In 2018, RROx became a signatory of the San Francisco Declaration on Research Assessment (DORA), which is a set of recommendations to improve the ways in which output of scientific research is evaluated. What I found exciting was that the RROx committee included one ECR from each division at the University, showing the importance of ECR involvement but also the realisation that ECRs will be important in driving reproducible research into the future. This was further highlighted by the work already done by ECRs in promoting what RROx is trying to achieve, an example of this is the ReproducibiliTiea initiative where ECRs inform and discuss the issue of reproducibility with other ECRs within divisions and departments. Dorothy’s talk was followed by Malika Ihle (coordinator or RROx) and Amy Warner (from the Bodleian library). Malike is an open-research ambassador and part of her role is to survey the awareness, practice and needs of researchers. What I found particularly interesting is that she has access to an inventory of current initiatives focusing on promoting reproducibility. Also good to see was the role she would be playing in liaising with stakeholders about relevant policies. Amy’s talk was showcasing the Bodleian library and their commitment to open access and an open research culture. This was really encouraging as the involvement of the Bodleian with RROx showed a shared interest of promoting reproducible research throughout the University.

screenshot 2020 03 23 at 08 44 34

The promotion of an open research culture beyond RROx and the University was made apparent by representation from UK Reproducibility Network (UKRN) and UK Research England (UKRI). The UKRN was represented by Laura Fortunato. The UKRN is a fairly new initiative with their first annual meeting held last year and, since then, they now have 10 member institutes as part of the network. Laura explained that UKRN provides the opportunity to improve research by reforming culture and practice, with the aim to work collaboratively, not competitively, and strive for broad disciplinary representation beyond sciences. But this raised the question - what does research reproducibility mean across disciplines? This is not a straightforward answer and the UKRN has implemented a conceptual framework to try and address what reproducibility means. The framework has three parts: 1) Methods of reproducibility, 2) Reproducibility of results and 3) Inferential reproducibility (to draw consistent conclusions from analysis and replication). At the moment UKRN are currently focusing on 1 and 2, and Laura explained that only when the first two parts of the framework have been resolved will the UKRN move on to 3. Laura also announced that the UKRN will be holding a conference from 9th – 11th September, entitled “Reproducibility, replicability and trust in science”, I suspect the conference will be stimulating and hopefully there will be a good attendance. The UKRI was represented by Claire Fraser. Claire is a senior higher education policy advisor and she discussed the role of open and reproducible research in the REF. To be honest I was somewhat surprised that reproducibility was on the radar of UKRI, to the extent that it would play a role in REF.  Apparently, the UKRI policy work (which includes open research) does cross over into REF and that open research does not just mean open publication but also a route to transparent, efficient and reproducible research. Another point that Claire made, one that was apparent throughout, was that incentives drive behaviours and that this impacts research and reproducibility. She discussed the role of funders and how they assess research excellence, which is important because this impacts behaviour of researchers and how they would approach reproducibility. The message was that research assessment should assess content of specific papers and not indices like impact factors and the like as surrogate measures. There was one point that Claire made that resonated with me and that is something that needs reassessing – all stakeholders need to measure what we value and play a role in facilitating cultural change. 


Figure 2. Brian Nosek highlighting that the 'Registered Reports' initiative is working and research is been cited by scientists.

The final two talks of the day were really interesting and were given by two researchers who are on the frontline pushing open and reproducible science. The first was by Stephen Curry, someone who’s online content I have been following for a while, and so I was really excited to hear him speak. Stephen is a Professor at Imperial College London and chair of the steering committee for DORA. In his talk, Stephen raised the point of value in research and that there needs to be a change in research culture and research evaluation. He proposed shared research values where: 1) research is reliable, rapidly communicated, accessible and of high quality; 2) researchers collaborate, have a duty of care to group members and colleagues, and a responsibility to the societies of which they are part of; and 3) the research system values the people within it, cares about their quality of life, and seeks out the creative vigour of diversity. What still astonishes me is that this is a proposal, whereas it should be the norm. It is an indication of a broken system that is our research culture, I believe (and have done so for some time) that it can be changed. I also believe many ECR’s feel the same way but it should not be up to the ECR’s alone to force this change. Stephen offered a practical solution of engaging in the open science discussion: Preprints, open peer review, open access generally and data sharing. Naturally this leads to important questions about democratic accountability but this is where DORA as a campaign addresses the way forward. I would highly recommend visiting their website to see the initiatives they are taking to ensure shared research values.

The final talk was given by Brian Nosek form the University of Virginia, co-founder and director of the Centre for Open Science, and leading contributor to the landmark paper showing the problem of lack of reproducibility in science. The message that Brian was indicating was that often incentives in research are for individual success, to get it published not get it right. He advocates for a change in culture and he says that this can be achieved and suggested steps to do so. First, if the infrastructure is made available then it makes it possible for the innovators to start the change. Secondly, through experiencing the change early adopters will promote the change. Then, through the community the change will be made normative and the early majority will take on the change. Through incentives, adopting the change will make it rewarding to the late majority and finally by initiating policy the laggards will be required to institute change. Brian gave an interesting example of an initiative that was implemented by the Psychological Science journal where a visual badge system was used to indicate openness of data and material and whether manuscripts were preregistered. Since the implementation of the system in 2013 one quarter of journal papers in 2019 have a preregistered badge, half have an open materials badge and 60% have an open data badge. Another initiative is Registered Reports, where peer review is done at the design and conception of the manuscript before data is collected and analysed. This will result in accuracy rather than beauty of reporting, publishing of negative results, conducting replications and that peer review focuses on the quality of the methods. Although still somewhat of a novel approach Brian did show some evidence that Registered Reports were working and being cited by scientists. What became apparent from Brian’s talk is that there is evidence of cultural change as more researchers are posting online materials and data while also preregistering analyses or hypotheses. However, he did stress that having grassroots campaigns are important. Visibility is essential for changing norms and we have to highlight open science as good practice and not as social identity. Two points Brian made is how I think we should do science - embrace incrementalism and cultivate intellectual humility. 

“We need to get it right over being right” – Brian Nosek, 2020.

Writtten by Jacques Deere

*Please note that at the time of writing this there has been no evidence of intent to falsify data or mislead by the party involved and the relevant investigations are still ongoing. Since the initial awareness of the problems surrounding the data were made, the approach taken by journals and collaborators that were impacted are made clear here and here.


 [AFR] Dit word al hoe duideliker dat daar meer gefokus moet word op reproduceerbare en deursigtige navorsing. In die lig van die onlangse* openbare blootstelling van wetenskaplike navorsing wat kommer oor reproduseerbaarheid en deursigtigheid van data geopper het, is ek as 'n vroeë loopbaannavorser (ECR) van mening dat hierdie 'onderwerp' aan die voorpunt moet wees van alle ECR's.

Die konsep van reproduceerbare en deursigtige navorsing was al 'n geruime tyd op my gedagtes en ek was gretig om die bekendstellingsgeleentheid vir 'Reproducible Research Oxford (RROx)' in Januarie by te woon ('n bietjie voorspoedig, gegewe onlangse gebeure). Dit was 'n baie interessante gebeurtenis wat nie net verduidelik het hoe hulle tot stand gekom het nie, maar ook mense uit verskillende dele van die navorsingsomgewing bymekaar gebring het om hul eie betrokkenheid by reproduceerbare navorsing te bespreek. Hierdie blog is een van twee. Hier gee ek 'n kort opsomming van die bekendstellingsgeleentheid en voeg 'n paar eie gedagtes by. Die tweede blog is deur Sara Middleton en sy sal haar siening as 'n ECR van die gebeurtenis en die onderwerp van reproduceerbaarheid en deursigtigheid van data bied.


Die eerste praatjie tydens die bekendstellingsgeleentheid is aangebied deur Dorothy Bishop. Sy open met die landmerkbevinding deur die Open Science Collaboration (gelei deur Brian Nosek; Open Science Collaboration. 2015. Science, 349, aac4716) dat slegs 1/3 tot 40% van sielkunde-studies dieselfde resultaat gelewer het toe dit gereproduseer is. Dit het 'n potensiële 'probleem' beklemtoon deurdat baie navorsingstudies moontlik nie reproduceerbaar is nie. Sy het gesê dat hierdie bevinding die aanhitser was om die kwessie van reproduceerbaarheid aan te spreek wat sy in 2015 saam met kollegas by die simposium van die Akademie vir Mediese Wetenskappe (AMS) geopper het. Na hierdie vergadering was daar 'n poging om die reproduceerbaarheidskwessie aktief aan te spreek en deur middel van befondsing, samewerking, opleiding en 'n groot ECR-betrokkenheid is RROx gestig ('n tydlyn van die saadfase vir die bekendstelling van RROx word in die tekskassie hieronder getoon). In 2018 het RROx 'n ondertekenaar geword van die San Francisco-verklaring oor navorsingsassessering (DORA), wat 'n stel aanbevelings is om die maniere waarop die uitsette van wetenskaplike navorsing beoordeel word, te verbeter. Wat ek opwindend gevind het, was dat die RROx-komitee een ECR van elke afdeling aan die Universiteit bevat, wat die belangrikheid van ECR-betrokkenheid toon, maar ook die besef dat ECR's belangrik sal wees om reproduceerbare navorsing in die toekoms te bevorder. Dit is verder beklemtoon deur die werk wat ECR's reeds gedoen het om die bevordering van wat RROx probeer bereik, 'n voorbeeld hiervan is die ReproducibiliTiea-inisiatief, waar ECR's die kwessie van reproduceerbaarheid met ander ECR's binne afdelings en departemente inlig en bespreek. Dorothy se praatjie is gevolg deur Malika Ihle (koördineerder of RROx) en Amy Warner (uit die Bodleiaanse biblioteek). Malike is 'n oop navorsingsambassadeur en deel van haar rol is om die bewustheid, praktyk en behoeftes van navorsers te ondersoek. Wat my veral interessant gevind het, is dat sy toegang het tot 'n inventaris van huidige inisiatiewe wat fokus op die bevordering van reproduceerbaarheid. Ook goed om te sien was die rol wat sy sou speel om met belanghebbendes oor die relevante beleid te skakel. Amy se praatjie toon die Bodleiaanse biblioteek en hul toewyding aan oop toegang en 'n oop navorsingskultuur. Dit was baie bemoedigend, aangesien die betrokkenheid van die Bodleian by RROx 'n gedeelde belangstelling getoon het om reproduceerbare navorsing regdeur die Universiteit te bevorder.


Die bevordering van 'n oop navorsingskultuur buite RROx en die Universiteit is duidelik gemaak deur verteenwoordiging van die UK Reproducibility Network (UKRN) en UK Research England (UKRI). Die UKRN is deur Laura Fortunato verteenwoordig. Die UKRN is 'n redelik nuwe inisiatief, hul eerste jaarvergadering was verlede jaar gehou en sedertdien het hulle nou tien lede-institute as deel van die netwerk. Laura het verduidelik dat UKRN die geleentheid bied om navorsing te verbeter deur kultuur en praktyk te hervorm, met die doel om saam te werk, nie mededingend te werk nie, en te streef na breë dissiplinêre verteenwoordiging buite die wetenskappe. Maar dit het die vraag laat ontstaan: wat beteken reproduceerbaarheid van navorsing oor dissiplines? Dit is nie 'n eenvoudige antwoord nie en die UKRN het 'n konseptuele raamwerk geïmplementeer om te probeer aanspreek wat reproduceerbaarheid beteken. Die raamwerk bestaan ​​uit drie dele: 1) Metodes van reproduceerbaarheid, 2) Reproduseerbaarheid van resultate en 3) Inferensiële reproduceerbaarheid (om konsekwente gevolgtrekkings uit analise en replikasie te maak). Op die oomblik fokus UKRN tans op 1 en 2, en Laura het verduidelik dat na die UKRN die eerste twee dele van die raamwerk opgelos het sal hulle aan 3 fokus. Laura het ook aangekondig dat die UKRN 'n konferensie van 9 tot 11 September gaan hou, getiteld “ Reproduseerbaarheid, herhaalbaarheid en vertroue in wetenskap ”, ek vermoed dat die konferensie stimulerend sal wees en dat daar hopelik goeie bywoning sal wees. Die UKRI is deur Claire Fraser verteenwoordig. Claire is 'n senior adviseur vir hoër onderwys en bespreek die rol van oop en reproduceerbare navorsing in die REF. Om eerlik te wees, was ek ietwat verbaas dat reproduceerbaarheid op die radar van UKRI was, in die mate dat dit 'n rol in REF sou speel. Die UKRI-beleidswerk (wat oop navorsing insluit) is oënskynlik oorgedra in REF en dat oop navorsing nie net oop publikasie beteken nie, maar ook 'n weg na deursigtige, doeltreffende en reproduceerbare navorsing. 'n Verdere punt wat Claire gemaak het, wat deurgaans duidelik was, was dat aansporings gedrag gedryf het en dat dit navorsing en reproduceerbaarheid beïnvloed. Sy het die rol van befondsers bespreek en hoe hulle navorsingsuitnemendheid beoordeel, wat belangrik is omdat dit die gedrag van navorsers beïnvloed en hoe hulle reproduseerbaarheid sou benader. Die boodskap was dat navorsingsassessering die inhoud van spesifieke referate moet beoordeel en nie indekse soos impakfaktore en dies meer as surrogaatmaatreëls nie. Daar was een punt wat Claire gemaak het wat met my resoneer, en dit is iets wat herevalueer moet word - alle belanghebbendes moet meet wat ons waardeer en 'n rol speel om kulturele verandering te vergemaklik.

Die laaste twee samesprekings van die dag was baie interessant en is aangebied deur twee navorsers wat op die voorste vlak oop en reproduceerbare wetenskap stoot. Die eerste is deur Stephen Curry, iemand wie se aanlyn-inhoud ek al 'n rukkie volg, en daarom was ek baie opgewonde om hom te hoor praat. Stephen is professor by Imperial College Londen en voorsitter van die bestuurskomitee vir DORA. In sy praatjie het Stephen die punt van waarde in navorsing geopper en dat daar 'n verandering in navorsingskultuur en navorsingsevaluering moet wees. Hy het gedeelde navorsingswaardes voorgestel waar: 1) navorsing betroubaar is, vinnig gekommunikeer, toeganklik en van hoë gehalte is; 2) navorsers werk saam, het 'n sorgsaamheidsplig teenoor groeplede en kollegas en is verantwoordelik vir die samelewings waarvan hulle deel is; en 3) die ondersoekstelsel waardeer die mense daarin, gee om oor hul lewensgehalte en soek na die kreatiewe lewenskragtigheid van diversiteit. Wat my nog steeds verbaas is dat dit 'n voorstel is, terwyl dit die norm moet wees. Dit is 'n aanduiding van 'n gebroke stelsel wat ons navorsingskultuur is, ek glo (en het dit al 'n geruime tyd gedoen) dat dit verander kan word. Ek glo ook dat baie ECR op dieselfde manier voel, maar dit behoort nie die ECR se verantwoordelikheid te wees om hierdie verandering te dwing nie. Stephen het 'n praktiese oplossing aangebied om deel te neem aan die oop wetenskaplike bespreking: Voorafdrukke, oop portuurbeoordeling, oop toegang in die algemeen en datadeling. Dit lei uiteraard tot belangrike vrae oor demokratiese aanspreeklikheid, maar dit is hier waar DORA as 'n veldtog die weg vorentoe aanspreek. Ek beveel sterk aan dat u hul webwerf besoek om te sien watter inisiatiewe hulle neem om gedeelde navorsingswaardes te verseker.

screenshot 2020 03 23 at 08 44 51


Die slotpraatjie is aangebied deur Brian Nosek van die Universiteit van Virginia, mede-stigter en direkteur van die Center for Open Science, en 'n toonaangewende bydraer tot die landmerkpapier wat die probleem van 'n gebrek aan reproduceerbaarheid in die wetenskap toon. Die boodskap wat Brian aandui, was dat aansporings in navorsing dikwels vir individuele sukses is, om dit gepubliseer te kry en nie reg te kry nie. Hy pleit vir 'n verandering in kultuur en sê dat dit bereik kan word en stel stappe voor om dit te doen. Eerstens, as die infrastruktuur beskikbaar gestel word, maak dit die innoveerders moontlik om met die verandering te begin. Tweedens vroeë aannemers sal die verandering bevorder deur die verandering te ervaar. Dan sal die verandering deur die gemeenskap normatief gemaak word en die vroeë meerderheid die verandering aanpak. Deur aansporings te neem, sal die aanvaarding van die verandering dit vir die laat meerderheid beloon, en uiteindelik deur die inisiëring van beleid, sal die agterstande nodig wees om verandering in te stel. Brian het 'n interessante voorbeeld gegee van 'n inisiatief wat deur die wetenskaplike tydskrif Psychological Science geïmplementeer is, waar 'n visuele kentekenstelsel gebruik is om openheid van data en materiaal aan te dui en of manuskripte vooraf geregistreer is. Sedert die implementering van die stelsel in 2013, het 'n kwart van die tydskrifartikels in 2019 'n vooraf-geregistreerde kenteken, die helfte het 'n kenteken vir oop materiaal en 60% het 'n kenteken vir oop data. 'n Ander inisiatief is Registered reports, waar eweknie-evaluering gedoen word by die ontwerp en konsepsie van die manuskrip voordat data versamel en ontleed word. Dit sal lei tot akkuraatheid eerder as om skoon te wees van verslaggewing, die publikasie van negatiewe resultate, replikasies uit te voer en dat ewekniebeoordeling op die kwaliteit van die metodes fokus. Alhoewel dit steeds 'n bietjie van 'n nuwe benadering is, het Brian tog bewys dat geregistreerde verslae deur wetenskaplikes werk en aangehaal word. Wat blyk uit Brian se bespreking, is dat daar 'n bewys is van kulturele verandering namate meer navorsers aanlynmateriaal en -gegewens plaas, terwyl hulle ook ontledings of hipoteses vooraf registreer. Hy het egter benadruk dat dit belangrik is om voetsoolvlak veldtogte te hou. Sigbaarheid is noodsaaklik vir die verandering van norme, en ons moet die oop wetenskap as goeie praktyk en nie as sosiale identiteit uitlig nie. Twee punte wat Brian gemaak het, is hoe ek dink ons ​​moet wetenskap doen - inkrementalisme omhels en intellektuele nederigheid kweek.

“Ons moet dit regkry erder as reg wees” – Brian Nosek, 2020.

Vertaal deur Jacques Deere

[Translated by Jacques Deere]

*Let daarop dat daar ten tyde van die skryf hiervan geen bewyse bestaan ​​van die voorneme om inligting te vervals of te mislei deur die betrokke party nie, en die betrokke ondersoeke is nog steeds aan die gang. Sedert die oorspronklike bewustheid van die probleme rondom die gegewens gemaak is, word die benadering wat deur wtenskaplike tydskrifte en medewerkers gebruik is, hier en hier duidelik gemaak.