Pascal Hitzler Department of Computer Science and Engineering Wright State University Dayton, OH, USA pascal.hitzler@wright.edu Krzysztof Janowicz Geography Department University of California, Santa Barbara Santa Barbara, CA, USA janowicz@ucsb.edu ABSTRACT The Semantic Web journal was established in 2010 and in the meantime became one of the primary journals in its field. Besides its focus on top quality research contributions, it is also distinguished by an alternative review process which emphasizes openness and transparency. 1 SCOPE AND RECOGNITION OF THE SEMANTIC WEB JOURNAL The Semantic Web [1,8] is a highly multidisciplinary field of research arranged around the broad vision of sharing, discovering, reusing, and integrating data and services on the Web by making them accessible and understandable to humans and machines alike. It brings together a broad variety of researchers focusing on theoretical aspects, core methods and tools, or on applications in a multitude of disciplines including the life sciences, the earth sciences, industrial information integration, assisted living, data and information management, media applications, and so forth. At its core is the shared need by many application domains to integrate massive amounts of data. Initiated in the 1990s by the inventor of the World Wide Web, Sir Berners-Lee, Semantic Web research has established, and is further developing, technologies and methods to address information integration at large scale. Central to the shared approach is the formalization of meaning of information using so-called ontologies, and the adherence to shared knowledge representation standards developed by the World Wide Web Consortium (W3C), e.g., the Web Ontology Language OWL [7]. The Semantic Web journal [2] publishes research contributions from the whole range of Semantic-Web-related research and applications. In particular, it is not only concerned with core technical contributions to the research area, but also with application-oriented contributions to other disciplines using Semantic Web technologies, with reports on high-impact tools and ontologies, and with theory-oriented contributions of primarily foundational interest. Since its inception, the journal has seen a significant rise in terms of volume of papers published (see Table 1). The journal’s main landing website has accumulated over 2,200,000 hits, and the 15 most viewed paper pages have over 10,000 hits each (http://www.semantic-web-journal.net/reviewedviews). The papers in the journal (http://www.semantic-web-journal.net/blog/swj-5-years-most-cited-papers) have Google Scholar citation counts similar to papers in top journals in comparable fields in Computer Science. Indeed, SCImago has recently ranked the journal 18th in Computer Science world-wide, 2nd within Computer Networks and Communications, 3rd in Information Systems, and 6th in Computer Science Applications (http://www.semantic-web-journal.net/blog/semantic-web-journal-ranking-scimago).  2 TRANSPARENT AND OPEN REVIEWING The Semantic Web journals’ alternative review process has two key aspects. On the one hand, reviewers sign their reviews, i.e., they are known to the authors and to the public; however we do provide an option for the reviewers so that they can explicitly choose to remain anonymous if desired. On the other hand, reviews are posted openly online, as are submitted papers, and any researcher can additionally contribute volunteered reviews to any submitted paper, in addition to the solicited reviews which are requested by the editors. In more detail, the review process can be described as follows. (1) After submission of a manuscript, the editors check whether the manuscript should be rejected without review. This is done for manuscripts which clearly, i.e., without requiring an in-depth assessment, will not meet the quality criteria (or the scope) of the journal. This applies to less than 7% of all submitted manuscripts. Such papers are rejected outright, and they are not posted in public on the journal website. (2) For papers which enter the review process, a handling editor or guest editor is assigned, and the paper and the editor name are put online. The editor solicits reviewers as usual, however the solicited reviewers remain anonymous at this stage. While the paper is under review, any researcher can provide comments or an additional review via the paper page’s comment function. However, these comments and volunteered reviews are not made public before the solicited reviews have been received. (3) After receipt of a sufficient number of quality reviews (usually three for a new submission and 5.4 on average before a paper gets accepted), the editor(s) together with the editors-in-chief make a decision regarding the paper, i.e. whether to accept, to reject, to require minor revisions, or to require major revisions. The decision letter is then sent to the authors, and at the same time both the solicited and the additional reviews, together with the editorial decision on the paper, are posted publicly (openly) on the paper’s webpage. Volunteered reviews are always non-anonymous, while solicited reviewers can choose to opt for anonymity. (4) After four weeks minimum after notification, authors of rejected papers can request that their papers be de-published. Resubmitted revised manuscripts re-enter the review process, preferably under the auspices of the same editor and by soliciting at least the reviewers from the previous round. (5) The printed manuscripts list the editors and reviewers in the head of the paper. Less than 11% of all papers receive a minor revision decision for the first round of reviews. In total, about 27% of submitted papers will be accepted after multiple revisions. The Semantic Web journal review process has been set up to provide benefits to all researchers involved in it. For authors, these include higher quality reviewing and insight (transparency) into the decision making process. It also means that manuscripts receive visibility, through the journal website, already upon submission – and, in fact, also after acceptance as the final author versions are available for free from the journal website indefinitely. For reviewers, it means that their efforts are publicly acknowledged as their names are listed on the paper webpage and in the final versions of accepted papers. For editors (and editors-in-chief), it also means public acknowledgement of their efforts as they are listed alongside the reviewers, and it also means that they receive higher quality reviews on which to base their decisions. In other words, the review process is not only open (reviewers, reviews, and reviewed papers are publicly available), but also transparent as the entire review process, e.g., assigned editor(s), solicited reviewers, turnaround time, manuscript history, and decision statuses, are visible. 3 OTHER INNOVATIONS The Semantic Web journal also supports innovation in other respects. Foremost to mention are alternative paper types (i.e., other than full research papers) which are published by the journal. Currently, these are surveys, dataset descriptions, software tools and systems, ontology descriptions, and application reports [3,4,5]. For surveys, the Semantic Web journal has a very high quality bar, and consequently some of the surveys published by the journal are among its most visible papers. Following a number of special calls, the journal has received a rather high volume of submissions in the software tools and systems [3], and in the dataset descriptions [5] categories. These types of papers do not describe direct research contributions, but are reports on computational artefacts (software, datasets) which are nevertheless of central importance for research advances. E.g., our currently most cited paper (as of April 2015), [9], describes a very widely used piece of software which is an enabler of both Semantic Web research and of transfer of Semantic Web research into applications. Creation and maintenance of such a piece of software requires significant investments, yet it is often difficult for the creators to receive academic credit for their work. By providing an opportunity to report on their work in a journal paper, the authors can be cited for their work in the traditional way, and thus receive corresponding credit. While we believe that established research assessment methods should be modified to directly take into account such work, i.e. without having to write a journal paper about it, corresponding changes are slow coming. In the meantime, the Semantic Web journal approach provides a workable and very popular proxy. Even more, these papers provide a good starting point for researchers to learn about important tools, systems, and datasets as these papers are often written with a broader audience in mind. The Semantic Web journal’s push towards transparency and openness has not gone unnoticed both within and outside the Semantic Web field [11] (https://www.youtube.com/watch?v=iqISnnGTls4). As a particular case in point, we would like to mention that the 19th International Conference on Knowledge Engineering and Knowledge Management, EKAW 2014, motivated by the journal’s successful review process, had adopted an option for authors to have their papers reviewed through the Semantic Web journal process, with minor modifications (http://www.ida.liu.se/conferences/EKAW14/callsfor/callsforpapers.html). The Semantic Web journal is furthermore a platform for the editors-in-chief to pursue research into scientometrics and to make all the journal's data available as Linked Data [10] (see also http://semantic-web-journal.com/SWJPortal/). 4 LESSONS LEARNED Installing our alternative process from the outset did, of course, bear significant risk. The journal was embarking on largely unknown territory, and the past is littered with failed attempts to instill significant changes on journal review processes. So we chose our set-up very carefully, anticipating major objections and adverse effects which may arise, and establishing process details to avoid most of them before they happened. One example of this is the option, for solicited reviewers, to choose anonymity. While this happens only for about a fifth of the solicited reviews, we believe this to be a very important option, because sometimes reviewers feel more comfortable (or at all able) to speak their mind if they are under the protective cover of anonymity – e.g. in cases of junior researchers reviewing a (poor) paper by an established authority in the field. Another example of this is that we screen papers pre-review, and reject them (and do not post them online) in case they are very clearly substandard for the journal. This removes, essentially, the possibility for authors to spam the journal with subpar papers in order to create some visibility for them through the prominent journal website. Of course, in addition to the benefits for researchers involved in the Semantic Web journal review process, there are also potential drawbacks, and we would like to discuss some of them. Authors, for example, run the risk of having their papers rejected, with the effect that their papers, together with the critique given in the reviews, are available to the public. It is for this reason that rejected papers are depublished on request, after a minimum period of four weeks after notification. At the same time, however, this risk on the side of the authors also means that authors are more reluctant to submit half-baked papers, which means that the ratio of submitted papers which are of adequate quality for the journal should be higher than for journals with a closed peer review. On the other hand, authors report that they explicitly select the Semantic Web journal for their newest and most timely work not only because of the visibility but as it provides immediate evidence of their research results through the publication of their manuscript during the review process. Reviewers, of course, have to deal with the fact that their reviews will be made public. This means that they may have to choose more careful wordings, and that it is not recommended to provide superficial reviews, e.g. under time pressure. We have installed the anonymity option for solicited reviewers as partial protection. Regarding wordings of reviews, we seemed to notice that editors sometimes do have to assess the reviews very carefully, as they tend to be written more constructively than we experience from other venues. However, this effect seems easily counterbalanced by the higher quality of the reviews received. Editors and editors-in-chief clearly have additional burdens under the journal’s review process. Most of this simply is because processes (and also how to assess alternative paper types) have to be explained over and over again to reviewers and guest editors. Furthermore, reviews have to be screened and assessed very carefully. Due to the transparency of the review process, occasional troubleshooting is also required, e.g. in the very rare cases where differences in opinion between reviewers, editors, and authors create conflicts: Since the setting is non-anonymous, careful mediation is required in such rare cases (in fact, 3 cases since 2011). On a more technical side, one of the main obstacles we had in establishing our alternative review process was that none of the available review support systems was suitable for our process. Initially, we worked with a public website in conjunction with a traditional, closed review system [11]. This created significant overhead, and so eventually we decided to create our own review system [6] with support by the publisher. In the meantime, the review system is also being transferred to other journals. And indeed, it has to be acknowledged that the alternative process which was established by the Semantic Web journal could not have been set up without significant support by the publishing house, IOS Press. 5 NEXT STEPS The Semantic Web journal has now been firmly established as a top tier journal in its field. As such, it gives witness to the fact that the alternative review process, which was set up from the outset, is suitable for sustaining a high-impact journal. At the same time, evidence for the exact impact of different review processes on review quality and outcomes remains largely anecdotal (There are only a few peer-reviewed publications addressing this topic, and additional documents on the Web which were not peer-reviewed. Generally speaking, there is not enough data at this stage to come to any hard conclusions). It would be very interesting indeed to pursue a quantitative assessment of the difference which the Semantic Web journal review process makes compared to traditional review settings. Yet currently, availability of data (aside of the journal's Linked Data) for comparison as well as lack of funding make this a tricky venture. It remains prominently on our minds, though. At the same time, we intend to remain agile and refine our process. A transfer to other journals or even to conferences, in the wake of EKAW 2014 reported above, would add further dynamics into the review process discussion, and as such would be very helpful in paving the way for improvements. We hope that this report will provide some stimulus for such changes. REFERENCES [1] T. Berners-Lee, J. Hendler, and O. Lassila. The Semantic Web. Scientific American, 284(5):34-43, 2001. [2] P. Hitzler and K. Janowicz. Semantic Web – Interoperability, Usability, Applicability. Semantic Web 1(1):1-2, 2010. [3] P. Hitzler and K. Janowicz. Semantic Web Tools and Systems. Semantic Web 2(1):1-2, 2011. [4] P. Hitzler and K. Janowicz. Semantic Web Surveys and Applications. Semantic Web 2(2):65-66, 2011. [5] P. Hitzler and K. Janowicz. Linked Data, Big Data, and the 4th Paradigm. Semantic Web 4(3):233-235, 2013. [6] P. Hitzler, K. Janowicz, K. Sengupta. The New Manuscript Review System for the Semantic Web Journal. Semantic Web 4(2):117, 2013. [7] P. Hitzler, M. Krötzsch, B. Parsia, P. F. Patel-Schneider, and S. Rudolph. OWL 2 Web Ontology Language: Primer (Second Edition). W3C Recommendation, 11 December 2012. [8] P. Hitzler, M. Krötzsch, and S. Rudolph. Foundations of Semantic Web Technologies. Chapman & Hall/CRC, 2010. [9] M. Horridge, S. Bechhofer, The OWL API: A Java API for OWL Ontologies. Semantic Web 2(1): 11-21, 2011. [10] Y. Hu, K. Janowicz, G. McKenzie, K. Sengupta, and P. Hitzler. A Linked Data-driven Semantically-enabled Journal Portal for Scientometrics. In: H. Alani, L. Kagal, A. Fokoue, P. Groth, C. Biemann, J.X. Parreira, L. Aroyo, N. Noy, C. Welty, and K. Janowicz (eds.), The Semantic Web – ISWC 2013. 12th International Semantic Web Conference, Sydney, NSW, Australia, October 21-25, 2013, Proceedings, Part II. Lecture Notes in Computer Science Vol. 8219, Springer, Heidelberg, 2013, pp. 114-129. [11] K. Janowicz and P. Hitzler. Open and transparent: the review process of the Semantic Web journal. Learned Publishing, 25(1):48-55, 2012.   |
E-Letter > STCSN-E-Letter-Vol-3-No-1 >