(2015-08-26) Hotchkiss Annotations As Peer Review

ScottHotchkiss wonders whether Annotation Systems could function as Peer Review.

I think the isolated scientific paper is a product of the 20th century, being imposed on the 21st purely because of inertia. A better solution would be to give a "living paper" to each general research project an individual researcher has. This living paper can then be updated as results change/improve. In such a system I would probably have ~5 living papers so far in my career, instead of ~20 old-style papers. Or, even better, would be a large Wiki edited, annotated, moderated and discussed by the science community as knowledge is gained. (cf WikiEbook)

Even if you to wish to keep "the paper" as how science is presented, I think that the journal system, while invaluable in the 20th century, also exists in the 21st century only due to inertia... Today, if a paper in your field is good, people talk about it. This gets discussed in emails amongst collaborators, which then disperses into departmental journal clubs and the information about the quality of the paper is disseminated like that. It's worth emphasising that, at least in high energy physics and cosmology, this often happens long before the paper is technically "published" via the slow, conventional Peer Review.

One way to achieve this would be to add comments/annotations to the arXiv. For various reasons, the people at arXiv are reluctant to do this... But this doesn't mean that the service isn't desired. And, now, in fact, something very near to this does exist. It is called HypothesIs and it is a web annotation tool... For such a system to work would require a critical mass of people using it before it becomes effective. At present, an annotation is either visible to just the author, or to anyone. If annotations could be restricted to sub-groups then people will be more inclined to write annotations. Then, the particular annotations that the group (e.g. a research group at a university) finds most useful can be made more publicly available, if desired. Also, the ability to be notified (e.g. via email) whenever annotations are written on specific webpages, or websites would be needed. At present I can only see the option to be notified when someone replies to my own annotation. This means that if an annotation is written on a paper at the arXiv, then nobody else knows until someone specifically chooses to look at that paper, meaning most annotations will lie unread for a long time; a time unbounded from above.*