(2018-04-07) The Scientific Paper Is Obsolete: Here's What's Next

James Somers: The Scientific Paper Is Obsolete. Here's What's Next. - The Atlantic

The scientific paper—the actual form of it—was one of the enabling inventions of modernity. Before it was developed in the 1600s, results were communicated privately in letters, ephemerally in lectures, or all at once in books. There was no public forum for incremental advances. (Scientific Method, Peer Review)

The more sophisticated science becomes, the harder it is to communicate results. Papers today are longer than ever and full of jargon and symbols. They depend on chains of computer programs that generate data, and clean up data, and plot data, and run statistical models on data. These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves.

Perhaps the paper itself is to blame. Scientific methods evolve now at the speed of software; the skill most in demand among physicists, biologists, chemists, geologists, even anthropologists and research psychologists, is facility with programming languages and “data science” packages. And yet the basic means of communicating scientific results hasn’t changed for 400 years. Papers may be posted online, but they’re still text and pictures on a page.

What would you get if you designed the scientific paper from scratch today? A little while ago I spoke to Bret Victor.

Victor gestured at what might be possible when he redesigned a journal article by Duncan Watts and Steven Strogatz, “Collective dynamics of ‘small-world’ networks.” He chose it both because it’s one of the most highly cited papers in all of science and because it’s a model of clear exposition

Victor’s redesign interleaved the explanatory text with little interactive diagrams that illustrated each step. In his version, you could see the algorithm at work on an example. You could even control it yourself.

Strogatz admired Victor’s design

What I’m studying is something dynamic. So the representation should be dynamic.”

This is, of course, the whole problem of scientific communication in a nutshell: Scientific results today are as often as not found with the help of computers. That’s because the ideas are complex, dynamic, hard to grab ahold of in your mind’s eye. And yet by far the most popular tool we have for communicating these results is the PDF—literally a simulation of a piece of paper. Maybe we can do better.

Stephen Wolfram published his first scientific paper when he was 15. As his research grew more ambitious, he found himself pushing existing software to its limit. He’d have to use half a dozen programming tools in the course of a single project.

At the heart of Mathematica, as the company’s flagship product became known, is a “notebook” where you type commands on one line and see the results on the next.

A Mathematica notebook is less a record of the user’s calculations than a transcript of their conversation with a polymathic oracle. Wolfram calls carefully authored Mathematica notebooks “computational essays.” (computational medium)

The notebook interface was the brainchild of Theodore Gray, who was inspired while working with an old Apple code editor.

The notebook is designed to turn scientific programming into an interactive exercise, where individual commands were tweaked and rerun

What made Mathematica’s notebook especially suited to the task was its ability to generate plots, pictures, and beautiful mathematical formulas

The vision driving that work, reiterated like gospel by Wolfram in his many lectures, blog posts, screencasts, and press releases, is not merely to make a good piece of software, but to create an inflection point in the enterprise of science itself.

In the mid-1600s, Gottfried Leibniz devised a notation for integrals and derivatives (the familiar ∫ and dx/dt) that made difficult ideas in calculus almost mechanical. Leibniz developed the sense that a similar notation applied more broadly could create an “algebra of thought.” Since then, logicians and linguists have lusted after a universal language that would eliminate ambiguity and turn complex problem-solving of all kinds into a kind of calculus.

Wolfram’s career has been an ongoing effort to vacuum up the world’s knowledge into Mathematica, and later, to make it accessible via Wolfram Alpha, the company’s “computational knowledge engine” that powers many of Siri and Alexa's question-answering abilities.

“I’ve noticed an interesting trend,” Wolfram wrote in a blog post. “Pick any field X, from archeology to zoology. There either is now a ‘computational X’ or there soon will be. And it’s widely viewed as the future of the field.” As practitioners in those fields become more literate with computation, Wolfram argues, they’ll vastly expand the range of what’s discoverable.

To write a paper in a Mathematica notebook is to reveal your results and methods at the same time; the published paper and the work that begot it.

Wolfram says that he’s surprised the computational essay hasn’t taken off. He remembers working with Elsevier, the scientific publishing giant, all the way back in the early ’80s.

I spoke to Theodore Gray, who has since left Wolfram Research to become a full-time writer. He said that his work on the notebook was in part motivated by the feeling, well formed already by the early 1990s, “that obviously all scientific communication, all technical papers that involve any sort of data or mathematics or modeling or graphs or plots or anything like that, obviously don’t belong on paper. That was just completely obvious in, let’s say, 1990,” he said.

In early 2001, Fernando Pérez found himself in much the same position Wolfram had 20 years earlier: He was a young graduate student in physics.

Pérez combined the three projects into one and took the reins. From the very beginning, the project, called IPython (the “I” for “Interactive”), was open-source.

He thought that if science was to be an open enterprise, the tools that are used to do it should themselves be open. Commercial software whose source code you were legally prohibited from reading was “antithetical to the idea of science,”

The idea for IPython’s notebook interface came from Mathematica. Pérez admired the way that Mathematica notebooks encouraged an exploratory style. “You would sketch something out—because that’s how you reason about a problem, that’s how you understand a problem.” Computational notebooks, he said, “bring that idea of live narrative out ... You can think through the process, and you’re effectively using the computer, if you will, as a computational partner, and as a thinking partner.” (Cyborg)

...built their notebooks as simple web pages. The interface is missing Mathematica’s Steve Jobsian polish, and its sophistication. But by latching itself to the web, IPython got what is essentially free labor: Any time Google, Apple, or a random programmer open-sourced a new plotting tool, or published better code for rendering math, the improvement would get rolled into IPython. “It has paid off handsomely,” Pérez said.

“I think what they have is acceptance from the scientific community as a tool that is considered to be universal,” Theodore Gray says of Pérez’s group. “And that’s the thing that Mathematica never really so far has achieved.” There are now 1.3 million of these notebooks hosted publicly on Github.

“There’s always chaos,” Gray said about open-source systems. “The number of moving parts is so vast, and several of them are under the control of different groups. There’s no way you could ever pull it together into an integrated system in the same way as you can in a single commercial product with, you know, a single maniac in the middle.” The maniac is, of course, Stephen Wolfram.

Maybe computational notebooks will only take root if they’re backed by a single super-language, or by a company with deep pockets and a vested interest in making them work. But it seems just as likely that the opposite is true. A federated effort, while more chaotic, might also be more robust—and the only way to win the trust of the scientific community.

It’ll be some time before computational notebooks replace PDFs in scientific journals, because that would mean changing the incentive structure of science itself. Until journals require scientists to submit notebooks, and until sharing your work and your data becomes the way to earn prestige, or funding, people will likely just keep doing what they’re doing.

Pérez told me stories of scientists who sacrificed their academic careers to build software, because building software counted for so little in their field: The creator of matplotlib, probably the most widely used tool for generating plots in scientific papers, was a postdoc in neuroscience but had to leave academia for industry.

Still, those who stay are making progress. Pérez himself recently got a faculty appointment in the stats department at Berkeley. The day after we spoke, he was slated to teach an upper-division data-science course, built entirely on Python and Jupyter notebooks. “The freshman version of that course had in the fall I think 1,200 students,” he said. “It’s been the fastest-growing course in the history of UC Berkeley. And it’s all based on these open-source tools.”

Paul Romer on Jupyter, Mathematica, and the Future of the Research Paper –

I had to learn the hard way why so many kept their distance from Mathematica. Now, I’m much more productive with Jupyter.

To their credit, Mathematica did open up a huge technical lead in the 1990s.

This technical engineering dimension is not the only one we should use to compare the proprietary and open models. There is an independent social dimension, where the metrics assess the interactions between people. Does it increase trust? Does it increase the importance that people attach to a reputation for integrity?

It is along this social dimension that open source unambiguously dominates the proprietary model. Moreover, at a time when trust and truth are in retreat, the social dimension is the one that matters.

Membership in an open source community is like membership in the community of science.

In 2015, I tried to share some research results in a Mathematica notebook.

So back in 2015, full of naive optimism, I set out to correct something that was wrong in a published paper.

On technical grounds, the Mathematica notebook was the perfect vehicle.

Wolfram made it hard to share a readable PDF version of a notebook because it wanted someone like me to distribute content in its proprietary file format, the CDF.

I stopped using Mathematica and gave up on notebooks, so it was only recently that I discovered how easy it is to use the Jupyter notebook to as a front end for Python libraries. It offers the best REPL I’ve ever used. It does a better job of delivering what Theodore Gray had in mind when he designed the Mathematica notebook. It lets me get quick feedback, via text or graphics, about what happens when I select a line of code and run it.

Now, Jupyter is the unambiguous technical leader.