(2019-03-30) Ink And Switch End User Programming

Szymon Kaliski, Adam Wiggins, James Lindenbaum of Ink and Switch: on End-user Programming. In their computing lives, power users often want simple extensibility.

Desktop programs, web apps, and smartphone apps are to various degrees like appliances with a hermetic seal to keep users out of their interior.

Some tools have a history of extensibility, such as vim and emacs

But most apps offer no such capabilities — and if they do, it’s usually by exposing programmatic API made for developers.

At Ink & Switch, we believe that software should be extensible in an easy, everyday manne

History offers us some great examples.

Unix pipes (pipeline) allows the user to chain together simple commands to make an ad-hoc program

The UNIX command line is the quintessential end-user programming environment

Perhaps equally as pervasive as Unix is the spreadsheet.

A beloved but less commercially-successful example than either of the first two examples is Hypercard.

A final success story is the world-wide web: specifically, HTML and some companion technologies like CSS, PHP, and Javascript. Combined with low-cost FTP shared hosts, Wordpress, Geocities, and Squarespace, this method of software creation has proven incredibly accessible.

Clay Shirky uses the term “situated software” to describe apps written for one person or just a small group of people.

End-user programming is done by a motivated power user who may or may not be a professional programmer, modifying or creating small ad-hoc tools for their own use

There shouldn’t be a chasm that the user has to cross in order to customize the behavior of their software. Going from using to inspecting to modifying the system should be a gradual process where each of the steps is small enough to be easily discoverable

We suggest three qualities of tools that support end-user programming: embodiment, living systems, and in-place toolchains.

We need ways to require less abstract reasoning and less of the user’s mental bandwidth to be given to the mechanics of modeling their program

One way to do this is embodiment. This is where a tool makes the elements of a working program concrete, usually via visual representations onscreen, to the user.

Embodying the program syntax and flow can be found in “visual programming” tools such as Scratch,

or realtime audio/visual environments like MaxMSP or OpenMusic

One can also use a traditional code-based programming language such as Javascript or Swift while providing embodiment of the output as a micro-visualization.

A second quality that we propose for end-user programming is interaction with a living system.

shortening the save/build/run loop, ideally until it collapses to nothing. (REPL)

Spreadsheets show their pioneering approach

This liveness is visible in another end-user programming success story: SQL, a language for interacting with databases

Taking this idea further, there is a whole academic community around “live coding.”

Ink & Switch explored the topic of live coding for scientific notebooks with the Livebook project

Adding live coding to programming tools is a huge technical and design challenge

But the “living systems” quality of end-user programming is broader than the fast feedback loops of live coding. It also includes the ability for the system to change itself from within

A small example here is is the developer tools console available in most desktop web browsers. Here the user can grab any element on the page and change it: its color, its size, whether it appears at all.

The lineage of Smalltalk, Squeak, and Pharo takes this idea to its furthest: if a system is fully written in itself (sometimes called “self-hosted”), the user can change absolutely anything about the system

A third quality that we think is necessary for end-user programming is an in-place toolchain. The user should be able to edit their programs without installing additional tools or programs.

The Google Docs script editor is available to any user from the Tools menu inside all Gsuite apps. Shown here is a script to export a Google Doc to Markdown format.

“A humane-interface principle is that the system itself should be built out of the same kind of pieces with which you are familiar from your everyday use of the system.” — The Humane Interface, chapter 5

Zapier and IFTTT offer tantalizing glimpses of end-user accessible automation for the web and cloud APIs. But by the full measure of the in-place toolchain idea, these fail

A similar problem of “tool in its own standalone universe” exists in academic examples of end-user programming. Eve, Sketch-n-Sketch, and Pharo are technical design marvels. But they require the user to enter an entire new world

With the context of those three qualities and what has come before, our research lab set out to do some end-user programming experiments. Our testbed was a tablet thinking tool called Capstone.

Capstone is a prototype by Ink & Switch which offers mixed-media cards (card deck) and freeform inking (sketching) on a shared canvas. It’s intended as a place for creative professionals to develop their ideas through research collection, sketching, and mood boards

what programming or automation capabilities can we offer that would give them more power, flexibility, and customization within that context?

As a simple starting place, we built an editor inside the Capstone system that allows the user to edit the underlying code of a given card. Think of this as similar to the web browser’s “View Source” option, but with write capability

However, we found the actual use of this approach uninspiring. It felt obvious, not a bold new direction that hasn’t been tried before. From the user’s standpoint, it felt editable but not inviting to the user. Would it be better to start from a user need/value gap?

*For our next experiment, we wanted to embrace the cards+canvas model and touch interface of Capstone, while also borrowing some ideas from Unix pipes.

What would this look like for a more visual environment like that of Capstone’s cards-on-canvas?*

Our approach was to allow each card within Capstone to take input and provide output, allowing the user to chain them together

In the examples below, we’ve borrowed a use case from this Emacs literate programming example

Many attempts at more accessible programming languages are weakly typed, under the hypothesis that types are unforgiving to newcomers. Our team’s instinct is the opposite: strong typing, with the right interface, can be friendlier for newcomers by making program components “snap” together like building blocks.

Unix pipelines continue to be the reigning champion for composability — something not yet replicated in GUI environments. We feel that this CSV pipeline experiment produced positive findings supporting the value of strong typing (here as uses/exposes), and showed a potential interface for in-place toolchains that don’t break out of the touchscreen interface.

On the other hand, visual embodiment of the data pipeline created some problems. It adds visual clutter

Furthermore we found a tradeoff between grouping cards together in a way that makes the program flow clear (e.g. pipeline goes top-to-bottom or left-to-right) versus grouping cards in a way that reflects how the user wants to think about their content more generally.

For our third experiment, we decided to relax the constraint of an in-place toolchain in exchange for better results on other dimensions. In particular, we wanted to see if the REPL (read-evaluate-print loop) used in many programming systems would be of value.

REPLs are traditionally built on wire protocols. That is, the user’s console sends commands to the runtime system over the network. But Capstone uses a synchronizing data model which keeps all of the visible elements stored in a live document. This allowed us to build the REPL by writing directly into the document from a session on another device.

The data layer for Capstone is based on a peer-to-peer CRDT system called Hypermerge.

We absolutely loved the resulting feel of this experiment.

we also quickly ran into what would likely be the biggest chunk of work in making a system like this real: API design.

Our finding here is that for a living system to work, the internal and external APIs need to be mostly the same.

We also noted that living systems produce a tension between hackabilty and the danger of user breakage. For example, the user can change a card’s background color just as easily as executing a command that would discard every card onscreen or even put the system into a crashed state or infinite loop. What to allow, how to surface errors, and how to recover are deep and challenging questions we did not explore in the course of this experiment. (undo)

Hooks bring a significant downside: the computation is no longer visible

Our next move was to extend the REPL with hooks for system events, such as the user dragging a card around the canvas... With a hook for card dragging

For our final experiment we wanted to explore how the user could create long-running programs (or daemons) inside of Capstone that solved the embodiment problem of hooks.

Typically when we say embodiment we mean a visual element onscreen. But we took inspiration from the world of chat bots: what if computation was embodied as something with a bit of personality

Although creating bot code does not satisfy the in-place toolchain goal, embodying them as cards has its own set of in-place toolchain benefits. In the same way a Unix user can manipulate a script file like any other file, a Capstone user can manipulate a bot like any other file.

We noticed user behavior like creating a board full of their favorite bots and mirroring those cards onto the shelf to take them to the board where they wanted it.

Our positive findings from these experiments include the strength of combining strong typing (uses/exposes) with visual program flows; the magic feeling of interacting with a living system; and embodiment of long-running computation as bots, visually represented as cards with properties the user already knows.

Our negative findings included that our best experiment (REPL) still required an external/separate toolchain; that visual arrangement of computation cards can get messy quickly; and that API design and documentation will be a huge challenge for a powerful end-user system.

We chose Google Chrome OS as our platform for Capstone over more mature options like the iPad or Surface largely because web technologies offer vastly more possibilities for end-user programming.

Some examples of apps that make great use of the web’s native extensibility:
the Atom programming editor
the Hyper command-line terminal
online code sandboxes like CodePen, JSFiddle, and CodeSandbox.

Overall, the web is the only full-featured platform ever created that allows instant download and execution of a program written by a stranger (by visiting a website URL). But the perfect sandboxing of the Javascript runtime means that this action is almost completely safe. This is a truly stunning technical achievement and means that the web is a promising place for end-user programming capabilities.