(2019-08-05) Kling Andy Matuschak On Books And Learning

Arnold Kling: Andy Matuschak on Books and Learning.

Our topic for today is a fascinating essay of his called "Why books don't work. (2019-06-01) Matuschak Why Books Don't Work

So, perhaps a better title for this essay would have been something like "Why Books Don't Reliably Contain Detailed Information."

I think one of the challenges of your essay, which is a challenge of this whole area, is: What do we really mean by 'learn'? Or, what do we really mean by 'remember'? Or, what do we mean by 'grasp'? But, at the heart of the challenge that you identify with books is something, a framework I had not heard before called transmissionism. Explain what that is.

And you make a parallel which I find quite interesting about lectures.

often a very short period of time after that lecture has passed, or if you've read the book, you don't remember a single thing

We're not expecting to learn how to master a subject. We want to be exposed to the working of a mind, someone's mind that appeals to us or that we think will inspire us. You know, I think about a weekly sermon as an example where, you're not really expected to learn anything. It's supposed to touch your heart, perhaps

they can convey cultural norms, cultural values, or perhaps even the single individual's values really quite effectively.

where I agree with you is that much of it is, I would call it, a pleasant way to pass time. We call that entertainment sometimes: I think that's somewhat demeaning

I go to a lecture to get something closer to wisdom.

Now, one thing that we don't know--we, kind of referring to scientists as a field--is the degree to which that kind of declarative knowledge, the information as you call it, has to be deeply internalized before a person is susceptible to the wisdom. Uh, mmnnn, lots of differing views on that: it's kind of difficult to study empirically. But it does seem that to some degree, you need to be familiar with the core components of whatever topic of wisdom is attempted and to be conveyed.

I think the challenge that we face as our time gets more valuable--and I know you are aware of this, so I'd like your opinion: The temptation to reduce everything to a multiple choice exam tends to force us toward spit-back information and move us away from wisdom, is, I think, part of the challenge. It's certainly the elementary-to-high-school educational problem.

Wittgenstein has this wonderful quote, 'a wheel that turns, although nothing turns along with it, is not part of the mechanism.' [Philosophical Investigations, Ludwig Wittgenstein--Econlib Ed.] And so, if you've simply memorized a whole bunch of piece of atomic information and yet they don't connect to each other, you are not able to combine them into anything.

Learning science gives us a couple of structures for thinking about this that I find helpful. One of them is Bloom's taxonomy for thinking about tasks. And, it begins with being able to recall things, reasonably; and it moves up through being able to, say, apply things, take some procedural knowledge, perform a procedure. Perhaps initially a very rote procedure and then perhaps one with some flexibility later. Finally it moves through being able to evaluate and make critical arguments. And ultimately to synthesize or create something anew

My favorite course evaluation I ever received was, I got a 1 out of 5 from a student who said, 'Professor Roberts is a terrible professor. He expects us to apply the material to things we've never seen before.' Of course, I warn people on Day 1, that's the essence of the class. And I think that is great teaching, when you can pull it off. It's not easily done. Not every student can do it. And I'm ashamed at how the techniques that I used when I was in the classroom to convey that process--my standard technique was: 'Here are some really hard problems. Think about them.' And I was not able to give the feedback that they needed often to help them see why their incorrect answers were incorrect. And that's extremely important, obviously is, to make progress. And we'll talk a little bit about that general challenge of feedback, because I think it's a fascinating one. And it's increasingly challenging as people's time gets more valuable.

People write me sometimes, 'What econ textbooks should I read?' And the answer is, 'None of them.' They are not designed to be read. They are designed to complement an often extremely dry, multiple-choice-driven class. And they are explicitly not designed to convey information.

When I was a Khan Academy, my R&D group worked on this lengthy project to try to, instead of having multiple choice questions for everything, actually have open-ended activities for things. We were very focused on AP Advanced Placement Humanities.

the challenge is, as you point out, isn't teaching people what the right answer is. It's teaching them how to think and weigh evidence related to answers that are inherently not well-defined as correct or false.

if that class is a seminar run by a competent facilitator, then you might get something pretty wonderful, where, the contents of that class, day to day, are discussions. And debates.

And, indeed, the students will surprise each other. And so, it kind of got me--we kind of tried to create that kind of thing online.

first I want to kind of describe what it's not. So, other people have tried to do open-ended learning activities online before. And often what that looks like is a peer-to-peer interaction. But, it's grading

we think that developing understanding looks a little bit more like that classroom, where there's an exchange of ideas and exchange of values. Now, we don't want, kind of the blind leading the blind type thing happening. So, we don't--actually many teachers do--just create a discussion forum--and say, 'Students please discuss between classes.' And sometimes that works. The evidence suggests that mostly it doesn't.

So, the solution that we build kind of offers structured scaffolds where students are presented with each others' arguments and then they are given almost like the little hand of cards, for, like, things to try. 'Oooh. I'm going to not disagree with you or point out a limitation of your argument, but actually one of the things I'm allowed to, card I'm allowed to play on your argument, is: I'm going to add a piece of evidence.' Or, 'I'm going to suggest a different implication of the argument that you are making that actually makes it even more powerful.'

So, we finished that project mid last year. And, it's kind of on the queue. But, Kahn Academy is a pretty small nonprofit. At the moment my understanding is that they are focusing mostly on middle-school math, where this particular set of skills is not quite the priority.

I was teaching economists how to convey economic ideas in writing.

had the opportunity to spend an hour with Orson Scott Card.

his suggestion was to have the students give each other feedback. Not literally grade, but critique the writing of the other students

It's a win-win, because, I think being a great writer requires being a great editor. And, it's very hard to learn how to edit one's own work. And, by editing the work of others, you learn how to edit your own work

you get instead is a teacher who writes in the margin in red pencil, 'Awkward'

I think the idea of learning how to fix an awkward sentence--and of course, in an even better class would be, would be, just write 'Awkward,' you write your better version of the sentence that would improve it. Because then it teaches you how to fix your own sentences

how little of that is in our current education system, how little of it is in our books, lectures. It's all--it's transmissionism is all the way down.

It seems to me that interaction and grappling with the material is the only way we really learn.

there's a growing recognition that even when one is learning math, this kind of interaction is essential. People are not going to really understand what, say, a negative number is simply by learning how to perform a set of procedures related to it. They need to get a feel for it, the sort of term of art is sense-making. And that involves experimentation and play.

this idea of grappling with problem-solving--and by problem-solving, it isn't just getting the right answer. In math it often is, and I think that's extremely important. But in economics it might be just to understand the factors that might be the ones you want to consider. And there's no "right answer" as to why something is happening in the world right now. That, to me, is the economic way of thinking

scientists actually kind of know a bunch of what's necessary to form memories reliably. They're just not necessarily what we'll naturally do, day to day. It's a little hard to pull off. Memories are formed more reliably, say, if you are exposed to things multiple times; if they anchored to powerful emotional experiences; if you have a variety of encodings for that memory--for instance, you know, both visual and auditory and perhaps relating to places. But, um, you know, the repetitions thing is a particularly easy and mechanistic approach

spacing effect. (SRS)

modern systems are also being used. It's a little niche right now. Often by, for instance, med students trying to prepare for the MCAT [Medical College Admission Test] and still in particular by language learners. But, my colleague Michael Nielson and I have been working on trying to integrate these systems more deeply into kind of any informational nonfiction texts you could possibly imagine.

one of the things that's sort of shocking about the spacing effect and the way that it's instantiated, is that normally, when we do things--say, professionally as knowledge workers or as thinkers--we get diminishing returns

But the weird thing of the spacing effect is it's the opposite of that. It's actually exponential.

we made this book on quantum computation. And we've chosen that topic in part because that's a topic that has a lot of notation, a lot of details

we kind of follow up; and it's an interactive reading experience where as you are reading, you get these little prompts--like, 'Are you remembering this?' If you are not, we are going to kind of come back to you a little sooner

So, the first chapter takes about 4 hours to read. And, it contains 112 details that we track your memory of

that isn't a cost which is equivalent to re-reading the book. In fact, it's very quick. It's 5 or 6 seconds per question, so there's about 112 of them. And so, it costs, ehh, 10-15 minutes for readers to do each of those repetitions

When you say 112 things--that's jargon, I assume. It's--may be some connections of things. But it's not the deeper understanding. Am I right?

some of the questions were--some of them were involved. They would ask things like: 'Why is it difficult to store a qubit in this particular kind of matter?' for instance. And so you kind of have to come up with a justification. Now, after you practice that a few times, it'll become a spit-back thing

some of those 112 details are somewhat less spitback. But I think the really important thing is what those 112 details let you think next.

We are experimenting with this thing we call the mnemonic medium, which is different from a bunch of flashcards; and it's also different from a book insofar as it's structured information embedded in a narrative. Because narrative can have ethos. Narrative can have pathos. And it can do those jobs that those wonderful books you described did. And, perhaps, also, do the job of leaving you with some detailed knowledge of what you read.

Let's talk about meaning. Why are we reading any of this stuff, anyway? What's the point of learning all this stuff, anyway? For me, it's--it's about the moments that spark joy. The moments that fill the delight behind my eyes, where I feel connected to the eternal

So, let's talk about how to make things better

in the essay I talk about ways in which it's possible to intentionally design media so that their grains bend in a particular way

you are conducting a class that is not so much informational but which you want to use to help students think in a particular way, you are shaping the grain of the discussion in that class by asking questions in a certain way

One example that I'll share that I'm just totally in love with: When I talk about seminar-oriented classes, people often say that, 'That's great for college,' or, 'That's really great for the Humanities; it's not going to work for Arithmetic.' And, there's this wonderful professor, Constance Kamii, who wrote a book called Young Children Reinvent Arithmetic, wherein she documents her experiments of having students do exactly that. She had a class of 5-year-olds in First Grade where she simply gave them a large library of games they could play, which she invented, such that the grain of the games kind of involved the students' learning; and in fact inventing for themselves arithmetic.

I think there is a challenging mix between engagement and content acquisition

games are great. Kids love games. If you can get them involved in games, they are going to get more engaged. The more time you spend on the gaming part, the less time they spend, maybe, learning. But maybe what they do learn is acquired more deeply

Kahn Academy, I think, in the early days made a conscious decision to have what I would call low production values--not fancy, very straightforward, not particularly entertaining. The main entertainment opportunity was to increase the speed for some kids who were kind of already getting it, maybe, or who could acquire it a little more quickly

Is it measurable in any sense? And, the extraordinary part is that feedback. I think that students don't just watch it and hear someone tell them how to do it. They are actually forced to interact with the material.

This idea of instant feedback--it goes way, way, way back. And, it's very valuable for this kind of declarative knowledge and rote procedural knowledge. And, it's been hugely helpful. There actually are great efficacy studies on Kahn Academy, demonstrating pretty interesting benefits. It's not a savior; it's not a silver bullet, at all. It's done some good. It's great. I think that model is very valuable. I think there's a lot of work to do to scale it to things like we discussed earlier, like, you know, 'What was the primary cause of World War I?' Getting there, rather than how to multiply two-digit numbers, is going to be an extra thing.

One other challenge for that model that we engaged a lot with over the years and haven't yet cracked is building the conceptual understanding. Like, you might understand how to multiply those two-digit numbers. But, do you really know what's going on? I'll share this example that really struck fear into my heart from research a couple of years ago. We were doing some kind of explanation-oriented stuff, where we wanted kids to generate explanations for how stuff worked. And, not just, you know, use a procedure. And so, we chose a bunch of kids who had gotten every single question right about a particular topic. Like, as far as the system was concerned, they knew everything there was to know about this topic. And then we asked the simple follow-up question like, 'Can you explain, you know, why this formula gives you the area of this kite?' A kite is like a geometric shape. And, we got this, like, horrifying set of answers: 'IDK': 'I Don't Know.' 'Math is weird.' 'I dunno[?]' 'Blank, blank, mmm, blank.' Zero students--it was several dozen students who had answered every question correctly--could explain that.

maybe the greatest tragedy of all to me, which is that: Why learn about math? Let's put kind of political disagreements about forced education aside. Math is sooo beautiful. Maybe I'm not going to force you to learn it; but, oh, my gosh--understanding something really deeply--I'm not talking about arithmetic, but, for instance, why does e show up in so many places? It's so beautiful. And that's because you are connecting to something fundamental and true. These are Platonic objects. Okay--that is a controversial idea. And none of that, none of that is what is accessed in traditional K-12 education. You are learning it because you have to, or because I told you so, or perhaps because it will help you get a job; or perhaps, in a very enlightened environment, because it will help you think analytically and critically and abstractly, and those are skills which are useful in a variety of domains.

Just for fun today I asked--the Adam Cifu episode came out today on Medical Conservatism, and I asked people what's one thing you learned from this episode, drawing on A. J. Jacobs' idea that you should have a 'One Thing' document where you list one thing that you might want to remember. And, you know, this is a great idea. You should do this for every episode

There's kind of a classic textbook on instructional design called Understanding by Design. And it introduces this concept that I found very helpful called Essential Questions. And, it argues that we should try to structure our curricula and structure our courses to really focus on accessing essential questions. What those are, are the questions which are central to the discipline. They are the questions which are alive. They are the questions which are not resolved. They are the questions which you, the novice, and I, the expert, we are all engaging in.

So, one 'One Thing' that might be fun for your listeners is if you or your guests introduce an essential question from their field or from their work, which then the listeners can discuss amongst themselves or think of individually.

challenge, though, and this is where I'm going to ask you to try to come up with this, is: What's the forum? So, we have a feature on the website called Extras, where we have--we raise questions related to the episode. Most people don't go to them. Almost by definition.

First, one thing we learned from designing the sort of open-ended activity tool at Kahn Academy was that, not all discussions are equally productive or constructive.

so, places like Reddit and HackerNews have kind of figured out mechanism which allow the most fruitful or generative discussions to rise to the top

What is the relationship between one's emotional connection to knowledge and one's detailed understanding of that knowledge? That's a question which a learning scientist can engage with extremely deeply. But it's also a question which everyone can engage with.

Just a couple other models that come to mind. So, one of them is moderated-ish. And, it's used by a lot of bloggers. So, many bloggers who write series of essays on their blog will write something; it will include some provocations. The blog comment section will have a lot of discussions going on. The person who wrote the thing might engage and facilitate those discussions slightly. But the more important thing that they'll do is kind of pull some of those comments, the important ones up into the body itself. Or, to address them in the next essay.

It's hard to, you know, get each episode out just already. But, to add to that might be too demanding. But, the idea of using postmortems as a way to enhance--certainly my understanding--is a good thing.

I would give a one-page feedback form every lecture. Every lecture. And, in that one page was the following: 'How was this lecture on a scale of 1 to 5?'--I think it probably asked, 'How much did you get out of this lecture on a scale of 1 to 5?' 'What was the most interesting thing or most valuable thing you learned?' And, 'What was the most confusing?' And what was fascinating about that is that I didn't anticipate what those were

using it as a way for followup and enhancing learning I think would be fantastic.

That way there are stakes. You'll know that, 'Perhaps Andy or Russ is going to reply to this. It will mean something.'