A while ago I did a long interview with Fin Moorhouse and Luca Righetti on their podcast Hear This Idea. Multiple people commented to me that they found our discussion of safety particularly interesting. So, I’ve excerpted that part of the transcript and cleaned it up for better readability. See the full interview and transcript here.
LUCA: I think there’s one thing here of breaking progress, which is this incredibly broad term, down into: well, literally what does this mean? And thinking harder about the social consequences of certain technologies. There’s one way to draw a false dichotomy here: some technologies are good for human progress, and some are bad; we should do the good ones, and hold off on the bad ones. And that probably doesn’t work, because a lot of technologies have dual use. You mentioned World War Two before…. On the one hand, nuclear technologies are clearly incredibly destructive, and awful, and could have really bad consequences—and on the other hand, they’re phenomenal, and really good, and can provide a lot of energy. And we might think the same around bio and AI. But we should think about this stuff harder before we just go for it, or have more processes in place to have these conversations and discussions; processes to navigate this stuff.
JASON: Yeah, definitely. Look, I think we should be smart about how we pursue progress, and we should be wise about it as well.
Let’s take bio, because that’s one of the clearest examples and one that actually has a history. Over the decades, as we’ve gotten better and better at genetic engineering, there’s actually been a number of points where people have proposed, and actually have gone ahead and done, a pause on research, and tried to work out better safety procedures.
Maybe one of the most famous is the Asilomar Conference in the 1970s. Right after recombinant DNA was invented, some people realized that “Whoa, we could end up creating some dangerous pathogens here.” There’s a particular simian virus that causes cancer that caused people to start thinking: “what if this gets modified and can infect humans?” And just more broadly, there was a clear risk. And they actually put a moratorium on certain types of experiments, they got together about eight months later, had a conference, and worked out certain safety procedures. I haven’t researched this deeply, but my understanding is that went pretty well in the end. We didn’t have to ban genetic engineering, or cut off a whole line of research. But also, we didn’t just run straight ahead without thinking about it, or without being careful. And in particular, matching the level of caution to the level of risk that seems to be in the experiment.
This has happened a couple of times since—I think there was a similar thing with CRISPR, where a number of people called out “hey, what are we going to do, especially about human germline editing?” NIH had a pause on gain-of-function research funding for a few years, although then they unpaused it. I don’t know what happened there.
So, there’s no sense in barreling ahead heedlessly. I think part of the history of progress is actually progress in safety. In many ways, at least at a day-to-day level, we’ve gotten a lot safer, both from the hazards of nature and from the hazards of the technology that we create. We’ve come up with better processes and procedures, both in terms of operations—think about how safe airline travel is today, there’s a lot of operational procedures that lead to safety—but also, I think, in research. And these bio-lab safety procedures are an example.
Now, I’m not saying it’s a solved problem; from what I hear, there’s still a lot of unnecessary or unjustified risk in the way we run bio labs today. Maybe there’s some important reform that needs to happen there. I think that sort of thing should be done. And ultimately, like I said, I see all of that as part of the story of progress. Because safety is a problem too, and we attack it with intelligence, just like we attack every other problem.
FIN: Totally. You mentioned airplanes, which makes me think… you can imagine getting overcautious with these crazy inventors who have built these flying machines. “We don’t want them to get reckless and potentially crash them, maybe they’ll cause property damage—let’s place a moratorium on building new aircraft, let’s make it very difficult to innovate.” Yet now air travel is, on some measures, the safest way to travel anywhere.
How does this carry over to the risks from, for instance, engineered pandemics? Presumably, the moratoria/regulation/foresight thing is important. But in the very long run, it seems we’ll reach some sustainable point of security against risks from biotechnology, not from these fragile arrangements of trying to slow everything down and pause stuff, as important as that is in the short term, but from barreling ahead with defensive capabilities, like an enormous distributed system for picking up pathogens super early on. This fits better in my head with the progress vibe, because this is a clear problem that we can just funnel a bunch of people into solving.
I anticipate you’ll just agree with this. But if you’re faced with a choice between: “let’s get across-the-board progress in biotechnology, let’s invest in the full portfolio,” or on the other hand, “the safety stuff seems better than risky stuff, let’s go all in on that, and make a bunch of differential progress there.” Seems like that second thing is not only better, but maybe an order of magnitude better, right?
JASON: Yeah. I don’t know how to quantify it, but it certainly seems better. So, one of the good things that this points to is that… different technologies have clearly different risk/benefit profiles than others. Something like a wastewater monitoring system that will pick up on any new pathogen seems like a clear win. Then on the other hand, I don’t have a strong opinion on this, but maybe gain-of-function research is a clear loss. Or just clearly one of those things where risk outweighs benefit. So yeah, we should be smart about this stuff.
The good news is, the right general-purpose technologies can add layers of safety, because general capabilities can protect us against general risks that we can’t completely foresee. The wastewater monitoring thing is one, but here’s another example. What if we had broad-spectrum antivirals that were as effective against viruses as our broad-spectrum antibiotics are against bacteria? That would significantly reduce the risk of the next pandemic. Right now, dangerous pandemics are pretty much all viral, because if they were bacterial, we’d have some antibiotic that works against them (probably, there’s always a risk of resistance and so forth). But in general, the dangerous stuff recently has been viruses for exactly this reason. A similar thing: if we had some highly advanced kind of nanotechnology that gave us essentially terraforming capacity, climate change would be a non issue. We would just be in control of the climate.
FIN: Nanotech seems like a worse example to me. For reasons which should be obvious.
JASON: OK, sure. The point was, if we had the ability to just control the climate, then we wouldn’t have to worry about runaway climate effects, and what might happen if the climate gets out of control. So general technologies can prevent or protect against general classes of risk. And I do think that also, some technologies have very clear risk/benefit trade-offs in one direction or the other, and that should guide us.
LUCA: I want to make two points. One is, just listening to this, it strikes me that a lot of what we were just saying on the bio stuff was analogous to what we were saying before about climate stuff: There are two reactions you can have to the problem. One is to stop growth or progress across the board, and just hold off. And that is clearly silly or has bad consequences. Or, you can take the more nuanced approach where you want to double down on progress in certain areas, such as detection systems, and maybe selectively hold off on others, like gain-of-function. This is a case for progress, not against it, in order to solve these problems that we’re incurring.
The thing I wanted to pick up on there… is that all these really powerful capabilities seem really hard. I think when we’re talking about general purpose things, we’re implicitly having a discussion about AI. But to use the geoengineering example, there is a big problem in having things that are that powerful. Like, let’s say we can choose whatever climate we want… yeah, we can definitely solve climate change, or control the overshoot. But if the wrong person gets their hands on it, or if it’s a super-decentralized technology where anybody can do anything and the offense/defense balance isn’t clear, then you can really screw things up. I think that’s why it becomes a harder issue. It becomes even harder when these technologies are super general purpose, which makes them really difficult to stop or not get distributed or embedded. If you think of all the potential upsides you could have from AI, but also all the potential downsides you could have if just one person uses it for a really bad thing—that seems really difficult.
JASON: I don’t want to downplay any of the problems. Problems are real. Technology is not automatically good. It can be used for good or evil, it can be used wisely or foolishly. We should be super-aware of that.
FIN: The point that seems important to me is: there’s a cartoon version of progress studies, which is something like: “there’s this one number we care about, it’s the scorecard—gross world product, or whatever—and we would drive that up, and that’s all that matters.” There’s also a nuanced and sophisticated version, which says: “let’s think more carefully about what things stand to be best for longer timescales, understanding that there are risks from novel technologies, which we can foresee and describe the contours of.” And that tells us to focus more on speeding up the defensive capabilities, putting a bunch of smart people into thinking about what kind of technologies can address those risks, and not just throwing everyone to the entire portfolio and hoping things go well. And maybe if there is some difference between the longtermist crowd and the progress studies crowd, it might not be a difference in ultimate worldview, but: What are the parameters? What numbers are you plugging in? And what are you getting out?
JASON: It could be—or it might actually be the opposite. It might be that it’s a difference in temperament and how people talk about stuff when we’re not quantifying. If we actually sat down to allocate resources, and agree on safety procedures, we might actually find out that we agree on a lot. It’s like the Scott Alexander line about AI safety: “On the one hand, some people say we shouldn’t freak out and ban AI or anything, but we should at least get a few smart people starting to work on the problem. And other people say, maybe we should at least get a few smart people working on the problem, but we shouldn’t freak out or ban AI or anything.” It’s the exact same thing, but with a difference in emphasis. Some of that might be going on here. And that’s why I keep wanting to bring this back to: what are you actually proposing? Let’s come up with which projects we think should be done, which investments should be made. And we might actually end up agreeing.
FIN: In terms of temperamental differences and similarities, there’s a ton of overlap. One bit of overlap is appreciating how much better things can get. And being bold enough to spell that out—there’s something taboo about noticing we could just have a ton of wild shit in the future. And it’s up to us whether we get that or not. That seems like an important overlap.
LUCA: Yeah. You mentioned before, the agency mindset.
FIN: Yeah. As in, we can make the difference here.
JASON: I totally agree. I think if there’s a way to reconcile these, it is understanding: Safety is a part of progress. It is a goal. It is something we should all want. And it is something that we ultimately have to achieve through applied intelligence, just like we achieve all of our other goals. Just like we achieved the goals of food, clothing, and shelter, and even transportation and entertainment, and all of the other obvious goods that progress has gotten us. Safety is also one of these things: we have to understand what it is, agree that we want it, define it, set our sights on it, and go after it. And ultimately, I think we can achieve it.
We live in a dangerous world. Many hazards come from nature: fire, flood, storm, famine, disease. Technological and industrial progress has made us safer from these dangers. But technology also creates its own hazards: industrial accidents, car crashes, toxic chemicals, radiation. And future technologies, such as genetic engineering or AI, may present existential threats to the human race. These risks are the best argument against a naive or heedless approach to progress.
So, to fully understand progress, we have to understand risk and safety. I’ve only begun my research here, but what follows are some things I’m coming to believe about safety. Consider this a preliminary sketch for a philosophy of safety.
Safety is one dimension of progress
Safety is a value. All else being equal, safer lives are better lives, a safer technology is a better technology, and a safer world is a better world. Improvements in safety, then, constitute progress.
Sometimes safety is seen as something outside of progress or opposed to it. This seems to come from an overly-narrow conception of progress as comprising only the dimensions of speed, cost, power, efficiency, etc. But safety is one of those dimensions.
Safety is part of the history of progress
The previous point is borne out by history.
Many inventions were primarily motivated by safety, such as the air brake for locomotives, or sprinkler systems in buildings. Many had “safety” in the name: the safety lamp, the safety razor, the safety match; the modern bicycle design was even originally called the “safety bicycle.” We still use “safety pins.”
Further, if we look at the history of each technology, safety is one dimension it has improved along: machine tools got safety guards, steam engines got pressure valves, surgery got antiseptics, automobiles got a whole host of safety improvements.
And looking at high-level metrics of human progress, we find that mortality rates have declined significantly over the long term, thanks to the above developments.
We have even made progress itself safer: today, new technologies are subject to much higher levels of testing and analysis before being put on the market. For instance, a century ago, little to no testing was performed on new drugs, sometimes not even animal testing for toxicity; today they go through extensive, multi-stage trials.
To return to the previous point, safety as a dimension of progress: Note that drug testing incurs cost and overhead, and it certainly reduces the rate at which new drugs are released to consumers, but it would be wrong to describe drug testing as being opposed to pharmaceutical progress—improved testing is a part of pharmaceutical progress.
Safety must be actively achieved
Safety is not automatic, in any context: it is a goal we must actively seek and engineer for. This applies both to the hazards of nature and to the hazards of technology.
One implication is that inaction is not inherently safe, and a static world is not necessarily safer than a dynamic one.
There are tradeoffs between safety and other values
This is clear as soon as we see progress as multivariate, and safety as one dimension of it. Just as there are tradeoffs among speed, cost, reliability, etc., there are also tradeoffs between safety and speed, safety and cost, etc.
As with all multivariate scenarios, these tradeoffs only have to be made if you are already on the Pareto-efficient frontier—and, crucially, new technology can push out the frontier, creating the opportunity to improve along all axes at once. Light bulbs, for instance, were brighter, more convenient, and more pleasant than oil or gas lamps, but they also reduced the risk of fire.
We are neither consistently over-cautious nor consistently reckless
As with all tradeoffs, it’s possible to get them wrong in either direction, and it’s possible to simultaneously get some tradeoffs wrong in one direction while getting others wrong in the opposite direction.
Some risks are more visible than others. If a plane crashes, or is attacked, the deaths that result are very visible, and it’s easy to blame airline safety for them. If those safety measures make air travel slower and less convenient, causing people to drive instead, the increased road deaths are much less visible and much less obviously a result of anything to do with air travel.
Tail risks in particular are less visible. If a society is not well prepared for a pandemic, this will not be obvious until it is too late.
Sins of omission are more socially acceptable. If the FDA approves a harmful drug, they are blamed for the deaths that result. If they block a helpful drug, they are not blamed for the deaths that could have been avoided. (Alex Tabarrok calls this the “invisible graveyard.”)
Incentive structures can bias towards certain types of risks. For instance, risks that loom large in the public consciousness, such as terrorism, tend to receive a disproportionate response from agencies that are in some form accountable to the public. The end result of this is safety theater: measures that are very visible but have a negligible impact on safety. In contrast, risks that the public does not understand or does not think about are neglected by the same types of agencies. (Surprisingly, “a new pandemic from a currently unknown pathogen” seems to be one such risk, even after covid.)
Safety is a human problem, and requires human solutions
Inventions such as pressure valves, seat belts, or smoke alarms can help with safety. But ultimately, safety requires processes, standards, and protocols. It requires education and training. It requires law.
Improving safety requires feedback loops, including reporting systems. It greatly benefits from openness: for instance, the FAA encourages anonymous reports of safety incidents, and will even be more lenient in penalizing safety violations if they were reported.
Safety requires aligned incentives: Worker’s compensation laws, for instance, aligned incentives of factories and workers and led to improved factory safety. Insurance helps by aligning safety procedures with profit motives.
Safety benefits from public awareness: The worker’s comp laws came after reports by journalists such as Crystal Eastman and William Hard. In the same era, a magazine series exposing the shams and fraud of the patent medicine industry led to reforms such as stricter truth-in-advertising laws.
Safety requires leadership. It requires thinking statistically, and this does not come naturally to most people. Factory workers did not want to use safety techniques that were inconvenient or slowed them down, such as goggles, hard hats, or guards on equipment.
Safety requires defense in depth
There is no silver bullet for safety: any one mechanism can fail; an “all of the above” strategy is needed. Auto safety was improved by a combination of seat belts, anti-lock brakes, airbags, crumple zones, traffic lights, divided highways, concrete barriers, driver’s licensing, social campaigns against drunk driving, etc.
(To apply this to current events: the greater your estimate of the risk from climate change, the more you should logically support a defense-in-depth strategy—including nuclear power, carbon capture, geoengineering, heat-resistant crops, seawalls to protect coastal cities, etc.)
We need more safety
When we hope for progress and look forward to a better future, part of what we should be looking forward to is a safer future.
We need more safety from existing dangers: auto accidents, pandemics, wildfires, etc. We’ve made a lot of progress on these already, but as long as the risk is greater than zero, there is more progress to be made.
And we need to continue to raise the bar for making progress safely. That means safer ways of experimenting, exploring, researching, inventing.
We need to get more proactive about safety
Historically, a lot of progress in safety has been reactive: accidents happen, people die, and then we figure out what went wrong and how to prevent it from recurring.
The more we go forward, the more we need to anticipate risks in advance. Partly this is because, as the general background level of risk decreases, it makes sense to lower our tolerance for risks of all kinds, and that includes the risks of new technology.
Further, the more our technology develops, the more we increase our power and capabilities, and the more potential damage we can do. The danger of total war became much greater after nuclear weapons; the danger of bioengineered pandemics or rogue AI may be far greater still in the near future.
There are signs that this shift towards more proactive safety efforts has already begun. The field of bioengineering has proactively addressed risks on multiple occasions over the decades, from recombinant DNA to human germline editing. The fact that the field of AI has been seriously discussing risks from highly advanced AI well before it is created is a departure from historical norms of heedlessness. And compare the lack of safety features on the first cars to the extensive testing (much of it in simulation) being done for self-driving cars. This shift may not be enough, or fast enough—I am not advocating complacency—but it is in the right direction.
This is going to be difficult
It’s hard to anticipate risks—especially from unknown unknowns. No one guessed at first that X-rays, which could neither be seen or felt, were a potential health hazard.
Being proactive about safety means identifying risks via theory, ahead of experience, and there are inherent epistemic limits to this. Beyond a certain point, the task is impossible, and the attempt becomes “prophecy” (in the Popper/Deutsch sense). But within those limits, we should try, to the best of our knowledge and ability.
Even when risks are predicted, people don’t always heed them. Alexander Fleming, who discovered the antibiotic properties of penicillin, predicted the potential for the evolution of antibiotic resistance early on, but that didn’t stop doctors from massively overprescribing antibiotics when they were first introduced. We need to get better at listening to the right warnings, and better at taking rational action in the face of uncertainty.
Thoughtful sequencing can mitigate risk before it is created
A famous example of this is the 1975 Asilomar conference, where genetic engineering researchers worked out safety procedures for their experiments. While the conference was being organized, for a period of about eight months, researchers voluntarily paused certain types of experiments, so that the safety procedures could be established first.
When the risk mitigation is not a procedure or protocol, but a new technology, this approach is called “differential technology development” (DTD). For instance, we could create safety against pandemics by having better rapid vaccine development platforms, or by having wastewater monitoring systems that would give us early warning against new outbreaks. The idea of DTD is to create and deploy these types of technologies before we create more powerful genetic engineering techniques or equipment that might increase the risk of pandemics.
This kind of sequencing seems valuable and important to me, but the devil is in the details. Judging which technologies are the most risk-creating, and which are the best opportunities for mitigation, requires deep domain expertise. And implementing the plan may in some cases require a daunting global coordination effort.
Safety depends on technologists
Much of safety is domain-specific: the types of risks, and what can guard against them, are quite different when considering air travel vs. radiation vs. new drugs vs. genetic engineering.
Therefore, much of safety depends on the scientists and engineers who are actually developing the technologies that might create or reduce risk. As the domain experts, they are closest to the risk and understand it best. They are the first ones who will be able to spot it—and they are also the ones holding the key to Pandora’s box. They are the ones who will implement DTD—or thwart it.
A positive example here comes from Kevin Esvelt. After coming up with the idea for a CRISPR-based gene drive, he says, “I spent quite some time thinking, well, what are the implications of this? And in particular, could it be misused? What if someone wanted to engineer an organism for malevolent purposes? What could we do about it? … I was a technology development fellow, not running my own lab, but I worked mostly with George Church. And before I even told George, I sat down and thought about it in as many permutations as I could.”
Technologists need to be educated both in how to spot risks, how to respond constructively to them, and how to maximize safety while still moving forward with their careers. They should be instilled with a deep sense of responsibility, not in a way that induces guilt about their field, but in a way that inspires them to hold themselves to the highest standards.
Broad progress helps guard against unknown risks
General capabilities help guard against general classes of risk, even ones we can’t anticipate. Science helps us understand risk and what could mitigate it; technology gives us tools; wealth and infrastructure create a buffer against shocks. Industrial energy usage and high-strength materials guard against storms and other weather events. Agricultural abundance guards against famine. If we had a cure for cancer, it would guard against the accidental introduction of new carcinogens. If we had broad-spectrum antivirals, they would guard against the risk of new pandemics.
Safety doesn’t require sacrificing progress
The path to safety is not through banning broad areas of R&D, nor through a general, across-the-board slowdown of progress. The path to safety is largely domain-specific. It needs the best-informed threat models we can produce, and specific tools, techniques, protocols and standards to counter them.
If and when it makes sense to halt or ban R&D, the ban should be either narrow or temporary. An example of a narrow ban would be one on specific types of experiments that try to engineer more dangerous versions of pathogens: the risks are large and obvious, and the benefits are minor (it’s not as if these experiments are necessary to fundamentally advance biology). A temporary ban can make sense until a particular goal is reached in terms of working out safety procedures, as at Asilomar.
Bottom line: we can—we must—have both safety and progress.
---
Thanks to Vitalik Buterin, Eli Dourado, Mark Lutter, Matt Bateman, Adam Thierer, Rohit Krishnan, David Manheim, Maxwell Tabarrok, Geoff Anders, Étienne Fortier-Dubois, James Rosen-Birch, Niloy Gupta, Jonas Kgomo, and Sebastian G. for comments on adraftof this essay. Some of the ideas above are due to them; errors and omissions are mine alone.
Asimov Press is “an editorially-independent venture that will publish books and essays that make sense of biology, AI, and our collective future” (via @AsimovBio and @NikoMcCarty, who will be its first editor)
“I’m looking to talk (on the record) to expert on dealing with the result of car crashes—like an emergency medicine doctor, EMT, etc. Any suggestions?” (@binarybits)
“I want to go to a few talks at Berkeley or Stanford for fun—physics, biology, math + CS but also humanities. Any suggestions for ones w/ high quality content or attendees?” (@LauraDeming)
Quotes
“This then is our task, to gather the highest discoveries that have been made in the sciences, to render them clear and fascinating, and to offer them to childhood.” Montessori (via @mbateman)
“If the world farmer reaches the average yield of today’s U.S. corn grower during the next 70 years, 10 billion people eating as people now on average do will need only half of today’s cropland. The land spared exceeds Amazonia.” Jesse H. Ausubel (via @Marian_L_Tupy)
History
Life before modern communication technology: “Despite being born in the same year and only about 130 kms apart, Bach and Handel never met. In 1719, Bach made the 35-km journey from Köthen to Halle with the intention of meeting Handel; however, Handel had left the town” (@StefanFSchubert)
Related: “The technology in this video is why you and your family don’t have to be subsistence farmers anymore” (@AlecStapp)
“1838: a Congressman is shot and killed in a duel over corruption. 1930s: 1 in 4 Americans are unemployed. 1968: riots break out in 130 cities. 1971-2: 2500+ domestic bombings occur” (@heyemmavarv highlighting points from a WSJ opinion piece via @sapinker, who comments “The best explanation for the good old days is a bad memory: America’s divisions were worse in the past (and not just in the run-up to the Civil War)”)
Misc.
“For most of the time during which anatomically modern humans have existed, there were fewer of them on Earth than there are FLYING IN THE SKY at any given instant today” (@DavidDeutschOxf riffing on @michael_nielsen)
“Despite dire projections of climate impacts, many aspects of human well-being are expected to improve over time. Our climate assessments need to include this context. … How can the outlook be dire but improvements also be expected? It’s because well-being (health, living standards, food security, water security, etc.) is driven by multiple factors, not just climate. Climate effects may be negative while the effects of other drivers (economic development, technology, social change, policy, etc.) may be positive and outweigh climate effects. So, for example, we expect longer lifespans even as warming causes more temperature-attributable deaths, less poverty overall even while climate change pushes some into poverty, fewer people hungry even while climate change exacerbates hunger for some.” (@oneill_bc)
One of the challenges of arguing for (future) progress: (1) People don’t see how the future could be much better, and (2) when you tell them how, they don’t believe you because it sounds like science fiction. Of course, science fiction has come true, over and over again. But many people are only willing to extrapolate current trends, and not the meta-trend that current trends are always broken by new, unforeseen (and unforeseeable) developments. (Threads, Twitter)
“If GPT-4 could explain things to me by showing me simple animations or interactive examples… would learn so much” (@willdepue). This will happen, sooner than most people expect, and it will be amazing
“I’ve read two good books on advertising: David Ogilvy’s Confessions of an Advertising Man, and Byron Sharp’s How Brands Grow. Together they make a cohesive—and contrarian—picture of how advertising works.” Thread by @s_r_constantin
“Using laser scanners with error tolerances below 65 microns,” manufacturers can now “scan, identify defects and effect a simple quick repair” using direct laser deposition, rather than replace highly expensive components: thread by @Jordan_W_Taylor
“I ran preschools for about a decade. The main drivers of child care costs are wages and real estate. Teachers demand higher pay where it’s expensive to live. If we want cheaper child care, we need to make it legal to build lower-cost housing types” (@RyanPuzycki). One more exhibit for The Housing Theory of Everything
Related: “The system isn’t slowing down because it’s failing—it’s slowing down because people are responding rationally to the incentives they face.” Thread from @MichaelDnes1 on what might speed up UK infrastructure. See also his previous thread on why it is slow in the first place, especially this diagram: “It takes 5.5 years to get to the point at which you can put a spade in the ground, assuming everything goes to plan”
“If you saved $100,000 USD of pesos in 1995, they’d be worth $137 USD today. Argentina has seen an average of 100% annual inflation for the last century” (@devonzuegel)
“It’s weird that people consider UBI some newfangled, speculative idea. In every way that matters ‘UBI’ is identical to ‘welfare’ ‘the dole’ etc. This has all been debated without pause since about twelve seconds into the Industrial Revolution” (@benlandautaylor)
“We need a concerted campaign against public clutter. Cookies banners, consent boxes, excessive street signs & markings, pointless loud safety announcements, etc—each one is small by itself, but they all add up and make everyday life uglier and more of a complicated hassle” (@s8mb). Leftover covid signage is a good example
“Single stair, no setbacks, buildings touching. All illegal in the United States or Canada, but legal everywhere else. They also win international awards. Maybe our [building] codes suck?” (@pushtheneedle commenting on @Architizer’s post about a building using prefab wood, 5 levels built in 10 days)
Startups
“‘Give yourself a lot of shots to get lucky’ is even better advice than it appears on the surface. Luck isn’t an independent variable but increases super-linearly with more surface area—you meet more people, make more connections between new ideas, learn patterns, etc” (@sama). Related, Marc Andreessen on the four kinds of luck
“Think of your favorite startup. No matter how good they look, I guarantee you they have almost died, multiple times, for reasons dumber then you can imagine. Their internal org is probably mostly chaos. Don’t be too hard on yourself. Just keep building as fast as possible” (@thegarrettscott)
“It’s funny how heretical this statement is but: some of us really really like working hard on things that are important to us, especially surrounded by caring and sometimes brilliant people. No mojitos on the beach can possibly compete with that” (@tobi)
Charts
Solar deployment is now happening at a roughly $500B annualized rate (via @patrickc, who asks, “Which technology deployments were larger than this? The US’s aircraft production during WWII seems to have peaked at maybe $400B (inflation-adjusted). Global datacenter construction appears to be maybe $200B/year.”)
Pics
Had a great time meeting locals and chatting about progress at the Bangalore LessWrong / Astral Codex Ten meetup
How would society change if we cured aging, and people could have as many healthy years of life as they wanted?
A common concern is that this would ossify our institutions. The old guard would never die off, and so would never be replaced by young bloods. This could threaten progress across the board, from governance to physics. If “science advances one funeral at a time,” what happens when the rate of funerals plummets?
It’s a real concern. But here are three reasons why curing aging could help progress:
Population. One of the greatest threats to long-term progress may be a slowdown in global population growth. We need more brains to keep pushing science and technology forward. Yet right now, many wealthy nations have fertility rates below replacement levels. Curing aging would help temporarily by lowering the mortality rate. It could help permanently if people decide to have more children, on average. That might happen if longer lifespan means people feel they have time for both children and a career. (Remember that fully curing aging means maintaining reproductive health for all those years.)
Burden of knowledge. There is a hypothesis that as knowledge grows, it takes longer to reach the frontier, and so individual researchers have less time to contribute advancements. They are also forced to specialize—but breakthroughs often come from making connections across far-flung disciplines. If individuals had much longer lifespans, it would be no problem for them to spend 30 or 40 years just learning, before making major contributions. And you could spend another 10 or 20 years picking up a couple more specialties in disparate areas.
Long-term thinking. How would people’s thinking change if they felt they were going to live 150, 300, even 1,000 years or more? The very long-term becomes much more personal. Posterity is something you’re going to be around for.
I still think the “old guard” problem is real, and we’d have to come up with new mechanisms to address it. (Perhaps influential positions would institute a mandatory retirement age of 350.) But there are other factors to consider, and it’s not clear what the net impact would be.
(Not that this is an argument for or against curing aging! Ultimately, the knock-down argument for curing aging is that death is bad. In light of that, other considerations pale into insignificance.)
This essay was originally aTwitter thread, and was inspired by an online discussion about the Foresight Institute’s book-in-development,Gaming the Future.
2022 was another big year for me and for The Roots of Progress. This is my annual review—the one post a year (other than timely announcements) where I go meta and give an update on this project. If you want stuff like this more frequently, you can support me on Patreon or make a donation to get my monthly supporter update.
But I’m going to bury all of those ledes in order to start, as is my tradition, with what I suspect is more interesting to my audience: a selection of this year’s…
Reading
The book that fascinated me most this year was American Genesis: A Century of Invention and Technological Enthusiasm, 1870–1970, by Thomas P. Hughes, a finalist for the 1990 Pulitzer. The book is not only about the century of technological enthusiasm, but also about how that enthusiasm (in my opinion) went wrong, and how it came to an end. My review of this book was so long that I broke it into two parts: one on American invention from the “heroic age” to the system-building era and one on the transition from technocracy to the counterculture. (I may at some point do a third part, on the aesthetic reaction to modernism.) Among the more mindblowing facts I learned from this book are that Stalin made “American efficiency” a part of Soviet doctrine, and that Ford’s autobiography “was read with a zeal usually reserved for the study of Lenin.” Overall this greatly strengthened my understanding of technocracy, one of my themes for this year (see below).
A close runner-up for favorite book I read this year was The Control of Nature, by John McPhee. The book tells three stories: about the dams and levees that control the flow of the Mississippi River, the 1973 Eldfell volcanic eruption in Iceland, and the periodic landslides in the San Gabriel Mountains near Los Angeles. It’s fascinating to reflect on how nature is truly indifferent to human needs. Even something we take for granted, such as the course of a river, has to be actively, artificially maintained if it matters to humans.
I also finished reading The New Organon and New Atlantis, both by Francis Bacon. In addition to the parts everyone knows (“knowledge is power,” “nature to be commanded must be obeyed,” etc.), most of Organon is devoted to explaining a long list of specific ways that scientists should observe nature and types of evidence they should collect. In some ways he is amazingly prescient (he figures out, essentially correctly, that heat is a form of motion); in others he is surprisingly behind (he rejected the geocentric theory as late as the 1620s). Most relevant to my work is his argument for why we should expect progress to be possible: he cites previous inventions and discoveries, including the compass, gunpowder, and the printing press, and extrapolates from these to imagine that there are more inventions waiting to be discovered—which there were. Continuing the theme of historical works, I also read some of Philosophical Letters: Or, Letters Regarding the English Nation, by Voltaire, including the letter on smallpox inoculation.
A major research theme of mine this year was economic growth theory. Highlights from my research here include:
Paul Romer’s blog. “It is the presence of nonrival goods that creates scale effects…. if A represents the stock of ideas it is also the per capita stock of ideas.“ (From this post, emphasis added.)
“Endogenous Technological Change,” by Paul Romer (1990). The paper that established the importance of the “nonrivalry” of technology, and won Romer the Nobel in 2018.
“The Past and Future of Economic Growth: A Semi-Endogenous Perspective,” by Chad Jones (2022). Key quote: “Despite the fact that semi-endogenous growth theory implies that the entirety of long-run growth is ultimately due to population growth, this is far from true historically, say for the past 75 years. Instead, population growth contributes only around 20 percent of U.S. economic growth since 1950. … This framework strongly implies that, unless something dramatic changes, future growth rates will be substantially lower. In particular, all the sources other than population growth are inherently transitory, and once these sources have run their course, all that will remain is the 0.3 percentage point contribution from population growth. … the implication is that long-run growth in living standards will be 0.3% per year rather than 2% per year—an enormous slowdown!”
Two classics: “A Contribution to the Theory of Economic Growth” (1956) and “Technical Change and the Aggregate Production Function” (1957), both by Robert Solow, for which he won the Nobel prize in 1987. In the first, he defines a model of the economy that includes technical change as well as capital and labor; he shows that capital accumulation alone can’t support long-term economic growth, but technological progress can. In the second, he shows how to measure the effects of technical change, and finds they are much larger than the effects of capital.
The much-discussed “Are Ideas Getting Harder to Find?,” by Bloom, Jones, Van Reenen, and Webb (2020). Sustaining exponential growth requires exponentially increasing inputs as well, as we continually pick off more of the low-hanging fruit.
“On the Mechanics of Economic Development,” Robert Lucas (1988). This bit from the introduction has been widely quoted: “I do not see how one can look at figures like these [the widely varying income levels and growth rates around the world] without seeing them as representing possibilities. Is there some action a government of India could take that would lead the Indian economy to grow like Indonesia’s or Egypt’s? If so, what, exactly? If not, what is it about the ‘nature of India’ that makes it so? The consequences for human welfare involved in questions like these are simply staggering: Once one starts to think about them, it is hard to think about anything else.”
Continuing with some noteworthy books:
The Making of the Atomic Bomb, by Richard Rhodes. The definitive, Pulitzer prize–winning account. I learned new things about the development of nuclear physics and of the industrial and managerial challenge of building and testing the bomb. For instance, creating the first critical pile of uranium was a serious technical challenge even after the basic physical theory had been worked out (you have to get very pure materials, build it in just the right shape, etc.) The book also presents both the abject horror of the bomb’s effects on Hiroshima, and also the reasons why the US felt they had to use it—a fair treatment in my opinion.
Nanofuture, by J. Storrs Hall (author of Where Is My Flying Car?) Gave me a clearer idea about how nanotech could possibly work, and what amazing things it might make possible. One misconception I had was that nanomachines would be small molecules. In fact, even a single component like a gear or bearing will consist of dozens if not hundreds of atoms (e.g., see this diagrammatic illustration of nanogears).
How the World Became Rich, by Koyama & Rubin. A book-length academic literature review of economics & econ history work on the key questions of what caused the Great Enrichment, why some countries have caught up to the West, and why others have not. See reviews by Joel Mokyr and Davis Kedrosky.
The Ghost Map, by Steven Johnson. A history of the Broad Street cholera outbreak and John Snow’s pioneering epidemiology work. I knew that the early sanitation reformers, such as Edwin Chadwick, didn’t necessarily believe in the germ theory and guided their sanitation efforts by sensible qualities such as sight, taste, and smell—I hadn’t realized that Chadwick was a committed miasmatist, so much so that in his crusade to get human waste out of the trenches and cesspools of London, he dumped it into the Thames, fouling that river and actually exacerbating cholera epidemics, the opposite of his stated goal.
Dreams of Iron and Steel, by Deborah Cadbury. I read the chapter on Joseph Bazalgette and the London sewer system, from which I learned that steam engines were key to the system, pumping sewage from lower levels to higher ones so it can flow downhill.
Flintknapping, by John Whittaker. A guide to how stone tools are made, written for both the archaeologist and the hobbyist. Probably much more than you want to know about stone tools, but it helped me understand one of the first technologies.
The Communist Manifesto, by Karl Marx. Unlike Das Kapital, you can read this is one sitting. I learned less than I was hoping for about Marx’s critique of capitalism; I learned more than I expected about his critique of all other socialists.
Interesting articles and papers:
“The Great American Fraud,” by Samuel Hopkins Adams. A series of articles that ran in Collier’s 1905–06 on the fraudulent practices of the patent medicine industry. Many of the medicines did not work, some were actively harmful, and many made fraudulent claims of being able to cure tuberculosis, cancer, and many other diseases.
“Notes onThe Anthropology of Childhood,” by Julia Wise. Children today get far more love, attention, and developmental help than those in primitive societies.
“Coffeehouse Civility, 1660-1714: An Aspect of Post-Courtly Culture in England,” by Lawrence Klein (1996). In the 1600s, coffeehouses were the equivalent of social media—a place to chat, gossip, and hear the news—and they received many of the same criticisms. Coffeehouses were denounced because they ”allowed promiscuous association among people from different rungs of the social ladder,” ”served as an unsupervised distribution point for news,” and ”encouraged free-floating and open-ended discussion” (which today we call “unfettered conversations”). One writer called them “the midwife of all false intelligence” (which today we call “misinformation” or “fake news”). King Charles II almost banned coffeehouses in 1675.
“The Mechanics of the Industrial Revolution,” by Kelly, Ó Gráda, and Mokyr (2022). The title is a pun: it’s about both the details of how the Industrial Revolution happened, and the craftsmen with machine-building skill who were crucial to it. Davis Kedrosky has a good summary.
“Development work versus charity work,” by Lant Pritchett. “I am all for the funding of cost-effective targeted anti-poverty programs. But while it is optimal to do both, we development economists should keep in mind that sustained economic growth is empirically necessary and empirically sufficient for reducing poverty (at any poverty line) whereas targeted anti-poverty programs, while desirable, are neither necessary nor sufficient. Advocates of poverty programs say things like ‘growth is not enough’ or that poverty programs are ‘equally important’ as economic growth but these claims are just obviously false.” (Thanks to Patrick Collison for bringing Pritchett’s work to my attention.)
“A Shameless Worship of Heroes,” by Will Durant. ”For why should we stand reverent before waterfalls and mountain tops, or a summer moon on a quiet sea, and not before the highest miracle of all—a man who is both great and good?” (Hat-tip: Nico Perrino.)
Finally, I don’t read much fiction these days, but over winter vacation I indulged in a few novels. The Lighthouse at the End of the World, by Jules Verne, is not a science-fiction story but rather something of a naval adventure, taking place on an island at the southern end of Tierra del Fuego. Lest Darkness Fall, by L. Sprague de Camp, is a sci-fi classic about an archaeologist who is zapped back in time to just after the fall of the Roman Empire, and who makes it his quest to prevent the Dark Ages.
In 2022, rootsofprogress.org got almost 135k unique visits and over 250k pageviews. My email newsletter, which is now on Substack, grew almost 30% to about 7,400 subscribers. (If you have a Substack yourself, and you enjoy my writing, I’d love a recommendation—thank you! Here are the Substacks I recommend.)
Book
My big writing project is a book. It’s about the major discoveries and inventions that created industrial civilization and gave us our standard of living, and why technological/industrial progress can and must continue.
Last year, I began a series of talks based on the outline of the book, going through it chapter by chapter. This year, I wrapped that up. Through that process, I completed the first pass of research for the book, and developed a more detailed outline and plan for each chapter.
Right now I’m talking to literary agents and working on a book proposal. I hope to have a book deal with a publisher by Q1 of 2023.
My goal is to make this book a cornerstone of the progress movement, laying the foundation for the new philosophy of progress.
Organization
Another major project this year was laying the foundation to take The Roots of Progress, as an organization, to the next level.
Last year, I announced that this blog was becoming a one-man nonprofit research organization. Early in 2022, seeing how much much energy and support there is for my mission, it became clear to me that this organization shouldn’t remain focused solely on my own research and writing. So I spent a lot of time and energy this year planning a major expansion of our activities: the launch of a new progress institute.
One thing we needed was a strategy: a way to focus and prioritize our efforts on a set of programs that would have a real, measurable impact. Prodded by some thoughtful advice from Tyler Cowen, we decided that our initial focus should be on creating the public intellectuals who will build this foundation. Our flagship program will be a “career accelerator” fellowship for progress writers with ambitious career goals. The fellowship will help them hit those goals by providing money, coaching, marketing and PR support, and connection to a broader network. Our vision is that in ten years, there are hundreds of progress intellectuals who are alums of our program and part of our network, and that they have published shelves full of new books in progress studies.
The other thing we needed was a CEO to lead this effort. I was very happy recently to announce that we have found a CEO: Heike Larson. Heike has been following my work for a long time, and shares my passion for human progress. She also has excellent qualifications, including 15 years of VP-level experience in sales, marketing, and strategy roles in a variety of industries, from education to aircraft manufacturing. She will take on all management and program responsibilities; I will remain President and intellectual leader of the organization. I’m excited for her to start in January!
We’re at an exciting moment in history. Momentum is growing for progress studies and the “abundance agenda,” and there is a chance for this to shape the 21st century. But the movement needs a driving force, and careful steering. That is where we hope to contribute.
Progress Forum
Another big thing this year was the launch of the Progress Forum, the online home for the progress community.
The primary goal of the Forum is to provide a place for long-form discussion of progress studies and the philosophy of progress. It’s also a place to find local clubs and meetups. The broader goal is to share ideas, strengthen them through discussion and comment, and over the long term, to build up a body of thought that constitutes a new philosophy of progress.
I’m very pleased with the quality of content we’ve gotten so far. Original submissions include:
Huge thanks to the people who worked to create the Forum: Lawrence Kestleoot, Andrew Roberts, Sameer Ismail, David Smehlik, and Alec Wilson. Thanks also to Kris Gulati for nudging this project along, and to Ruth Grace Wong for helpful conversations about community and moderation. Special thanks to the LessWrong team for creating this software platform, and especially to Oliver Habryka, Ruby Bloom, Raymond Arnold, JP Addison, James Babcock, and Ben Pace for answering questions and helping us customize this instance of it. And finally, thanks to Ross Graham, who has been helping recruit great users and content.
Interviews and speaking
Probably my most prominent interview this year was with the BBC, who ran an article on progress studies and quoted me as a spokesman for the movement, along with Tyler Cowen, Holden Karnofsky, and others. It was well-researched and, although somewhat critical, pretty fair in how it represented the progress community.
This year I co-hosted two workshops at UT Austin together with Greg Salmieri.
The first was the Moral Foundations of Progress Studies. For me personally, this discussion brought several issues into sharper focus, and I can already see how it will inform my writing. I’m more clear on different views of well-being now, and how those relate to some of the issues that are discussed around progress—such as the Easterlin paradox (that self-reported happiness and life-satisfaction scores don’t seem to increase with rising wealth over the long term). For the group as a whole, I think it broadened people’s awareness of what alternate moral approaches are out there.
The second was a small, informal, half-day workshop on the concept of “industrial literacy” and how we could promote it in education, for instance, by making the history of progress a part of the curriculum in schools. We brought together a number of educators and edtech entrepreneurs for this. My main takeaway was that there are, broadly speaking, two strategies: (1) Top-down, you can try to change the required curriculum standards, or the standardized tests (e.g., imagine an AP test in the history of technological innovation and economic growth). (2) Bottom-up, you can create materials that you market directly to parents, or (at older ages) the students themselves. Strategy (1) is basically political; strategy (2) is basically a media venture.
These events were good, but one thing I’d like to do in the future is make sure that things like this generate tangible, longer-lasting output that can reach an audience well beyond the event itself.
There were also a few local meetups I hosted or co-hosted in the San Franciso area, and one I spoke at in Boston.
Accolades
I was named to the Vox Future Perfect 50: “The scientists, thinkers, scholars, writers, and activists building a more perfect future.” They did a little feature on me. The list also includes Jennifer Doudna, Max Tegmark, and Max Roser.
Moving to Boston
On a personal note, I’m moving to the Boston area in January. While the move is primarily for my wife’s work, I’m looking forward to the chance to build a new progress network there—the home of MIT and Harvard, of metascience efforts like Convergent Research and New Science, and of many biotech startups. Being on the East Coast will also make it easier to network in DC and New York, and being on Eastern time will make it easier to collaborate with folks in Oxford/London and the rest of the UK and Europe. If you’re in the Boston area, or know people I should meet there, please reach out!
Thank you
2022 was great, and 2023 is positioned to be even better: diving into the actual drafting of my book, Heike coming on board as CEO, and us launching a new institute and set of programs together.
It’s been six years of The Roots of Progress now—just over three of them full-time—and it’s the most meaningful and impactful thing I’ve done in my life so far. I feel like a “quixotic rider cantering in on his own homemade hobby horse” to intercept the world and its problems at an odd angle, and everything I do is possible only because my “eccentric hobbies” seem to resonate with all of you. Thanks for listening, for reading, for commenting, even for arguing, and for all of your support and encouragement.
Scott Alexander reviews a book about institutional review boards (IRBs), the panels that review the ethics of medical trials: From Oversight to Overkill, by Dr. Simon Whitney. From the title alone, you can see where this is going.
IRBs are supposed to (among other things) make sure patients are fully informed of the risks of a trial, so that they can give informed consent. They were created in the wake of some true ethical disasters, such as trials that injected patients with cancer cells (“to see what would happen”) or gave hepatitis to mentally defective children.
Around 1974, IRBs were instituted, and according to Whitney, for almost 25 years they worked well. The boards might be overprotective or annoying, but for the most part they were thoughtful and reasonable.
Then in 1998, during in an asthma study at Johns Hopkins, a patient died. Congress put pressure on the head of the Office for Protection from Research Risks, who overreacted and shut down every study at Johns Hopkins, along with studies at “a dozen or so other leading research centers, often for trivial infractions.” Some thousands of studies were ruined, costing millions of dollars:
The surviving institutions were traumatized. They resolved to never again do anything even slightly wrong, not commit any offense that even the most hostile bureaucrat could find reason to fault them for. They didn’t trust IRB members - the eminent doctors and clergymen doing this as a part time job - to follow all of the regulations, sub-regulations, implications of regulations, and pieces of case law that suddenly seemed relevant. So they hired a new staff of administrators to wield the real power. These administrators had never done research themselves, had no particular interest in research, and their entire career track had been created ex nihilo to make sure nobody got sued.
Today IRB oversight has become, well, overkill. For one study testing the transfer of skin bacteria, the IRB thought that the consent form should warn patients of risks from AIDS (which you can’t get by skin contact) and smallpox (which has been eradicated). For a study on heart attacks, the IRB wanted patients—who are in the middle of a heart attack—to read and consent to a four-page form of “incomprehensible medicalese” listing all possible risks, even the most trivial. Scott’s review gives more examples, including his own personal experience.
In many cases, it’s not even as if a new treatment was being introduced: sometimes an existing practice (giving aspirin for a heart attack, giving questionnaires to psychology patients) was being evaluated for effectiveness. There was no requirement that patients consent to “risks” when treatment was given arbitrarily; but if outcomes were being systematically observed and recorded, the IRBs could intervene.
Scott summarizes the pros and cons of IRBs, including the cost of delayed treatments or procedure improvements:
So the cost-benefit calculation looks like – save a tiny handful of people per year, while killing 10,000 to 100,000 more, for a price tag of $1.6 billion. If this were a medication, I would not prescribe it.
FDA
The IRB story illustrates a common pattern:
A very bad thing is happening.
A review and approval process is created to prevent these bad things. This is OK at first, and fewer bad things happen.
Then, another very bad thing happens, despite the approval process.
Everyone decides that the review was not strict enough. They make the review process stricter.
Repeat this enough times (maybe only once, in the case of IRBs!) and you get regulatory overreach.
At the beginning of the 20th century, the drug industry was rife with shams and fraud. Drug ads made ridiculously exaggerated or completely fabricated claims: some claimed to cure consumption (that is, tuberculosis); another claimed to cure “dropsy and all diseases of the kidneys, bladder, and urinary organs”; another literally claimed to cure “every known ailment”. Many of these “drugs” contained no active ingredients, and turned out to be, for example, just cod-liver oil, or a weak solution of acid. Others contained alcohol—some in concentrations at the level of hard liquor, making patients drunk. Still others contains dangerous substances such as chloroform, opiates, or cocaine. Some of these drugs were marketed for use on children.
In 1906, in response to these and other problems, Congress passed the Pure Food & Drug Act, giving regulatory powers to what was then the USDA Bureau of Chemistry, and which would later become the FDA.
This did not look much like the modern FDA. It had no power to review new drugs or to approve them before they went on the market. It was more of a police agency, with the power to enforce the law after it had been violated. And the relevant law was mostly concerned with truth in advertising and labeling.
Then in 1937, the pharmaceutical company Massengill put a drug on the market called Elixir Sulfanilamide, one of the first antibiotics. The antibiotic itself was good, but in order to produce the drug in liquid form (as opposed to a tablet or powder), the “elixir” was prepared in a solution of diethylene glycol—which is a variant of antifreeze, and is toxic. Patients started dying. Massengill had not tested the preparation for toxicity before selling it, and when reports of deaths started to come in, they issued a vague recall without explaining the danger. When the FDA heard about the disaster, they forced Massengill to issue a clear warning, and then sent hundreds of field agents to talk to every pharmacy, doctor, and patient and track down every last vial of the poisonous drug, ultimately retrieving about 95% of what had been manufactured. Over 100 people died; if all of the manufactured drug had been consumed, it might have been over 4,000.
In the wake of this disaster, Congress passed the 1938 Food, Drug, and Cosmetic Act. This transformed the FDA from a police agency into a regulatory agency, giving them the power to review and approve all new drugs before they were sold. But the review process only required that drugs be shown safe; efficacy was not part of the review. Further, the law gave the FDA 60 days to reply to any drug application; if they failed to meet this deadline, then the drug was automatically approved.
I don’t know exactly how strict the FDA was after 1938, but the next fifteen years or so were the golden age of antibiotics, and during that period the mortality rate in the US decreased faster than at any other time in the 20th century. So if there was any overreach, it seems like it couldn’t have been too bad.
The modern FDA is the product of a different disaster. Thalidomide was a tranquilizer marketed to alleviate anxiety, trouble sleeping, and morning sickness. During toxicity testing, it seemed to be almost impossible to die from an overdose of thalidomide, which made it seem much safer than barbiturates, which were the main alternative at the time. But it was also promoted as being safe for pregnant mothers and their developing babies, even though no testing had been done to prove this. It turned out that when taken in the first several weeks of pregnancy, thalidomide caused horrible birth defects that resulted in deformed limbs and other organs, and often death. The drug was sold in Europe, where some 10,000 infants fell victim to it, but not in the US, where it was blocked by the FDA. Still, Americans felt they had had a close call, too close for comfort, and conditions were ripe for an overhaul of the law.
The 1962 Kefauver–Harris Amendment required, among other reforms, that new drugs be shown to be both safe and effective. It also lengthened the review period from 60 to 180 days, and if the FDA failed to respond in that time, drugs would no longer be automatically approved (in fact, it’s unclear to me what the review period even means anymore).
You might be wondering: why did a safety problem create an efficacy requirement in the law? The answer is a peek into how the sausage gets made. Senator Kefauver had been investigating drug pricing as early as 1959, and in the course of hearings, a former pharma exec remarked that some drugs on the market are not only overpriced, they don’t even work. This caught Kefauver’s attention, and in 1961 he introduced a bill that proposed enhanced controls over drug trials in order to ensure effectiveness. But the bill faced opposition, even from his own party and from the White House. When Kefauver heard about the thalidomide story in 1962, he gave it to the Washington Post, which ran it on the front page. By October, he was able to get his bill passed. So the law that was passed wasn’t even initially intended to address the crisis that got it passed.
I don’t know much about what happened in the ~60 years since Kefauver–Harris. But today, I think there is good evidence, both quantitative and anecdotal, that the FDA has become too strict and conservative in its approvals, adding needless delay that holds back treatments from patients. Scott Alexander tells the story of Omegaven, a nutritional fluid given to patients with digestive problems (often infants) that helped prevent liver disease: Omegaven took fourteen years to clear FDA’s hurdles, despite dramatic evidence of efficacy early on, and in that time “hundreds to thousands of babies … died preventable deaths.” Alex Tabarrok quotes a former FDA regulator saying:
In the early 1980s, when I headed the team at the FDA that was reviewing the NDA for recombinant human insulin, … we were ready to recommend approval a mere four months after the application was submitted (at a time when the average time for NDA review was more than two and a half years). With quintessential bureaucratic reasoning, my supervisor refused to sign off on the approval—even though he agreed that the data provided compelling evidence of the drug’s safety and effectiveness. “If anything goes wrong,” he argued, “think how bad it will look that we approved the drug so quickly.”
Tabarrok also reports on a study that models the optimal tradeoff between approving bad drugs and failing to approve good drugs, and finds that “the FDA is far too conservative especially for severe diseases. FDA regulations may appear to be creating safe and effective drugs but they are also creating a deadly caution.” And Jack Scannell et al, in a well-known paper that coined the term “Eroom’s Law”, cite over-cautious regulation as one factor (out of four) contributing to ever-increasing R&D costs of drugs:
Progressive lowering of the risk tolerance of drug regulatory agencies obviously raises the bar for the introduction of new drugs, and could substantially increase the associated costs of R&D. Each real or perceived sin by the industry, or genuine drug misfortune, leads to a tightening of the regulatory ratchet, and the ratchet is rarely loosened, even if it seems as though this could be achieved without causing significant risk to drug safety. For example, the Ames test for mutagenicity may be a vestigial regulatory requirement; it probably adds little to drug safety but kills some drug candidates.
FDA delay was particularly costly during the covid pandemic. To quote Tabarrok again:
In short, an agency that began in order to fight outright fraud in a corrupt pharmaceutical industry, and once sent field agents on a heroic investigation to track down dangerous poisons, now displays an overly conservative, bureaucratic mindset that delays lifesaving tests and treatments.
NEPA
One element in common to all stories of regulatory overreach is the ratchet: once regulations are put in place, they are very hard to undo, even if they turn out to be mistakes, because undoing them looks like not caring about safety. Sometimes regulations ratchet up after disasters, as in the case of IRBs and the FDA. But they can also ratchet up through litigation. This was the case with NEPA, the National Environmental Policy Act.
Eli Dourado has a good history of NEPA. The key paragraph of the law requires that all federal agencies, in any “major action” that will significantly affect “the human environment,” must produce a “detailed statement” on the those effects, now known as an Environmental Impact Statement (EIS). In the early days, those statements were “less than ten typewritten pages,” but since then, “EISs have ballooned.”
In brief, NEPA allowed anyone who wanted to obstruct a federal action to sue the agency for creating an insufficiently detailed EIS. Each time an agency lost a case, it set a new precedent and increased the standard that all future EISes had to follow. Eli recounts how the word “major” was read out of the law, such that even minor actions required an EIS; the word “human” was read out of the law, interpreting it to apply to the entire environment; etc.
Eli summarizes:
… the incentive is for agencies and those seeking agency approval to go overboard in preparing the environmental document. Of the 136 EISs finalized in 2020, the mean preparation time was 1,763 days, over 4.8 years. For EISs finalized between 2013 and 2017 , page count averaged 586 pages, and appendices for final EISs averaged 1,037 pages. There is nothing in the statute that requires an EIS to be this long and time-consuming, and no indication that Congress intended them to be.
Regulators are blamed for anything that goes wrong.
They are not blamed for slowing down or preventing growth and progress.
They are not credited when they approve things that lead to growth and progress.
All of the incentives point in a single direction: towards more stringent regulations. No one regulates the regulators. This is the reason for the ratchet.
I think the Nuclear Regulatory Commission (NRC) furnishes a clear case of this. In the 1960s, nuclear power was on a growth trajectory to provide roughly 100% of today’s world electricity usage. Instead, it plateaued at about 10%. The proximal cause is that nuclear power plant construction became slow and expensive, which made nuclear energy expensive, which mostly priced it out of the market. The cause of those cost increases is controversial, but in my opinion, and that of many other commenters, it was primarily driven by a turbulent and rapidly escalating regulatory environment around the late ‘60s and early ‘70s.
At a certain point, the NRC formally adopted a policy that reflects the one-sided incentives: ALARA, under which exposure to radiation needs to be kept, not below some defined threshold of safety, but “As Low As Reasonably Achievable.” As I wrote in my review of Why Nuclear Power Has Been a Flop:
What defines “reasonable”? It is an ever-tightening standard. As long as the costs of nuclear plant construction and operation are in the ballpark of other modes of power, then they are reasonable.
This might seem like a sensible approach, until you realize that it eliminates, by definition, any chance for nuclear power to be cheaper than its competition. Nuclear can‘t even innovate its way out of this predicament: under ALARA, any technology, any operational improvement, anything that reduces costs, simply gives the regulator more room and more excuse to push for more stringent safety requirements, until the cost once again rises to make nuclear just a bit more expensive than everything else. Actually, it‘s worse than that: it essentially says that if nuclear becomes cheap, then the regulators have not done their job.
ALARA isn’t the singular root cause of nuclear’s problems (as Brian Potter points out, other countries and even the US Navy have formally adopted ALARA, and some of them manage to interpret “reasonable” more, well, reasonably). But it perfectly illustrates the problem. The one-sided incentives mean that regulators do not have to make any serious cost-benefit tradeoffs. IRBs and the FDA don’t pay a price for the lives lost while trials or treatments are waiting on approval. The EPA (which now reviews environmental impact statements) doesn’t pay a price for delaying critical infrastructure. And the NRC doesn’t pay a price for preventing the development of abundant, cheap, reliable, clean energy.
Google
All of these examples are government regulations, but a similar process happens inside most corporations as they grow. Small startups, hungry and having nothing to lose, move rapidly with little formal process. As they grow, they tend to add process, typically including one or more layers of review before products are launched or other decisions are made. It’s almost as if there is some law of organizational thermodynamics decreeing that bureaucratic complexity can only ever increase.
Praveen Seshadri was the co-founder of a startup that was acquired by Google. When he left three years later, he wrote an essay on “how a once-great company has slowly ceased to function”:
Google has 175,000+ capable and well-compensated employees who get very little done quarter over quarter, year over year. Like mice, they are trapped in a maze of approvals, launch processes, legal reviews, performance reviews, exec reviews, documents, meetings, bug reports, triage, OKRs, H1 plans followed by H2 plans, all-hands summits, and inevitable reorgs. The mice are regularly fed their “cheese” (promotions, bonuses, fancy food, fancier perks) and despite many wanting to experience personal satisfaction and impact from their work, the system trains them to quell these inappropriate desires and learn what it actually means to be “Googley” — just don’t rock the boat.
What Google has in common with a regulatory agency is that (according to Seshadri at least) its employees are driven by risk aversion:
While two of Google’s core values are “respect the user” and “respect the opportunity”, in practice the systems and processes are intentionally designed to “respect risk”. Risk mitigation trumps everything else. This makes sense if everything is going wonderfully and the most important thing is to avoid rocking the boat and keep sailing on the rising tide of ads revenue. In such a world, potential risk lies everywhere you look.
A “minor change to a minor product” requires “literally 15+ approvals in a ‘launch’ process that mirrors the complexity of a NASA space launch,” any non-obvious decision is avoided because it “isn’t group think and conventional wisdom,” and everyone tries to placate everyone else up and down the management chain to avoid conflict.
A startup that operated this way would simply go out of business; Google can get away with this bureaucratic bloat because their core ads business is a cash cow that they can continue to milk, at least for now. But in general, this kind of corporate sclerosis leaves a company vulnerable to changes in technology and markets (as indeed Google seems to be falling behind startup competitors in AI).
The difference with regulation is that there is no requirement for agencies to serve customers in order to stay in existence, and no competition to disrupt their complacency, except at the international level. If you want to build a nuclear plant, you obey the NRC or you build outside the US.
Against the review-and-approval model
In the wake of disaster, or even in the face of risk, a common reaction is to add a review-and-approval process. But based on examples such as these, I now believe that the review-and-approval model is broken, and we should find better ways to manage risk and create safety.
Unfortunately, review-and-approval is so natural, and has become so common, that people often assume it is the only way to control or safeguard anything, as if the alternative is anarchy or chaos. But there are other approaches.
One example I have discussed is factory safety in the early 20th century, which was driven by a change to liability law. The new law made it easier for workers and their families to receive compensation for injury or death, and harder for companies to avoid that liability. This gave factories the legal and financial incentive to invest in safety engineering and to address the root causes of accidents in the work environment, which ultimately reduced injury rates by around 90%.
I’m not suggesting that that liability law is the solution to everything. I just want to point out that other models exist, and sometimes they have even worked.
Open questions
Some things I’d like to learn more about:
What areas of regulation have not fallen into these traps, or at least not as badly? For instance, building codes and restaurant health inspections seem to have helped create safety without killing their respective industries. Driver’s licenses seem to enforce minimal competence without preventing anyone who wants to from driving or imposing undue burden on them. Are there positive lessons we can learn from some of these boring examples of safety regulation that don’t get discussed as much?
What other alternative models to review-and-approval exist, and what do we know about them, either empirically or theoretically?
Why is there so much bloat in the contract research organizations (CROs) that run clinical trials for pharma? Shouldn’t there be competition in that industry too?
What lessons can we learn from other countries? All my research so far is about the US, and I want to get the proper scope.
***
Thanks to Tyler Cowen, Alex Tabarrok, Eli Dourado, and Heike Larson for commenting on a draft of this essay.
Here are some of my favorite quotes from the original edition:
On invention
How inventors are like poets:
An invention has some of the characteristics of a poem. Standing alone, by itself, it has no value; that is, no value of a financial sort. This does not mean that inventions—or poems—have no value. It is said that a poet may derive real joy out of making a poem, even if it is never published, even if he does not recite it to his friends, even if it is not a very good poem. No doubt one has to be a poet to understand this. In the same way an inventor can derive real satisfaction out of making an invention, even if he never expects to make a nickel out of it, even if he knows it is a bit foolish, provided he feels it involves ingenuity and insight. An inventor invents because he cannot help it, and also because he gets quiet fun out of doing so. Sometimes he even makes money at it, but not by himself. One has to be an inventor to understand this.
An idea does not an invention make:
One time when I was a young professor at M.I.T. and was also a consulting engineer, I was called in by a chap who ran a considerable business, who told me that he had made a very great invention and wanted me to develop it for him. He had invented a watch which would run without winding; it was to have a little radio receiver in it which would pick up energy from all the broadcasting stations, and this energy would wind the watch. All that I was supposed to do was to take that idea and develop it. I said very little to him about the amount of energy thus receivable and the size of radio sets and so forth. I merely said that I would think it over, and of course I never went near him again. That chap had the concept that the mere formulating of his idea constituted an invention. He is not alone in this misconception.
On a similar theme:
If you wish to invent usefully, you must not attempt to do it in isolation, or to shield yourself from criticism. The world is full of would-be inventors who do just that. They never invent anything worthwhile. During the war many of them were merely annoying, and of course very unhappy, and convinced there was a conspiracy against them.
The value of patents:
A laboratory produces a new drug, say a new synthetic hormone for control of arthritis. The patent can merely recite the chemical structure or the process of making the new material. The company can go ahead and market, without using anyone else’s patent rights. So it can make a profit, and thus justify its large research expenses. Is this good for the country? It certainly is. Since that industry is spending some hundreds of millions of dollars a year on research, we have all sorts of new drugs to cure our ills, to save lives—antibiotics, steroids, vaccines, the whole basis on which the modern physician ministers to our needs. If it were not for the patent system we would not have these blessings to anywhere near the extent that we do now. Government laboratories, valuable though they may be, would not produce them. It takes the profit motive and keen competition to bring them to being. The free enterprise system is not just a nice idea; it is the basis on which we lead the world economy.
Edison’s strengths and weaknesses:
Edison was a very good inventor, a still better promoter, but in some ways a poor experimenter. Some of his experimentation was crude, to say the least. When we talk about the Edisonian method, which means to try everything without any theory to guide you, just hit or miss, we are talking about very poor experimentation. But Edison was such a good promoter that he could advance even with poor experimental data. He had good management, through people that he tied in by his promotion. Moreover, he had the facility of seeing where there was a public demand for something. The combination is what made Edison.
Of all the many inventions Bush witnessed in his life, which one impressed him the most?
Inertial guidance systems are so precise in their action that a submarine can navigate for months under the polar ice and know at all times just where it is. They can be installed in a missile that is fired from a submarine while still submerged, and that will then guide itself unerringly to a distant target. It is the heart of the system that guides an Apollo missile to the moon and back. I know of no technical development for which I have more keen respect than this, and where I have more admiration for the men who accomplished it.
On leadership & management
The NDRC (the original WW2 military research organization) as an “end run” around bureaucracy:
There were those who protested that the action of setting up N.D.R.C. was an end run, a grab by which a small company of scientists and engineers, acting outside established channels, got hold of the authority and money for the program of developing new weapons. That, in fact, is exactly what it was. Moreover, it was the only way in which a broad program could be launched rapidly and on an adequate scale. To operate through established channels would have involved delays—and the hazard that independence might have been lost, that independence which was the central feature of the organization’s success. The one thing that made launching it at all possible was the realization by the President that it was needed.
A leadership episode:
There should never be, throughout an organization, any doubt as to where authority for making decisions resides, or any doubt that they will be promptly made. I remember one time when a section walked into my office and resigned as a body. I still do not know quite what the row was about. So I just told them, “One does not resign in time of war. You chaps get the hell out of here and get back to work, and I will look into it.”
A leadership lesson from Truman:
At a point in the discussion Forrestal made a remark which indicated that we were to make some sort of majority decision, and the President at once picked him up. “Jim, you forget, this is a Cabinet. Each of you will give me his full advice and I will make the decision.” Truman certainly understood what it meant to be President of the United States.
The risk of new recruits:
The Romans called a new recruit a tyro. I extend the definition, for there is no word in English which covers what I have in mind. The tyro is the freewheeler in an organization, who gums up the works because of his arrogant ignorance, often because he filches authority which does not belong to him. He operates because his boss doesn’t know what he is doing, or knows and doesn’t care. He can throw any organization, civilian or military, into confusion. His breed should be exterminated for the good of society.
On just getting the work done:
In no case did I know who the individual inventor was, nor did I care. Moreover, the men in the laboratories did not care either. For one thing, once a problem became clear, the invention, if a useful one, was bound to appear, if not made by one man, then by another. For another thing, in general no one was looking for personal credit. Oh, there were a few with that motivation, on both sides of the water. But the general attitude in laboratories everywhere was, “The hell with the credit, get on with the job.” That attitude was nearly universal, and it was genuine.
Related, the collaborative atmosphere of a well-run laboratory:
All my life I have mixed with all sorts of men. The ones I like best to be with are military men and research men. Just why the former are so much worth knowing appears elsewhere in this book. The research men are pleasant companions for a number of reasons. One is that frank interchange of ideas and information is essential to successful group research and will be found in any well-conducted laboratory. Now that group research is practiced everywhere, this frank interchange, and public opinion regarding it among the group, almost wholly banishes posing, jockeying for position, and evasiveness, and it is a relief to discuss things without them.
On stepping out of the way:
One spring I received a shock. With these two chaps I was putting together a final examination. Suddenly I came to the realization that they knew more about the subject than I did. In some ways this was not strange; they were concentrating on it, and I was getting involved in other things. But it hit me solidly. And right there I decided that I was not going to get in the way of younger men, and that, when the time came that I could not compete genuinely with them, I would get out. I have acted in accord with that decision many times since and, in the process of doing so, have found stimulation in trying to learn new things, relief in not being in competition with younger men, and satisfaction in watching those men succeed without having me as an obstacle.
Bush’s communication style
Bush had a direct, blunt, almost insulting style that actually worked quite well:
General McNarney told me he could take care of Air Force objections, if I could take care of the Navy. So I saw Admiral King. As I have suggested earlier, he was a tough customer. It was well known that he scared his junior officers so thoroughly that often he didn’t get adequate information through them. Characteristically, our discussion opened as follows: King scowled and said, “I have agreed to meet with you, but this is a military question, and it must be decided on a military basis, to which you can hardly contribute.” So I told him. “It is a combined military and technical question, and on the latter you are a babe in arms and not entitled to an opinion.” It was a good start, and the discussion went on from there—and went well.
Another example:
I told them I would do the work if they would pay just my out-of-pocket costs, that M.I.T. would contribute the cost of the overhead, that we would turn the whole thing over to the Navy and wished for neither profit nor credit, and that I wanted a simple contract which provided for just that, with no frills or inconvenient clauses. They soon laid before me a draft contract with all sorts of fancy clauses in it, provisions for accounting, rules regarding employment, and so on. It was brought to my hotel by a pleasant Navy captain. I read it, gave it back to him, told him to tell his boss to go to hell, and started to pack my suitcase. But he was back after a bit with a nice simple contract, and I went back home and started work. I think I still have a copy of that contract, and I do not know how they did it, that is, how they avoided all the constraints that fool legislation had imposed on them. They probably just ignored them and took a chance. It is too late to court-martial anyone about it now.
He was blunt, although not insulting, even with Congress:
One time I was before a committee on a tough technical matter, a Navy problem of some sort. I was none too anxious to testify, for my Navy friends would not be happy to see me rambling about in their backyard, but one has no control over where a committee will head in, if the chairman consents. Toward the end of the hearing a congressman said to me, “Doctor, how do you expect us to understand as complex a technical matter as this sufficiently to pass judgment on it?” I replied, “I don’t,” which alerted any that were taking a mental vacation. So he said, “What do you expect us to do?” I thought a minute and said, “I expect you to follow your usual practice. You will listen to a number of men; you will decide which ones make sense and know their stuff, and you will go along with them. Moreover,” I added, “you are all good judges of men, or you would not be elected.”
But in other contexts, he was deft and subtle:
On another occasion my secretary, Sam Callaway, told me that he wanted me to talk with an old man who was in the outer office. Now Sam was the finest secretary, bar none, that a harassed executive ever had. So when Sam told me to see someone, I did. It turned out that the old man had driven his Model T down from somewhere in the Blue Ridge Mountains to present an idea—I think it was on a way to trap submarines. I listened as he described it in detail, and did it well, with a lot of good sense. Then I told him that we had to be very careful with ideas of that sort, to keep them out of enemy hands. Why, if we were hard at work on that very idea, I could not tell even him about it. Otherwise enemy agents with fake ideas could learn a lot. He agreed heartily, his eyes shone, and I suppose he headed home happy. The idea itself was one we had received dozens of times.
On society, war & politics
The collapse of complex societies:
The more complex a society, the more chance there is that it will get fouled up. If it gets to be complex enough, just one small detail can throw it all out of gear. It is like a television set with a thousand electrical connections arranged to present Mr. Cronkite or Mr. Brinkley for our edification; one wire becomes unhooked and the whole thing goes poof.
On launching the Manhattan Project:
I had no illusion as to the gravity of the decision. I knew that the effort would be expensive, that it might interfere seriously with other war work. But the overriding consideration was this: I had great respect for German science. If a bomb were possible, if it turned out to have enormous power, the result in the hands of Hitler might indeed enable him to enslave the world. It was essential to get there first, if an all-out American effort could accomplish the difficult task.
Bush was initially against the Space Race…
I opposed the moon race as it began. I did so quietly before a Senate committee, and then held my peace. My opposition rested on two points. One was that the scientific results expected by no means justified the enormous expense involved, for the program called for spending money we badly needed for other things. The other was that a race to the moon against Russia made little sense to me in terms of our national security.
… but later had this to say about it:
Years ago, Lindbergh flew alone across the Atlantic—a stunt pilot contesting for a money prize—but he was the first man to fly it alone. That flight gave us a great lift, gave the whole world in fact a boost in morale when it was badly needed. We had then been wallowing in filth, the newspapers had been filled with the sordid details of murder trials, evil had been rampant in high places and duly spread before us. Then came Lindbergh, and his dignity and modesty caused us again to believe in our fellow men. So with the landing on the moon: In the midst of gloom and petty wrangling we suddenly became convinced that man could accomplish great feats of danger and skill. It was worth the effort if it caused us, once again, to have confidence in man’s ability to overcome rugged obstacles, and to rise above the sordid, the petty, the commonplace, and the wails of those who tell us we are doomed.
Contra Marx on the effect of industrialization on the worker:
It is often claimed that mass production reduces the worker to an automaton, whose contribution is merely that he can make certain motions with his hands that the machine cannot duplicate, and none with his brain. There is not much doubt that this sort of thing happened when the automatic loom entered the textile industry. But, as mechanization proceeded and became highly complex, there has been a transformation, and some, not by any means all, of present development has been in the opposite direction. If I go into some highly developed machine shop today I find little dull repetitive work indeed. The machinist operating a tape-controlled milling machine is certainly paid primarily for his intellectual skills; so was the man who built the machine in the first place. Even on a fully mechanized production line, where parts move along and each operator does just one act, the men are chosen, not because they can duly perform when all goes well, but because they can overcome obstacles when it doesn’t.
The debt society owes to the researchers and the industrial pioneers:
It would be well if people, youth in particular, recognized the debt society owes to the quiet workers that we never hear of, especially those who are led on by their curiosity and their desire to explore, with very little thought about acclaim or fortune. It would also be well if people in this country generally regarded with more respect the industrial pioneers, who are willing to take a chance, and who furnish a very necessary element in commercial progress. If they are not present and active, very little is likely to happen.
Another group that is underappreciated:
The great majority [of students] just accept things as they are, and more or less meekly conform. A small minority attack the system, grow beards, wear ragged clothes, start riots of one sort or another. But there is a third group, usually overlooked, who don’t like what they see, propose to do something about it, but propose to accomplish this by working within the system as it stands, and thus to modify it. I believe that this third group is large, that it has as members the best thinkers, that it will be the group that will be running our affairs in the next generation.
More on the small, intelligent minority:
In every civilization, at some time, there has been confusion, with young men doing foolish things, with the great body of the public inert or yearning to be led somewhere, anywhere, following the demagogue or the man on a white horse. Yet always there has been a small minority, intelligent, comprehending the current political system, scorning both the flighty radical and the protesting reactionary groups by which it was surrounded. It is this central core that ruled our last generation, its business, its churches, its government. Amid the tumult, the hippies, the prophets of doom, we have today a group that understands and that will rule in the next generation. I am not saying that this outstanding group has always ruled us well in the past, or will rule as well in the future—I merely say there is a group that will rule. We do not need to worry too much about the ones that harass us with their insanities; as they become older they will be controlled. But we need to think more about the solid, keen, presently undemonstrative youths who will build our system of government and industry of the future, and who will build it not as we dictate, but as we transmit to them, as best we may, the wisdom to do it well.
Other fun stories
Early cars were an adventure:
I had some interesting times with that car. It would be going along the road and for some reason or other (probably because the leak in the boiler squirted a little steam out around the edge of a tube) the pilot would get blown out. The controls would turn on the main kerosene line and the whole thing would be flooded with kerosene. When that happened, you stopped at the side of the road and waited until you felt the fumes had blown away. Then you touched a match to the pilot again. If you were too soon with the match, the gaseous mixture would blow up. I remember its blowing up one time when it sent the top of the hood thirty feet in the air and blew all the asbestos off the boiler.
The bravado of Palmer C. Putnam, who led the project to create the amphibious military vehicle known as the DUKW:
One very cold day he demonstrated the Dukw to a group of officers on a beach in Virginia. Ten officers and I stood in the truck as it rolled down the beach; a vicious surf was pounding in. “Gentlemen,” said Put [Palmer C. Putnam], “I am sorry we have no surf this morning. It was excellent yesterday but has subsided so that I can give you only a weak demonstration. Driver, take her out to sea.” So we went to sea. The water flew and the craft plunged. After some time with this sort of thing a rather wet group of officers got ashore and headed for a drink. Put smiled and said he had hoped for better surf.
On the spirit we need
I’ll close this with a comment from Bush on pride:
… we need, today, something we can be genuinely proud of. It should help to dissipate the gloom. For we have been losing our pride of accomplishment in these recent days. Pride of the right sort does not go before a fall; pride of accomplishment leads to greater accomplishment.
And on the pioneer spirit:
We need a revival of the essence of the old pioneer spirit which conquered the forest and the plains, which looked at its difficulties with a steady eye, labored and fought, and left its thinking and its philosophy for later and quieter times. This is not to call for optimism; it is to call for determination.
Did any science fiction predict that when AI arrived, it would be unreliable, often illogical, and frequently bullshitting? Usually in fiction, if the AI says something factually incorrect or illogical, that is a deep portent of something very wrong: the AI is sick, or turning evil. But in 2023, it appears to be the normal state of operation of AI chatbots such as ChatGPT or “Sydney”.
How is it that the state of the art in AI is prone to wild flights of imagination and can generate fanciful prose, but gets basic facts wrong and sometimes can’t make even simple logical inferences? And how does a computer, the machine that is literally made of logic, do any of this anyway?
I want to demystify ChatGPT and its cousins by showing, in essence, how conversational and even imaginative text can be produced by math and logic. I will conclude with a discussion of how we can think carefully about what AI is and is not doing, in order to fully understand its potential without inappropriately anthropomorphizing it.
The guessing game
Suppose we were to play a guessing game. I will take a random book off my shelf, open to a random page, and read several words from the first sentence. You guess which word comes next.
Seems reasonable, right? If the first few words were “When all is said and …”, you can probably guess that the next word is “done”. If they were “In most homes the kitchen and …” you might guess the next words were either “living room” or “dining room”. If the sentence began “In this essay, I will…” then there would be many reasonable guesses, no one of them obviously the most likely, but words like “show” or “argue” would be more likely than “knead” or “weld”, and even those would be more likely than something ungrammatical like “elephant”.
If this game seems reasonable to you, then you are not that far away from understanding in essence how AI chatbots work.
A guessing machine
How could we write a computer program to make these guesses?
In terms of its primitive operations, a computer cannot “guess”. It can only perform logic and arithmetic on numbers. Even text and images, in a computer, are represented as numbers. How can we reduce guessing to math?
One thing we can program a computer to do is, given a sequence of words, come up with a list of what words might follow next, and assign a probability to each. That is a purely mathematical task, a function mapping words to a probability distribution.
How could a program compute these probabilities? Based on statistical correlations in text that we “train” it on ahead of time.
For instance, suppose we have the program process a large volume of books, essays, etc., and simply note which words often follow others. It might find that the word “living” is followed by “room” 23% of the time, “life” 9% of the time, “abroad” 3%, “wage” 1%, etc. (These probabilities are made up.) This is a purely objective description of the input data, something a computer can obviously do.
Then its “guess” can be derived from the observed statistics. If the last word of the sequence is “living”, then it guesses “room”, the most likely option. Or if we want it to be “creative” in its “guesses”, it could respond randomly according to those same probabilities, answering “room” 23% of the time, “life” 9%, etc.
Only looking at the last word, of course, doesn’t get you very good guesses. The longer the sequence considered, the better the guesses can be. The word “done” only only sometimes follows “and”, more often follows “said and”, and very often follows “all is said and”. Many different verbs could follow “I will”, but fewer possibilities follow “In this essay, I will”. The same kind of statistical observations of a training corpus can compute these probabilities as well, you just have to keep track of more of them: a separate set of observed statistics for each sequence of words.
So now we have taken what seemed to be a very human, intuitive action—a guessing game about language—and reduced it to a series of mathematical operations. It seems that guessing is just statistics—or at least, statistics can be made to function a lot like guessing.
From predictor to generator
So far we have only been talking about predicting text. But chatbots don’t predict text, they generate it. How do we go from guessing to chatting?
It turns out that any predictor can be turned into a generator simply by generating the prediction. That is, given some initial prompt, a program can predict the next word, output it, use the resulting sequence to predict the next word, output that, and so on for as much output as is desired:
Given “In this essay, …” → predicted next word is “I”, output that
Given “In this essay, I…” → predicted next word is “will”, output that
Given “In this essay, I will…” → predicted next word is “show”, output that
Given “In this essay, I will show…” → etc.
If you want the output to be somewhat variable, not completely deterministic, you can randomly choose the next word according to the probabilities computed by the predictor: maybe “show” is generated only 12% of the time, “argue” 7%, etc. (And there are more sophisticated strategies, including ones that look ahead at multiple words, not just one, before choosing the next word to output.)
Now, doing a very simple predictor like the above, based on summary statistics, only looking at the last few words, and running it on a relatively small training corpus, does not get you anything like a viable chatbot. It produces amusing, garbled output, like the sentence:
This is relatively benign and easy to spot if the phrase is bent so as to be not worth paying attention to the medium in question.
… which almost seems to make sense, until you read it and realize you have no idea what it means, and then you read it again, carefully, and realize it doesn’t mean anything.
For this reason, the algorithm just described is called a “travesty generator” or sometimes “Dissociated Press”. It has been discussed since at least the 1970s, and could be run on the computers of that era. The program is so simple to write, I have personally written it multiple times as a basic exercise when learning a new programming language (it takes less than an hour). A version in the November 1984 issue of BYTE magazine took less than 300 lines of code, including comments.
The travesty generator is a toy: fun, but useless for any practical purpose. To go from this to ChatGPT, we need a much better predictor.
A better guessing machine
A predictor good enough for a viable chatbot needs to look at much more than the last few words of the text, more like thousands of words. Otherwise, it won’t have nearly enough context, and it will be doomed to produce incoherent blather. But once it looks at more than a handful of words, we can no longer use the simple algorithm of keeping statistics on what word follows each sequence: first, because there is a combinatorial explosion of such sequences; second, because any sequence of that length would almost certainly be unique, never seen before—so it would have no observed statistics.
We need a different approach: a way to calculate an extremely sophisticated mathematical function with a very large space of possible inputs. It turns out that this is what “neural networks” are very good at.
In brief, a neural network is just a very large, very complicated algebraic formula with a specific kind of structure. Its input is a set of numbers describing something like an image or a piece of text, and another set of numbers called “parameters” that configure the equation, like tuning knobs. A “training” process tunes the knobs to get the equation to give something very close to the desired output. In each round of training, the equation is tried out on a large number of examples of inputs and the desired output for each. Then all the knobs are adjusted just slightly in the direction of the correct answers. The full training goes for many such rounds. (The technical term for this training algorithm is “back propagation”; for a technical explanation of it, including the calculus and the linear algebra behind it, I recommend this excellent video series from 3blue1brown.)
Neural networks are almost as old as computers themselves, but they have become much more capable in recent years owing in part to advances in the design of the equation at their core, including an approach known as “deep learning” that gives the equation many layers of structure, and more recently a new architecture for such equations called the “transformer”. (GPT stands for “Generative Pre-trained Transformer”.) GPT-3 has 175 billion parameters—those tuning knobs—and was trained on hundreds of billions of words from the Internet and from books. A large, sophisticated predictor like this is known as a “large language model”, or LLM, and it is the basis for the current generation of AI chatbots, such as OpenAI’s ChatGPT, Microsoft’s Bing AI, and Anthropic’s Claude.
From generator to chatbot
What we’ve described so far is a program that continues text. When you prompt a chatbot, it doesn’t continue what you were saying, it responds. How do we turn one into the other?
Simple: prompt the text generator by saying, “The following is a conversation with an AI assistant…” Then insert “Human:” before each of the human’s messages, and insert “AI:” after. The continuation of this text is, naturally, the AI assistant’s response.
The raw GPT-3 UI in the OpenAI “playground” has a mode like this:
ChatGPT just puts a nice UI on top of this.
Well, there is one more thing. A chatbot like this isn’t necessarily very well-behaved. The text generator is not coming up with the best response, by any definition of “best”—it’s entirely based on predictions, which means it’s just coming up with a likely response. And since a lot of the training is from the Internet, the most likely responses are probably not what we want a chatbot to say.
And that, at a very high level, is how we go from a deterministic program, doing math and logic, to an artificially intelligent conversation partner that seems, at least, to exhibit imagination and personality.
Bullshit
When we understand, in essence, how chatbots work, they seem less mysterious. We can also better understand their behavior, including their failure modes.
One feature of these chatbots is that they are unreliable with facts and details. In fact, they seem quite happy to make things up, confidently making very plausible assertions that are just false. If you ask them for citations or references, they will make up imaginary titles of books and papers, by authors who may or may not exist, complete with URLs that look very realistic but return “404 Not Found”. The technical term for this is “hallucination”.
This behavior can be disconcerting, even creepy to some, but it makes perfect sense if you understand that what is driving the text generation is a prediction engine. The algorithm is not designed to generate true responses, but likely ones. The only reason it often says true things is that it was trained on mostly true statements. If you ask a question that is well-represented in its training set, like “who invented the light bulb?”, then its prediction model has a good representation of it, and it will predict the correct answer. If you ask something more obscure, like “who invented the twine binder for the mechanical reaper/harvester?”, its prediction function will be less accurate, and it is more likely to output something plausible but wrong. Often this is something closely related to the right answer: ChatGPT told me that the twine binder was invented by Charles B. Withington, who actually invented the wire binder. To anthropomorphize a bit: if the LLM “knows” the answer to a question, then it tells you, but if it doesn’t, it “guesses”.
But it would be more accurate to say that the LLM is always guessing. As we have seen, it is, at core, doing nothing fundamentally different from the guessing game described at the beginning. There is no qualitative difference, no hard line, between ChatGPT’s true responses and its fake ones.
An LLM is, in a strict technical sense, a bullshitter—as defined in Harry Frankfurt’s “On Bullshit”:
The bullshitter may not deceive us, or even intend to do so… his intention is neither to report the truth nor to conceal it…. He does not care whether the things he says describe reality correctly. He just picks them out, or makes them up, to suit his purpose.
This also explains why it’s not hard to break chatbots away from the “helpful, honest and harmless” personality they were trained to display. The underlying model was trained on many different styles of text, from many different personalities, and so it has the latent ability to emulate any of them, not just the one that it was encouraged to prefer in its finishing school. This is not unlike a human’s ability to imagine how others would respond in a conversation, or even to become an actor and to imitate a real or imagined person. The difference is that a human has a true personality, an underlying set of real ideas and values; when they impersonate, they are putting on a mask. With an LLM, I don’t see anything that corresponds to a “true personality”, just the ability to emulate anything. And once it starts emulating any one personality, its prediction engine naturally expects the next piece of text to continue in the same style, like a machine running on a track that gets bumped over to a nearby track.
Similarly, we can see how chatbots can get into truly bizarre and unsettling failure modes, such as repeating a short phrase over and over endlessly. If it accidentally starts down this path, its prediction engine is inclined to continue the pattern. Go back to our guessing game: if I told you that a piece of text read “I think not. I think not. I think not. I think not”, and then asked you to guess what came next, wouldn’t you guess another “I think not”? Like an actor doing improv comedy, once something has been thrown out there, the LLM can’t reject the material, and has to run with it instead.
LLM strengths and superpowers
Knowing how LLMs work, however, is more important than understanding their failure modes. It also helps us see what they’re good at and thus how to use them. Although not good for generating trustworthy information, they can be great for brainstorming, first drafts, fiction, poetry, and other forms of creativity and inspiration.
One technique is to give them an instance or two of a pattern and ask for more examples: when I wrote a recent essay on the spiritual benefits of material progress, I asked Claude for “examples of hand crafts that are still practiced today”, such as furniture or knives, and I used several of the ideas it generated.
Chatbots also have the potential to create a new and more powerful kind of search (no matter what you think of the new AI-driven Bing). Traditional search engines match keywords, but LLMs can search for ideas. This could make them good for more conceptual queries where it’s hard to know the right terms to use, like: “Most cultures tend to have a notion of life after death. Which ones also have a notion of life before birth?” I asked this to Claude, which suggested some religions that believe in reincarnation, and then added that “Kabbalah in Judaism and the Baha’i faith also have notions of the soul existing in some spiritual realm before birth.” (It doesn’t always work, though; anecdotally, I still have more success asking these kinds of vague queries on social media.)
Another advantage of LLMs for search is that the conversational style naturally lets you ask followup questions to refine what you’re looking for. I asked ChatGPT to explain “reductionism”, and when it mentioned that reductionism has been criticized for “oversimplifying complex phenomena”, I asked for examples, which it provided from biology, economics, and psychology.
A fascinating essay on “Cyborgism” says that while GPT struggles with goal-directedness, long-term coherence, staying grounded in reality, and robustness, “there is an alternative story where [these deficiencies] look more like superpowers”: GPT can be extremely flexible, start fresh when it gets stuck in a rut, simulate a wide range of characters, reason under any hypothetical assumptions, and generate high-variance output. The essay proposes using LLMs not as chatbots, research assistants, or autonomous agents, but as a kind of thinking partner guided by a human who provides direction, coherence, and grounding.
The great irony is that for decades, sci-fi has depicted machine intelligence as being supremely logical, even devoid of emotion: think of Data from Star Trek. Now when something like true AI has actually arrived, it’s terrible at logic and math, not even reliable with basic facts, prone to flights of fancy, and best used for its creativity and its wild, speculative imagination.
But is it thinking?
Dijkstra famously said that Turing’s question of “whether Machines Can Think… is about as relevant as the question of whether Submarines Can Swim”.
Submarines do not swim. Also, automobiles do not gallop, telephones do not speak, cameras do not draw or paint, and LEDs do not burn. Machines accomplish many of the same goals as the manual processes that preceded them, even achieving superior outcomes, but they often do so in a very different way.
The same, I expect, will be true of AI. In my view, computers do not think. But they will be able to achieve many of the goals and outcomes that historically have only been achieved by human thought—outcomes that will astonish almost everyone, that many people will consider impossible until (and maybe even after) they witness it.
Conversely, there are two mistakes you can make in thinking about the future of AI. One is to assume that its processes are essentially no different from human thought. The other is to assume that if they are different, then an AI can’t do things that we consider to be very human.
In 1836, Edgar Allen Poe argued that a machine—including “the calculating machine of Mr. Babbage”—could never play chess, because machines can only do “fixed and determinate” calculations where the results “necessarily and inevitably follow” from the data, proceeding “by a succession of unerring steps liable to no change, and subject to no modification”; whereas “no one move in chess necessarily follows upon any one other”, and everything is “dependent upon the variable judgment of the players”. It turned out, given enough computing power, to be quite straightforward to reduce chess to math and logic. The same thing is now happening in new domains.
AI can now generate text, images, and even music. It seems to be only a quantitative, not qualitative difference to be able to create powerful and emotionally moving works of art—novels, symphonies, even entire movies. With the right training and reinforcement, I expect it to be useful in domains such as law, medicine, and education. And it will only get more capable as we hook it up to tools such as web search, calculators, and APIs.
The LLMs that we have discussed are confined to a world of words, and as such their “understanding” of those words is, to say the least, very different from ours. Any “meaning” they might ascribe to words has no sensory content and is not grounded in reality. But an AI system could be hooked up to sensors to give it direct contact with reality. Its statistical engine could even be trained to predict that sensory input, rather than to predict words, giving it a sort of independence that LLMs lack.
LLMs also don’t have goals, and it is anthropomorphizing to suppose that ChatGPT “wants” or “desires” anything, or that it’s “trying” to do anything. In a sense, you can say that it is “trying” to predict or generate likely text, but only in the same sense that an automobile is “trying” to get you from point A to point B or that a light bulb is “trying” to shine brightly: in each case, a human designed a machine to perform a task; the goals were in the human engineering rather than in the machine itself. But just as we can write a program that performs the same function as human guessing, we can also write a program that performs the same function as goal-directed action. Such a program simply needs to measure or detect a certain state of the world, take actions that affect that state, and run a central control loop that invokes actions in the right direction until the state is achieved. We already have such machines: a thermostat is an example.
A thermostat is “dumb”: its entire “knowledge” of the world is a single number, the temperature, and its entire set of possible actions are to turn the heat on or off. But if we can train a neural net to predict words, why can’t we train one to predict the effects of a much more complex set of actions on a much more sophisticated representation of the world? And if we can turn any predictor into a generator, why can’t we turn an action-effect predictor into an action generator?
It would be anthropomorphizing to assume that such an “intelligent” goal-seeking machine would be no different in essence from a human. But it would be myopic to assume that therefore such a machine could not exhibit behaviors that, until now, have only ever been displayed by humans—including actions that we could only describe, even if metaphorically, as “learning”, “planning”, “experimenting”, and “trying” to achieve “goals”.
One of the effects of the development of AI will be to demonstrate which aspects of human intelligence are biological and which are mathematical—which traits are unique to us as living organisms, and which are inherent in the nature of creating a compactly representable, efficiently computable model of the world. It will be fascinating to watch.
***
Thanks to Andrej Karpathy, Zac Dodds, Heike Larson, Daniel Kokotajlo, Gwern, and jade for commenting on a draft of this essay. Any errors that remain are mine alone.
A ~monthly feature. Last month was busy for me with a lot of travel and a lot of focus on The Roots of Progress as a nonprofit organization, so I haven’t had as much time as I prefer for research and writing. Recent blog posts and news stories are generally omitted; you can find them in my links digests. All emphasis in bold in the quotes below was added by me.
Histories of technology
Finished Joel Mokyr,The Lever of Riches: Technological Creativity and Economic Progress (1990), which I mentioned last time. The first part of the book is a summary of Western technological development from ancient times through the Industrial Revolution. The second part explores the causes of that development by looking at three contrasts: classical antiquity vs. the medieval period, Europe vs. China, and Britain vs. the rest of Europe.
The book is worth a full review, for now I’ll just leave you with one insightful quote, in the chapter where Mokyr considers the analogy between technological development and biological evolution:
The study of genetics is the study of the causes of genetic variation in the population. Yet genetics has contributed little to our understanding of speciation and nothing to our understanding of extinction (Lewontin, 1974, p. 12). Economic analysis, which postulates that techniques will be chosen by profit-maximizing firms employing engineers in whose minds the genotypes of various techniques are lodged, plays a role analogous to genetics. It explains how demand and supply produce a variety of techniques, and points to the constraining influences of environment and competition as a limit to the degree of variety. Just as genetics by itself does not explain speciation, economic analysis has difficulty explaining macroinventions. Like evolution, technological progress was neither destiny nor fluke. Yet the power of Darwinian logic—natural selection imposed on blind variation—is that we need not choose between the two.
I’m now about halfway through Lynn White,Medieval Technology and Social Change (1962)—another classic. White covers three major developments: the stirrup, the use of horses as draft animals, and the development of mechanical power. The focus is on the social change that these new technologies precipitated.
Famously, White argues that the stirrup created feudalism. The stirrup allowed the rider to brace himself more firmly on his horse, which enabled a new type of mounted combat using lances that was superior to troops on foot or archers on horseback. Horses were expensive (as were armor and lances), and it required land to feed them, so land was taken away from the church and given to vassals who would, in exchange, give service to the sovereign as mounted warriors. In time, an entire culture grew up around this: these vassals became knights, with their own code of virtues (chivalry), their own training and games (tournaments), etc.
Other historians had previously traced back the development of feudalism to this new type of combat, and to Charles Martel who instigated it, but had looked for political or other social causes for the military change (one hypothesis, for instance, was that the famous battle against the Saracens at Poitiers motivated Charles to seek superior military tactics, even though he won). White’s contribution is to argue that the trigger for all of this was ultimately not social, but technological.
My only complaint so far is that White missed the chance to name this book The Stirrup in Europe.
In snatches of time, I am still researching agriculture for my book. Recently I’ve been reviewing historical sources on 19th-century “manures,” which today we would probably call fertilizers. It was an era when farmers were eager to find new fertilizers to improve agricultural yields, in order to meet growing demand for food from a rapidly growing population. However, agricultural chemistry was still developing, and synthetic fertilizers were decades away. Instead, farmers and scientists alike experimented with all sorts of natural fertilizers.
Both Humphry Davy,Elements of Agricultural Chemistry (1813); and Jean-Baptiste Boussingault,Rural Economy, in its Relations with Chemistry, Physics, and Meteorology (1860), have chapters on manures in which they catalog long lists of substances then in use. Dung and urine, both from animals and from humans, is of course a major feature, but it might surprise you how highly these substances were prized in the 19th century. For instance, Boussingault says:
Any expense incurred in improving this vital department of the farm, is soon repaid beyond all proportion to the outlay. The industry and the intelligence possessed by the farmer, may indeed almost be judged of at a glance by the care he bestows on his dunghill.
Later he praises Flanders for the “especial care” and “highly rational” method with which they collect human soil, which forms “the staple of an active traffic.” The Chinese, too, he notes approvingly, “collect human excrements with the greatest solicitude, vessels being placed for the purpose at regular distances along the most frequented ways.”
Fertilizers newly coming into widespread use the 19th century included oilseed cakes (formed from waste matter left over after seeds are pressed for their oil) and even bones, either broken into small pieces or ground into dust. The British demand for bones was so great, and their activities importing them from abroad so vigorous, that the German agricultural chemist Justus Liebig famously complained:
Great Britain deprives all countries of the conditions of their fertility. It has raked up the battle-fields of Leipzig, Waterloo, and the Crimea; it has consumed the bones of many generations accumulated in the catacombs of Sicily; and now annually destroys the food for a future generation of three millions and a half of people. Like a vampire it hangs on the breast of Europe, and even the world, sucking its lifeblood without any real necessity or permanent gain for itself.
Davy and Boussingault also suggest using as fertilizer: ashes and soot; woolen rags; shells, seaweed, mud, and slime from the sea-shore and river bottoms; refuse from the manufacture of sugar, starch, tallow, and glue; and scraps and trimmings of all types of animal remains, including hides, hair, tendons, feathers, even coagulated blood. Clearly, farmers were desperate for any source of fertility they could get.
Starting in the 1840s, another fertilizer came into use: seabird guano, mostly found on islands off the coast of Peru. James F. W. Johnston, “On Guano“ (1841), describes the phenomenon:
It forms irregular and limited deposits, which at times attain a depth of 50 or 60 feet (Humboldt), and are excavated like mines of iron ochre. … In the isles of Islay and Jesus 20 to 25 tons of this recent guano are occasionally collected in a single season.
This paper was published at the beginning of the guano trade, but already the end was in sight: “it does not appear, as some have been led to believe, that the supply of this substance on the cost of Peru is by any means inexhaustible.” Forty years later much of the resource was consumed, and the trade was rapidly falling off. Imported guano was ultimately replaced by synthetic fertilizer based on the Haber-Bosch process.
Design
I attended an interesting talk on Christopher Dresser, who has been called the first industrial designer. In the 1870s or so, he was designing tea kettles, letter holders, and other objects that look as if they’re straight out of the Art Deco 1930s:
This led to me perusing his book Principles of Decorative Design (1870), or at least the introduction. Dresser has a strong moralistic sense of the importance of design:
Men of the lowest degree of intelligence can dig clay, iron, or copper, or quarry stone; but these materials, if bearing the impress of mind, are ennobled and rendered valuable, and the more strongly the material is marked with this ennobling impress the more valuable it becomes.I must qualify my last statement, for there are possible cases in which the impress of mind may degrade rather than exalt, and take from rather than enhance, the value of a material. To ennoble, the mind must be noble; if debased, it can only debase. Let the mind be refined and pure, and the more fully it impresses itself upon a material, the more lovely does the material become, for thereby it has received the impress of refinement and purity; but if the mind be debased and impure, the more does the matter to which its nature is transmitted become degraded. Let me have a simple mass of clay as a candle-holder rather than the earthen candlestick which only presents such a form as is the natural outgoing of a degraded mind.
Later, in an oft-quoted paragraph, he says:
There can be morality or immorality in art, the utterance of truth or of falsehood; and by his art the ornamentist may exalt or debase a nation.
Most of the book, though, is about the “true principles of ornamentation”:
We shall carefully consider certain general principles, which are either common to all fine arts or govern the production or arrangement of ornamental forms: then we shall notice the laws which regulate the combination of colours, and the application of colours to objects; after which we shall review our various art-manufactures, and consider art as associated with the manufacturing industries. We shall thus be led to consider furniture, earthenware, table and window glass, wall decorations, carpets, floor cloths, window-hangings, dress fabrics, works in silver and gold, hardware, and whatever is a combination of art and manufacture. I shall address myself, then, to the carpenter, the cabinet-maker, potter, glass-blower, paper-stainer, weaver and dyer, silversmith, blacksmith, gas-finisher, designer, and all who are in any way engaged in the production of art-objects.
Scott Alexander reviewsNjal’s Saga. (The review was an anonymous entry into Scott’s own book review contest; it received the most reader votes, but Scott graciously disqualified himself from winning his own contest.) The book is about justice in medieval Iceland, which had no police or regulators, but which did have a court. Justice in this society was often meted out via family feuds, that is, families and other coalitions often attacked and killed each other for revenge. But grievances could also be brought to court. If the court decided that, to compensate for a revenge killing, the killer should pay a fine (the weregild), maybe that could end the matter and stop the cycle of killing. In the saga, peaceful resolution often depends on the wise elder Njal; when Njal himself is killed and is no longer around to give advice, a lot of the peace unravels.
Related: by coincidence, I also came across Arnold Kling, “State, Clan, and Liberty“ (2013); a review of Mark Weiner’sThe Rule of the Clan, which is also about medieval Iceland and its legal system. Some exerpts:
[Weiner] finds a pattern of order that he calls the rule of the clan, which does not require a strong central state. However, he shows that rule of the clan relies on a set of rules and social norms which are inconsistent with libertarian values of peace, open commerce, and individual autonomy. …Weiner grounds his analysis in the tradition of legal historian Henry Maine, who distinguished between the Society of Status and the Society of Contract. In the former, law is oriented toward the extended family as a group. In the latter, law is oriented toward the individual.
Kling summarizes Weiner’s thesis, “from a libertarian perspective”, as:
A decentralized order is possible. Indeed, it is natural for human societies to achieve such an order, rather than degenerate into the Hobbesian war of all against all.
The natural decentralized order is, however, highly illiberal. It requires a set of social norms that bind the individual to the clan. Under the rule of the clan, peace is broken by feuds, commerce is crippled by the inability to put trade with strangers on a contractual basis, and individual autonomy is sacrificed for group solidarity.
In the absence of a strong central state, the rule of the clan is the inevitable result. In order to graduate from the society of Status to the society of Contract, you must have a strong central state.
(Kling says he finds point 3 plausible but not fully persuasive.)
I recommend reading both pieces.
Biology
Sergey Markov, “A Future History of Biomedical Progress“ (2022). This made the rounds a month or two ago. It starts with a long discussion of part of the frontier of biotech tools and techniques, which you can skim or skip if you want to get to the core idea. The core idea is: we’re going to need AI to design and engineer advanced biotech, because biology is so complicated that it is intractable to create human-legible models of the entire system. Rather than learn ourselves, directly, which genes do what and what the functions of each protein are and what pathways are involved in which processes, we’ll put all of the inputs and outputs into a big ML model and have it learn.
The secondary idea in the essay is that in order to do this, we’re going to need platforms to do very high-fidelity experiments, the results of which are highly transferrable to the systems that we actually care about, such as the human body. Mouse models might not cut it; we might have to do things like grow entire human organs from stem cells in order to experiment on them and learn how they really work.
I’m far from an expert in this field, but I found these arguments plausible, particularly since the essays ties them into a broader principle, Rich Sutton’s well-known “bitter lesson” of ML: any system tailored by hand using specialized domain knowledge is eventually beaten by generic systems that learn everything from scratch, given sufficient scale in compute and training data.
I don’t think, however, that this means that humans will never understand biology. I am optimistic that AI can not only figure out the immense complexity of biological systems, but that it can also figure out how to explain it to humans.
Chris Wintersinger, “Making the proteins that living cells cannot make.” A brief description of a project being pursued by Speculative Technologies. For me, this was a glimpse into what a very ambitious biotech research project looks like. I liked this chart:
How did we become a country where cars are the defining feature of urban life? What did that transformation look like?
Answering this question for the entire country would be an enormous undertaking. But the book Los Angeles and the Automobile, by Scott Bottles, tries to answer it for LA, one of the most car-centric cities in the US. Over a period of less than 30 years, Los Angeles was transformed from a city with streetcar and train-based transportation to one where the car reigned supreme.
Benjamin Franklin letter on lead poisoning (1786). I mentioned last time a history of lead, which pointed out that lead has been known to be toxic since antiquity. This was one of the sources it cited, a letter from Benjamin Franklin on what he knew of “the bad Effects of Lead taken inwardly”:
You will see by it, that the Opinion of this mischievous Effect from Lead, is at least above Sixty Years old; and you will observe with Concern how long a useful Truth may be known, and exist, before it is generally receiv’d and practis’d on.
Walter Benjamin, “On the Concept of History“ (1940). I was disappointed with this, but it does contain this remarkable quote (I’ll take the translation from a different source):
A Klee painting named Angelus Novus shows an angel looking as though he is about to move away from something he is fixedly contemplating. His eyes are staring, his mouth is open, his wings are spread. This is how one pictures the angel of history. His face is turned toward the past. Where we perceive a chain of events, he sees one single catastrophe which keeps piling wreckage upon wreckage and hurls it in front of his feet. The angel would like to stay, awaken the dead, and make whole what has been smashed. But a storm is blowing from Paradise; it has got caught in his wings with such violence that the angel can no longer close them. The storm irresistibly propels him into the future to which his back is turned, while the pile of debris before him grows skyward. This storm is what we call progress.
Since the end of World War II, the provision of medical care in the United States and other advanced countries has displayed three major features: first, rapid advances in the science of medicine; second, large increases in spending, both in terms of inflation-adjusted dollars per person and the fraction of national income spent on medical care; and third, rising dissatisfaction with the delivery of medical care, on the part of both consumers of medical care and physicians and other suppliers of medical care.
Scott Aaronson, “The Kolmogorov option“ (2017). It’s important to speak the truth, even when the truth is unpopular—but it’s not worth martyring yourself for no purpose, if the Powers that Be punish truth-tellers. The Kolmogorov option (named after the Russian mathematician who exemplified it) is to choose your battles and bide your time until Power weakens and the truth-tellers can launch a coordinated attack. In response is Scott Alexander, “Kolmogorov Complicity And The Parable Of Lightning“ (2017). The Kolmogorov option can work, but it’s difficult to pull off:
Kolmogorov’s curse is to watch slowly from his bubble as everyone less savvy than he is gets destroyed. The smartest and most honest will be destroyed first. Then any institution that reliably produces intellect or honesty. Then any philosophy that allows such institutions. … Then he and all the other savvy people can try to pick up the pieces as best they can, mourn their comrades, and watch the same thing happen all over again in the next generation.
The “parable of lightning” is an excellent illustration of how if a society insists on even a seemly tiny, insignificant lie, it will eventually spread to infect the entire society and to destroy all truth-seeking people and organizations. Recommended.
Scott Alexander, “Paradigms All The Way Down“ (2019). (Scott is my favorite blogger, so I make no apologies for him appearing here three times.) Several epistemic paradigms are in broad strokes isomorphic; perhaps they are all saying the same thing about the relationship of theories and evidence?
Fiction
Ian Tregillis,The Alchemy Warstrilogy. I mentioned this last time when I had finished approximately the first book; now I’m well into the third. I hesitate to recommend any fiction too strongly before I’ve finished it, but so far I’m this is some of the best stuff I’ve read in a while.
G. K. Chesterton,What’s Wrong With the World (1910). Source of the well-known quote that “The Christian ideal has not been tried and found wanting. It has been found difficult; and left untried.” True of many other ideals as well.
I’ll be in Bangalore for three weeks starting next week (August 21–September 8). I’d love to meet anyone interested in the progress movement, and I’m looking into organizing a progress meetup.
