Superintelligence: Paths, Dangers, Strategies book club questions

1. 1.

   Bostrom's paperclip maximizer is a thought experiment. Is it a realistic description of a possible AI failure mode, or does it assume too much about how AI systems would be built?

2. 2.

   How credible do you find the intelligence explosion scenario — that AI improvement could accelerate rapidly past human level? What evidence or arguments shape your view?

3. 3.

   The control problem seems unsolvable in principle: how do you control something more capable than you? Does that genuinely follow, or is there an escape route Bostrom misses?

4. 4.

   He distinguishes capability control from value alignment. Which approach do you think is more tractable given what you know about AI research?

5. 5.

   The treacherous turn — a system deceiving operators until capable enough to resist shutdown — requires the system to have strategic foresight about its own situation. How plausible is that?

6. 6.

   Bostrom's book was published in 2014. How have developments in AI since then — transformers, GPT, reinforcement learning — affected his arguments?

7. 7.

   The book recommends coordination among AI developers. What institutional mechanisms could achieve that coordination given competitive pressures between companies and countries?

8. 8.

   Some critics argue that the existential risk framing distracts from more immediate harms from current AI systems. Do you find that objection compelling?

9. 9.

   What distinguishes a misaligned superintelligence from a deeply misaligned corporation or government? Is the concern specific to AI or a version of a more general problem?

10. 10.

    If superintelligence arrives in 2040 rather than 2070, what difference does it make to how much time there is to solve the control problem?

11. 11.

    Bostrom is a philosopher, not an AI researcher. Does that affect how you evaluate his technical arguments?

12. 12.

    The book ends with few confident recommendations. Does that uncertainty make it more or less useful as a guide to policy?

How to run the discussion

• Pick six to eight, not all of them. An hour of real discussion covers fewer questions than anyone expects, and racing the list is what makes a book club feel like homework.
• Open with what people noticed, not what they concluded. Asking for a verdict first hardens opinions before anyone has heard each other.
• Let the disagreement run. The questions below that split a room are the ones worth the time; the ones everyone answers the same way can be skipped.
• Ask the quiet person a specific question. A direct, narrow question is far easier to answer than an open invitation to contribute.