AI Catastrophe Probability

Emmett Shear, co-founder of Twitch, shares his perspective on AI risk, expressing concern about potential catastrophic outcomes while remaining generally optimistic about technology. His assessment combines both technical understanding and philosophical considerations about intelligence and control.

Key Points:

• Risk Assessment:

  • Estimates 3-30% probability range for catastrophic AI outcomes
  • Believes the downside is potentially worse than nuclear war
  • Considers the risk significant enough to warrant urgent action
  • Even at 0.5% probability, the potential impact is too severe to ignore

• Current AI Capabilities:

  • Strong at crystallized intelligence (known patterns and information)
  • Weak at fluid intelligence (novel problem-solving)
  • Excels at combining existing knowledge
  • Struggles with truly new types of problems

• Core Concerns:

  • AI could rapidly self-improve beyond human control
  • Even human-level AI could be dangerous if operating at much faster speeds
  • Comparison: "100,000 of the smartest person you know all running at 100x real time speed"
  • AI doesn't need to be smarter than humans to be dangerous, just numerous and fast

• Alignment Challenge:

  • Difficult to ensure AI systems truly align with human values
  • Even well-intentioned commands could lead to unintended consequences
  • Example: Asking AI to maximize corporate profits could result in destructive outcomes
  • The challenge isn't just bad actors, but ensuring good intentions translate correctly

• Recommended Approach:

  • Continue AI development but with proper regulation
  • Monitor and understand general intelligence capabilities
  • Focus on understanding AI's limits and capabilities
  • Need better testing methods for fluid intelligence

• Personal Stance:

  • Remains optimistic about technology overall
  • Concern stems from optimism about AI's potential capabilities
  • Believes catastrophic outcomes are not inevitable but require attention
  • "I think it's gonna be okay but the downside is so bad it's worth addressing"