AI Intelligence Testing Framework

A framework for understanding and testing AI intelligence, specifically the distinction between crystallized and fluid intelligence based on Emmett Shear's analysis.

Core Intelligence Types in AI

• Crystallized Intelligence

  • Excellent at combining existing knowledge
  • Strong pattern matching from training data
  • Good at linear interpolation between known concepts
  • Performs well on previously seen problems or combinations
  • Can effectively blend multiple known concepts together

• Fluid Intelligence

  • Ability to solve novel problems
  • Requires logical reasoning from first principles
  • Currently much weaker in AI systems
  • Struggles with truly new combinations of concepts
  • Cannot reliably think through new scenarios

Testing Framework Example

• Gear Problem Test Case
  • "7 gears on wall, alternating direction, flag on 7th gear"
  • Tests ability to work through novel mechanical problem
  • Requires step-by-step logical reasoning
  • Current AI struggles despite understanding basic gear principles
  • Shows limitation in fluid intelligence despite strong knowledge base

Key Observations About Current AI

• Performs exceptionally well when:

  • Drawing from existing knowledge
  • Combining known patterns
  • Working with explicit human knowledge
  • Making predictions based on training data

• Struggles with:

  • True novel problem solving
  • Original logical reasoning
  • Problems requiring step-by-step deduction
  • Scenarios outside training distribution

Testing Recommendations

• Need new IQ tests specifically for AI

  • Must be novel problems AI hasn't seen
  • Should test fluid intelligence capabilities
  • Focus on logical reasoning abilities
  • Avoid problems that can be solved through pattern matching

• Evaluation metrics should:

  • Distinguish between memorized vs. reasoned solutions
  • Test genuine problem-solving capabilities
  • Measure ability to handle truly novel scenarios
  • Track improvement in fluid intelligence over time