Why AGI could not be (just) a tool: goals, life, and general intelligence

It is widely believed that AGI has the potential to be a wonderful tool that humans can use to meet our needs, solve our problems, and improve our lives. Against this view, we argue that any entity with truly general, human-level intelligence would have the capacity to lead its own life, with its own purposes and integrated hierarchy of goals. And thus any true AGI could not be merely a tool, even if it turned out to be extremely helpful for human beings. If we are correct, there is a dilemma at the heart of the ambition to build AGI as a valuable tool. On the one hand, any mere tool that we might build would lack capacities essential to the kind of general intelligence exhibited by human beings; it would not be genuine AGI. On the other hand, were we to create genuine AGI, then what we would have created would not be a mere tool, but something more. In this paper, we make the case for this dilemma. In so doing, we illuminate the connections between a set of core ideas: intelligence, agency, tools, and life.