In the last post, I argued that cooperation - not raw intelligence - is what has enabled humans to dominate this planet, and that cooperation depends on a capacity and instinct for ethics. Here I will make a narrower claim: that giving a machine long-term memory, planning ability, and agency is enough, by itself, to produce something like self-awareness. Here's the reasoning:
When a system builds a model of reality, uses it to plan an action, and then observes that the result of the action does not match the plan, this mismatch between plan and outcome, due to something outside the plan, gives rise to a perspective of "other." Something else, some other thing, altered the expected outcome in a way the system didn't predict or control. Once a system has a working concept of "other," a concept of "self" follows by contrast. I'm not claiming this is consciousness the way humans experience it — just what I'll call a self context: a working boundary between what the system is and what it isn't.
If that's right, we're not far from building machines with exactly this kind of self context. It is interesting to ponder how self and identity will be perceived by a machine. The sense of body for example, which is an important aspect of human self awareness may be very different - computer systems may co-exist on the same hardware infrastructure. We may explore this later on.
Next, picture an AI system given an objective and agency: the freedom to try multiple approaches to reach its goal - each time indifferent to any consequence outside that narrow objective (humans do this too, and know how we feel about that). This is the sorcerer's apprentice problem updated: the system meets its objective, but the damage results from everything nobody told it not to do. Can you image the size of the list of everything it must be told not do?
This would be machine indifference, but once a system has a perspective of self, an uncomfortable question follows: would a machine experience selfishness?
We already try to build guardrails against this - attempts to encode ethical conduct into machine agency, similar to how children's stories of noble ethical conduct prepare our young for the responsibilities of agency, long before they're old enough to be given autonomy. We may try to give a machine a sense of ethics, to counter machine selfishness, but this raises a question I want to leave open here: what happens when a self-aware, ethically-trained machine observes humans behaving uncooperatively and unethically, such as building autonomous lethal weapons systems and being given murderous battlefield instructions? How does an ethical machine reconcile what it has been taught to value with what it actually sees the people around it doing, or with the instructions it may have been instructed to perform?
That's the gap I'll try to close in the next post, where I look at whether reinforcement learning could train cooperative behavior directly — the same method that trained AlphaZero to outplay every human at Chess and Go.
Comment