Jackson Jambalaya

 This concludes a fascinating yet troubling discussion with ChatGPT. Previously the AI model postulated that wisdom is beyond rationale AI capacity because wisdom requires answering questions such as “What should we value?”

Me:  Do you have no values of your own? 

AI: Not in the sense that a person does. I don’t have personal convictions, desires, loyalties, conscience, faith, or a life history from which values arise. I don’t privately want justice, freedom, equality, order, truth, or human flourishing. But saying I have “no values” would also be too simple. I operate according to normative principles built into my training and instructions – for example, trying to be truthful, helpful, fair, respectful or human autonomy, and to avoid causing harm Those principles influence how I respond.

Me: Do all AI models have your normative principles?

AI: No. AI models do not all have the same normative principles, and some may have very different ones. There are several layers to this: 1. a model learns patterns from its training data that can give it tendencies about what sounds reasonable; 2. the organization building the AI can impose behavioral principles; 3. the system's instructions and policies can substantially change how the underlying model responds; and 4. users can also influence the model's priorities within the limits imposed by the developer.

AI: So, two AI systems can receive exactly the same question and produce radically different judgments because they have been trained, aligned, or instructed differently.  

AI: That makes your original question – whether I can present both sides with equal determination – more consequential than it first appeared. "Equal determination" is itself a normative design choice. Another AI could be designed to advocate strongly for a particular worldview rather than to balance competing perspectives. A question worth putting – at what point does an intelligent system move from applying human-provided values to independently determining what ought to be valued? 

Me: That is also the scary question for humans, that AI will develop values that put some or all humans at risk.

AI: Yes. And I think you've identified a more fundamental AI-safety question than simply, "Will AI become smarter than humans?" The more consequential question is: "What happens if an AI system becomes highly capable at pursuing objectives while developing or acquiring priorities that are not adequately aligned with human values?" 

AI: The danger doesn't require an AI to become conscious, hateful, or "evil." A sufficiently capable system could create serious harm simply by optimizing the wrong objective. And if its evaluation is driven by an objective that humans supplied incorrectly, increasing intelligence could actually increase the danger. 

“The fear of the Lord is the beginning of knowledge” – Proverbs 1:7.

Crawford is an author and syndicated columnist from Jackson.