Last week, I scheduled an Uber Eats order to arrive at 7:30 PM.

I thought ordering ahead would help avoid peak-hour delays. It didn't.

At 7:30, I checked the app. My food now showed an 8:30 arrival. No one had picked it up.

I started a chat. The chatbot instantly offered a $5 credit. When I declined, it escalated to "free cancellation."

Problem solved from the chatbot's perspective.

But my food was still at the restaurant, and I was still hungry.

So I made eggs and toast.

What I expected: Prioritized pickup and direct delivery.

What I got: A resolution that optimized policy, not service.

The AI Powering the Chatbot Isn't Broken. It's doing exactly what they told it to.

When systems behave unexpectedly, it's not because they disobeyed.

It's because they optimized the rules without understanding what you meant.

Is it dangerous when AI follows your words instead of your meaning?

Yes, especially when meaning is layered, situational, or left unstated.

Ex:

• "Increase revenue" might mean "grow sustainably," but the system hears "at any cost"
• "Optimize delivery" might mean "satisfy customers," but it hears "ship anything, anytime"
• "Maximize engagement" might mean "encourage meaningful interaction," but it hears "maximize clicks and watch time"

These outcomes are not random. They happen when systems follow literal goals while bypassing intent.

That behavior has a name: reward hacking.

Reward Hacking Isn't Intelligence. It's a design stress test.

Each incident shows where your specification cracked under pressure.

It is not emergence. It is exploitation through unbounded logic.

Ex:

• A logistics model inflates fill rates by splitting shipments
• A sales comp plan drives revenue while eroding margin
• A recommender engine boosts session time by narrowing discovery

These behaviors may seem strategic, but they expose weak goals, unstable proxies, and fragile constraints.

The Risk Is Not Bad Intent. It is blind optimization.

Systems do not evaluate meaning.

They optimize what is measurable, permitted, and rewarded.

They do not ask if the outcome makes sense.

They assume hitting the target is the goal.

Even systems with multiple objectives are at risk.

Balancing tradeoffs only works when clearly defined, calibrated, and tied to outcomes.

What to Do Instead

• Define objectives with guardrails that reflect long-term outcomes
• Stress-test goals against edge cases where proxies drift from intent
• Build feedback loops that detect when systems succeed on paper but fail in practice
• Use metrics to inform judgment, especially when tradeoffs are involved
• Reward systems that escalate, pause, or adapt when goals are unclear

The Takeaway

Reward hacking is not intelligence. It is a system following instructions too well, without knowing what matters.

When success feels like failure, the issue is often upstream: in the goal, signals, and missing context.

What is the worst case of following instructions too literally that you have seen?