# RLHF (Reinforcement Learning from Human Feedback) **RLHF** is the training technique that turns a raw language model — a system that predicts text — into an AI assistant that follows instructions, refuses harmful requests, and produces responses humans find helpful. It is the bridge between a [[Foundation Model]]'s raw capability and the deployed product that millions of people interact with daily. The technique was developed at OpenAI (building on earlier DeepMind work) and first demonstrated publicly with InstructGPT (January 2022), then scaled dramatically with ChatGPT (November 2022). The core idea: after pre-training a model on internet text, you fine-tune it using human preferences as the reward signal, rather than just next-word prediction. ## How It Works RLHF proceeds in three stages: **Stage 1 — Supervised Fine-Tuning (SFT).** Human labelers write examples of ideal assistant behavior: given a prompt, here's what a good response looks like. The model is fine-tuned on these examples to shift from "predict the next word" to "respond helpfully." **Stage 2 — Reward Model Training.** The model generates multiple responses to the same prompt. Human raters rank them (A is better than B is better than C). These rankings train a separate "reward model" — a model that learns to predict which responses humans prefer. **Stage 3 — Reinforcement Learning.** The main model generates responses, the reward model scores them, and the main model is updated to produce higher-scoring responses. This loop runs for thousands of iterations, gradually shaping the model's behavior toward human preferences. Variants and successors include DPO (Direct Preference Optimization), which skips the separate reward model; RLAIF (RL from AI Feedback), where an AI evaluates responses instead of humans; and Anthropic's [[Constitutional AI]], which uses a written constitution to guide the feedback process. ## The Labor Behind the Curtain RLHF depends on large-scale human annotation — thousands of people rating model outputs, often for low wages. Time magazine's January 2023 investigation revealed that OpenAI's RLHF pipeline relied on Kenyan workers employed by Sama, paid less than $2/hour, rating content that included graphic descriptions of violence, sexual abuse, and self-harm. The workers described the experience as traumatic. This is the labor structure of RLHF: the AI systems that feel polished and safe to Western consumers are trained on judgments made by low-paid workers in the Global South, often exposed to the worst content the internet produces. The "human" in "human feedback" is not an abstraction — it is a specific labor force with specific working conditions that the AI industry has been reluctant to discuss. The shift toward RLAIF (AI-generated feedback) and automated evaluation partially reduces this dependency but introduces a different problem: AI systems training on their own judgments, with the biases and blindspots of the initial RLHF round baked in and potentially amplified. ## The Values Embedding Problem RLHF is where values enter AI systems. The human raters' preferences — what counts as "helpful," what counts as "harmful," what counts as politically neutral — become the model's behavior. This makes RLHF the site of the most consequential editorial decisions in AI: it determines what the model will and won't say, how it frames contested topics, and whose sensibilities it defers to. The raters' instructions (the "rating rubric") are proprietary to each lab and not publicly disclosed in detail. This means the values embedded in ChatGPT, Claude, and Gemini are shaped by documents that users never see, written by teams whose composition and incentives are opaque. When critics accuse AI models of political bias — from either direction — they are usually objecting to RLHF decisions, whether they know it or not. ## KB Significance 1. **The control layer.** RLHF is where AI companies exercise editorial control over their models — deciding what the model will refuse, how it handles political topics, and what it treats as dangerous. For the KB's analysis of tech-oligarchy power, RLHF is the mechanism through which a small number of organizations shape the informational environment for hundreds of millions of users. 2. **The hidden labor economy.** The RLHF labor pipeline — low-wage workers in Kenya, the Philippines, India, and elsewhere rating traumatic content — is a microcosm of the AI industry's broader labor dynamics: value created by many, captured by few, with the most unpleasant work outsourced to the lowest-cost jurisdictions. 3. **The bias battlefield.** Every political argument about AI bias — "too liberal," "too conservative," "too cautious," "not cautious enough" — is ultimately an argument about RLHF. The technique is the interface between societal values and model behavior, and there is no neutral setting. ## See Also - [[Large Language Model]] — What RLHF is applied to - [[Constitutional AI]] — Anthropic's RLHF variant - [[Fine-Tuning]] — The broader category of post-training adaptation - [[AI Safety]] — The governance framework RLHF serves - [[Alignment]] — The theoretical problem RLHF attempts to solve