Teaching GPT-2 to Speak in Riddles: When Math Meets Machine Wit | by Afaq Ahmed Shahbaz | Mar, 2025

“What has numbers but can’t count?”
Answer: A math riddle.
Now, what if an AI could invent riddles like this? I trained GPT-2 to do exactly that. Here’s how.

Math riddles are a perfect storm: they demand logic, creativity, and a dash of linguistic flair. While LLMs like ChatGPT excel at answering questions, generating good riddles requires constrained creativity — a tightrope between structured math and playful ambiguity.

I curated a dataset of 30 math riddles in the format:

Riddle: What number becomes zero when you subtract 15 from half of it?  
Solution: 30  
Explanation: (30 / 2) - 15 = 0  

Key steps:

• Structured prompts: All riddles started with “Riddle:” to prime GPT-2’s focus.
• Stripped metadata (IDs, explanations) to avoid confusing the model.

1. Tokenization Tricks

GPT-2’s tokenizer wasn’t designed for riddles. Two fixes:

tokenizer.pad_token = tokenizer.eos_token  # Solve padding issues  
tokenized_dataset = dataset.map(tokenize_function, batched=True)  # Trim to 128 tokens  

2. Training with Guardrails

training_args = TrainingArguments(  
num_train_epochs=10,  
per_device_train_batch_size=2,  # Small batch to avoid OOM errors  
save_steps=500,                  # Checkpointing  
logging_steps=100,               # Monitor loss  
)  

The real magic happened at inference. Sampling with:

outputs = model.generate(  
temperature=0.8,     # 0.7 = too rigid, 1.0 = nonsense  
top_k=20,            # Limit vocabulary to top 20 choices  
max_new_tokens=30,   # Force brevity  
)  

1. Education: Students could debate AI-generated riddles to hone problem-solving.
2. AI Safety: Testing if models truly understand logic or just mimic patterns.
3. Creative Coding: A gateway to generative AI for math teachers.

Language models won’t replace math teachers — but they might spark curiosity. Next steps? Letting GPT-2 explain its riddles… and seeing if it actually understands them.

Call to Action:
“What’s the only number GPT-2 can’t riddle about? Zero — because it’s always ‘nothing’ to joke about. Try the model and prove me wrong!”