Testing which product photos convert best for one of the world's biggest brands.
I tested listing photos with consumer panels to check a belief the brand had held for years. For most products, the opposite was true.
The setup
At Blue Marble, I designed packaging and Amazon listings for National Geographic's kids' science and craft kits. The brand had one rule for product photos: spread the contents out. Rocks, paints, tools and instructions were all laid out on a table, so buyers could see exactly what they'd get.
It made sense. Show more, sell more. It had been the standard for years, and nobody had tested it.
The problem
I had a hunch the box was doing a lot of the work. National Geographic's packaging is striking, with the yellow border, bold type and great photography. On a crowded Amazon results page, a clean box might stand out more than a table full of stuff.
Testing that on live listings was too risky. These were top sellers on Amazon, Target, Walmart and other stores. If I was wrong, sales would drop. And the brand believed in the contents photo so strongly that an untested hunch wouldn't get far with leadership.
I needed a way to test without touching anything live.
The approach
I found PickFu, a site that runs image polls with consumer panels. People sign up to give feedback on products, so they aren't real shoppers. The results wouldn't be perfect, but they'd be completely safe.
I set up testing in two stages. First, PickFu polls let me try lots of ideas quickly and cheaply. Then, when a result looked strong, I checked it with real shoppers using Amazon's own A/B testing.
If you were shopping for a kids' dig kit, which one would you buy?
Each person picks one and writes a sentence on why.
PickFu (external panel)
Fast and cheap, with no risk to live sales. Used to try many photo ideas and find data-backed insights.
Amazon A/B testing
Real shoppers and real sales. Only used for results that held up in Stage 1. What we learned went to every store.
What I tested
The first question was simple. For a kit where the box art is the star, does the box alone beat the contents spread out on a table?
I started with the geode kits. The box was bold and clean. The contents photo was a pile of rocks. I expected the box to win.
What I found
Across ~15 panel tests, the box photo won ~70% of the time. The contents photo still had a place, just for fewer products than the brand thought.
It came down to how the contents looked. When they were shiny or colorful, the contents photo did better. For most products, the box stood out more.
Box wins
When the box art is the story
Geode kits and rock painting. The photography on the box was more appealing than a pile of items on a table.
Contents photo wins
When the contents are the story
Shiny things like crystals and gems, and very colorful or textured contents. These looked better than the box.
When a panel result looked strong, I ran the same comparison in Amazon's A/B testing with real shoppers. The panel results held up, and the sales data gave leadership what they needed to make the change.
What changed
Before these tests, every kit led with the contents photo. After, most listings led with the box, on Amazon, Target, Walmart and the brand's own site. You can still see it in the stores today.
Testing also became part of every launch. Each new kit got a quick panel test, box or contents, so we weren't guessing.
What I learned
The testing setup mattered more than any one result. Panels let me try risky ideas quickly, so by the time a test reached real shoppers, we knew it was worth running.
The brand believed in the contents photo because no one had ever tested it. Safe, early tests let me bring evidence to a decision that had been made on gut feel.
I didn't track these tests formally at the time, so the ~70% and ~15 tests are my best memory.