A Culture of Testing is Not Enough
SPOILER ALERT – The more beautiful page lost by 37.8%, but that is not all.
In late 2017, Our testing team received a beautiful redesign for a hotel page from design. This new design had gone through user research, including user testing, and would replace the current design after we tested it. Executive support was high, and about a year’s worth of work was represented in these high fidelity mock-ups. However, all of this took place without involving the testing team.
I asked them to completely change it before we even tested it. Needless to say, the designer was not happy with my “outlier” opinion on the page, especially since he spent so much time on it. In truth, it was a very beautiful page. But I took one look and knew it was a bad design when it comes to conversion rates.
The good news? I was able to veto anything I felt would not be a good test.
While I could veto a bad test, this design had a year’s worth of work done and would all be a guaranteed waste if I did not test it. To honor that work, we tested it.
So we set out to test this fancy redesign. At the same time, we had another designer build a completely different page based on the thought sequence I believed the user was expecting.
The Original Design
Notice where the calendar is on the page. What would you consider the three most important things happening on the page?
The Beautiful Redesign
Notice where the calendar is now. What do you consider are the three most important things happening on this page?
The Second and Third Takes of the Redesign
(but with a very specific thought sequence to them)
Do you see how the hotel name and info sit above the image with white space to help the user see the name and image beneath it first? This is because the user needs to verify they are on the page they expected to land on before all else. Notice that as soon as they know they are on the right page, they can see what they can do on the page? They can search dates.
Results – Most Important Rule of Good Data, Don’t Argue With Good Data
The results were a stark contrast from each other. The only experiences that performed near identically were the second and third takes of the redesign. (This was important to show the lack of impact the brand of the page had on our users)
Here are the results as they relate to the original page –
Not only did we create a 29% increase in online orders (resulting in millions of extra dollars to the company) in-spite of the traditional redesign approach, but we prevented millions in lost revenue by testing out our changes first.
If we were able to start the process with testing first, we could have saved even more money in expensive resources – by requesting a redesign with specific parameters surrounding thought sequence and strategic design before jumping into the design itself. You do not need a “testing culture”, you need a “Test first culture”.
A testing culture can prevent some loss in revenue and increase profit, but the way to prevent the most loss and even see the most gains, is testing first.
Scientific Web Optimization 