Situation:
When we ran our second form page test, and it too did not succeed, I was discouraged, yet hopeful. you see, I had already done some work on the form page in question and it had already given us some pretty significant lifts, so it made sense that it would be difficult to succeed here. Time was running low though, we only had 3 more months to run another test, and the form page was the easiest to run tests on because it was a single page.
I suggested we move away from the form page for the time being so we could focus on areas that have not seen significant testing done yet. The team agreed but there were a couple problems. First off, no single landing page of ours produced enough traffic to provide statistical significance, and we did not have many other testing opportunities. That problem has to do with our rockstar marketing strategies but should be saved for another discussion. Secondly, we only had a few months to set up the test and let it run.
The good news is, we have an expert team of testers, project managers, data scientists, user experience designers, and developers. Running what we call a Multi-Page Experiment is anything but simple, but with the right team, you can do just about anything.
The Team
A while back, I did an analysis of the requirements to run an effective Website Optimization team. I discovered a lack of specific resources on our team and when presenting the evidence and potentially adding those resources to the team to the Director of Demand Generation, I was turned down. Head count is almost always a last resort and for good reason.
Next I gave an alternative to hiring head count, outsourcing. This in and of itself was a long journey, but it was approved. Yay! Now the real work begins. We found the organization that seemed to be a cut above the rest, MECLABS, they are the research organization that fuels MarketingSherpa and seem very capable and methodical at all areas of testing and optimization.
The Multi-Page Experiment Monster
Having decided on testing the landing pages, which we had hoped to update in a template format; we discovered the template would not work the way we wanted, with the changes we needed. That’s when I asked the question “well, how many pages DO we need to get statistical significance in a reasonable amount of time?” “45 Pages” was the answer… well, 45 pages is a lot less than 1,000 so I was happy.
Data Driven Decisions
With MECLABS running the data analysis, we were confident that spending the time on these 45 pages would be worth it. I quickly got to work. We worked with MECLABS in forming the design, it went though a few internal revisions till we were satisfied with the thought sequence (Thought Sequence is the design of the conversation between the marketer and the person on the website. Upon request I can write up an article about this too) behind the design and that the design accurately reflected that thought sequence.
We Built out the landing page from scratch, and started duplicating the page for each landing page we would be testing (because we needed to test multiple pages at once to get statistical significance). Then we migrated the content that would not be changing from the control over to the treatment. This was a slow methodical process that involved myself along with other resources internally. If you are confident in the data behind the decision, spending the time building the test is a no-brainer.
Post Test Launch
We launched the Test. Sure, the countless hours of test set up are over; and now begin the countless hours of tracking and understanding test results (all the while setting up the other tests we have queued up for production).
Test tracking is vital, that is if you want to learn anything from your tests. With The right segments built into our analytics platform, we could accurately measure the results and gain a learning. With that learning sometimes come positive, sometimes negative, and the worst kind – “meh” results. This our 3rd test with MECLABS has proven to give us Positive results as well as some fantastic learnings.
Biggest Result:
- We saw a software related lead generation relative increase of 52.3% with statistical significance of 99.7%! What does that do to our bottom line? $1,543,320 Million in Sales pipeline growth from only our top 22 software related pages!
Learnings:
- Grouping resources by type, better orients visitors, helps them find what they are looking for, and reduces friction.
- Hover over descriptions of the resources provide more clarity around product content, which will increase the perceived value exchange.
- Providing more supporting value to the videos by displaying them as thumbnails increases their appeal and encourages interaction.
- The use of social media portals is not consistent with visitor motivation or thought process, adding friction (or clutter) to the page.
At the end of the day, to exist as a business, we need to see a return on investment. But the best way to see that return is through customer wisdom. The better we know the customer, the better we can predict what they’ll want, the better we can see a return on that investment. The real win for this test is that now we can apply what we learned to other parts of the site and see significant wins there. And we will.
Scientific Web Optimization 