Discovery:

• User Testing on Control Pages:
  The landing page squad began by performing a Q&A user flow analysis via Usertesting.com on the current North American and European landing pages in both the United States and United Kingdom markets, respectively. The purpose of these user tests was to identify any issues with the current flow, gather insights into what customers are looking for at this stage of their purchasing process, learn what's working and not working on the current pages, and to gather general thoughts on layout and aestheics.
• Collecting Additional Customer Insights:
  The cross-functional team reviewed the timeine of upcoming product option releases and shared customer insights collected via the user flow analysis (managed by landing page team), customer interviews (managed by product team), website metrics (managed by site team) and more.
• Additional Research:
  A survey of competitor marketing efforts were reviewed and considered.

Prototyping, Testing & Analysis:

• Prototype Development:
  The cross-functional team took the learnings from the discovery phase and split into groups to sketch low-fidelity prototypes themed in one of three main [learned] customer values: Quality, Ease and Inspiration. After a team-wide critique, each group circled back and designed a single high-fidelity prototype based on team feedback.
• Focus Groups:
  The cross-functional team presented customers with the finalized prototypes via a video-conferenced focus group. Focus group participants were screened to include a diverse group of both men and women from a variety of industries, Vistaprint customers and non-customers, and online and offline print customers. These focus groups were done in both the United States and United Kingdom markets.
• Qualitative Analysis & Redesign:
  After reviewing the qualitative feedback from both the discovery phase and the focus group discussions, the cross-functional team was able to validate the priortized global themes and customer needs found from the intial audit. The team was able to toss out one of the prototypes altogether (Inspiration) and revisit the other designs in preparation for an A/B/C live test.
• Content Preference Test:
  One signficant piece of qualitative data revealed that the design/content on the business cards played a significant role in a customer's connection with the page. Popular designs from an array of industries were identified by the product owners and designers placed several combinations of business card designs on new photography that was shot specifically for the prototype redesigns.
  
  A casual survey of Vistaprint Finance and HR employees narrowed down the sets to four, which were then placed on Usertesting.com for a preference test. The test was made available to both the United States and United Kingdom markets yielding a clear winner (coincidentally, the same set for both markets).
• Eye-Tracking Software:
  As a final check before beginning the live, in-market test, the team ran the challenger pages through a machine-learning eye-tracking software called EyeQuant. This software yielded a heat map of the pages and also evaluated the cleanliness of the page. A couple very minor adjustments were made based on the output of this tool.

In-Market Testing & Analysis:

The A/B/C test ran until statistical significance was achieved in Vistaprint's top ten markets for business cards.

• Quantitative Analtyics:
  The Vistaprint Analytics Team used Adobe Analytics to compare three main success metrics: revenue, conversion rate and average order value (AOV). The test revealed that both challenger pages beat the control in AOV, but that revenue and conversion rate were down ever-so-slightly (suggesting that while fewer people were buying, those who were buying were buying bigger). The two challengers were near indentical in revenue, but one challenger (Ease) was slightly higher in conversion.
  
  A deeper dive across all metrics allowed our Analytics Team to notice a significant increase in bounce rate among the challengers. Comparing the gap between bounce rate and revenue allowed them to hypothesize that because customers were recieving a more holistic view of the product options, they were likely exiting the page because they got the information they needed to make a purchasing decision at a later time.
  
  The other discovery from the deep dive was a testing conflict in three of the major locales during a portion of the test. The Analytics Team was able to identify that this conflict likely had a negative impact on conversion, but were not able to discern the degree of the impact.
• Qualitative Analytics:
  On top of Adobe Analytics, the team used FullStory to review a sample of user sessions. Watching in-market user behavior allowed Vistaprint to cross-reference some of the aforementioned hypotheses and note some additional patterns.
• Calling the Test:
  The cross-functional team was able to accomplish four of the five project goals, falling short in financially outperforming the control pages. Despite the narrowly missed target, the team felt like the value added to the customers by accomplishing the other goals was enough move in the direction in one of the challengers.
  
  The Analtyics Team also felt confident that by implementing the undisputed learnings from the live test (and having removed any source for conflicts) that either of the updated challengers would make up for any revenue loss seen during the A/B/C test. The team altered the page with some "quick wins" and, since revenue loss was equal among challengers, rolled out the one with the higher conversion (Ease).
• Next Steps:
  The cross-functional team was set to furuther evaluate learnings from the live test to identify areas for potential optimizations, which would be evaluated for upcoming multivariate tests. The next multivariate test is likely to include the product options area which saw higher clicks on the challenger that was not rolled out (Quality).