The app that could not be tested

The home and the results page were both fixed layouts, so nothing on them could vary and nothing could be learned from them. Rebuilding both as systems let the team run a continuous testing programme.

Role
Senior Product Designer, covering UI / UX, native mobile and brand
What I owned
The design of both rebuilds, the modular home and the dynamic card framework, across iOS and Android, along with the single brand language holding them together. On the home I designed the cards, the rules for who saw each one, and how they looked on both platforms. Within the testing programme, I owned the variants that ran on those two screens.
The team
Product set each test’s goal. Engineering rebuilt the screens, and analytics ran the tests. Loyalty looked after Secret Prices and Rewards, and research tested prototypes in the lab. I sat with the iOS team in London, and reviewed the Android builds sent from Budapest. I also built an overlay that drew the text styles and the outlines over a live build, so a review could point at the exact difference.
When
2016 to 2019, iOS and Android

The problem

Around 165,000 bookings a day, in 90 points of sale, and every one of them started on those two screens.

Hotels.com sells in 41 languages and lists properties in over 200 countries. The two screens were a dated, static home, and a search results page on a fixed card.

Both were arrangements rather than systems. Nothing on either could respond to who was looking or what they were in the middle of, and because nothing could vary, no question about what a traveller actually needed to see could be answered. The app shipped on judgement and defended it afterwards.

A booking made again in the app avoided a cost we often paid on the first one, through Google or a comparison site. Hotel chains wanted those same travellers, and they offered member rates. The home barely showed the two reasons to come back: Secret Prices, and Rewards.

On the old iOS home, 15% of visits reached a hotel page, 12% reached sign in, and 12% reached an upcoming booking. That is the control group from the iOS test, October to November 2016.

How it was worked out

One programme for both screens, instead of a separate launch for each idea.

The home moved from a fixed layout to a modular and scalable one, with a single brand language holding it together across iOS and Android, and it was the first personalisation work the company had done. Search results moved from static cards to dynamic ones, so that a card could carry different amounts of content in the same slot. Android went first, in April 2016, with nine personalised cards, on the hope that engagement would lift bookings. iOS followed in October, moving onto the same cards with a narrower aim: hold the booking rate.

What ran through them was a standing programme across product, design, engineering and analytics, built to fail fast and iterate. Its operating principle, in the words used at the time: create a hypothesis to give us an opportunity to learn, not to predict. A losing test can still teach something, which is why the one below stayed in.

The 2019 refresh pushed the same idea further: a global rebrand with parity across platforms, machine learning behind modules like recommended destinations and travel guides, and a shift towards content before, during and after a trip rather than only at the point of booking.

Three tests, and what each one settled

  1. One test on the Hotels.com property page. The control, on the left, shows a single fixed photograph badged one of 87. Variant 1, on the right, shows a three-image gallery strip badged 87, which lifts the price, rating and Book Now button up into the first screenful.One test on the Hotels.com property page. The control, on the left, shows a single fixed photograph badged one of 87. Variant 1, on the right, shows a three-image gallery strip badged 87, which lifts the price, rating and Book Now button up into the first screenful.

    A richer image gallery, on Android

    Won. One test, with a single fixed photograph as the control and one variant showing three. Conversion moved, and gallery interaction moved by a great deal more. Travellers had wanted the photographs, and the old arrangement had been showing them one. It also sent better-qualified shoppers through to the booking form, not simply more of them.

  2. The iOS search recap test: a control, and two variants adding check-in and check-out details above the lowest-priced room, with conversion, engagement, attention and confidence figures beneath each.The iOS search recap test: a control, and two variants adding check-in and check-out details above the lowest-priced room, with conversion, engagement, attention and confidence figures beneath each.

    A search recap carrying check-in details, on iOS

    Lost, and then lost harder. The first variant added check-in and check-out, and the second added the times as well and did worse on every measure. The problem was summarising back to travellers something they had told us thirty seconds earlier.

  3. The sign-in module test: a control home, a variant placing a sign-in prompt above the rewards module, and a second variant that also swaps in a carousel hero image. Every measure still running.The sign-in module test: a control home, a variant placing a sign-in prompt above the rewards module, and a second variant that also swaps in a carousel hero image. Every measure still running.

    A sign-in module in the results stream

    Unresolved. It was still running when it was written down, and it never produced a verdict. A test without an answer stays in the record.

The two screens

A home whose modules changed with context, and a results card that could hold a different amount of content in the same slot.

  • The Hotels.com home on a Samsung Galaxy S3 and an iPhone 5s. A blue "Where are you going?" card sits over a full-bleed destination photograph, with a recently viewed property card below it.
  • The Hotels.com home on Android and an iPhone 6. The search field has moved up into the red header, with a list of recent searches below it and a "Need a hotel tonight?" card under that.
  • The Hotels.com home on Android and an iPhone X. A named greeting and a rounded search field sit over a city photograph, followed by a guide card for the destination and the upcoming booking.

A search card laid over a destination photograph, with recently viewed beneath it.

Five years and three homes. The 2014 screen sells at you. The 2016 screen is the card system, with search and what you last looked at first. By 2019 the same cards open on the trip you have already booked, and a destination guide sits with it.

On the home, modules carried recommended destinations, travel guides and trip content, and the mix changed with context rather than staying fixed. What appeared, and in what order, became the variable, where before it had been drawn once and shipped. Each card had a rule. Secret Prices showed on a first visit, until dismissed. Rewards asked people to join or sign in, then showed the nights they had stayed. Recent searches and hotels showed after browsing. Trip cards carried the stay before, during and after. Need a hotel tonight showed to everyone. Search stayed in a bar above the cards, in place of the large form. Trip cards came to iOS later, so the first test could cover the rebuild on its own.

  • Hotels.com search results before the rebuild, on an iPhone X. A red header over a white bar reading "London, England, United Kingdom" with an Edit button, then "3322 properties" and a grey "Hotel Name..." field. Each result is a fixed row: name, star rating, price and an "Excellent 4.4/5" chip, with a square photograph and a pin icon at the trailing edge.
  • The rebuilt results page with filter chips reading "Below £800", "Above 4-stars" and "Above 8.0" above a "Property name..." field. Each card runs its text down the left with a square photograph at the trailing edge: property name, a Gold Exclusive badge, star rating, distance to the city centre, price and a review score out of ten. Three results fit on screen.
  • The same three properties with the card reversed: the photograph sits on the leading edge and the text on the right, with the review score above a right-aligned price. Taller cards, so fewer than three results fit on screen.
  • The same properties again with a full-width photograph across the top of each card and the text beneath it. The Gold Exclusive badge is overlaid on the image rather than sitting in the text. One and a half results fit on screen.
  • A later results page opening on a photograph of London at dusk, with the dates and "81% booked for your travel dates" over it. Filter chips and "7,886 properties" sit below, then full-bleed cards showing a green review score and a price broken down as "for 1 room for 1 night including taxes and fees". A floating Filters and Map control sits above a five-icon tab bar.

Before: one fixed row, with a rating out of five and no filters above the list.

The layout that existed, the three built to be tested against each other, and where results went next. The middle three are the comparison: same chrome, same filters, same three properties, and the only variable is how much space a result is worth. Three fit on screen in the text-led layout, and one and a half in the full-bleed one.

On results, three adaptive layouts were built and tested against each other, each carrying a different balance of price, image and review content in the same slot. That gave us one card in three densities, with the framework underneath deciding which one a given result deserved.

Impact

The home’s booking numbers come from two tests in 2016. The iOS result is the clear one.

+2.5
extra bookings for every 100, on iOS
+4.0
extra bookings for every 100 people new to the app, on iOS
+85%
more accounts created, on iOS
$3.6M
estimated extra gross profit a year, from the iOS result

Hotels.com analytics readouts MVT-3480 (Android, 11 April to 12 June 2016, 1.3 million users, 91% confidence) and MVT-4513 (iOS, 3 October to 15 November 2016, 2.1 million users, above 99%). Booking tiles are extra bookings for every 100 on the old home screen. Android’s change for everyone was +0.6 and could be chance: the readout’s range includes no change. People new to the Android app were +1.5, worked out from the published counts. The profit estimate assumes a flat 12% margin. Also counted from the home screen: sign-in visits +25% on Android and +11% on iOS, Rewards visits +33% and +17%, and visits to the sign-up form +191% on Android.

On iOS, bookings rose by 2.5 for every 100, and by 4.0 among people new to the app. Accounts created rose 85%. Android’s result for everyone could be chance, so the $3.6 million estimate stays with iOS. On Android, visits to the sign-up form rose 191%.

On the new iOS home, the share of visits that reached a hotel page moved from 14.7% to 18.9%, and sign in from 12.3% to 15.8%. The share that stopped at search results moved from 36.6% to 34.2%. Android’s search share moved from 55.5% to 57.0%, sign in from 6.4% to 8.0%, and account creation from 0.6% to 1.8%. The two tests counted visits differently, so the change on each platform is what compares.

The search rebuild’s own result was reported in direction rather than as a figure: a strong positive, led by the US point of sale, with more customers returning within 30 days and more shortlisting.

Next