AI onboarding for Vrbo hosts

Hypothesis

Vrbo hosts have to write their own headlines and descriptions when listing a new property.

We know this causes friction for hosts and quality issues for Vrbo including:

  • non-compliant text that results in friction for hosts,

  • localization at scale is an issue when hosts create text that’s not localizable,

  • and new hosts feel intimidated writing headlines and descriptions to market their listings.

These obstacles don’t just affect Partners, EG Travelers rely on high-quality listings to make their travel decisions.

Our hypothesis is that by providing hosts with generated headlines and descriptions that include traveler-relevant data

My Role: Content Design Director - product concept, demo, and UX writing

Approach

Me and the PM pitched the SVP of Product on using ChatGPT to generate headline and description text for hosts as a test and learn experiment.

Our learning metrics:

  • % increase in host onboarding conversion

  • % increase of headline and description step completion

  • % of headlines and descriptions adopted with no edits

Demo

To get executive support and align the team quickly on the vision, I created a demo with our PM and UX Designer.

  • I wrote the story direction, script, and illustration direction for the demo

  • The demo not only touches on the base functionality of a headline and description feature, but future personalization opportunities using GPT in combination with EG Traveler data and shopping signals

LLM & evals

As the content design director, I worked with an ML and a CD on my team on the prompt and evaluation metrics for this feature including:

  • text classification for property descriptors

  • troubleshooting data modeling issues between onboarding data and output text

  • creating a definition of good for headline and description including training set examples

  • defining eval metrics and grading outputs on appropriateness, style, granularity, representativeness, accuracy, tone and more

We actually had to explore a structured content (NLP) intermediary step to categorize some of the onboarding data into categories the LLM could use more effectively for relevant outputs.