top of page

Indeed Check

  Project Brief  

Explaining Sponsored Jobs with GenAI

Researching and designing a GenAI chatbot to help employers understand the value in sponsoring jobs on Indeed.

Chatbot mock 02.png
Chatbot mock 01.png

Project Summary

Discovery research uncovered significant user confusion with the concept of sponsoring jobs, especially with SMB employers. I designed around this pain point to create a GenAI solution that helped explain the value and underpinnings of job sponsorship in a personalized way to that user.

Role

I led user research and design from our discovery phase through the end of the project. Research consisted of qualitative interviews and an AI powered analysis of chat transcripts. I designed and iterated through several solutions and coordinated with external teams on our final design strategy.

Hypothesis

A GenAI powered chatbot can enhance employer decision making around Sponsored Jobs on Indeed by providing personalized feedback using the user's data and clarifying the underpinnings of Indeed's recommended budget, resulting in increased revenue for Indeed.

Process Overview

I partnered with a product manager, an engineer, a data scientist, and an engineering manager to define an initial test and an MVP for this project.

  Usability Test  

  User Interviews  

User Interviews

GenAI really shines when it comes to explainability and distilling complex topics into digestible information. For this project, my PM and I aimed to leverage this capability of GenAI in a way that improved the job posting experience for employers. To start, we had to identify compelling user problems. 

User Interviews

While we knew we wanted to utilize GenAI in a potential solution, we didn't know what problems we were addressing. I led user interviews with Indeed employers, while my PM led interviews with customer service reps at Indeed to identify common pain points. 

Research Questions

  1. What pain points do employers encounter posting jobs from Indeed?

  2. Where do employers experience anxiety or angst when posting jobs?

  3. What insights might improve their hiring experience on Indeed?

Methodology

Using Zoom, I led 4, 45 minute interviews with employers that post jobs on Indeed. Ideally, we wanted to talk to more employers but due to the timeline and some cancellations we weren't able to. However, even with these 4 employers, we began to see patterns emerging.  

Key Findings from Employers:

  • Sponsoring jobs on Indeed was the primary source of confusion for all 4 employers.​

  • When employers outspent their budgets, they felt they didn't understand Sponsored jobs.  

  • Employers didn't have a solid understanding of the outcome of sponsoring a job.

  • Employers wanted to understand how a sponsored job would compared to a job with no budget.

  • Employers were interested in having more actionable data but were sensitive to information overload.

Vector.png

HR Manager

Non-profit

Vector.png

HR Manager

Aviation

Vector.png

HR Manager

Summer Camp

Vector.png

HR Manager

Tourism

Customer Service Rep Interviews

My PM simultaneously led interviews with 4 customer service reps at Indeed to get their take on the most common problems they addressed with employers and how they went about solving them.

CSRep.png
CSRep.png
CSRep.png
CSRep.png

Key Findings from CS Reps:

  • Employers lack understanding of market conditions and competition on Indeed.

  • Employers are often intimidated and unaware of available analytics.

  • Proactivity in resolution builds trust with clients

  • Clients may not see the value in spending if they've had success with posting for free.

  • Clients have tight budgets and need to see clear ROI to justify spending. 

4

Employer Pain Points

We heard 4 key pain points repeatedly throughout these 8 interviews. These pain points gave us confidence that there compelling user problems to solve within the Sponsored Jobs product. 

noun-explosion-4991309 1.png

Employers do not have a clear understanding of Sponsored Jobs on Indeed and its ROI. 

noun-explosion-4991309 1.png

Employers have tight budgets and are sensitive to overspending.

noun-explosion-4991309 1.png

Employers want actionable data to inform decisions but are sensitive to data overload.

noun-explosion-4991309 1.png

Employers don't have a deep understanding of competition and market conditions and how that affects Sponsored Jobs on Indeed.

Research Readout

I compiled findings and insights into a slide deck and held a research readout for the team. These interviews helped us identify pain points that we could design an MVP around.

  ChatGPT Analysis  

UXR with ChatGPT

The user interviews helped us understand that employers struggled to understand and see the value in job sponsorship. We still needed to understand the magnitude of this problem and how customer service reps helped users work through it. 

Methodology

Our team sourced 4k chat transcripts from the customer service team from interactions that started on the Sponsored Jobs page. I created a research plan outlining our key research questions and partnered with a developer to leverage ChatGPTs API to synthesize the high level findings.

Research Questions

  1. What are the most frequent themes?

  2. What are the most common questions about Sponsored Jobs?

  3. What were the themes in positively resolved conversations?

Docs.png

4k chat transcripts

AI.png

ChatGPT API

Key Findings

  • Employers were mainly interested in understanding daily budgets, pricing models, budgets types, and how to attract quality candidates.

  • About 50% of these conversations were about the recommended budget of Sponsored Jobs.

  • In 97% of positively resolved conversations, the support reps provided examples.

  • When reps provided examples, users were more likely to spend the recommended budget. 

Product Direction

The findings of this analysis paired with the interview insights painted a clear picture that Sponsored Jobs was a difficult concept for clients. There clearly was an opportunity here to leverage GenAI's finesse in explainability to help clients understand how Sponsored Jobs work and how it can help them hire effectively.

  MVP Ideation  

MVP Ideation

Following our discovery research, I started ideating on ways we could leverage GenAI powered explainability on the Sponsored Jobs page.

Embedded Explainability Mocks

My initial ideation explored the concept of embedded explainability. I went through a few different ideas and shared them with the team and stakeholders after coming up with about 8 different variants. Below are some of them:

Sponsorship Guide

I designed my first idea around the user desire for understanding how a sponsored job compares to a free job. This mock leverages GenAI to explain how Indeed came to the conclusions regarding how many clicks they can expect.

Alert

My first idea seemed to dominate the sponsorship page with information that should be secondary. My second idea featured a sponsorship guide that only appeared when the user lowered their budget below Indeed's recommendation.

MonAI Initial Mocks 01.png
MonAI Initial Mocks 03.png

Data Visualization I

One limitation with GenAI's explainability finesse is that it's entirely copy. The first variants relied a lot on text, and for this one I tried to leverage data visualizations to complement the text. Additionally, I made the "AI Sponsorship Guide" expandable so the user could toggle it open if it sparked their interest.

Data Visualization II

The chart in this idea was designed to change as the user adjusted their budget below or above the recommended budget to illustrate how that affects their job's competitiveness. It also had the "AI Guide" that users could open or close as they needed to reduce how much space it took up. 

Competitive Data and Explainability.png
MonAI Initial Mocks 02.png

Socializing Designs

I shared the ideation with our core team and one of our GenAI leads. While the mocks were well received, they ended up prompting a more existential question about whether or not this type of "sponsorship guide" was the best way to present this data to the user.

Design Concerns

When socializing these designs, the team brought up two main points of concern. The first was whether or not we even needed GenAI to do this type of implementation. The engineers felt that this design could be achieved by leveraging programmatic rules and didn't necessarily need GenAI.

 

The second was that this was a lot of information to explain to the user on a single page and could feel overwhelming. These concerns pushed the team to think of other ways we could address user confusion around Sponsored Jobs.

  Chatbot Ideation  

Chatbot Ideation

Due to the amount of information needed to effectively explain Sponsored Jobs and the desire to make the answer feel personalized, the team pivoted our approach to a GenAI chatbot instead of continuing with the "embedded explainability" approach.

Chat Entry Point

One of the big concerns with the chatbot approach was that users normally start chats when they feel they have a problem. Our research indicated that users may not fully understand Sponsored Jobs, but they may not necessarily feel that this is a problem. I ideated on some ways we could nudge users to start a chat.

Alert Variant

My first idea was to introduce a banner on the page that a user could start a chat from. If the user lowered their budget below the recommendation, this banner turned into an alert to nudge the start a chat to understand why. 

Initial entrypoint ideation 04.png
Initial entrypoint ideation 02.png

Pop Up Variant

This iteration popped up a chat window whenever the user would lower their budget below the recommendation. This window also had an alert explaining why the chat appeared. While this would help us achieve our goal of getting more users to start a chat, it felt a bit intrusive and also hard to execute from an eng perspective. 

Prompt Variant

Past user research indicated that prompts were effective methods to nudge users into a chat. For Indeed's job seeker facing AI chatbot, almost 90% of interactions started with a micro-prompt within the chat experience. I wanted to template this known success with prompts and introduce it into our experience.

 

We ended up aligning on this entry point design for our test. 

Initial entrypoint ideation 03.png

  Usability Test  

Chatbot Usability Test

After aligning on a entry point, I wanted to conduct a usability test on the design to gauge employer's receptiveness to this feature and ensure there weren't any usability concerns I overlooked.  At this point in the project, we knew that we were going to 

Methodology

Our team sourced 4k chat transcripts from the customer service team from interactions that started on the Sponsored Jobs page. I created a research plan outlining our key research questions and partnered with a developer to leverage ChatGPTs API to synthesize the high level findings.

Research Questions

  1. What are the most frequent themes?

  2. What are the most common questions about Sponsored Jobs?

  3. What were the themes in positively resolved conversations?

Vector.png

HR Manager

Non-profit

Indeed Check

Usability Test
User Int
ChatGPT UXR
MVP Ideation
Chatbot Ideation
Variant B
bottom of page