The 5-star rating system is terrible for gig workers. Here’s a simple solution

As consumers, we are accustomed to rating almost all the products and services we pay for. From toilet paper and tacos, to vacation rentals and online courses, a star rating is the status quo for reviewing pretty much any customer experience. But for platform-based gig workers who work to provide all kinds of everyday services, these ratings are nightmare fuel.

Taking consumers mere seconds to dole out, anything below the full five out of five stars can completely upend a gig worker’s income and access to work. Academics from around the world have found that negative reviews often serve as disciplinary tools that can reduce a worker’s pay, can generate an “inexplicable” shortage of gigs, or prompt a sudden and unjustifiable suspension or deactivation from the platform.

Throw in the many subtle and overt racial biases that influence a user’s approach to ratings, and it’s no wonder gig workers, who make up somewhere between 10% and 38% of the U.S. workforce, are plagued by paranoia and insecurity. 

At the will of ever-changing, inequitable user review processes, performance metrics and opaque algorithms, one thing is clear: Workers are grappling with invisible digital overlords, just to make enough to scrape by. A new study published in the scientific journal Nature finally offers proof that these star ratings are trash—and suggests that gig workers could get a better shot at fair work with a straightforward design tweak. 

A small gap with huge consequences

Existing research has shown that when customers submit evaluations, individual workers from ethnic minority groups are more likely to be negatively evaluated, even if their performance and quality is the same. Less is known, however, about how to eradicate, or redress these biases, especially in the gig economy landscape.

However, Tristan L. Botelho, Katherine DeCelles, Demetrius Humes, and Sora Jun, all academics at North American universities, had a hypothesis that by switching to a binary rating scale (think: thumbs up/thumbs down) these biases could be reduced. 

An avenue to test this presented itself to the team because the gig economy platform they were collaborating with, which matches North American homeowners with small business entrepreneurs for domestic repairs, decided to simplify its customer ratings from the common five-star system to a straightforward up versus down vote scale.

It was a snap decision, with no prior warning to customers or workers. Ratings were conditional on having a job completed, so any customers who held more explicit racist attitudes may have already cancelled. When examining approximately 70,000 customer ratings, collected before and after the sudden change, a distinct pattern emerged.

Under the five-star scale, workers of color received slightly lower ratings on average (4.72 stars) than white workers (4.79 stars). But this small gap had huge consequences. Since ratings determined pay, workers of color earned just 91 cents for every dollar white workers made—for the same work. After the switch to a thumbs up versus thumbs down scale, the rating gap and therefore the income gap disappeared.

“Any deviation from five stars is problematic as platforms rely so heavily on these ratings, but this was the first time I saw it linked to an outcome for workers, and despite the decades I’ve spent studying inequality in evaluation processes, it shocked me,” says Tristan L. Botelho, coauthor of the paper and Associate Professor of Organizational Behavior at the Yale School of Management. 

“If you leave a four-star book review, you’re not telling someone to read 80% of the book, yet many platforms allow for the evaluator to create their own rubric for what it means,” explains Botelho. “Your four is different from my four, and with all this noise, switching the focus onto good versus bad offers less room for subtle biases to creep in.”  

Toward a more level playing field

The shift isn’t about letting customers off the hook. More fairly designed systems could, in theory, mitigate racial, gender, and language biases (among others) and the negative effect they have on workers, leading to more accurate evaluations. Star ratings also make it hard to identify and address bad experiences, creating more barriers than solutions. In contrast, upvote/downvote ratings directly ask if the service met customer standards. If not, platforms could follow up to improve. A redesigned rating system could offer a route to supporting worker security, and making platforms seem more responsive and fair.

Many evidence-based solutions require training, investment, and expertise. But implementing a simple binary system would be easy for most firms—even starting with small tests in certain markets. “At the end of the day, the many platforms I’ve talked to are interested in making their evaluation processes fair and accurate.” And since the paper’s publication in February, multiple organizations have expressed interest in learning more.

“We often take for granted how apps are designed,” says Botelho. “I’d encourage platforms and companies to step back and ask: What is the goal of this evaluation process? Is it actually helping us gather useful information?”

Moving away from star ratings won’t solve every problem faced by gig economy workers—only more robust regulation will do that—but it’s certainly a step towards taming those cruel digital gods.

View the full article