As I’m sure you’re aware by now, Star Wars Episode VIII The Last Jedi has an abysmal Rotten Tomatoes audience score which is in direct conflict with its high critics score. This by itself is rather unusual, but this is also Star Wars, the most popular movie franchise in existence, so it’s a big surprise that the conversation around the film isn’t just about its merits, but why so many people strongly dislike the film. But the real question shouldn’t really be why it’s disliked, but if that’s a true reflection of the film’s audience.
Let’s start with a polling conversation. It’s important for this editorial to distinguish between “opt in polling” and “random sample polling”. Random sample polling tends to be the most accurate and scientifically valuable because it’s most reflective of a given population. Random sample polling is exactly what it sounds like: respondents are selected at random (just like picking a name out of a hat) with a process that assures that the different respondents have an equal opportunity to be selected. This process allows the pollster to calculate a margin of error or see how different the poll is from the population it’s representing. Opt in polling is also like it sounds: a pollster contacts many people and those respondents who agree to participate become part of your poll. This is the most common form of internet polling and also the way exit polls (a poll that determines behavior after a specific activity is done) are achieved. Opt in polling will not match the population as well as random polling does and it’s often easy to see bias and skewed data in such surveys.
However, there are ways to adjust the data from opt in polling to help it be more reflective of a population, so it’s important for our purposes here to state that opt in polls that make these adjustments to reduce bias and attempt to accurately reflect a population are far more valuable than opt in polls that don’t make any adjustments. The polls that don’t adjust have very little scientific value and can often contain a bias called self-selection bias. This occurs when the respondents select themselves into the test group, often to create a skewed poll.
This is what happened to The Last Jedi on Rotten Tomatoes and Metacritic. A group of individuals sought out these polls and gave the movie a negative score with the desire to create a negative consensus. This activity is called “review bombing” and is more frequent in video game user polls. The result these individuals are looking for is to cause harm to a product’s sales by assuming potential customers will use the poll to determine the product’s quality. Steam, a popular video game platform, has been combating review bombing for several years and recently changed their review system to try to combat this. They now graph their user reviews over time so customers can take the extra step and glance at a graph to see if there’s any cluster of negative reviews.
How do we know this was review bombing and not an organic collection of opinions? I pulled some information about the individuals who reviewed The Last Jedi on Rotten Tomatoes and Metacritic. I need to first point out that both sites don’t make this easy. Neither site had any breakdown on who was responding the way imdb does, Rotten Tomatoes apparently only lets you see the most recent 50 pages of user reviews and Metacritic would frequently give me an error message when cycling through different pages of user reviews. Why? Well, imdb needs to have their data be as valid as they can make it since they use that data as part of their paid imdb pro service. Rotten Tomatoes and Metacritic don’t sell their data, so they are less inclined to put safeguards in place to make their data more scientifically sound.
Still, I randomly pulled 100 “poor reviews” (score of one or less) from both sites, and noted when they first started reviewing content and how many reviews they have contributed.
As you can see, there were a lot of respondents who signed up to review The Last Jedi and with Rotten Tomatoes, and a third of the respondents deleted their account or had their account deleted after registering. This suggests an effort was made to create a negative self-selecting bias because these new users chose to register with the site just to have their review counted in the overall Rotten Tomatoes and Metacritic score for the title. There are some other issues with these polls – there’s no guarantee that the respondents actually watched the film and there’s also the ability for the same unique individuals to vote multiple times, and I could see in my data pull the same review pop up a few times or the same name pop up a few times. I also noticed, but could not include this data since, again, neither site collected demographic information, that an overwhelming majority of the respondents had male names. An educated guess would be 90% or more.
Even if the gender split on the user reviews is an educated guess, it’s a pretty strong indication (along with the low score in general) that the data from these websites is skewed and not a true reflection of the actual audience that saw the film. How do we know the audience wasn’t 90% male? Because there’s a company called comScore that provides exit poll surveys that, while it is an opt in survey, is one that adjusts to reflect the movie going population. Their data showed the audience did skew male, but not as high as 90% or more.
comScore surveys almost a thousand audience members in 20 markets immediately after they view the film, including The Last Jedi. So, while Rotten Tomatoes and Metacritic showed audience scores in the 50s or lower, comScore showed a score almost at 90 which was in line with both The Force Awakens and Rogue One. comScore was kind enough to allow me to provide you with some of their exit polling data on these three films:
Lots of good data to unpack here, but most importantly, this data shows the audience liked The Last Jedi a great deal. Rating a movie as either “excellent” or “very good” tends to be the key metric in most Hollywood evaluation of content, both in testing and in exit polls. Anything over a score of 80 in the “top two boxes” is concerned a great score, and The Last Jedi has a top two box score of 89. So, if 89% really liked the film, the amount of people who would score the film “poor” or the equivalent of a zero or one score on Rotten Tomatoes or Metacritic would be a very small, single digit percentage.
This bears repeating – the audience who saw this film opening weekend and hated it was the smallest portion of the audience possible so the individuals registering at Rotten Tomatoes and Metacritic to post negative reviews and try to skew the audience score make up a tiny minority of the film’s actual audience. Also, the main complaint of the film, which you can see in any number of internet petitions to Disney, is that the film betrayed aspects about the franchise that they felt were essential. This point is directly relevant to the expectations one had going into a film and you can see in the comScore data that 93% of the audience felt the movie met or exceeded their expectations. Again, we can reflect that the people starting and supporting these petitions are a small minority of The Last Jedi’s attendees.
I think its worth pointing out here (and I assume this will come up in the comments) that just because it’s a tiny percentage of the audience, it doesn’t make a negative opinion of The Last Jedi wrong. We can all agree opinions of films can’t be right or wrong. What I’m trying to demonstrate is that the consensus of most of the audience (almost 90%) really liked the film. That’s the key thing here, not that the only way to feel about the movie should be positive. The number of people who truly hate this film is not statistically significant and the Rotten Tomatoes and Metacritic scores are not reflective of the film’s actual audience.
I want to conclude by pointing to some recent documentation from the good people over at 538.com of similar content that was affected by similar activities. This could suggest some motivation for review bombing movies. Again, it’s important to point out there are people out there who saw The Last Jedi and didn’t like it, so these observations are not meant to suggest they apply to everyone who hated The Last Jedi, but only those who are intentionally trying to skew the data to the negative.
538.com has posted three recent studies regarding imdb and its user scores:
- First, there was this one that talked about the gender split on TV shows and how men scores were negatively skewing shows directed towards women.
- Then this one that talked about how men were review bombing the Ghostbusters remake.
- Finally, this one about An Inconvenient Sequel and how it was reviewed bombed, even before its release to the public.
All three discuss similar conclusions to the ones mentioned in this editorial, but I think it’s relevant to point out that in all three studies, the targeted content contained traditional liberal ideology – either content that represented diverse and more inclusive female voices or a documentary about global warning. Considering there was an attempted boycott of The Force Awakens over diverse casting and a female central character, I’ll submit that the motivation behind the review bombing of The Last Jedi is not solely motivated by opinions about quality and faithfulness to the franchise, but also as a protest to how Disney has recently attempted to create more diverse characters and also cast more diverse actors and actresses in their tentpole content, including Star Wars.
Ultimately the narrative around the film has yet to be finished. It had the second largest domestic opening of all time and is likely on its way to the third largest domestic gross ever and the largest release of 2017 (but not enough to salvage 2017 from being a down year overall). However too much of the conversation this weekend and early this week was about the audience score at Rotten Tomatoes without reflecting on whether that score was truly representative of its audience. By giving these voices more weight than they deserve, these voices actually accomplished their goal even if Disney will consider the comScore data the accurate representation on how audiences received the movie. These individuals created a conversation, and it was a conversation that reflected negatively on a film with an incredibly positive consensus of quality. This observation cannot be lost and its behooves us to consider how significant user reviews that use opt-in polling without population adjustments should be valued in film criticism.