Question 1

How to cancel your subscription

Accepted Answer

If you wish to cancel your subscription, please follow the steps below.

Go to https://app.thumbnailtest.com/settings/team
Click on the "Cancel Subscription" button

If you cancel within 14 days of when you subscribed, you will be automatically refunded.

NOTE: If the red button does not appear, that means the subscription is already cancelled!

Question 2

Understanding the Significance Calculations

Accepted Answer

Stats Used for Calculations

We primarily use two key metrics for our significance calculations:

Views: The number of times your video was watched.
Impressions: The number of times your thumbnail was shown to potential viewers.

We use these because:

They directly measure the performance of your thumbnail or title.
Views represent "successes" (when someone clicked and watched).
The difference between impressions and views represents "failures" (when someone saw but didn't click).

This allows us to calculate a Click-Through Rate (CTR) and analyze its significance.

Probability of Winning

This number shows how likely it is that a particular variation is the best performer. It's expressed as a percentage:

A higher percentage (closer to 100%) means that variation is more likely to be the winner.
A lower percentage (closer to 0%) means it's less likely to be the best.

For example, if Thumbnail A has a 75% probability of winning, it means there's a 75% chance it's the best performing option among all the variations tested.

Credible Interval

This is a range that shows how confident we are about the true performance of each variation:

The lower number is the "worst case" scenario.
The higher number is the "best case" scenario.
We're 95% confident that the true performance falls within this range.

For instance, a credible interval of [2.5%, 3.5%] for click-through rate (CTR) means we're 95% sure the true CTR for this variation is between 2.5% a

Question 3

CTR

Accepted Answer

Basics:

Stands for "Click-Through Rate". This is how many people who see your thumbnail/title, click on the video. If 100 people see your thumbnail/title on youtube, and 4 click, your CTR is 4.00

Learn:

This is probably the most misunderstood stat across YouTube.

Even Youtube employees have said that it's not as useful/intelligent as they had hoped.

In reality, this number changes a lot… day to day, hour to hour, depending on where people watched the video from, etc.

When A/B testing, do not put all your eggs in this basket.

You can have a high CTR, but low views, and vice versa.

Unfortunately, succeeding on YouTube is not as simple as making one number go up.

But hopefully with better understanding and data, you can make better decisions.

Gotchas:

CTR is not exactly #views/#impressions - some views don't count towards CTR.

For example, External views (views from twitter/etc) don't count towards CTR.

CTR also differs between view sources - you can have a high Browse CTR, but a low Suggested CTR.

CTR is useful in very particular scenarios, for example if a video had very low CTR, and very high AVD/impressions, you might consider changing the thumbnail/title, as it's possible that the packaging (thumbnail/title) is not strong enough to get people to click.

But, even this requires some nuance - if you have a high CTR, but low views, it's possible that the thumbnail/title is good, but the video itself is not good enough to keep people watching.

The big

Question 4

Impressions

Accepted Answer

Basics:

This is how many times your video showed up on YouTube. In other words, how many times people saw this thumbnail/title show up on their screen on youtube.com.

learn:

Impressions are a good stat. The more the algorithm pushes the video, the more you'll see a spike in impressions.

When impressions begin to flatline, it's a good indicator that the algorithm is no longer pushing the video, and might be a good time to make a thumbnail/title change.

Many videos will get a large spike in impressions on the first few days after upload, as YouTube pushes your video out to your subscribers.

Gotchas:

Pay very close attention to impressions when you're A/B testing.

If one thumbnail got significantly more impressions than another, it's possible this is a much better thumbnail, as YouTube is pushing it more.

However, if that happened to be the thumbnail you showed first, it's possible that YouTube was just pushing the video more because it's new, and this data can be misleading.

Be sure to look at the dates/times shown for each thumbnail to help yourself understand the data better.

Also, any time someone sees your video outside of YouTube, it doesn't count towards impressions.

For example, if someone sees your video on twitter, and clicks on it, that's a view, but not an impression. So it is possible to have thousands of views and 10 impressions.

Advanced:

You can actually see impressions by source, which can be very useful.

We don't yet support this in a

Question 5

How soon after uploading a video can I start my test?

Accepted Answer

You can start the test at midnight PST any day. By default, we will start swapping thumbnails as soon as the clock hits midnight PST after you start your test, or on the hour.

You can also start a test within 2 hours.

If you scheduled the upload, you can set the test to go out at the future time of publish

but you would set up the test before the video goes live

If you go to start a test right now, it'll always be delayed a couple hours to make sure it gets properly synced with YouTube servers (ex: so you don't start a test at 11:59 and it doesn't get saved until 12:01 so all the data is wrong)

Question 6

Returning Viewers

Accepted Answer

Basics

Returning Viewers are viewers who have watched something on your channel before.

Learn

This does not mean "subscribed views". This means "people who have watched a video on your channel before". How you view this stat depends on the video - if the video is just trying to engage your "existing audience", this stat is important. In a way, this can be a more helpful number than your subscriber count. This tells you how many people watching this video are likely interested in your content - likely to watch another video (if your topics are consistent).

Gotchas

This isn't really a stat to optimize for, it's more informational. But, if you notice "Returning Viewers" is consistently low, you might not be building a loyal audience.

Advanced

If you want to start a business off of your YouTube channel, it can help to have a loyal audience. If you're getting a lot of returning viewers, it's a good sign that you're building that loyal audience.

Frequently Read Articles

Browse All Categories