Overview
The Vidyard for Pardot integration has a step in the setup process which requires you to create/use a Campaign ID that you can associate to Vidyard. This will ensure that the lead source is set to Vidyard for these new prospects.
Symptom
You are noticing that the lead source for prospects is being set to Vidyard, although the prospect hasn't watched any videos.
Cause
Although it appears to be a Vidyard issue, it's more likely that you have a Pardot script embedded on a page which is referencing the Campaign ID that you associated to Vidyard.
This would mean that any forms that capture the details of a prospect who lands on your page will associate their record to the Vidyard Campaign.
Solution
You will need to ensure that your Pardot scripts are not referring to the Vidyard Campaign ID. It's unnecessary to use this Campaign ID anywhere on your page, even if you have a Vidyard Player embedded.
Vidyard will automatically make the association to the Campaign ID for those who have watched videos based on the Pardot account details in the Integrations page.