Bug? Not all processed files go to recycling bin?

marvin-miller

New member
Hi Folks;

I'm running the latest VAP but I've been seeing this issue for a while.

Recently a Star Trek Episode was recoreded - Errand of Mercy.

When I look at the processed file, it's too small (512k) so something went wrong. Not a problem, I'll go to the recycling bin and restore it and re-process it. This is where it gets funny, the file does not exist in the recycling bin. There's tons of other processed files but not that one.

I haven't a clue how to track this down or find out where the original recording went but I see this fairly often and it's always after a file was processed and it's too small (something went wrong).

It seems that, when it's done processing the file it encounters some kind of error. When that happens you get a partial processed file that's been renamed and filed properly but it's too small. Whatever that error is that caused the issue seems to also cause it to not put the file in the Recycling bin.
 

dlflannery

Moderator
Have you looked in the failed files folder in the monitored folder?

What does the VAP log say when this is happening?

What kind of files and what process flow?
 

marvin-miller

New member
Yes, the file does not go into either stalled or the other folder (the one where it gives up, I can't remember it's name).

The VAP log never seems to show anything relevant. Basically it seems to say that it's running comskip and the next line indicates it's starting a new file when it should be saying things like cutting / renaming or saving. It's as if if skips right over and starts looking at the next file (at least, this is what the log file seems to show).

The files are all H.264 .WTV files. The process flow is it that it runs Comskip on the files, cuts out the commercials, re-names the file according to metadata and then files in the proper folders.

What I do see each time is that a partial file is written out and that file has been renamed. So it has completed comskip, renamed the file, and is writing out the final finished file. The next step is to send the original file to the recycling bin. This is where the problem is, between the last two steps. When I look at the file that was written out it is never full sized (42 mins or so). Today, one file was 512k. So I see a properly named file but it is not complete. When that happens the next step (send the original to the recycling bin) does not happen. So something is happening between the last two steps. When writing out the cut file something happens and then VAP just deletes the original file.

I can post up screen pics of VAP tomorrow so you can see the exact settings/flow but that is where the problem lies. As to what the problem is when it writes out the cut file, I don't know. But when it happens (and this is not often) then VAP will always delete the file and not send it to the bin. That should give you an idea where the problem lies?
 

dlflannery

Moderator
First I am confused as to where you stand on the Comskip version issue. With your latest VAP installation are you no longer getting the message saying you can't use Comskip versions before 81.xxx? And please observe the log messages at VAP startup to verify there is a specific line that says you are using a donator (early access) version. The donator and free versions have exactly the same filenames and file lengths and version numbers, so it is easy to get them confused -- the message in the VAP log is the only way to be sure. (Or you can run Comskip in a command window, with no arguments, and observe the startup messages which will include text saying it's donator/early access. In effect that is what VAP does at startup.)

So assuming we've verified the above, am I correct you still have the problem with bad output and missing input files? And does this happen for any WTV file that contains H.264 video or only some? If it happens for all such files, then I have suitable test files on hand. Otherwise, The best way for me to debug this is to have one of the wtv files that exhibits this problem -- is that feasible? I do need to know your exact process flow either by description or screen shots. If I can't get a sample file, I need to know the video specs of an offending file either by mediaInfo ('text' display) or by loading into VRD and hitting Ctrl-L.
 
Last edited:

marvin-miller

New member
With the latest (fixed) version of VAP I no longer see the line in the log file about having to use a Comskip version higher then xxx (even though I was using a Comskip version higher then that). So it looks like that issue was fixed :) The only log entries I'm seeing for the latest VAP are;

Comskip files present, so Comskip commercial detection is enabled.
Comskip is early access (Donator) version

In other words, it looks like the bug with the previous version of VAP was fixed - thank you :)

Since I get my Comskip releases from the donator section of the site these are all I use. I don't use the free version. So there's no question about which I have installed - it's always the paid version.

Having cleared that up, I do still see (intermittently) an issue with files that are not processed correctly. When that happens, instead of sending the original file to the recycling bin they get deleted. It will be tough to find (I think) because it does not happen often. I can easily send over the butchered file but because the original file gets deleted - it's gone. So I'm assuming there's an issue in the VAP code where, when a cut file is being written out, and something happens (I don't know what) it then incorrectly deletes the original file thinking it's done. The issue is, it does not send it to recycling bin, it deletes it outright even though my settings are set to send them all to the recycling bin.

Is it possible my recycling bin settings are causing the issue?

Attached are some pictures of my settings
Picture1.gif
Picture2.gif
Picture3.gif
Picture4.gif
 
Last edited:

marvin-miller

New member
Found something. Today two Mayday episodes were recorded. Only one is in the recycling bin. Because two were recorded, there should be two. Looking in the VAP logs shows no stall or other errors for either of these files. VAP logs show they were recorded correctly and processed correctly. The log also shows that both were sent to the recycling bin but only one is actually present.

So it looks like this issue has nothing to do with processing errors during writing out the cut file but instead happens even when everything is processed fine. This should be easier for me to track as I can look in the recycle bin daily and compare it with the days recordings. Ie, if there were 5 recordings that day there should be 5 files in the bin.
 

marvin-miller

New member
Update #2 - I increased the recycling bin size on the processing drive. That may be all there was to it. I'll check it daily and reconcile the contents with the recordings and let you know if they match. If so, then perhaps what was happening was that the bin limit was being reached on that drive?
 

dlflannery

Moderator
Check for log file messages indicating some kind of error message following where it says it's starting to move the input file to the recycled folder. The code is designed to provide that but have to admit I've never verified it works.
 

marvin-miller

New member
Looking in the VAP logs shows no stall or other errors for either of these files. VAP logs show they were recorded correctly and processed correctly. The log also shows that both were sent to the recycling bin but only one is actually present.
It could well be that limitations on the recycling bin for the processing drive were causing the issue. I'll check it daily for the week and reconcile the two and post back my findings. At least it will be easy to verify :)
 

dlflannery

Moderator
It could well be that limitations on the recycling bin for the processing drive were causing the issue. I'll check it daily for the week and reconcile the two and post back my findings. At least it will be easy to verify :)
That's good. If that's the problem and my error messaging system isn't working, I can probably have VAP check the recycle bin to see if there is enough room and then give a message and not try to move the input file there.

However, I can't see how this issue would cause truncated output files to be produced. VAP doesn't begin moving the input file until after processing completes.
 

marvin-miller

New member
Good point...BUT the two may be mutually exclusive issues. They might not be related. The only reason I twigged on to the fact that not all files were going to the recycling bin was because I would see an improperly processed file (too small) in my recordings. At that point I would go and look for it in the recycling bin and more often then not, it was not there.

So they might not be related at all. It might be that periodically files that are bring processed correctly, like the one today, also don't show up in the bin, like the one today. Getting VAP to log a message if there is not enough room in the bin would be a good idea.

Like I said though, I will check it each day through the week and that should tell us if the bin size was the problem or if there is another issue.
 

dlflannery

Moderator
Did a little googling on the Recycle Bin and found this:
http://www.dummies.com/computers/operating-systems/windows-10/how-to-use-the-recycle-bin-in-windows-10/
And a quote from that page:
Your Recycle Bin keeps your deleted files until the garbage consumes about 5 percent of your computer’s available space. Then it purges your oldest deleted files to make room for the new. If you’re low on hard drive space, shrink the bin’s size by right-clicking the Recycle Bin and choosing Properties. Decrease the Custom Size number to purge the bin more quickly; increase the number, and the Recycle Bin hangs onto files a little longer
According to this there should never be a problem due to the size of the bin. Is it possible you are low on disk space in general? That might explain both failure to put files in the Recycle Bin and truncated output files. VideoReDo needs space for temp files sometimes.
 

marvin-miller

New member
It could be, we'll see what happens over the week. Like I said, I can reconcile the days recordings with the bin very easily. I just expanded the bin so perhaps everything will be smooth. It was a small drive (60 gigs?) as it's an SSD so perhaps that will be root cause. Would be nice if it was!
 

marvin-miller

New member
Update, I'm seeing a discrepancy between what's been processed and what's in the bin (ie, the bin is missing some items). I will check the size of the bin to be sure it's not a case of the bin filling up and it's deleting some as it goes. Will update further once I know
 

dlflannery

Moderator
Please remove low disk space as a possible factor. In explorer left window (folder tree), right click on the drive containing Output Folder and select Properties. How much free space? If you use a different drive for the Temp(QSF) folder, do the same for it.

If low disk space is not a factor, is there some common property of the video files that have problems that distinguishes them from those that don't? And/or is there anything different in the log file when processing those files? Other than this I'm going to need actual example wtv files to debug. I've tested with wtv files and your process flow and can't duplicate the problem.
 

marvin-miller

New member
Temp / QSF folders are on the same drive which is an SSD.

The recycling bin never shows contents from that drive which is D:

Recordings are done on G: drive - this is the drive that the recycling bin shows the contents are from. So when you restore any of those files they go back G:

I've set the bin on G: to half the drive size (250 GB) and I'll check it again. So far, it's looking like there are still files missing that should be in the bin.....
 

dlflannery

Moderator
I don't believe setting the recycle bin that large is going to change or help anything. What is the disk free space on the G: and D: drives, with the recycle bin set to a more normal size? If VideoReDo doesn't have enough space on either D: or G: during processing, it may cause strange effects, e.g. failure of the processing.

Also, what drive is the Monitored Files folder on and what is its free space? If processing fails and there isn't enough space free on the drive containing the Monitored Files folder, VAP would be unable to move the failed input file to the Failed Files folder which is a subfolder of the monitored files folder.
 
Top Bottom