Codec based filtering

dlflannery

Moderator
ummmm..... life is getting very complicated for VAP!

Yes, I was aware of the new COM function Dan mentioned. I can't do much advance work on it until I know the COM function prototype and it's actually released in a VRD version so I can test with it.

Two additional factors have emerged that will mean VAP might require a major recode, however:

1. Dan's statement in post #3 in the thread you referenced that you should not QSF to a different codec. This is news to me and VAP's process flows are based on the assumption it is OK to do that. Thus if a user just wanted to recode an input file, this was done by configuring QSF (no adscan) and selecting the desired Output Profile. If I understand what he's saying correctly, this case should actually be done as a two step process:
a) QSF to the same codec as the input file, outputting to a temp file.
b) Load the temp file and save to the desired new output profile (recoding).

2. The whole concept of VAP automatically choosing output profiles based on the input file codec. (This is what you are asking for in your WTV cases, it appears.) This is totally new and raises all kinds of issues both for the VAP GUI and for it's process flow logic. An added complication here is that video container (e.g., .wtv vs .ts) are distinct things from video codec (h.264 vs. mpeg2), so this creates an additional combinatorial explosion of possibilities.

I'm currently reeling in shock from all these new complications. ;) Hopefully after some time thinking about it and clarifying issues (such as Dan's post #3) things won't seem so confusing.
 

dbjorck

New member
Hi!

I'm sure you can iron it out. I was thinking myself of the possible collisions between extension filtering and codec filtering. What are the people that use extension filtering using it for? Could it possibly turn out to be a 1 to 1 situation in reality? (Where my need is 1 to 2) If so the filtering could just be changed to be based on codec instead. Another option is to make the rules as AND rules, in my case one rule for wtv AND H.264 and one rule for wtv AND MPEG2.

Brgds

Danny
 

dlflannery

Moderator
Until these major changes are made, I suggest you give the -forceWTV command line option another chance. Yes it will create either .ts or .dvr-ms files in the Temp(QSF) folder, but that is nothing to be concerned about -- they will be deleted when processing of the file is completed, and you should get a WTV output file containing the same codec type as the input file, and all operations should be fast because they don't involve recoding.

For details on -forceWTV see the VAP-ReadMe.pdf change log comments for 20 August 2010, Ver. 0.47, but ignore the concern about which required profiles are present in VRD -- all of them are distributed in current VRD versions.

-forceWTV uses an Ad Hoc method for determining if a WTV file contains H.264 or mpeg2. It isn't 100% reliable but close I think. There is a Windows utility program that will convert WTV files to DVRS files but since the latter can only contain MPEG2 video the utility fails immediately if the input file contains H.264. VAP runs the program and detects the immediate failure ( or deletes the converted DVRMS file).

Be warned it's been a long time since -forceWTV has had any attention and I don't know if anyone has been using it. If you run into problems, they may require a little fix up.
 

dbjorck

New member
Oh. I used to use -forceWTV, but took it out recently when you said I don't need to... I've now put it back in. I haven't noticed any direct problems with it. I've included .ts and .dvr-ms files clean up in my Housekeeping app too.

Brgds

Danny
 

dlflannery

Moderator
Hi!

I'm sure you can iron it out. I was thinking myself of the possible collisions between extension filtering and codec filtering. What are the people that use extension filtering using it for? Could it possibly turn out to be a 1 to 1 situation in reality? (Where my need is 1 to 2) If so the filtering could just be changed to be based on codec instead. Another option is to make the rules as AND rules, in my case one rule for wtv AND H.264 and one rule for wtv AND MPEG2.

Brgds

Danny
I assume by "extension filtering" you mean the match-string profile feature. Note that this isn't just for extension-based filtering. The match string can be any substring of the input file name.

I prefer not to base design decisions on assumptions about how people are using a feature, although that sometimes does happen. For example, I suspect there are users who want to take WTV/MPEG2 inputs and transcode them to WTV/H.264 outputs (to reduce file size). Many of the container types (i.e., file extensions) that VRD handles can contain more than one type of video. WTV, .ts, .m2ts and .mkv files are examples.

I'm not quite following your 1:1 and 1:2 concepts. Perhaps the above discussion explains why -- or not. ;)
 

dbjorck

New member
Hi!

I assume by "extension filtering" you mean the match-string profile feature. Note that this isn't just for extension-based filtering. The match string can be any substring of the input file name.
Yes - but normally codec isn't part of either. When it was dvr-ms, it was always the same codec I think? I'm not sure though, I don't know much about these things. But I'm thinking it's like mpg or mp2 files are always MPEG2 encoding. So then the switch from extension to codec would be one-to-one; if that's the reason why extension matching was introduced. This was before "my time", so I don't know why the extension matching was introduced on top of filename matching, but I assumed someone had a specific issue, and that was the solution.

I suspect there are users who want to take WTV/MPEG2 inputs and transcode them to WTV/H.264 outputs (to reduce file size).
That sounds plausible. But - the key factor there is again the codec, not the extension.

Many of the container types (i.e., file extensions) that VRD handles can contain more than one type of video. WTV, .ts, .m2ts and .mkv files are examples.
And this practice seems to be increasing. Extensions used to say something about the codec, no longer. Again a reason why a codec aware VAP is probably better than an extension aware version.

I'm not quite following your 1:1 and 1:2 concepts. Perhaps the above discussion explains why -- or not. ;)
Perhaps I explained it a bit better above with mp2 always used to be MPEG2. We don't have that situation anymore. If .mp2 was used solely in order to identify it was MPEG2 encoding, then it is a one-to-one situation in moving across to a codec based application from extension based.

Brgds

Danny
 

dbjorck

New member
-forceWTV and stall functionality

Hi!

Does the stalling recognition "live outside" the -forceWTV functionality? I have this file now which stalls, but VAP doesn't notice it. It's been running for 12 hours and the dvr-ms temp file isn't growing.

Brgds

Danny
 

dlflannery

Moderator
Hi!

Does the stalling recognition "live outside" the -forceWTV functionality? I have this file now which stalls, but VAP doesn't notice it. It's been running for 12 hours and the dvr-ms temp file isn't growing.

Brgds

Danny
Stall recognition should work with -forceWTV .... may be a bug .... I will look into it.

What does the log file say about this?
Did it go on processing other files after the stall?
Did it move the file into the stalled files subfolder?
Can you cut and post the section of the xml file pertaining to this video?
 

dbjorck

New member
Hi!

What does the log file say about this?
The VAP log says:
10-10-2011 21:36:03 Started QSF, input: 60 Minutes_TV2 DK_2011_10_10_02_41_00.wtv
10-10-2011 21:36:04 Use of auto-setting of dimensions for QSF is disabled in Advanced Configuration settings
10-10-2011 21:36:04 No filter dimensions apply
10-10-2011 21:36:10 VideoReDo said: INFO: VideoReDo version 4.20.7.628 - Sep 28 2011
10-10-2011 21:36:10 VideoReDo said: INFO: No QSF filter to apply
11-10-2011 09:17:33 VideoReDo said: INFO: Aborted QSF: C:\temp\60 Minutes_TV2 DK_2011_10_10_02_41_00.dvr-ms
11-10-2011 09:18:04 Operator aborted VideoReDo processing
Did it go on processing other files after the stall?
Nope, as you can see in the log above, it just hanged around doing nothing for twelve hours. Funny thing is, it was not dead, I could select other files and everything. It hadn't realized there was a stall condition at all. I aborted it by pressing the Stop button and then Start, and it restarted the processing. It was the exact same, nothing had happened after 2 hours. I checked the VRD status, and it had stopped at 13% after a couple of minutes, and the time remaining number kept increasing; from being a few minutes at that time to after two hours 12 hours remaining but still only 13% done, and the temp file it was writing was only 328 MB. So I stopped again and blocked processing of that file, then restarted, and so it continued doing the other files in the queue while I looked at the file.

Did it move the file into the stalled files subfolder?
Nope, it did not detect there was a stall at all.

Can you cut and post the section of the xml file pertaining to this video?
Unfortunately not, because after some tinkering I managed to get the file processed. Speaking of which; how and when do you clean up <StalledFileData>? I see files in there which are no longer stalled; either they were unsalvagable and I deleted them, or I managed to coax them through manually, but they remain under StalledFileData.

Brgds

Danny
 
Last edited:

dlflannery

Moderator
Stall detection is based on CPU usage, as you know. It's at least conceivable that VRD could be sitting there in some kind of endless loop, using CPU time but not outputting any more video file. If you run into this again, check Task Manager to see what CPU usage for VideoReDo is.

I guess you've deleted the original file but if not, how does it do in VRD GUI?

What did you do to finally get it processed?
 

dbjorck

New member
Hi!

Stall detection is based on CPU usage, as you know. It's at least conceivable that VRD could be sitting there in some kind of endless loop, using CPU time but not outputting any more video file. If you run into this again, check Task Manager to see what CPU usage for VideoReDo is.
Yes, I forgot to check that. But I find it hard to believe that CPU was moving without anything being created. But it is conceivable.

I guess you've deleted the original file but if not, how does it do in VRD GUI?
What did you do to finally get it processed?
I was hoping you wouldn't ask that (which is why I withheld it from the post), because it is truly weird. By now you know I always end up with weird things happening. I simply attract goblins. But perhaps you'd get tickled by a mystery that not even Sherlock could explain. But first; the weird behaviour of VRD does not explain why VAP didn't catch the stalling condition... :rolleyes:

OK, so it first ran in VAP without anything happening for 12 hours. Then again in VAP without anything happening for 2 hours. So I opened it in VRD and pressed Play, to see with my own eyes if there was a problem with the file itself. After 7 minutes 34 seconds, it started looping. It replayed the last second over and over again indefinitely. Apparently some sequence error in the file then. I wanted to know if this was unique to VRD or a general real corruption of the file, so I played it in Media Center. Ooops, no looping... I opened it in VRD again, and played back the same period - and here the weird things started; now it suddenly did not loop after all. I reactivated it in VAP, and hey presto, QSF was done within a couple of minutes. My thinking then was that Media Center had perhaps fixed the problem itself while playing it back. I take backup copies of all recordings before they even get to VAP. So I opened the backup copy in VRD... and whatchaknow... no looping on the backup copy either, which had not been touched by anything since recording. Spooky.

I was going to post it on the VRD forum, but since I then suddenly couldn't repeat it, I decided against it. The VRD log has the same thing as VAP, it just stops logging anything whatsoever for 12 hours until I abort it. Which is why I'm thinking it's hard to believe that the CPU was actually moving.

Brgds

Danny
 
Last edited:

dlflannery

Moderator
Actually a loop that uses CPU time but doesn't do anything else that is observable is quite easy to program. Just create a while loop that repeatedly calls a random number generator for example. Make the end condition something that is never satisfied and there you have it!

When you said VRD was looping, that just smacks of something like that happening. The intermittent behavior is of course very weird and I would be amazed if Media Center "fixed" a file ... but I've been amazed before. ;)
There is so much going on in all these programs (many threads, interaction with video cards, dependence on and setting of Registry values) that varying behavior on the same input file is not inconceivable.
 

dlflannery

Moderator
Hi!

I'm sure you can iron it out. I was thinking myself of the possible collisions between extension filtering and codec filtering. What are the people that use extension filtering using it for? Could it possibly turn out to be a 1 to 1 situation in reality? (Where my need is 1 to 2) If so the filtering could just be changed to be based on codec instead. Another option is to make the rules as AND rules, in my case one rule for wtv AND H.264 and one rule for wtv AND MPEG2.

Brgds

Danny
Hi!


Yes - but normally codec isn't part of either. When it was dvr-ms, it was always the same codec I think? I'm not sure though, I don't know much about these things. But I'm thinking it's like mpg or mp2 files are always MPEG2 encoding. So then the switch from extension to codec would be one-to-one; if that's the reason why extension matching was introduced. This was before "my time", so I don't know why the extension matching was introduced on top of filename matching, but I assumed someone had a specific issue, and that was the solution.


That sounds plausible. But - the key factor there is again the codec, not the extension.


And this practice seems to be increasing. Extensions used to say something about the codec, no longer. Again a reason why a codec aware VAP is probably better than an extension aware version.


Perhaps I explained it a bit better above with mp2 always used to be MPEG2. We don't have that situation anymore. If .mp2 was used solely in order to identify it was MPEG2 encoding, then it is a one-to-one situation in moving across to a codec based application from extension based.

Brgds

Danny
I have an idea for handling the extension/codec complexity in VAP ... I think it may be what you were getting at before -- in that case you can have the credit!

The Profile Match String feature becomes the Profile Match String-AND-codec feature. I'll add a Codec column on the match string page with a pull down where the user selects among the possible input codecs, including a choice of "any" or similar.

The codec and extension entered in the last two columns (as now) will be used IF:
1. The match string is contained in the input file name (which can be used simply for selecting by file extension, e.g. ".wtv").
AND
2. The input codec matches what the user selected in the codec column. Unless "any" was selected in which case the whole scheme reverts back to the original matching just based on the string fragment.

What do you think about that?
 

dbjorck

New member
Hi!

The Profile Match String feature becomes the Profile Match String-AND-codec feature. I'll add a Codec column on the match string page with a pull down where the user selects among the possible input codecs, including a choice of "any" or similar.
Yes, that was exactly what I was thinking. "any" could be the default for people upgrading who already have Profile Match Strings.

But the neatest way would still be if VRD could create a "No Video Recoding" profile. Then the Match Profile String wouldn't need to change at all. With the VAP change, I would have to create two pillarbox profiles in VRD; H.264 pillarbox and MPEG2 pillarbox. And then for every show where I have the aspect ratio problem, I would also need to create up to two Match Profiles; one H.264 and one MPEG2. Many of my shows are on several different channels, and could be encoded differently. Off hand I know one which I'm recording, which is both on an H.264 channel and an MPEG2 channel. But if there were a "No Video Recode" profile, I could use that as the basis for my pillarbox profile, and just use that on all those shows in Match Profile without duplicating anything.

Brgds

Danny
 
Last edited:
Top Bottom