VideoReDo-Autoprocessor Program

tannebil

New member
I've started encountering a problem where the adscan process sometimes hangs in the process of creating the drax file. The first adscan message appears but not the second. In the log extract below, you can see a successful run for an episode of Hawaii Five-0 followed by a hang on an episode of The Closer. This is bad since VRD metadata processing follows chaptering and doesn't get done. VAP started a VideoReDo process but the VRD process isn't consuming any CPU cycles. Manually stopping processing in VRD doesn't kill the VRD process if that helps at all.

I did a reset on the episode in VAP and am rerunning it to see if it happens again.

Update: It ran fine the second time which makes me think it's the problem we've already seen where the COM interface fails to open the file. However, as I recall, that always threw a VRD error in the past. But I didn't record many new shows in December so it's hard to be sure.

1/4/2011 1:15:36 AM theTVDB metaData search succeeded for: Hawaii Five-0 - ''Ke Kinohi'' (Recorded Jan 3, 2011, KIRODT).TiVo
1/4/2011 1:15:36 AM Started QSF, input: Hawaii Five-0 - ''Ke Kinohi'' (Recorded Jan 3, 2011, KIRODT).TiVo
1/4/2011 1:15:37 AM No filter dimensions apply
1/4/2011 1:15:46 AM VideoReDo said: INFO: VideoReDo version 4.20.6.610 - Nov 18 2010
1/4/2011 1:15:58 AM VideoReDo said: INFO: No QSF filter to apply
1/4/2011 2:25:27 AM VideoReDo completed QSF on input: D:\Tivo\Hawaii Five-0 - ''Ke Kinohi'' (Recorded Jan 3, 2011, KIRODT).TiVo
1/4/2011 2:25:27 AM Started Ad Scan/chapter marks, input: Hawaii Five-0 - ''Ke Kinohi'' (Recorded Jan 3, 2011, KIRODT).TiVo
1/4/2011 2:25:45 AM VideoReDo said: INFO: VideoReDo version 4.20.6.610 - Nov 18 2010
1/4/2011 2:26:13 AM VideoReDo said: INFO: Edit mode set to cut
1/4/2011 2:33:46 AM VideoReDo completed CHAPMARKS on input: O:\beyondtv_vap\Hawaii Five-0 - ''Ke Kinohi'' (Recorded Jan 3, 2011, KIRODT).mp4
1/4/2011 2:33:46 AM Created Drax Chapter File
1/4/2011 2:33:46 AM Atomic Parsley started
1/4/2011 2:33:49 AM Using thumbnail image: Hawaii Five-0 (2010)_thumbnail.jpg
1/4/2011 2:33:49 AM Atomic Parsley completed
1/4/2011 2:33:49 AM User post process batch file started
1/4/2011 2:36:07 AM User post process batch file completed
1/4/2011 3:00:29 AM theTVDB metaData search succeeded for: The Closer - ''An Ugly Game'' (Recorded Jan 3, 2011, TNTPHD).TiVo
1/4/2011 3:00:29 AM Started QSF, input: The Closer - ''An Ugly Game'' (Recorded Jan 3, 2011, TNTPHD).TiVo
1/4/2011 3:00:30 AM No filter dimensions apply
1/4/2011 3:00:38 AM VideoReDo said: INFO: VideoReDo version 4.20.6.610 - Nov 18 2010
1/4/2011 3:00:48 AM VideoReDo said: INFO: No QSF filter to apply
1/4/2011 4:08:28 AM VideoReDo completed QSF on input: D:\Tivo\The Closer - ''An Ugly Game'' (Recorded Jan 3, 2011, TNTPHD).TiVo
1/4/2011 4:08:28 AM Started Ad Scan/chapter marks, input: The Closer - ''An Ugly Game'' (Recorded Jan 3, 2011, TNTPHD).TiVo
1/4/2011 4:08:39 AM VideoReDo said: INFO: VideoReDo version 4.20.6.610 - Nov 18 2010
1/4/2011 9:05:23 PM theTVDB metaData search succeeded for: No Ordinary Family_KOMODT_2011_01_04_19_59_00.wtv
 
Last edited:

dlflannery

Moderator
I've started encountering a problem where the adscan process sometimes hangs in the process of creating the drax file. The first adscan message appears but not the second. In the log extract below, you can see a successful run for an episode of Hawaii Five-0 followed by a hang on an episode of The Closer. This is bad since VRD metadata processing follows chaptering and doesn't get done. VAP started a VideoReDo process but the VRD process isn't consuming any CPU cycles. Manually stopping processing in VRD doesn't kill the VRD process if that helps at all.

I did a reset on the episode in VAP and am rerunning it to see if it happens again.

Update: It ran fine the second time which makes me think it's the problem we've already seen where the COM interface fails to open the file. However, as I recall, that always threw a VRD error in the past. ...........
Yes, I believe there would be a logged error message for the "classic" :)rolleyes:) file open failure.

Would you be able to run your custom ad scan script on this file? It would be interesting to repeatedly start it on this file (then shut down if it starts OK) and see what the odds of failure are.

Just as a sanity check, VAP creates a diagnostics file named "arglogs.txt" in the VAP data folder each time it launches a VBS script. The file contains the arguments passed to the script process. You could verify that there is no difference in these args for the successful versus failed cases, which would tend to indicate the problem isn't caused by VAP.
 

tannebil

New member
OK, got another failure.

1. Nothing interesting in the arguments.

2. I ran Process Monitor to see what the VRD process was doing and it just kept repeating the same set of registry actions. I've attached a PML file with the events (renamed .log so that I could upload it).

3. I ran the vbs file you gave me 10 time and it worked every time. I did notice that the mini-icon that shows up in the tray for Ad-Detective did not exist for the hung process. Maybe that says something about where in the program it's hanging.

4. I stopped and started VAP and it picked up with the "Started Ad Scan/chapter marks" message in the log, i.e. VAP either thinks it didn't start yet or was smart enough to restart the step. The Ad-Detective icon appeared in the tray and the normal log entries showed up so it seems to be to be working.
 

Attachments

Last edited:

dlflannery

Moderator
OK, got another failure.

1. Nothing interesting in the arguments.
I would have been very surprised if there was anything here, but thanks for checking.
2. I ran Process Monitor to see what the VRD process was doing and it just kept repeating the same set of registry actions. I've attached a PML file with the events (renamed .log so that I could upload it).
Apparently the PML file is a binary file in a format that only Process Monitor can read. From all other symptoms I think it is just showing some problem in VRD COM code, so doubt I would learn much more from it. I may install process monitor and take a look however.
3. I ran the vbs file you gave me 10 time and it worked every time. I did notice that the mini-icon that shows up in the tray for Ad-Detective did not exist for the hung process. Maybe that says something about where in the program it's hanging.
In a recent post, a VRD staff member said that actually the VRD functions open the file several times during one operation, so it's probably failing at one of those points.

I think only VRD staff might be able to debug this. I hope we get some of their attention. Your recent thread on the "classic" open failure hasn't had any response from them yet -- not a good sign.
 

dlflannery

Moderator
tannebil,

Please ignore this if your Ad Scan hangs have gone away.

Please track these VRD hang failures. What percentage of the time do they happen? Are they only when running Ad Scans? Do you run Ad Scans for any purpose other than Drax chapters? If so do they also occur on those runs? Actually, would you mind running a few Ad Scan's other than for Drax purposes (e.g., QSF then AdScan w/o AutoCuts), just to see if it could be the VAP code specific to Drax Ad Scans that is somehow at fault?

Your example was for a TiVo original file. Do the hangs occur for other file types? Of course I assume the Ad Scan is running on a mp4 resulting from the QSF.

Are your TiVo files TS or PS ?

Will be interesting to see if the hangs are particular to build 610.
 
Last edited:

tannebil

New member
OK, I'll track them going forward.

I have not had any in the last few days. As far as I recall, all the hang-ups were with TiVo files. I'm about 80% WTV files the last couple of days but it's normally more 50-50 between WTV and TiVo. The AdScans are running against mp4 files.

I don't run AdScans for any reason other than chaptering but I can do so. Are you saying that I should just turn off Drax chaptering in advanced options?

I don't understand the question about my TiVo files being TS or PS. My TiVo files are 720p and 1080i TiVo files
 

dlflannery

Moderator
.......I don't run AdScans for any reason other than chaptering but I can do so. Are you saying that I should just turn off Drax chaptering in advanced options?

I don't understand the question about my TiVo files being TS or PS. My TiVo files are 720p and 1080i TiVo files
Yes turn off Drax chaptering and configure QSF and Ad Scan but not AutoCut after adscan. After VAP processes a file and it goes into AWAITING CUTS status, just reset it.

I had the idea that Aussie TiVo's and/or Premiere's could or did provide Transport Stream (TS) rather than Program Stream (PS). Media Info or the VRD File Info (Ctrl-L) should tell which. It probably has no bearing on the hangs. I was just curious.
 

tannebil

New member
I tried a batch of three TiVo files and it hung on the first one. Stopping and starting VAP and AdScan completed the first one and the other two worked fine.

I ran the same batch of three TiVo files with Drax chaptering turned off and AdScan turned on. It processed all three without a problem.

Not really seeing much, if any, pattern. I should note that my process flow is different for WTV files as a second instance on VAP watches the "Recorded TV" folder and does a QSF/copy of the file (without transcoding and while retaining WTV format) to the TiVo folder.
 

dlflannery

Moderator
I tried a batch of three TiVo files and it hung on the first one. Stopping and starting VAP and AdScan completed the first one and the other two worked fine.

I ran the same batch of three TiVo files with Drax chaptering turned off and AdScan turned on. It processed all three without a problem.

Not really seeing much, if any, pattern. I should note that my process flow is different for WTV files as a second instance on VAP watches the "Recorded TV" folder and does a QSF/copy of the file (without transcoding and while retaining WTV format) to the TiVo folder.
Could you run the one file that hung several times with Drax off and AdScan on to see if it ever hangs?
 

tannebil

New member
VAP is beginning to hang on the AdScan/chaptering process more frequently (maybe 1 in 3) and fails with both TiVo and WTV files. VAP shows the action as "Ad Detective Scan for Chapter Marks" with a completely clear progress bar. A VideoRedo process exists but the tray icon does not. The log message "Video ReDo said: INFO: VideoReDo version 4.20.6.610 - Nov 18 2010" but does not have the "VideoReDo said: INFO: Edit mode set to cut" message. Stopping and starting VAP causes a new VRD process and Adscan/chaptering completes normally.

Maybe there's an issue in the bit that's setting the edit mode. Can you give me a test version without that bit of code that just uses the current setting from VRD?

Has anybody reported the problem when doing AdScan/AutoCut? Is that done in a significantly different way than AdScan/Chaptering?
 

dlflannery

Moderator
VAP is beginning to hang on the AdScan/chaptering process more frequently (maybe 1 in 3) and fails with both TiVo and WTV files. VAP shows the action as "Ad Detective Scan for Chapter Marks" with a completely clear progress bar. A VideoRedo process exists but the tray icon does not. The log message "Video ReDo said: INFO: VideoReDo version 4.20.6.610 - Nov 18 2010" but does not have the "VideoReDo said: INFO: Edit mode set to cut" message. Stopping and starting VAP causes a new VRD process and Adscan/chaptering completes normally.

Maybe there's an issue in the bit that's setting the edit mode. Can you give me a test version without that bit of code that just uses the current setting from VRD?

Has anybody reported the problem when doing AdScan/AutoCut? Is that done in a significantly different way than AdScan/Chaptering?
The VAP code and the VBS code being used are the same code used for AdScan without autocuts. The only thing different could be the arguments passed to Adscan.VBS and logged to arglog.txt, and you said these were identical whether a test file hung or not.

If you look at AdScan.vbs, the hang point has to be when it opens the file or when it sets the edit mode. You can create the version of AdScan.vbs that doesn't set the edit mode by just deleting or commenting out the following lines:
Code:
'Set edit mode to cut mode
editmode = Cint(1)
on error resume next
VideoReDo.SetCutMode(editmode) 
if err.number = 0 then
	Wscript.StdOut.WriteLine("INFO: Edit mode set to cut")
else
	Wscript.StdOut.WriteLine("INFO: Error setting cut mode")
end if
on error goto 0
Commenting is done with either ' or rem

How hard would it be for you to try running with whatever VRD build you used before 610 ?

One practical way the Chapter Mark scan differs from most previous experience, I suspect, is that the Ad Detective scan is being run on an H.264 video whereas typically it runs on a mpeg2 file. This shouldn't cause a problem with VRD but ........
 

tannebil

New member
I'll run with the edit mode code commented out for a few weeks to see if the problem goes away.

I'm going to be traveling most of the next month and will have limited access to my home system while I'm gone so "no news" just means "no news" rather than "good news".
 

laserfan

New member
Sorry if this is discussed already, but I dunno how to search on the questions I have. I've been using TivoPlaylist and TivoDecode in lieu of Tivo Desktop since...forever--but in particular I had trouble in the early daze with VRD and Tivo Desktop, and Playlist/Decode combo never failed for me, so I've stuck with it. Lately I'm finding that, to avoid doing a whole lot of work editing a Tivo transfer and then getting an error/crash out of VRD on "Save" (due to some error(s) in the stream), I've decided I need to QSF each/every file out of my TiVos that I want to edit. Currently this is a PIA to say the least:

1. Transfer from TiVo to PC using TiVoPlaylist

2. Invoke TivoDecode to make a VRD-readable (unencrypted) .mpg file

3. Run a QSF on the unencrypted .mpg file

4. Edit and Save my program to make it commercial-free

The above involves three transfers of sometimes monstrous files (for example a Super Bowl .tivo file is 35Gb!) between hard drives, and a boatload of hdd space of course.

So does the above get alot easier w/TVAP and TD installed i.e. transfer-and-decode-and-QSF in one swell foop? Does TD now, and always, allow perfect handling by VRD (it did not originally, which drove me to TiVoPlayList and TiVoDecode which have always worked perfectly)?

I recall Tivo Desktop was some pretty ugly bloatware but if you tell me it's gonna make my life alot easier than I may try it again! Of course, if there's a way to use TVAP with TivoPlayList and TivoDecode, I'd love to hear that even more.
 

dlflannery

Moderator
laserfan,

First the program is VAP, not TVAP. (TVAP was a previous program that only worked on TiVo files and is no longer supported.) Just to clarify, VAP automates running VRD. It doesn't run any other software other than what you might choose to do in a post-process batch file.

VAP will work fine with any mechanism you want to use for depositing files (including TiVo files) into its monitored folder. That includes Desktop and TiVoPlayList, both of which I use.

I don't understand the question: "Does TD now, and always, allow perfect handling by VRD?" Whether VRD has a problem with a file depends on a lot more than whether you use TD or TiVoPlayList to download it.

You can set up VAP to QSF all Tivo files that appear in the monitored folder, outputting to any format supported by the version of VRD you are using. If you desire you can also have it run Ad Detective or ComSkip on the QSF'ed file, creating a project (.vprj) file. Then TVAP waits for you to load the project file and review/adjust the commercial cuts. As soon as you save the modified project file, VAP then runs VRD again to produce the final output file. (Or you can just separately load the QSF'ed files into VRD and manually edit and save them if you prefer.)

Please see link in signature and the VAPreadMe.pdf file for more info and/or post further questions.
 

laserfan

New member
VAP will work fine with any mechanism you want to use for depositing files (including TiVo files) into its monitored folder. That includes Desktop and TiVoPlayList, both of which I use.

I don't understand the question: "Does TD now, and always, allow perfect handling by VRD?" Whether VRD has a problem with a file depends on a lot more than whether you use TD or TiVoPlayList to download it.
I'm asking not about "depositing files", but about the de-encryption aspect. In the earliest daze of VRD and TiVo files, it used to be that the combo of VRD and TD did not always work perfectly (maybe only for me, but regardless...) so I download w/TPL and decode w/TivoDecode as I said.

Simplest way I can ask this: is your program totally dependent on Tivo Desktop to decrypt the .tivo files before QSF? Or is there a way I can use Tivo Decode instead? I'm talking about this program that integrates with Tivo Play List:

http://tivodecode.sourceforge.net/
 

dlflannery

Moderator
I'm asking not about "depositing files", but about the de-encryption aspect. In the earliest daze of VRD and TiVo files, it used to be that the combo of VRD and TD did not always work perfectly (maybe only for me, but regardless...) so I download w/TPL and decode w/TivoDecode as I said.

Simplest way I can ask this: is your program totally dependent on Tivo Desktop to decrypt the .tivo files before QSF? Or is there a way I can use Tivo Decode instead? I'm talking about this program that integrates with Tivo Play List:

http://tivodecode.sourceforge.net/
Any way you can deposit a file that can be opened by VRD in the VAP monitor folder is fine. By default VAP will attempt to run VRD on any such file found in the monitor folder. But there is a file-ignore feature in VAP (again easily configured via the GUI). If, for example, you put ".tivo" in the file ignore list, then VAP would not process .tivo files, but would process .mpg (for example). AFAIK, VRD can open (and thus implicity decode) .TiVo files transferred by TD but if you prefer to do your own decoding it should be no problem, as just described.

The short answer is "No, VAP doesn't depend on TD to do TiVo decoding but that's because VAP itself doesn't do any decoding." Now, VRD as I understand it does depend on TD being installed in order for it to input (decode) .TiVo files. But you don't have to use TD to transfer from the TiVo if you prefer not to. Also, in case you didn't know, the only reason you need to keep the TD "server" running is to transfer files back to the TiVo.

Is metadata a consideration for you? VAP reads metadata from tivo, .mp4, .m4v, .wtv and .dvr-ms input files and will insert it into .mp4, .m4v, .wtv, and .dvr-ms output files. VAP also will optionally create a pyTiVo metadata file if possible, based on a command line switch. However for a .mpg input file, the only way VAP gets metadata is an automatic search of theTVDB.com performed for TV shows only. Thus letting VAP process the .tivo input instead of the decoded .mpg input has an advantage. (But this requires having TD installed so VRD can open the .tivo file.)
 
Last edited:

laserfan

New member
I think I've got it, thanks. I can use TPL to retrieve the .tivo file(s) I want, and tivodecode to decrypt them, then VAP to watch the "decrypted" folder for automatic QSFing by VRD, though since TivoDesktop would allow me to skip the tivodecode step I may bite-the-bullet and try it.

No I don't need metadata--I make all my recordings into Blu-ray discs for storage/safekeeping/settopBDplayer playback.
 

tannebil

New member
I'll run with the edit mode code commented out for a few weeks to see if the problem goes away.

I'm going to be traveling most of the next month and will have limited access to my home system while I'm gone so "no news" just means "no news" rather than "good news".
OK, I've now run the process against 100+ files and it's working pretty well. Maybe 15-20% of the files have some kind of a problem with commercial detection where an autocut file would have been missing part of the program so chaptering is definitely a better approach for me.

I continue to have the same two intermittent issues. The first is where VRD throws an error about being unable to open the input file for the Ad-Detective step. VAP/VRD continues properly after clicking the error message so it's not a big deal unless I don't login for a few days and processing gets backed-up The second issue is where VRD doesn't release the output file before running Drax so Drax throws and error. The fastest way for me to fix the problem is kill the VRD process if it's still running, delete the mp4 output file, and re-run the process in VAP.

I see one issue or the other every few days so that's about 10% of the files.

Both these problems feel like VRD problems rather than VAP problems. VAP 0.67 has been stable and I've seen no issues with it.
 
Top Bottom