NextPVR - Three questions concerning VAP and having it Parse from XML files. Thanks!

dlflannery

Moderator
@jtcreate,

VRD will not load the .ts example files you provided, yielding error: "No PIDs specified when trying to open a transport stream". For example one of the files successfully processed per your VAP log file:

Entertainment Tonight_20170608_19302000.ts

I noticed this when I tried to process it in VAP, but it will not load directly into the VRD program either, producing the error message as above.

I suspect you created small test files clipped from your much larger originals and something must have gone wrong in that process. This isn't critical to me. I can create a small .ts file containing h.264 video and rename copies of it as needed for testing. However, unless this issue can be corrected, there would be no point in including the .ts file samples in any future samples you may provide.

Are you planning to use the BeyondTV output xml files? I see that CL option in your VAP log at startup. I don't believe they provide nearly as much metadata as the pyTivo .txt output files. I can provide a custom-designed xml output format -- I just would like to have it firmly specified so that major code revisions aren't frequently needed.
 

jtcreate

Member
@jtcreate,

VRD will not load the .ts example files you provided, yielding error: "No PIDs specified when trying to open a transport stream". For example one of the files successfully processed per your VAP log file:

Entertainment Tonight_20170608_19302000.ts

I noticed this when I tried to process it in VAP, but it will not load directly into the VRD program either, producing the error message as above.I suspect you created small test files clipped from your much larger originals and something must have gone wrong in that process. This isn't critical to me. I can create a small .ts file containing h.264 video and rename copies of it as needed for testing. However, unless this issue can be corrected, there would be no point in including the .ts file samples in any future samples you may provide.
Ok, I forgot about that! Came across that issue too. Easy fix! The PID error is a VideoRedo created issue, not a .ts problem. I've already talked with Videoredo and they gave me a simple setting adjustment. Open your Videoredo app. Click "Tools". Hold down shift key and click "options" in VideoRedo. Ramp up your "number of times PMT has to be read" setting to "50". Then Go to "Stream parameters" in options. Make sure the box "Ignore transport stream Maps" is not checked. That should completely solve the problem. You should be able to open practically any .ts file at any state of length.

Are you planning to use the BeyondTV output xml files? I see that CL option in your VAP log at startup. I don't believe they provide nearly as much metadata as the pyTivo .txt output files. I can provide a custom-designed xml output format -- I just would like to have it firmly specified so that major code revisions aren't frequently needed.
Completely agree with you on finalizing the XML format to avoid future revisions. I've actually been focusing my ideas on that same goal. BTV XML seems to be more than suitable and you have already done the work on that(seems Pytivo txt mirrors the same info as BTV). VAP doesn't need any additional information from the original XML file. But I do think keeping any extra data in the original XML is important in case I ever add any future arguments/data to the XML file from the NEXTPVR. However, that added data wouldn't have anything to do with VAP's processes. It would just need VAP to detect anything extra and write it into the BTV element. I hope that is not too complex.


You mentioned CL option? Do you mean command line? I use command line for the additional handbrake processing as X264 seems to handle motion better than the VideoRedo MainConcept encoder when doing the final squeeze down to MP4. I don't use it to contain or process XML info.
 

dlflannery

Moderator
Ok....Potential is definitely there. A few bugs and a file maintenance issue I overlooked. Sorry for the long list.


1# {title} tag is not populating. Instead, the "EpTitle" information is appearing in the title tag instead. This is occuring even when I parse from the .ts file name in advanced settings (i.e. .ts {title}_{wildcard} ). The Season and Episodes numbers are not double digits each. s02e08 instead of s2e8. Could cause issues when organizing a library by alpha. Perhaps VAP can make a simple correction when it detects a single digit.

This is my current naming convention:

{title} ~s{seasonnum}e{epnum}~ OAD {year} {month}-{day} ''{eptitle}'' [RDT- {callsign} {wildcard}]

This is what VAP is outputting.

A Problem Like Maria ~s1e17~ OAD 2017 03-08 ''A Problem Like Maria'' [RDT- 20170613_20002100]


2# Original XML file fate should be the same as the Original .ts file. If VAP deletes or moves the original .TS file from the 'monitor' folder to a another destination folder or recycle bin, the same should happen with the original XML file too.

3# Some XML files like "Access Hollywood" are missing "EPtitle" info and TVDB has no listing. I'm still thinking this one over. Perhaps copy and paste title into eptitle as a last resort.

4# VAP Missing a "Callsign" tag.


5# I did notice that VAP was trying to do TVDB searches on the .ts files before the XML file arrived into the 'Source' directory. Perhaps the NEXTPVR switch is necessary so that VAP knows to wait for the XML file to arrive once the TS file finishes transfering. Although once in a blue moon, a TS file may not have an XML companion. As a failsafe, VAP can procede TS file processing without the XML file after a certain amount of time has passed.



I made some more samples and put them in the link below and I am including my VAP log from the test to help. Let me know if there is something I can do. Thanks.

https://1drv.ms/f/s!AnHKMcbNlt04pAVfUJow_hYSLAXf
Try the test executable (not an installer) contained in the zip downloaded from this link:

VAPexe133T3.zip

I think this addresses most of the issues quoted above except:
1. Having to wait for the xml file to turn up is something I was not aware of previously. However I think there is a simple solution:
On the "Other" tab of the Advanced Configuration form, set "Delay time after detection before processing a file in monitor folder" to some time you think is more than enough for the xml file to show up. Note the time is in minutes but you can use decimals, e.g., 0.2 would be 12 seconds..

2. Need to clarify the issue about season and episode numbers needing to be two digits wide.
 
Last edited:

jtcreate

Member
QUOTE=dlflannery;125496]Try the test executable (not an installer) contained in the zip downloaded from this link:

VAPexe133T3.zip

I think this addresses most of the issues quoted above except:
1. Having to wait for the xml file to turn up is something I was not aware of previously. However I think there is a simple solution:
On the "Other" tab of the Advanced Configuration form, set "Delay time after detection before processing a file in monitor folder" to some time you think is more than enough for the xml file to show up. Note the time is in minutes but you can use decimals, e.g., 0.2 would be 12 seconds..[/QUOTE]



NICE! Operates very well. I think we almost have it. Also, your recommendation to have VAP delay the file processing in Advance Settings does just fine.

I did notice something that I never saw before. I'm kind of curious why VAP is moving deleted inputs files before the Handbrake post-process is completed. It would probably be best to wait until Handbrake was completed before moving the the original files to delete. One of the things I really liked about VAP is that if the computer crashes or the power drops and the system restarts, VAP starts over and reprocesses the file. The delete process is definitely something that should be executed once all the other processess are finished.

2. Need to clarify the issue about season and episode numbers needing to be two digits wide.

As far as the EP numbering, the problem is that the networks are not very consistent with one another (Especially MTV and BET who are not even consistent with themselves one week to the next! ). Some inconsistencies are unavoidable and I can work around those. Season/Episode numbering inconsistencies are more frequent. One BET show might report s2e3 for a couple of weeks and then switch to double digit standard s02e04. It seems to depend who at BET is reporting the schedule. Very lazy if you ask me. Amazing! We can put a man on the moon but we can't have a decent EPG standard? Also, Some networks like ABC will stick to a strict double digit standard s02e03, while AMC might just do s2e3. So the best way to correct this is to force all episode numbers to one standard. The one I see the most is double digit.

The episode number mixup messes up how the files are viewed the directory. Mainly when it comes to organizing by Season number. I made a pic sample below.

File Sample.png
 

dlflannery

Moderator
2-digit season and episode numbers, and delaying input file move or deletes until after post-process script completion are in the VAP test executable (not an installer) contained in the zip downloaded from here:

VAPexe133T4.zip

I'm curious, what delay time did you settle on for "Delay time after detection before processing a file in monitor folder"?
 

jtcreate

Member
Damn! Just when I think we got it, one more bug shows up. If VAP doesn't detect an element for Season or Episode in the original XML, it just puts zeros into SeasonNum and Epnum. I don't think its checking TVDB as an alternative. Check out the XML for American Ninja Warrior. Its one of the samples I send you.
 

dlflannery

Moderator
Actually according to theTVDB.com the episode of ANW you provided is season 0, episode 24. My VAP log says theTVDB.com search is succeeding -- it's using the series title and the OAD date as the search keys. The processed output file name I get using your templates is:

American Ninja Warrior-S00E24 (KNBC) 20170604_20002300.mkv

Which is as expected, correct?

However, I discovered that with Ver. 1.33T4, missing (blank, empty string) season or ep numbers do result in 00 whereas the intended behavior always has been to just insert empty strings (e.g., SE). Or if the first letter of the template tag is capitalized, to not rename the file at all -- read the help on the config tab where the templates are entered. However this issue did not affect the ANW episode in question. The season actually is zero, so S00 is correct.

The VAP test executable (not an installer) in the zip downloaded from here:

VAPexe133T5.zip

corrects the issue introduced in the previous test version where blank season or episode numbers would be given as 00.

Still curious as to what delay time you settle on for the xml files.
 

jtcreate

Member
Actually according to theTVDB.com the episode of ANW you provided is season 0, episode 24. My VAP log says theTVDB.com search is succeeding -- it's using the series title and the OAD date as the search keys. The processed output file name I get using your templates is:

American Ninja Warrior-S00E24 (KNBC) 20170604_20002300.mkv

Which is as expected, correct?

However, I discovered that with Ver. 1.33T4, missing (blank, empty string) season or ep numbers do result in 00 whereas the intended behavior always has been to just insert empty strings (e.g., SE). Or if the first letter of the template tag is capitalized, to not rename the file at all -- read the help on the config tab where the templates are entered. However this issue did not affect the ANW episode in question. The season actually is zero, so S00 is correct.

The VAP test executable (not an installer) in the zip downloaded from here:

VAPexe133T5.zip

corrects the issue introduced in the previous test version where blank season or episode numbers would be given as 00.

Still curious as to what delay time you settle on for the xml files.

Right now, about 30 seconds. I'm playing it safe for the moment and seeing how it all performs. Hopefully, we get some solid results. I look forward to testing out your build this weekend.
 

jtcreate

Member
Right now, about 30 seconds. I'm playing it safe for the moment and seeing how it all performs. Hopefully, we get some solid results. I look forward to testing out your build this weekend.

So far, its worked very very well. No crashes. Very stable. I did see a couple recordings over the weekend where VAP didn't parse and replace the filename. I'm going to run some more checks to see if the XML file is arriving to VAP monitor folder. I am parsing the Ts filename as a backup so I can at least obtain the recording information. I mainly use the {title}_{wildcard} as I did with the .wtv files.
 

jtcreate

Member
Latest test

Ok, aside from this one issue, VAP seems to be solid in its performance. I accidentally deleted a couple test, but I made the latest one over the weekend. I did a recording sample of "Hang'em High" where VAP errors and doesn't change the filename from the original. I believe if I recorded it through MediaCenter, VAP would have change the filename. I just started to brain storm this a bit. Here is the XML sample for you to take a look. Thanks

View attachment Hang Em High_20170624_08301100.xml
 

dlflannery

Moderator
Ok, aside from this one issue, VAP seems to be solid in its performance. I accidentally deleted a couple test, but I made the latest one over the weekend. I did a recording sample of "Hang'em High" where VAP errors and doesn't change the filename from the original. I believe if I recorded it through MediaCenter, VAP would have change the filename. I just started to brain storm this a bit. Here is the XML sample for you to take a look. Thanks

View attachment 2248
Please expand description of "VAP errors". If you see error messages in the log file, please provide them. Is it failing to produce output BTV xml files?

Also, please provide:
1. Movie Output file name/path renaming template
2. TV Series Output file name/path renaming template
3. Templates for parsing metadata from input file names

One problem I see is that VAP has no way to know the movie year for a movie when the input file has no metadata, unless it is in the NextPVR xml file (it isn't) or is parsed from the input file name. Currently VAP has to have a movie year in order to know the video is a movie. I can fix this by using the fact that the NextPVR xml files seem to have "movie' as one of the genres (for movies). But I want to be clear exactly what erroneous performance you were seeing so I can be sure I'm working on the right problem.
 

jtcreate

Member
Ok. I've let the system run for a couple of weeks and there is an issue with some of the XML info. Its not showing up in the log from what I can see. But, there were more shows not being renamed than I expected. I zipped up some xml file samples that are not working and added them here. I added one movie XML as well.

View attachment XML issues 20170713.zip
 

dlflannery

Moderator
4 of the 5 xml samples are neither movies nor series episodes. VAP has never produced metadata outputs for those types of videos. "The Patriot" is the only one that has "Movie" in the genre list. As I mentioned before I can mod VAP to detect that, and it appears most nextPVR movie input xml files have the movie year (which VAP needs to determine if it is a move) encoded in the Original Air Date.

I've modified VAP to use this logic and did a test run using the "Suicide Squad" xml you previously furnished. Here is the resulting BTV output xml:

Code:
<?xml version="1.0" encoding="utf-8"?>
<episode-properties>
  <property name="EpisodeDescription">Figuring they're all expendable, a U.S. intelligence officer decides to assemble a team of dangerous, incarcerated supervillains for a top-secret mission. Now armed with government weapons, Deadshot (Will Smith), Harley Quinn (Margot Robbie), Captain Boom</property>
  <property name="Title">Movies</property>
  <property name="DisplayTitle">Movies</property>
  <property name="MovieYear">2016</property>
  <property name="OriginalAirDate">20160101</property>
  <property name="TargetStart">131370089380000000</property>
  <property name="Genre">Movie / Action / Adventure / Fantasy</property>
  <property name="StationCallSign">HBOHDP</property>
  <property name="Channel">7696</property>
</episode-properties>
Does this look acceptable for your purposes?
 

jtcreate

Member
4 of the 5 xml samples are neither movies nor series episodes. VAP has never produced metadata outputs for those types of videos. "The Patriot" is the only one that has "Movie" in the genre list. As I mentioned before I can mod VAP to detect that, and it appears most nextPVR movie input xml files have the movie year (which VAP needs to determine if it is a move) encoded in the Original Air Date.

I've modified VAP to use this logic and did a test run using the "Suicide Squad" xml you previously furnished. Here is the resulting BTV output xml:

Code:
<?xml version="1.0" encoding="utf-8"?>
<episode-properties>
  <property name="EpisodeDescription">Figuring they're all expendable, a U.S. intelligence officer decides to assemble a team of dangerous, incarcerated supervillains for a top-secret mission. Now armed with government weapons, Deadshot (Will Smith), Harley Quinn (Margot Robbie), Captain Boom</property>
  <property name="Title">Movies</property>
  <property name="DisplayTitle">Movies</property>
  <property name="MovieYear">2016</property>
  <property name="OriginalAirDate">20160101</property>
  <property name="TargetStart">131370089380000000</property>
  <property name="Genre">Movie / Action / Adventure / Fantasy</property>
  <property name="StationCallSign">HBOHDP</property>
  <property name="Channel">7696</property>
</episode-properties>
Does this look acceptable for your purposes?


Yeah. It looks fine.

4 of the 5 xml samples are neither movies nor series episodes. VAP has never produced metadata outputs for those types of videos.

Is there anyway to go to TVDB as a back up using the title and the OAD? There is also IMDB and TV.com sources too. Some of those shows are reality series shows that are not being fulling detailed in the XML.
 
Last edited:

dlflannery

Moderator
Did you notice the Title and DisplayTitle in the example I posted above are both just "Movies" ? That seems questionable to me. I would think one or both of them should be the title of the movie. What would you prefer?

VAP already searches theTVDB based on the info it gets from an input NextPVR xml file. But theTVDB only includes series TV shows. Specials (unless considered part of a TV Series) and movies are not included.

I author another program that will search theMovieDB.org for movies. It will find most of the movies you've sent as examples but not generally specials: metagenerator-version-3

It would be a somewhat major exercise to integrate the movieDB searches into VAP, even more for any other source -- probably more than I wish to do.

However, perhaps some good news: As you may have noticed in another thread here I'm about to stop supporting the BeyondTV xml output. This means I can re-purpose it to be a customized "jtcreate" xml output, with the likelihood of improving response to your needs. More metadata and/or different xml formatting are possible, with the caveat that the metadata has to be obtainable by VAP from these sources:
1. TVDB search (series TV shows only)
2. Embedded in input file (.mp4, .wtv or .tivo)
3. Input NextPVR xml file
4. Parsed from input video file name using user-defined template.
 
Last edited:

jtcreate

Member
Did you notice the Title and DisplayTitle in the example I posted above are both just "Movies" ? That seems questionable to me. I would think one or both of them should be the title of the movie. What would you prefer?
I took a second look and thought it over. funny thing is that its not something that bothers me either way. I'll leave that to your descretion. Whatever makes things easier for you.

VAP already searches theTVDB based on the info it gets from an input NextPVR xml file. But theTVDB only includes series TV shows. Specials (unless considered part of a TV Series) and movies are not included.
Ah! I didn't fully realise that TVDB has limits when it comes to TV specials. hmmm. Ok. In those cases, all I really need VAP to do is make the filename change. The output BeyongTV xml file can stand as it is. There is still useful data in the original XML to help with the file name change. So far, all the original XML files still have the "title", "OAD", "Channel" and "decription". They tend to miss the "Subtitle" or "Eptitle". When we first stated this endeavour, I mentioned that if the "subtitle" or "Eptitle" was not available, I was fine with you filling that missing data with the "title". I should have gone with part of the "Description" instead that is capped at a particular character length (probably around 55).

I author another program that will search theMovieDB.org for movies. It will find most of the movies you've sent as examples but not generally specials: metagenerator-version-3
It would be a somewhat major exercise to integrate the movieDB searches into VAP, even more for any other source -- probably more than I wish to do.
Understandable. Less is more in this case.

However, perhaps some good news: As you may have noticed in another thread here I'm about to stop supporting the BeyondTV xml output. This means I can re-purpose it to be a customized "jtcreate" xml output, with the likelihood of improving response to your needs. More metadata and/or different xml formatting are possible, with the caveat that the metadata has to be obtainable by VAP from these sources:

1. TVDB search (series TV shows only)
2. Embedded in input file (.mp4, .wtv or .tivo)
3. Input NextPVR xml file
4. Parsed from input video file name using user-defined template.
That could be good news. How come you are dropping the support for Beyond TV?
 
Top Bottom