Verbose feedback during Library Update

[vic3mila]

Would be nice to have a visual feedback because XBOX update library can take a lot of time (about one hour here).
On the top of the main screen there is a little box with two progress bar but the upper one is always empty (is this by design? depend from chosen skin?)
Maybe is possible to show current dir name / tv series name / movie name / song artist / ... ?
Something like:
- scanning {share name as assigned from XBOX} / {current dir} of {total dir on that share}
- found {total numer of objects}, analyzing {current one}, (eventually adding {name})

Example: (having "tv" as \\storage\video\tv series)
- scanning tv:
- found 1234 dirs,
- - current dir: NCIS has 42 seasons, analyzing #01
- - - analyzing tv/NCIS/season42/NCIS-42x01 That name.avi
- - - - added ...

All of above packed in two/three lines of current library update progress display, maybe without progress bar but with "{share name} - {current dir}/{total &} - {current season}/{total &} - {current episode}/{total &}" for tv series and something different for movies (there is no "season" on a movie). Not yet tried to add my music to library but something similar can be replicated there: {dir/total} - {song/total}.

Actually file name appear only when new...

I also added a transparent proxy (squid) between xbox and gateway (router) to speedup update process but it is still very slow
Last added {movies/tv episodes} is also really slow (about one minute). It depend from xbox HDD or from sqlite query? How can I speedup (if possible)?

The very best would be user selectable: no progress / bar only (like now) / verbose (this crazy stuff)
What about this idea? Should I ask to XBMC main project or can be a XBOX only enhancement?
Thanks for you time.

--
Riccardo.

[Dan Dar3]

I'd say most people with large libraries would use a media manager application to generate the .nfo files and thumbs on the PC - e.g. Media Companion http://mediacompanion.codeplex.com/ - so that information is stored next to the media itself, and XBMC4Xbox would be scanning that instead of an online scan. That way you can save quite a bit of time when starting with a fresh install. On the other hand, you can probably manually export the media information, or if you're doing this when updating XBMV4Xbox, you can look into the XBMC SVN Installer plugin I wrote, that would copy your userdata in between versions.

If you still want the visual update, it's probably best to bounce it off the main project - if you get more people behind it on the forums and will get implemented, I'd say will definitely make it easier to backport it here.

[vic3mila]

I'd say most people with large libraries would use a media manager application to generate the .nfo files and thumbs on the PC - e.g. Media Companion http://mediacompanion.codeplex.com/ - so that information is stored next to the media itself, and XBMC4Xbox would be scanning that instead of an online scan.

Thanks a lot for hint but... "Media Companion v3.500 and above requires .NET 4.0 Client" Any sh/perl/python script around? Maybe the same inside XBMC running standalone? Or a powered "scan for new content" that do an "update library only for this serie"? Anyway the slow part seems to be scanning share (or decoding XML or accessing sqlite) and not accessing online information. Can some user with XBMC on a different platform confirm this? Is a CPU or a network problem? Is aTV2 faster or is better to buy 2-3 spare XBOX?

Just lurked into squid logs:
- update started at Thu Nov 1 17:25:35 CET 2012
- update ended at Thu Nov 1 18:40:00 CET 2012
(about one hour and a quarter)
Every episode zip downloaded in 400/700/1000ms when accessing net (not cached) and 5-10ms when already on cache, for a total time of less of two minutes (89".636).
I wrote a script for pre-caching .../series/{id}/all/it.zip but the time remained the same
The other 43 minutes are lost between scanning share / uncompressing zip / decoding XML / accessing sqlite? How can I identify the slowest process (and try to speedup)?
(not a server problem, series are on high speed disk that can saturate 100Mbit switch w/out problems, no need for a gigabit switch because XBOX has a 100Mbit LAN)

That way you can save quite a bit of time when starting with a fresh install. On the other hand, you can probably manually export the media information, or if you're doing this when updating XBMV4Xbox, you can look into the XBMC SVN Installer plugin I wrote, that would copy your userdata in between versions.

The slow process is with an already populated library, think about adding 7-8 episodes on 3-4 different tv series. Yes, I know, is much faster using "scan for new content" on each changed serie, but that command doesn't refresh missing episodes title on series not yet completed, right? I already export (about once a week) library as separates files just in case of fault of XBOX and I obviously use automatic update process of XBMC using your wonderful XBMC SVN installer 1.8beta3 (used yesterday to update to nightly build r31572 ). Maybe HDD of XBOX is the slow ring? (no more the original one, now an 80GB for XBMC and thumbs, series are on external server ). Maybe I need to cleanup thumbs and other things on XBOX HDD?

If you still want the visual update, it's probably best to bounce it off the main project - if you get more people behind it on the forums and will get implemented, I'd say will definitely make it easier to backport it here.

And what about a flag (like seen / not seen) for completed series? Something like (scan / not) for consolidated ones?

--
Riccardo.

[Dan Dar3]

I always had the impression it was the regular expressions used in scanning for file names and episode patterns - if you got a predefined set I think there was a way to specify them in the advancedsettings.xml, the old wiki is gone, maybe you can find something on the web.

I'd say enable Debugging in XBMC4Xbox System settings and watch the log, you might find the parts being slow by comparing timestamps between lines. Another way would be from tracing the dir / file scan activity on the server share side. You don't seem to be using Windows (or I would've suggested Process Monitor), so I can't advise on that.

[vic3mila]

I always had the impression it was the regular expressions used in scanning for file names and episode patterns - if you got a predefined set I think there was a way to specify them in the advancedsettings.xml, the old wiki is gone, maybe you can find something on the web.

XBMC use python for scanning? http://swtch.com/~rsc/regexp/regexp1.html

I'd say enable Debugging in XBMC4Xbox System settings and watch the log, you might find the parts being slow by comparing timestamps between lines. Another way would be from tracing the dir / file scan activity on the server share side. You don't seem to be using Windows (or I would've suggested Process Monitor), so I can't advise on that.

Some time ago I also tried a fresh install - fresh scan (loosing seen/not seen flag, just to test during incompatible sqlite db change). Do you remember that I export all library as separate files to shares? The scanning of update library was slow the same :'( If you are right and RE is so slow, how can we improve performance? I'm pretty sure that other people have more than "ten" episodes in their libraries...

--
Riccardo.

[Dan Dar3]

No, I believe the code is C, but you can change the patterns externally, to leave or change them to suit your filenames.

This is from mainline XBMC but I think it should work here as well - maybe try that see if it makes any difference.
http://wiki.xbmc.org/index.php?title=Ad ... ttings.xml

See a sample startup entry here (rename by removing template and customize with only the parts you need and place in Q:\UserData and probably restart XBMC4Xbox):
http://redmine.exotica.org.uk/projects/ ... ttings.xml

Turning on debug logging would show you timings in the xbmc.log so you can see which part is actually slow and then go from there. Maybe upload an excerpt to pastebin.com if you don't want to paste everything, ideally something from the top so we can see details about the version used and anything else, maybe a couple of scan results.

http://www.xbmc4xbox.org.uk/forum/viewt ... p?f=6&t=65

[vic3mila]

No, I believe the code is C, but you can change the patterns externally, to leave or change them to suit your filenames.

To speed up debugging phase I installed FreeBSD port of XBMC (to avoid ftp in/out setting and log files).
Ok, I know, it is not the same version (it show as Eden 11.0 Git:Unknown), but this way I can check changes to settings more fast and without having to share TV with my sons
Without advancedsettings.xml it took less than 3 minutes to complete (same share as XBOX but mounted NFS instead of SMB)
After adding this:

Code: Select all

<advancedsettings>
        <tvshowmatching append="no">
                <regexp>-([0-9]+)x([0-9]+) ([^\\/]*)$</regexp>
        </tvshowmatching>
</advancedsettings>

log file show the matching pattern, so RE would be correct:

DEBUG: VideoInfoScanner: Found episode match .../alphas/season01/alphas-1x01 pilot.avi (s1e1) [-([0-9]+)x([0-9]+) ([^\\/]*)$]

Just a question: why file name is showed as all-lowercase instead as found on disk (would be .../Alphas/season01/Alphas-1x01 Pilot.avi)?

Log debug is really full of info, just started ad update library on the real XBMC (the one on XBOX ) with debug on, after that I'll try again with reduced RE expression. BBL.

--
Riccardo.

[Dan Dar3]

I don't know why names are in lower case. Since you're experimenting and seem to know your stuff, I would try using FTP sources, see how they compare with SMB, might be faster. I know some people reported improved streaming with FTP vs SMB, might make a difference here as well.

Also, I could be wrong here, but I think I keep all episodes in a single tv show folder, and they still show up on scan. See if that is scanned faster than multiple season sub-folders.

For me, networking on XBMC4Xbox was always slower - I guess it's might be down to the SDK developed a good few years ago, and on top of that there are libraries to provide the protocols, although to be honest I never looked at that part of the source code. I am pretty sure though if it was possible to improve, Arnova and Buzz would've done something to improve it already. They've applied a lot of fixes to the SMB libraries and all that, the whole slowness though might be in the end down to the underlying TCP/IP implementation.

I don't know if this applies much to your case, but maybe you can make something out of it:
http://redmine.exotica.org.uk/issues/10

[vic3mila]

I don't know why names are in lower case. Since you're experimenting and seem to know your stuff, I would try using FTP sources, see how they compare with SMB, might be faster. I know some people reported improved streaming with FTP vs SMB, might make a difference here as well.

Never tried (yet) because FTP is not a streaming protocol. It may works for simple play (or forward/fast forward) but what happens on rewind? Ok, ok, going to try...

Also, I could be wrong here, but I think I keep all episodes in a single tv show folder, and they still show up on scan. See if that is scanned faster than multiple season sub-folders.

Not an option, I prefer a "clean source", usable even without XBMC.

For me, networking on XBMC4Xbox was always slower ... I don't know if this applies much to your case, but maybe you can make something out of it: http://redmine.exotica.org.uk/issues/10

SMB is a bad protocol, also when used between win* machines

update: today scan (still on SMB) was 24', with reduced RE only 21' (not a big improvement), only 2'30" for movies, the remaining for tv series. Some tv series path accessed 2-3 times (log entry Using authentication url smb://...). FPS at 50 all the time, CPU ranging from 50 to 70% (mainly between 55-60%), freemem never under 27MB.
Still seeing two progress bar on scan mini window (not happens on eden version). Maybe is a skin problem?

--
Riccardo.

update: tried ftp://
- installed SVN:31571 as upgrade of 31572 (copying userdata) and as clean install (no copy)
- adding a source with hostname doesn't resolve (I have a dhcp reservation for xbox mac-address, xbox get IP from dhcp but doesn't use domain part)
- adding a source with FQDN works but:
- - pointing a dir that end with / (slash) show as double //, like ftp://mystorage.my.home:21//video/
- - pointing a dir without ending / make it impossible to edit (doesn't show mini menu edit-remove), edit-remove still possible accessing userdata/mediasources.xml
- - editing an already configured FTP source show username and password on source list ( like ftp://myuser:mysecret@mystorage.my.home:21//video/ )
Verified also on Eden 11.0, same problems there
- tried confluence and confluence lite, first scan show movie/tv series names on mini windows and second (empty) progress bar doesn't appear
- first scan locked after 38 episodes :'(
- next scan found about half episodes but all series names (all episoded has .nfo files exported from previous run)
- every update library (after the first) found some other episodes, after 3-4 run I manually run refresh from inside some series, one using .nfo file w/out success, one refreshing from internet

... locked again, this time hard (switched off from main)

Maybe too many changes at the same time. Starting from scratch tomorrow

[vic3mila]

Sorry for a long silence but still working on XBOX (after some err.13 it doesn't switch on anymore, sigh).
Keeping a lot of time disconnected from mains restart, show X logo and start clicking (HDD is gone).
Thinking about a replacement (power supply and HDD) or a switch to a different platform...
Any user here with experience about ATV v2 (or another hw) please PM, I think this is a bit OT here.

The only good news is that last ftp backup is not so old...

--
Riccardo.