Author Topic: Offline Scantool (for X2 models)  (Read 82961 times)

Offline Lupissimo

  • Senior Member
  • ***
  • Posts: 395
  • Helpful Contribution Status: +6/-0
Re: Offline Scantool
« Reply #15 on: April 11, 2012, 05:23:43 PM »
Clarification about the rating...

Although the IMDB ratings only go up to 10, they do use one decimal point.  You can have a rating on IMDB of 8.5.  However, this will download to the xml files as 85 (not 8.5), so it appears the scale is to 100 (10.0).  In the movie jukebox, there is an option to show movies by ratings, 1-5, which is why Sanji divide the rating by 2 and round when storing the rating in the db.  The end result stored in the movie db is a whole number between 1 and 5, to match the jukebox option.

I do not see this when using Thumbgen to create the xml file. In my xml it is stored as  <rating>6.2</rating>.


Offline Unreal

  • B8a Tester - X2
  • Experienced Member
  • ****
  • Posts: 228
  • Helpful Contribution Status: +1/-0
    • UnrealRG Home Page
Re: Offline Scantool
« Reply #16 on: April 11, 2012, 05:32:22 PM »
As I have no insight into the mede8er code I can only guess that the hashcode is used to skip already existing movies by comparison while scanning. The timestamp is the" win last write" translated into unix time:
Win LastWrite =  LocalFileTimeToDateTime(LastWriteFileTime))
Local Time, not UTC
UnixTime  =  FileTimeToUnixTime(LastWriteFileTime))
for example:
Win LastWrite = 10.12.2010 21:25:26
UnixTime      = 1292016325
and is probably used to sort for actuality.

There is an option in the movie jukebox to "Sort Search by Latest", which probably uses this field for sorting.  I believe the intention is to show the movies added (most recently written to) first in the jukebox.
MED450X2 with PCIe Wireless                                 Firmware 3.0.2, NAS off, DLNA DMR&DMS enabled
Samsung LED HDTV (UN46C6300)                           Windows 7 Enterprise x64 SP1
Samsung USB Drives [2x2TB] (HX-DT020EB/B62)      Asus RT-N66U Router (FW 3.0.0.4.260)
Analog audio output to Pioneer Receiver (VSX-516)   HDMI audio output to HDTV

Offline Unreal

  • B8a Tester - X2
  • Experienced Member
  • ****
  • Posts: 228
  • Helpful Contribution Status: +1/-0
    • UnrealRG Home Page
Re: Offline Scantool
« Reply #17 on: April 11, 2012, 05:35:31 PM »
I do not see this when using Thumbgen to create the xml file. In my xml it is stored as  <rating>6.2</rating>.

This could be a scraper/skin difference then.  I use Y2M/YAMJ with this skin...

http://www.mede8erforum.com/index.php/topic,8279.0.html

...and it stores the rating in the XML file without the decimal point, so "8.5" is stored as "85".
MED450X2 with PCIe Wireless                                 Firmware 3.0.2, NAS off, DLNA DMR&DMS enabled
Samsung LED HDTV (UN46C6300)                           Windows 7 Enterprise x64 SP1
Samsung USB Drives [2x2TB] (HX-DT020EB/B62)      Asus RT-N66U Router (FW 3.0.0.4.260)
Analog audio output to Pioneer Receiver (VSX-516)   HDMI audio output to HDTV

Offline Unreal

  • B8a Tester - X2
  • Experienced Member
  • ****
  • Posts: 228
  • Helpful Contribution Status: +1/-0
    • UnrealRG Home Page
Re: Offline Scantool
« Reply #18 on: April 11, 2012, 05:45:42 PM »
You seemed to have figured out all of the movie related fields already.  If you are looking for info regarding the SQLite header fields, you may find this helpful...

http://www.sqlite.org/fileformat.html
MED450X2 with PCIe Wireless                                 Firmware 3.0.2, NAS off, DLNA DMR&DMS enabled
Samsung LED HDTV (UN46C6300)                           Windows 7 Enterprise x64 SP1
Samsung USB Drives [2x2TB] (HX-DT020EB/B62)      Asus RT-N66U Router (FW 3.0.0.4.260)
Analog audio output to Pioneer Receiver (VSX-516)   HDMI audio output to HDTV

Offline Lupissimo

  • Senior Member
  • ***
  • Posts: 395
  • Helpful Contribution Status: +6/-0
Re: Offline Scantool
« Reply #19 on: April 11, 2012, 05:47:03 PM »
Quoting Maasbommel:
 It should be the IMDB figure like this:

We get from IMDB Rating = '81'
so we divide by 2 and take it to the nearest full digit
So '81' will get a rating of '4' in the db

Unquote
which cannot be correct as 81 / 2 = 40.5 and the nearest digit would be 40 not 4.
Hopefully Maasbommel can clearify.
If it should be stored as 81 instead of 8.1 then ThumbGen needs to be corrected also.

Offline Unreal

  • B8a Tester - X2
  • Experienced Member
  • ****
  • Posts: 228
  • Helpful Contribution Status: +1/-0
    • UnrealRG Home Page
Re: Offline Scantool
« Reply #20 on: April 11, 2012, 06:24:05 PM »
...which cannot be correct as 81 / 2 = 40.5 and the nearest digit would be 40 not 4.

I suspect this is how it is done...

81 divided by 2 = 40.5

...forget the real location of the decimal and ALWAYS place it after the first digit, so...

40.5 becomes 4.05

...now round to the nearest single digit whole number, which is 4.  Obviously I do not know the actual algorithm, but they may only care about the first two digits, and use only the second digit (ignoring the third digit) to determine whether to round up or down the first digit.
« Last Edit: April 11, 2012, 06:36:16 PM by Unreal »
MED450X2 with PCIe Wireless                                 Firmware 3.0.2, NAS off, DLNA DMR&DMS enabled
Samsung LED HDTV (UN46C6300)                           Windows 7 Enterprise x64 SP1
Samsung USB Drives [2x2TB] (HX-DT020EB/B62)      Asus RT-N66U Router (FW 3.0.0.4.260)
Analog audio output to Pioneer Receiver (VSX-516)   HDMI audio output to HDTV

Offline Lupissimo

  • Senior Member
  • ***
  • Posts: 395
  • Helpful Contribution Status: +6/-0
Re: Offline Scantool
« Reply #21 on: April 11, 2012, 06:45:26 PM »
If you look into the mede8er.db you will find that it is not as simple as that. An imdb rating of 5.4 becomes a mede8er rating of 3. A rating of 6.1 becomes 4 and an 8.1 becomes 5.

Offline Unreal

  • B8a Tester - X2
  • Experienced Member
  • ****
  • Posts: 228
  • Helpful Contribution Status: +1/-0
    • UnrealRG Home Page
Re: Offline Scantool
« Reply #22 on: April 11, 2012, 07:10:07 PM »
Keep in mind the rating on IMDB is constantly changing, so be sure you are looking at the value in your movie.xml file (which could have a different value than what is currently on IMDB).

Also, for your examples...

54/2=27
2.7 rounded to the nearest single digit whole number is 3, so this looks correct.

For the others...

61/2=30.5
3.05 rounded to the nearest single digit whole number is 3, so this seems wrong.

81/2=40.5
4.05 rounded to the nearest single digit whole number is 4, so this seems wrong too.

Like already stated, make sure you are looking at the values in your XML files, not IMDB.  If you are looking at the values in the XML files, then this may be a Mede8er scanner bug.
« Last Edit: April 11, 2012, 07:21:08 PM by Unreal »
MED450X2 with PCIe Wireless                                 Firmware 3.0.2, NAS off, DLNA DMR&DMS enabled
Samsung LED HDTV (UN46C6300)                           Windows 7 Enterprise x64 SP1
Samsung USB Drives [2x2TB] (HX-DT020EB/B62)      Asus RT-N66U Router (FW 3.0.0.4.260)
Analog audio output to Pioneer Receiver (VSX-516)   HDMI audio output to HDTV

Offline Skylinc

  • B8a Tester - 1000X3D
  • Hero Member
  • ****
  • Posts: 519
  • Helpful Contribution Status: +2/-0
Re: Offline Scantool
« Reply #23 on: April 11, 2012, 08:19:11 PM »
61/2=30.5
3.05 rounded to the nearest single digit whole number is 3, so this seems wrong.

81/2=40.5
4.05 rounded to the nearest single digit whole number is 4, so this seems wrong too.
Unreal, why do you say these look wrong?
81/100 = 81% and 4/5 = 80%. They are both represented as 4 stars on M8R...
Product: Mede8er MED1000X3D f/w v2.0.0 (5Feb2013)
Audio: HDMI > LG 42LW4500
Audio setting: HDMI Passthrough / Night mode Comfort
Video: HDMI output > LG 42" LED Cinema3D 42LW4500
Video setting: 1080p 50Hz, 23Hz ON,16:9
Media Source: Wired Network > Win 8 Professional SMB shares, OpenMediaVault Shares
Hard Drive: 2TB Western Digital

Offline Lupissimo

  • Senior Member
  • ***
  • Posts: 395
  • Helpful Contribution Status: +6/-0
Re: Offline Scantool
« Reply #24 on: April 11, 2012, 08:39:01 PM »
We did a lot of comparison between mede8er.db values created by MED400X2 and our own algorithm (> 1600 movies) to find out an algorithm to match the med400X2 behaviour. And as the actual IMDB value is not important for the calculation but only the values MED400X2 calculates from a given rating in the xml I am pretty sure the cited values are correct ( They are the same Med400X2 comes up with)

Offline Unreal

  • B8a Tester - X2
  • Experienced Member
  • ****
  • Posts: 228
  • Helpful Contribution Status: +1/-0
    • UnrealRG Home Page
Re: Offline Scantool
« Reply #25 on: April 12, 2012, 01:20:33 AM »
@Skylinc

It was in response to the previous post by Lupissimo who stated the Meded8er converted 61 (6.1) to 4 stars and converted 81 (8.1) to 5 stars.  Proper  conversion from a base 10 (with a single decimal) to base 5 (single digit whole number) rating with rounding would be for 61 (6.1) to become 3 stars and 81 (8.1) to become 4 stars.  I was pointing out that either the Mede8er converted it wrong or Lupissimo was looking at the wrong values (IMDB current rating versus the rating stored in the XML file, since the online value could have changed since he last scraped with thumbgen).

From the last post by Lupissimo, it seems he has the right algorithm, so all is good.
MED450X2 with PCIe Wireless                                 Firmware 3.0.2, NAS off, DLNA DMR&DMS enabled
Samsung LED HDTV (UN46C6300)                           Windows 7 Enterprise x64 SP1
Samsung USB Drives [2x2TB] (HX-DT020EB/B62)      Asus RT-N66U Router (FW 3.0.0.4.260)
Analog audio output to Pioneer Receiver (VSX-516)   HDMI audio output to HDTV

Offline Lupissimo

  • Senior Member
  • ***
  • Posts: 395
  • Helpful Contribution Status: +6/-0
Re: Offline Scantool
« Reply #26 on: April 12, 2012, 11:59:41 AM »
In the meantime I have talked to the person who has helped me with programming my off-line scantool and we have agreed to publish it. It runs on any win32bit OS and scans directories accessible via the OS including network shares. Besides creating the mede8er.db in the scanned starting directory it checks and reports missing or empty xml files. Those who are interested can contact me via PM. 

Offline Lupissimo

  • Senior Member
  • ***
  • Posts: 395
  • Helpful Contribution Status: +6/-0
Re: Offline Scantool
« Reply #27 on: April 12, 2012, 02:14:52 PM »
I have checked again the Format of the rating created by ThumbGen.

https://docs.google.com/spreadsheet/ccc?key=0AspSJAZQmOIvdDJhLWZIOUt5ZC04Y1dBajNvZjJ3N2c&authkey=CLqc3ZsC&hl=en&authkey=CLqc3ZsC#gid=0

 There are two parameters possible :
1. <Rating>                : Numbers between 0 and 10 e.g. 6.8
2. <Ratingpercent>   : Numbers between 0 and 100 e.g. 68

I could not find a similar documentation for Y2M (YAMJ TO MEDE8ER) , so it is not clear to me which parameter is used there.
Could we please get an official answer what the Mede8er firmware is using <Rating> or  <Ratingpercent>?
 

Offline Unreal

  • B8a Tester - X2
  • Experienced Member
  • ****
  • Posts: 228
  • Helpful Contribution Status: +1/-0
    • UnrealRG Home Page
Re: Offline Scantool
« Reply #28 on: April 12, 2012, 03:40:26 PM »
Lupissimo,

It seems there may be an issue with the way you are parsing the xml files (possibly a difference in the way Y2M/YAMJ creates the xml compared to ThumbGen, or maybe a German/English difference), as the generated Mede8er.db was not correct.  To help you with debugging, I have attached the Mede8er.db file created by Sanji and by your scanner, along with the ini, protokoll, stmt, and xml files (all in Lupos_Test.zip).  Please let me know if you need anything further.
MED450X2 with PCIe Wireless                                 Firmware 3.0.2, NAS off, DLNA DMR&DMS enabled
Samsung LED HDTV (UN46C6300)                           Windows 7 Enterprise x64 SP1
Samsung USB Drives [2x2TB] (HX-DT020EB/B62)      Asus RT-N66U Router (FW 3.0.0.4.260)
Analog audio output to Pioneer Receiver (VSX-516)   HDMI audio output to HDTV

Offline Lupissimo

  • Senior Member
  • ***
  • Posts: 395
  • Helpful Contribution Status: +6/-0
Re: Offline Scantool
« Reply #29 on: April 12, 2012, 04:11:28 PM »
There are several issues:
1. You seem to have started the scan from a different directory than the mede8er scan, thats why the titles and consequently the hash are different
2. the rating parameter is different , see posting above
3. the genre ??? we are looking into it

Clearly ThumbGen uses differnt token definitions (see above) than Y2M. Before we start adapting we need to have the Y2M Format or the Format allowed by Sanji.
« Last Edit: April 12, 2012, 04:21:56 PM by Lupissimo »