Script to extract GPS Data from Novatek MP4

OP
OP
S

SergeiF

Active Member
Joined
Dec 20, 2015
Messages
120
Likes
95
Country
New Zealand
#81
Hey guys, I've just got a GS63H dash-cam which is using the same Novatek96660 chipset and tried to extract the gps data out of some .mp4 files that I captured yesterday ,unfortunately I have one out of 2 things happening while I run the script:
- a .gpx file is generated , but it's empty(it only contains the header)
- I get a parsing error(see above trace for details):

Code:
/usr/bin/python3 nvtk_mp42gpx.py -i sample_1.MP4 -o $GPS_TRACK.GPX
Queueing file 'sample_1.MP4' for processing...                                                  
Processing file 'sample_1.MP4'...                                                              
Found moov atom...
Found gps chunk descriptor atom...
Traceback (most recent call last):
File "nvtk_mp42gpx.py", line 184, in <module>main()
File "nvtk_mp42gpx.py", line 169, in main process_file(f)
File "nvtk_mp42gpx.py", line 157, in process_file                                                                      
gps_data.append(get_gps_atom(get_gps_atom_info(f.read(8)),f))                                                      
File "nvtk_mp42gpx.py", line 99, in get_gps_atom
active=active.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9b in position 0: invalid start byte
I have to mention I ran the script in Linux subsystem for windows after installing pyhon3 on it and that I am no programmer so I want to know if any of you experienced a similar thing and if you know how can I fix it ?

With regards,
k-tod.
I have fixed the script to handle weird atoms (it just skips them now). Let me know if you still have issues.

Slightly offtopic I had someone contact me about Soocoo S100 Pro camera that records GPS coordinates in weird format that I cannot correlate to anything real. This particular camera also had same issue (with garbage atom header).
So there is a chance that you can get random coordinates.

If you do so, I am interested in fixing it, but with the sample file from Soocoo S100 Pro provided I could not make the sense of it. They encode coordinates somehow (not arithmetic or degree vs minute thing). I even broke down the floats they store the coordinates in to make sure they behave like floats (and as far as I could see they did).
 

k-tod

New Member
Joined
Apr 15, 2018
Messages
8
Likes
0
Country
Netherlands
Dash Cam
Azdome G63H, YI Dash Cam
#82
Hi Sergei, thank you very much for your fast reply, I tried the new version of the script but it is still failing to extract the log:

Code:
/usr/bin/python3 nvtk_mp42gpx.py -i sample_1.MP4 -o $GPS_TRACK.GPX
Queueing file 'sample_1.MP4' for processing...
Processing file 'sample_1.MP4'...
Found moov atom...
Found gps chunk descriptor atom...
Traceback (most recent call last):
File "nvtk_mp42gpx.py", line 188, in <module> main()
File "nvtk_mp42gpx.py", line 173, in main process_file(f)
File "nvtk_mp42gpx.py", line 161, in process_file gps_data.append(get_gps_atom(get_gps_atom_info(f.read(8)),f))
File "nvtk_mp42gpx.py", line 103, in get_gps_atom active=active.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9b in position 0: invalid start byte
Does it worth to prepare a short video ? like 10..20 seconds long and upload it here, or maybe send it to you somehow ? would it help ?
 
OP
OP
S

SergeiF

Active Member
Joined
Dec 20, 2015
Messages
120
Likes
95
Country
New Zealand
#83
Hi Sergei, thank you very much for your fast reply, I tried the new version of the script but it is still failing to extract the log:

Code:
/usr/bin/python3 nvtk_mp42gpx.py -i sample_1.MP4 -o $GPS_TRACK.GPX
Queueing file 'sample_1.MP4' for processing...
Processing file 'sample_1.MP4'...
Found moov atom...
Found gps chunk descriptor atom...
Traceback (most recent call last):
File "nvtk_mp42gpx.py", line 188, in <module> main()
File "nvtk_mp42gpx.py", line 173, in main process_file(f)
File "nvtk_mp42gpx.py", line 161, in process_file gps_data.append(get_gps_atom(get_gps_atom_info(f.read(8)),f))
File "nvtk_mp42gpx.py", line 103, in get_gps_atom active=active.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9b in position 0: invalid start byte
Does it worth to prepare a short video ? like 10..20 seconds long and upload it here, or maybe send it to you somehow ? would it help ?

Are you sure you have new version of the script? Open the script in a text editor and see if it has "except UnicodeDecodeError:" on line 80.

Ahh, I see they have garbage in "active"... Let me think of a quick fix.
 
OP
OP
S

SergeiF

Active Member
Joined
Dec 20, 2015
Messages
120
Likes
95
Country
New Zealand
#84
I have put some exception handling around, could you please try again?

I suspect it will not work (as in GPX file will be either empty or contain garbage) as they have weird stuff in the GPS data.
 

k-tod

New Member
Joined
Apr 15, 2018
Messages
8
Likes
0
Country
Netherlands
Dash Cam
Azdome G63H, YI Dash Cam
#85
I tried again, no exception anymore, but all the coordinates are skipped the resulting file only contains the header.
 

k-tod

New Member
Joined
Apr 15, 2018
Messages
8
Likes
0
Country
Netherlands
Dash Cam
Azdome G63H, YI Dash Cam
#87
It looks like they changed the format of the GPS data again ..., I'll try to add a small sample this afternoon, as I have to leave now :(
 
OP
OP
S

SergeiF

Active Member
Joined
Dec 20, 2015
Messages
120
Likes
95
Country
New Zealand
#89
it is printed very often , in a 3 minutes file I get the message 180 times.
I will need a sample as I bet there are 180 GPS samples of ~ 180second long video (or close) and their format has changed again.
You could probably share it with me via google drive.
 

k-tod

New Member
Joined
Apr 15, 2018
Messages
8
Likes
0
Country
Netherlands
Dash Cam
Azdome G63H, YI Dash Cam
#90
Hi Sergei,
I made a short sample and I shared it with you via google drive, please let me know what you think,
Regards, k-tod.
 
OP
OP
S

SergeiF

Active Member
Joined
Dec 20, 2015
Messages
120
Likes
95
Country
New Zealand
#91
I am looking at your file and the GPS data is in completely different format to what it used to be.
Why Chinese manufacturers are doing this stuff is beyond me, it is borderline malicious....
Don't they understand the more "standard" this stuff is the more adoption there is?
This is second camera that went out of its way to make sure the format is unusable. Total utter garbage :(.

EDIT:

To elaborate further here is the packet:


Code:
002974b0  47 50 53 20 05 01 00 00  01 03 aa aa f2 e1 f0 ee  |GPS ............|
002974c0  54 54 98 9a 9b 92 9a 9e  9b 9f 9b 9b 99 9b 98 92  |TT..............|
002974d0  a6 8a 8a 8a 8a 8a 8a 8a  8a 8a aa aa aa aa aa a9  |................|
                        REMOVED ACTUAL COORDINATES
*
002975b0  aa aa aa aa aa aa aa aa  aa aa aa 00 00 00 00 00  |................|
002975c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
0029f4a0  00 00 00 00 00 00 00 00  00 00 20 00 66 72 65 65  |.......... .free|
I don't see anything resembling time, date or floats for GPS data.

Just for comparison here is a packet from a Viofo A119:

Code:
00bed660  47 50 53 20 4c 00 00 00  00 00 00 00 00 00 00 00  |GPS L...........|
00bed670  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00bed680  00 00 00 00 00 00 00 00  0e 00 00 00 15 00 00 00  |................|
00bed690  17 00 00 00 12 00 00 00  04 00 00 00 0e 00 00 00  |................|
00bed6a0  41 53 45 00 1d 58 63 45  8c 2c 88 46 85 eb 0d 42  |ASE..XcE.,.F...B|
00bed6b0  33 93 8a 43 00 00 00 00  00 00 00 00 00 00 00 00  |3..C............|
00bed6c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
If the first packet has the GPS data it is purposely obfuscated (not sure what kind of encoding will generate 0xAA for 0 and 0x9X for data). Although speaking of 0xAA being zero is it a simple transformation applied (XOR?)?


EDIT2: yep it is XOR with key 0xAA. Removed actual coordinates from the sample.
 
Last edited:

viofo

Well-Known Member
Manufacturer
Joined
Sep 29, 2015
Messages
634
Likes
1,048
Location
Shenzhen
Country
China
Dash Cam
WR1/A119/A119S, A118C/A118C2, G1W-C/G1W-CB/G1W-S
#92
Some company use special GPS module which can only use their own GPS player.
 
OP
OP
S

SergeiF

Active Member
Joined
Dec 20, 2015
Messages
120
Likes
95
Country
New Zealand
#95
I "cracked" it. It is XOR cipher with key 0xAA.

Code:
002974c0  54 54 98 9a 9b 92 9a 9e  9b 9f 9b 9b 99 9b 98 92  |TT..............|
Translates to:
20180415113128

UPDATE:

I can get meaningful data out of this, basically it is ASCII string encoded with XOR (key:0xAA). I am not entirely sure about specific fields (eg: +132-048-007), these might be heading. speed and altitude (not in that order)?
Since the structure is very different (yet simpler) from Viofo generated files I will need some time to write a "new" script. Format is probably azdome specific....
 
Last edited:

k-tod

New Member
Joined
Apr 15, 2018
Messages
8
Likes
0
Country
Netherlands
Dash Cam
Azdome G63H, YI Dash Cam
#96
Wow, nice... :)
I also don't understand why don't they just use a common standard. Would be nice to have a script for this camera as well, maybe other people would also need it :), I feel bad because I am not knowledgeable enough to make it but you can definitely count on me for testing it !
 
OP
OP
S

SergeiF

Active Member
Joined
Dec 20, 2015
Messages
120
Likes
95
Country
New Zealand
#97
I have fixed the script with addition of Azdome GPS data parser. It is the same script.

One note that the speed accuracy is compromised due to value stored is truncated at the decimal point (they do equivalent to int(speed)), eg 10.64 becomes 10. I guess good enough for cars.
I still haven't figured out what the "+098-002-002" numbers are, but they did not correlate to the speed. I don't use heading/elevation/g-sensor output for GPX so that is not important.

As of interested to anyone the payload is starting with XKZD (after decoding), which might hint to the manufacturer of the GPS device.
 
Last edited:

kamkar1

Well-Known Member
Joined
Oct 11, 2013
Messages
16,641
Likes
8,667
Country
Denmark
Dash Cam
Lukas LK-7500G / Innovv C3 / mobius / G1W / SG9665GC & more
#99
Most of widows users like to have a GUI
Indeed, actually GUI are what really kicked off home computing i think, and for this we need to thanks Motorola ( where Steve Jobs stole / got the GUI idea from ) and Microsoft / Bill Gates who stole the idea from Steve Jobs.

While i was a hardcore dos fan and only changed to windows with windows 98 se, then i have never been brave enough to take on linux though not a totally text based system.

Now i just do the sgt Murtaugh " im too old for this sh.." and might i add i do that quite often.
 
Thread starter Similar threads Forum Replies Date
RobTrehy A119 138

Similar threads