BlackVue Sync - A new portable, hands-off, automatic recording downloader for NAS or Docker

[blazn420]

I noticed that my camera disconnects from the cloud service but the file transfer is still active.

 

[leonnnn]

Seeing as the network traffic stops, my dash cam definitely drops from the network not just the cloud.

I noticed that my camera disconnects from the cloud service but the file transfer is still active.

 

[blazn420]

Seeing as the network traffic stops, my dash cam definitely drops from the network not just the cloud.

Checked this morning seems my camera disconnected from the network after about 9 hours of transfers. I actually had to go to my cam and manually turn WiFi off then back on again. I still have to do manual dumps from the SD card to my NAS, I have over an hour commute each way to work plus 8-9 hours of time lapse parking recordings every day so the connection is not fast enough. Still love this docker for the event and manual file dumps.

I was looking into getting this for the dashcam and my drone video dumps:

https://www.amazon.com/gp/B06X6H1LY6
It has an SD card slot and I can connect this directly to my NAS once a week. But I have to slow down on the spending, between the cam, battery, polarized filters, and sd card this is getting expensive.

 

[acolomba]

I see interruptions too with a UniFi setup. (It's one of the reasons I made blackvuesync: I don't think I saw another downloader that was resilient to failures.)

I have a semi-directional AP on the opposite side of the garage wall. It points inside the house, so it's not an optimal setup for the dashcam obviously. Signal strength is 85% with just about 10 packet retries/min (curiously only for packets to the dashcam.) The UniFi software reports 30-100Mbps speeds with 5min sampling, but effectively I see ~30Mbps download speeds in blackvuesync

I'm recording just the front camera at 100MB/min, so 30Mbps means I download a minute of footage in ~30 seconds, or a 2x speedup. I imagine a front+rear setup would get me close to a 1:1 ratio recording/download.

I just pushed a change to blackvuesync that logs effective download speeds to help evaluate these setups.

 

[leonnnn]

I dug around some more and observed that attempting to download a file that's in the process of being deleted or created (and the occasional corrupt file) causes a disconnect.
I'm thinking a simple "skip 1-2 oldest files" and "skip file which timestamp < 180s old" (max recording time is 3mins).
Only problem I see is if the time goes out of sync due to daylight savings. Retrieving the server time might work in this case?

I've also been toying around with a way to optimize downloads even further for those who aren't interested in footage at home.
If we log when the cam comes online and offline (delay by ~5-15mins before marking as offline due to intermittent disconnects) we can ignore downloads that are within the "online" timestamps. With this setup you just might be able to download a day's worth of footage in the 6h mimimum PMP timeout.

I reckon: M > E > P (outside of home) > N > P (at home)

 

[blazn420]

Has anyone had issues with the 750 after the latest firmware update? The docker doesn't seem to download files anymore after I updated.

 

[acolomba]

Has anyone had issues with the 750 after the latest firmware update? The docker doesn't seem to download files anymore after I updated.

I just upgraded my 750 and tried the docker container, and it seems to work fine.

snoopy:~/tmp > docker run -it --rm \
> -e ADDRESS=dashcam \
> -e VERBOSE=1 \
> -e RUN_ONCE=1 \
> -v $HOME/tmp:/recordings \
> --name blackvuesync \
> acolomba/blackvuesync
2019-04-10 01:06:18,896: INFO Downloaded recording : 20190408_193458_EF.mp4 (41Mbps)
2019-04-10 01:06:42,801: INFO Downloaded recording : 20190408_193558_NF.mp4 (41Mbps)

I would recommend trying running the container manually, enable verbosity level 1 or 2. If it's something that looks like a blackvue problem, post the logs and we'll take it from there.

 

[blazn420]

Ok it was a disk space error. But there is still over 800gb of space. I also have it set to 4 weeks retention shouldn't it delete the old files and resume?

 

[acolomba]

Old files are pruned before starting to download at every run. So you should not see any files with a date prior to 4 weeks ago.

As to disk space, blackvuesync by default exits if it sees more than 90% disk space used. If you have a NAS with, say, 10TB, blackvuesync will refuse to download when you have "only" 800GB free. It logs a message to that effect though.

 

[blazn420]

Will it prune only files that it downloaded? Or any file in that directory? Sometimes I dump the sd card straight into the nas.


Sent from my iPhone using Tapatalk

 

[acolomba]

It prunes files in the directory. It does not distinguish between files it downloaded and other files.

 

[Op911]

This is wonderful. Thanks for writing this and implementing this.

I was just wondering if others are noting a small issue I am encountering - I'm running this on a NAS (Synology) through Docker and I've noted an odd issue where if I try to access the share over the network it is very unresponsive and hangs Finder and the Blackvue application unless I turn off the script in Docker on the NAS. Once I turn off the script temporarily while accessing the files all is well, but I hadn't encountered this unresponsiveness in the file system in the past with the NAS and I'm wondering if it's a weird interaction between the script and the NAS.

 

[blazn420]

What are the specs of your NAS? CPU and memory specifically.


Sent from my iPhone using Tapatalk

 

[Op911]

What are the specs of your NAS? CPU and memory specifically.


Sent from my iPhone using Tapatalk

This is a Synology DS1513+, with 2GB of ram, an Intel Atom dual core processor running at 2.13GHz.
CPU utilization is never close to maxed out on the resource monitor.

 

[acolomba]

This is a Synology DS1513+, with 2GB of ram, an Intel Atom dual core processor running at 2.13GHz.
CPU utilization is never close to maxed out on the resource monitor.

The docker version of blackvuesync only runs every 15 minutes, idling the rest of the time. I can't think of any reasons just having that image running would make a difference.

If you want to track a slowdown, I'd recommend also looking at the load, not just the CPU. The "top" command will show the 1min, 5min and 15min load averages, giving you an idea about the pattern. A high load may be a result of many threads competing for the CPU (which is not the case for you), but also of many threads waiting for I/O (which is a possibility.)

 

[Op911]

Thanks.
The slowdown only seems to occur when the app is in the midst of downloading files, not at other times. The first few days I had loaded up blackvuesync it was busy downloading all of the stuff on a 64 gb card, which took several days. It seems to have caught up now, so there is less time spent where the share seems inaccessible.

I have screenshots of what top shows when it is downloading files to the NAS and the share seems to be mostly inaccessible:

 

[Op911]

... also:
System task manager when Blackvuesync is downloading and the SMB share is unresponsive:


And then when Blackvuesync has been halted, and the system is once again responsive:

 

[acolomba]

Your load average indicates 5 threads are waiting for CPU time. IO doesn't seem to be the culprit, since it's just 5%.

It'd be best to show the load from the host, not from docker, 5min after downloads have started, and 15 mins after you've stopped. Basically before/after. Otherwise we don't know how much blackvuesync is contributing. (For instance, I see DSM is taking 20% CPU, but I don't know what else it's doing.)

But to do a quick and dirty test first, try this from a command prompt. It will download the same file over and over.

while true; do wget http://192.168.1.148/Record/<filename>.mp4 ; done

Pick a filename from one you know it's on the camera. If you don't have wget, curl will work just as well.

If you see the same SMB unresponsiveness, it's your system bottlenecking somewhere. If not, try the above.

 

[Op911]

Thank you for taking a look.

This time I used SSH to get into the system, and didn't used the NAS's GUI interface (which is what the DSM thread is, I think).
Here are the results (4 separate screen caps, posted in two separate posts due to file size limitations)
#1) Doing the whiletrue thing. When accessing the SMB share I could see smbd spike in CPU to 55% or so for a while, but the share remained responsive and I was able to access the files on the share. (The following screenshot is after Finder successfully loaded the directory contents and SMBD didn't need to do anything any further)



#2) The following is the results of top while Blackvue sync is downloading

 

[Op911]

#3) The following is if I try to access the share through Finder while Blackvuesync is downloading (SMBD takes up a ton of CPU, finder hangs, and I have to force restart finder to get everything working properly again on both the client system and the NAS)


#4) The following is what happens in top when all downloading is complete