Linux Bash scripts to separate channels

OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#41
Performance Improvement

A few weeks ago, when looking up how to extract video from a specified time frame using ffmpeg, I encountered an option that greatly improves the performance on using ffmpeg. What used to take a few minutes now takes a few seconds if not shorter.

if you have already downloaded chansplit all add -c copy -copyts to the ffmpeg lines

Code:
        if [ $extractFront = "yes" ];then
            date +"%Y-%M-%d %H:%M:%S - Starting front channel " >> chanSplitAll.log.txt
            chan="-front"
            newName=$(echo $outDir/$fname_NoExt$chan.$ext|sed 's/--front/-front/g')
            rm $newName
            ffmpeg -i $f $sound -map v:0 -async 1  -c copy -copyts $newName
        fi
        if [ $extractRear = "yes" ];then
            date +"%Y-%M-%d %H:%M:%S - Starting rear channel " >> chanSplitAll.log.txt
            chan="-rear"   
            newName=$(echo $outDir/$fname_NoExt$chan.$ext|sed 's/--rear/-rear/g')
            rm $newName
            ffmpeg -i $f $sound  -map v:1 -async 1 -c copy -copyts $newName   
        fi
My google drive and box accounts have version 10.

The chansplit script files can be acquired from Google drive and box.net
https://drive.google.com/folderview?id=0B-YpHDCsVqFENDd3OWl0em9fNzA&usp=sharing
https://app.box.com/s/dlpz1rud480phgcca5htwp8l1s7fsz2a


It appears you may need a recent version of ffmpeg. I needed to update my version I had on Windows to get "-c copy -copyts" to work

This is the first work I did in over a year on chansplit all as my rear channel camera died on my two channel system and I have bought a few single channel systems since then.
 
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#43
Sweet! I'll have to add this to my vbscript as well!
On my windows PC, I was still on the version of ffmpeg from over two years ago, back when i noticed the discussion about your vbscript . Copy and copyts wasn't working on command line on windows box until I downloaded and installed the new version. It doesn't seem to improve the performance in Windows as much as it does in Linux.

Copy and Copyts works on MOST ffmpeg commands. I am using FFMPeg to extract the time frame I want from a clip and it works really well. It also works on merging\joining\concatenating two or more files. Sadly it doesn't work on altering the speed of clips.
 
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#45
The script (chanSplitAll.sh) runs smooth on OS X El Capitan using ffmpeg from HomeBrew (https://brew.sh/) (Thinkware F770 and F50 files).
That's cool to know about HomeBrew and that you are able to use the script. Do you require homebrew to use ffmpeg, because according to FFMpeg's page it runs OS X intel 64; OS X 10.5 and above?
 
Joined
Aug 18, 2014
Messages
43
Likes
14
Country
Colombia
#46
Yes, I do require it. This is what I get before installing from Homebrew: ./chanSplitAll.sh: line 122: ffmpeg: command not found

Maybe it's not installed by default.

Although I said the script runs smooth, it displays these messages when processing MP4 files:

[avi @ 0x7fbae1800000] Format avi detected only with low score of 1, misdetection possible! tempFile.avi: Invalid data found when processing input

Stream map 'v:1' matches no streams. To ignore this, add a trailing '?' to the map.

[mp4 @ 0x7fed32813c00] Non-monotonous DTS in output stream 0:0; previous: -24024, current: -24024; changing to -24023. This may result in incorrect timestamps in the output file.

[mp4 @ 0x7fed32813c00] Non-monotonous DTS in output stream 0:0; previous: -24023, current: -24024; changing to -24022. This may result in incorrect timestamps in the output file.

[mp4 @ 0x7fed32813c00] Non-monotonous DTS in output stream 0:0; previous: -24022, current: -24024; changing to -24021. This may result in incorrect timestamps in the output file.

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fdb23800000] unsupported color_parameter_type aclc Guessed Channel Layout for Input Stream #0.1 : mono


Do I have to customise the script to skip those messages? The video files I get after running the script are just fine, no sound which is what I want, muted videos.

That's cool to know about HomeBrew and that you are able to use the script. Do you require homebrew to use ffmpeg, because according to FFMpeg's page it runs OS X intel 64; OS X 10.5 and above?
 
Last edited:
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#47
Sorry for a late reply, its been one of those weeks.

You may want to experiment with ffmpeg outside of my script with one file from your camera.

Code:
ffmpeg -i YOURFILENAME.mp4 $sound  -map v:0 -async 1 -c copy -copyts  NEWName.mp4
where
  • YOURFILENAME.mp4 is the raw 2 channel file mp4 coming from your camera
  • NEWName.mp4 the filename you specify containing vidwo only the front camera
Yes, I do require it. This is what I get before installing from Homebrew: ./chanSplitAll.sh: line 122: ffmpeg: command not found

Maybe it's not installed by default.

It sounds like homebrew installs software and what I would call packages in the Linux world that don't come at the in the standard configuration for Mac OS. For FFmpeg, in Windows and Linux, one has download or acquire the software. I have done a recent Linux install and I had to get FFMpeg. What appears to be cool about Homebrew is that it bundles numerous things such as wget so you don't have to download and install them individually. In Windows, I have to download wget, but it comes default in most Linux distros. I would recommend going to ffmpeg's home page and downloading the newest version in Mac OS as i feel you may have an older version at it may be the source to some of the problems you are having.


Although I said the script runs smooth, it displays these messages when processing MP4 files:
[avi @ 0x7fbae1800000] Format avi detected only with low score of 1, misdetection possible! tempFile.avi: Invalid data found when processing input
The script makes a dummy AVI or MP4 files if neither exist. It appears Mac OS can not follow something in the statements below as it handles redirection or piping differently.
Code:
wcAvi=$(ls  |grep avi |wc     |awk '{print $1}')
wcMp4=$(ls  |grep mp4 |wc    |awk '{print $1}')

if [ $wcAvi -gt 0 ];then
    if [ $wcMp4 -eq 0 ];then
          firstAvi=$(ls *.avi |head -1)
          head $firstAvi|head -2 > tempFile.mp4
          tempFile=tempFile.mp4
      fi   
fi

if [ $wcMp4 -gt 0 ];then
    if [ $wcAvi -eq 0 ];then
          firstMp4=$(ls *.mp4 |head -1)
          head $firstMp4|head -2 > tempFile.avi
          tempFile=tempFile.avi
      fi   
fi
It appears MAC OS does not handle redirection the same way as Linux and Unix. It looks like its breaking on the head statement. You could comment out or remove head $firstMp4|head -2 > tempFile.avi and head $firstAvi|head -2 > tempFile.mp4

Here is the solution if you only have MP4's
  1. Make a backup/copy of the script in the event the below fails
  2. Remove the above code with, the WC statements and the "IF" statements that use them.
  3. Change the for statement removing the or pointed to AVI files, remove -o `ls *.avi`. That says or List avi files. You aren't look for avi files, so remove it
Code:
for f in `ls *.mp4` -o  `ls *.avi`

Stream map 'v:1' matches no streams. To ignore this, add a trailing '?' to the map.

[mp4 @ 0x7fed32813c00] Non-monotonous DTS in output stream 0:0; previous: -24024, current: -24024; changing to -24023. This may result in incorrect timestamps in the output file.

[mp4 @ 0x7fed32813c00] Non-monotonous DTS in output stream 0:0; previous: -24023, current: -24024; changing to -24022. This may result in incorrect timestamps in the output file.

[mp4 @ 0x7fed32813c00] Non-monotonous DTS in output stream 0:0; previous: -24022, current: -24024; changing to -24021. This may result in incorrect timestamps in the output file.

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fdb23800000] unsupported color_parameter_type aclc Guessed Channel Layout for Input Stream #0.1 : mono


Do I have to customise the script to skip those messages? The video files I get after running the script are just fine, no sound which is what I want, muted videos.
I don't know if the Non-montonous DTS is because of an old version of ffmpeg or an issue with your MP4 from your Thinkware camera. FFmpeg can be used for several things aside from just splitting channels and on the earlier versions of ffmpeg, i got a timestamp warning if I tried to cut a segment out of a source file, say 5 to 10 seconds out of a 30 second clip. I don't get that error on the new version of ffmpeg, However, I am getting a similar error when I try to combine two or more files if one file is at fast speed and the other at normal speed with files coming from my A119.

I would try running ffmpeg on the command line (terminal) on the files in a terminal
 
Joined
Aug 18, 2014
Messages
43
Likes
14
Country
Colombia
#48
@GTA Driver

Homebrew ffmpeg version is up to date:
ffmpeg version 3.3.3 Copyright (c) 2000-2017 the FFmpeg developers built with Apple LLVM version 8.0.0 (clang-800.0.42.1)

I ran the one line command, NEWName.mp4 file generated fine. Still getting Non-monotonous messages.

Edited script as per your instructions, got same silent files and these messages displayed:

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fcc24001a00] unsupported color_parameter_type aclc Guessed Channel Layout for Input Stream #0.1 : mono


Stream map 'v:1' matches no streams. To ignore this, add a trailing '?' to the map.
 
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#49
@Quidagis

It appears either FFMpeg has different version numbers for the different operating systems or you have a newer version than myself.

For me
ffmpeg version 2.8.10-0ubuntu0.16.04.1 Copyright (c) 2000-2016 the FFmpeg developers

The silent file matter is totally my fault and easy to fix.

Because I usually have a passenger with me when driving and want to exclude sound to eliminate conversations in the car the audio variable is set to no. So change chanSplitAll.sh to keep sound. In the script, change the sound variable from no to yes. This will be around the 28th line on version 10

Code:
#audio         - use sound from clip or make silent, yes is sound, no is silent
audio=yes    #either yes or no

#frontChan -extract front channel?
I neglected to consider I have a variable for audio when I cut and paste the line from the script when showing how to run in command line
Code:
ffmpeg -i YOURFILENAME.mp4 map a:0 -map v:0 -async 1 -c copy -copyts  NEWName.mp4
ffmpeg -i YOURFILENAME.mp4               -map v:0 -async 1 -c copy -copyts  NEWName.mp4
The first example keeps audio, the second one removes it.

I have three cameras, two of them being single channel and with all of them I get warnings or message I ignore. I get more messages with my two channel camera than my single channel camera. Most of the time these warnings\mesages have no impact on the files I am creating. The colour parameter warning concerns me a little, but the "This may result in incorrect timestamps in the output file" concerns me a lot. I had the same problem twice and made small video clips that ended up to be 20 or minutes or moving regular speed video to really slow motion. I got the timestamp warning when trying to something with speeding a video clip and then including it with a normal clip which then made the regular speed to be way off. I was planning on joining the ffmpeg forum to get help on this.

These timestamp warning you are getting could be because
  1. the files produced by your cameras
  2. Operating system matters in Mac OS
  3. a bug in ffmpeg,
 
Last edited:
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#50
I copied four old multi channel MP4 files that I am going to provide to @Quidagis to a folder before loading on box.com. It gave me the opportunity to see what impact copy and copyts have on ffmpegs performance and how much of an improvement version 10 has over version 9.

chanSplitAll.sh withOUT copy and copyts: 4 minutes and 26 seconds
chanSplitAll.sh WITH copy and copyts: 5 seconds

@Quidagis
to determine the source of the problem, I have uploaded four files that convert with minor warnings. You can find the files at
https://app.box.com/s/67vuugdeez68y08rhgsmd9i2ci556y25

The one starting with 14 (shot in 2014) has sound, the other three don't. I was able to combine the video from the last three files to produce a video on Youtube. I don't get a time stamp warning on these.

If you download my files and don't get the errors you are getting on files produced from your camera, we know its not an operating system issue but something with your multi-channel camera.
 
Last edited:
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#51
In seems like a new version of ffmpeg has come out since May or June. Here is how to get the latest version if ffmpege is already on your Ubuntu/Mint box
Code:
sudo add-apt-repository -y ppa:mc3man/trusty-media; sudo apt-get update; sudo apt-get install --only-upgrade ffmpeg
 
Joined
Aug 18, 2014
Messages
43
Likes
14
Country
Colombia
#52
Hey @GTA Driver

I executed v1.10 - 27 Jul 2016 of your script (unmodified) on your MP4 video files, and yes, not a single warning/error message. You're right, it's my MP4 video files. But that's fine with me, all I want is to remove audio from my videos, merge them all together and and get one big single silent video file. BTW, how about concatenating the files as an option in your script? Too much coding effort, maybe? Just a thought.

Thanks for your script and kindly help.
 
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#53
That's good to know about the Thinkware F770 and F50 files. Just wondering, but do either camera have an option to use another format such as AVI. Some cameras, such as the Mobius have that option but I am not aware of that for a dual channel camera. @Gibson99 's multi-channel camera writes in AVI. Also I have noticed some problems with my MP4 have gone away with a new version of ffmpeg. Hopefully new versions of ffmpeg will fix that.

Funny, for a month now I have been familiar with using ffmpeg to concatenate several files into one for files coming from my A119 but I never thought about making it an option for chanSplitall.sh. It would take several hours to program that but what I may do is show dash cam users how to this and several other useful things at the command line. The rear camera on my two channel died and I have another front camera and a single channel rear camera. I have even mixed files from Iroad 3300ch (at one time a multi channel) and my A119 using one of ffmpegs concatenation options. ALSO IS SO MUCH FASTER THAN USING A VIDEO EDITOR, AND IF YOU KNOW HOW TO USE REDIRECTION, MORE ACCURATE.

Here is how to do it at command line for both single and multi-channel cameras.
  1. Put all of the source files in a target folder
  2. Use ls and do redirection to a file
    Code:
    ls *.mp4 *.MP4 *.avi *.AVI > input.txt
    Ignore the warning about extensions you don't have such AVI or upper or lower case. This for two extensions I am aware of that support multi-channel.
  • Open input.txt, it will look something like this
    Code:
    file001.MP4 
    file002.MP4 
    file003.MP4
    or this (example files using my Iroad)
    Code:
    141105_072314_E1-front.mp4
    141105_072314_E1-rear.mp4
    151010_020020_I1-front.mp4
  • The file input.txt is alphabetical order and has all files from the ls statement in step #2. Remove files you don't want and change the order if you don't want it in alphabetical. If you want to include a clip twice, just copy and paste.
  • Add the single quotes before the file name and the word file. It would look something like
    Code:
    file 'file001.MP4'
    file 'file002.MP4'
    file 'file003.MP4'
    or this (example files using my Iroad)
    Code:
    file '141105_072314_E1-front.mp4'
    file '141105_072314_E1-rear.mp4'
    file '151010_020020_I1-front.mp4'
  • save and exit input.txt.
  • Execute either of the commands depending whether you want audio or not
    To remove audio:
    Code:
    ffmpeg -f concat -i input.txt -c copy -an output.mp4
    to keep audio :
    Code:
    ffmpeg -f concat -i input.txt -c copy  output.mp4
Oddly the above works somewhat with RAW NonSplit multi-channel video files ... kind of. If you only want the front channel it works, otherwise if you want the rear, you need to run chanSplitAll.sh and then steps 1 to 7 in the folder chanSplitAll.sh creates, which is called chanSplit.

By the end of the month, version 11 will include concatenation.

edit, the numbering for my step to step instructions got messed up.
Edit, the above works for Linux. I suspect Mac Os handles redirection differently.
 
Last edited:
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#54
A long shot, but perhaps you could convert the files from mp4 to avi and then run my script

ffmpeg -i source.mp4 -c copy -copyts target.avi
 
Top