Linux Bash scripts to separate channels

Gibson99

Well-Known Member
Joined
Jun 20, 2014
Messages
8,298
Likes
5,714
Location
Houston, we've had a problem, Texas
Country
United States
Dash Cam
(2x) A119, (2x) A118c, iTrue X3, Git1, Git2
#21
finally trying to run the script. made the .sh file executable, ran in konsole. no such file or directory. not sure if it can't find ffmpeg or somehow can't find the script, even though i'm running it straight from dolphin so it should already have the path. if i simply type ffmpeg from my home dir, i get command not found. locate finds tons of files with ffmpeg in the name, but none look like they're obviously the right one. i DO have libav-tools installed according to synaptic.

any suggestions, @GTA Driver ?

feeling like a total noob right about now. :oops:
 
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#22
if i simply type ffmpeg from my home dir, i get command not found. locate finds tons of files with ffmpeg in the name, but none look like they're obviously the right one. i DO have libav-tools installed according to synaptic.

any suggestions,
That is strange you have files for ffmpeg, but does not seem to execute.

In a terminal shell try
Code:
 whereis ffmpeg
Also maybe try installing ffmpeg it again.
 
Last edited:
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#23
During the holidays or perhaps during winter, I plan on changing the script so that it gives control on the target file name. Such as where in the target file name do you state front or rear channel (before or after the original (source) name), or making file names in numerical order of the original files names, but removing the original file from the files created in the target folder (01-front, 01-rear, 02-front, 02-rear). The last option would be useful to me as I am struggling associating the front and rear channel from the same clip in 20 file project as the original files are so long.
 
Last edited:
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#24
When ffmpeg is in your path, or installed / re-installed, the scripts need to be in your path.

If the scripts are in /home/yourname/bin, you need to add that directory to your path. Edit .bashrc in your home directory. You can then run the scripts from any where
 
Last edited:

Gibson99

Well-Known Member
Joined
Jun 20, 2014
Messages
8,298
Likes
5,714
Location
Houston, we've had a problem, Texas
Country
United States
Dash Cam
(2x) A119, (2x) A118c, iTrue X3, Git1, Git2
#25
I put the script in the same directory as the videos to eliminate that possibility. I then set outdir to . to make it dump the split files to the same dir. Will try your suggestions after work today. Thanks!
 
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#26
This evening, or more likely early tomorrow morning, I am going to write organized install, testing and using instructions. Also, I have made some changes since I last posted a change to the script file. I think there was an annoying script file error message that I fixed.

The newest version produces a log file or perhaps what it really is - a status file -, that shows the status of how far along it is before its complete. ffmpeg gives a lot info and the info regarding file N of NN is lost on the screen, but captured in a text file. I have that on a new version I have NOT posted yet

If using this on 20 or more files, you will have time to get a beer or coffee after you start it, depending on the time of day. It takes a minute and half per file on my files and system.
 
Last edited:

Gibson99

Well-Known Member
Joined
Jun 20, 2014
Messages
8,298
Likes
5,714
Location
Houston, we've had a problem, Texas
Country
United States
Dash Cam
(2x) A119, (2x) A118c, iTrue X3, Git1, Git2
#27
this machine is an i7 w/ 32gb ram (got it from work). in windows, it splits each file in a few seconds. i'm guessing you can use tail or whatever to watch the "log" file? wonder if you can kick off that process and not wait for it to finish, then come back to the bash script and actually do the splitting. then maybe write one more line to the end of the log file to let the user know it's done. as if the lack of movement in the log wouldn't be enough indication. ;)

looking forward to your docs. i'm no linux expert by any stretch of imagination, but i'm not totally new either. though sometimes i feel that way. :oops:
 
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#29
Installing / ensuring you have ffmpeg

Typing the command
Code:
 ffmpeg
should get
Code:
ffmpeg version N-76045-g97be5d4 Copyright (c) 2000-2015 the FFmpeg developers
built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04)

configuration: --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-avresample --disable-debug --enable-nonfree --enable-gpl --enable-version3 --enable-libopencore-amrnb --enable-libopencore-amrwb --disable-decoder=amrnb --disable-decoder=amrwb --enable-libpulse --enable-libdcadec --enable-libfreetype --enable-libx264 --enable-libx265 --enable-libfdk-aac --enable-libvorbis --enable-libmp3lame --enable-libopus --enable-libvpx --enable-libspeex --enable-libass --enable-avisynth --enable-libsoxr --enable-libxvid --enable-libvo-aacenc --enable-libvidstab

libavutil 55. 4.100 / 55. 4.100
libavcodec 57. 6.100 / 57. 6.100
libavformat 57. 4.100 / 57. 4.100
libavdevice 57. 0.100 / 57. 0.100
libavfilter 6. 11.100 / 6. 11.100
libavresample 3. 0. 0 / 3. 0. 0
libswscale 4. 0.100 / 4. 0.100
libswresample 2. 0.100 / 2. 0.100
libpostproc 54. 0.100 / 54. 0.100
Hyper fast Audio and Video encoder

usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

If you don't get the above then check, try the following for installing ffmpeg


Creating a script directory
You should have a directory (folder) to place your own script files. This could be a folder from /usr/local/bin or from your $HOME folder. I use my home folder as I don't want to sudo or change to root. I am the only user on my home folder.

You could call your script folder, scripts, bin, sh or cmd. I am not aware of any convetion. I perhaps wrongly call mine bin. My examples will use $HOME/bin

Code:
mkdir $HOME/bin

Adding script folder to your path
You need to edit the script folder created above to your PATH file

Code:
gedit .bashrc
Add the following two lines at the end


Code:
PATH=$PATH:/$HOME/bin

export PATH
Start a new bash shell to inherit the changes you just made. The next time you login, or start a terminal, these settings will be there, but you need to force bashrc to get the changes for now

Code:
bash
 
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#30
Acquiring latest version of the script script

Go to your script folder and start gedit. The example below using “bin” from your home folder
Code:
cd $HOME/bin
gedit chanSplitAll.sh

copy the following code and paste into your gedit with wi
Code:
# chanSplitAll.sh
# Script to split streams from ALL files in current directory with more than one video stream inside.  
# Useful for dashcams such as the Blacksys CF-100, and JaeWonCNC Iroad IOne-3300CH
# Highly recommended that script be located in a directory in your PATH

# You may need to customize some of the variables.  

# VERSION HISTORY
# v1.00 - 03 Oct 2015:     Initial Release
# v1.01 - 05 Oct 2015:     Looks for both MP4 and AVI in directory,
# v1.02 - 06 Oct 2015:     Sound map and video map improvements.  
#            In file name, Change from CH# to more clear "front" and rear channel.  
#                  Fixed bug in or statement
# v1.03 - 13 Oct 2015: Added counter, fixed OR bug, where there only 1 of 2 media type of files
# v1.04 - 21 Dec 2015: removes target file if already there



outDir=chanSplit
mkdir $outDir
#audo - use sound from clip or make silent
audio=no     #either yes or no
counter=0

wcAvi=$(ls  |grep avi |wc |awk '{print $1'})
wcMp4=$(ls  |grep mp4 |wc  |awk '{print $1}')

let numMediaFiles=wcAvi+wcMp4


if [ $wcAvi -gt 0 ];then
    if [ $wcMp4 -eq 0 ];then
          firstAvi=$(ls *.avi |head -1)
          head $firstAvi|head -2 > tempFile.mp4
          tempFile=tempFile.mp4
      fi   
fi

if [ $wcMp4 -gt 0 ];then
    if [ $wcAvi -eq 0 ];then
          firstMp4=$(ls *.mp4 |head -1)
          head $firstMp4|head -2 > tempFile.avi
          tempFile=tempFile.avi
      fi   
fi




if [ $audio = "yes" ];then
      sound='-map a:0'
fi   


date +"%Y-%M-%d %H:%M:%S - Starting script" > chanSplitAll.log.txt
counter=1
for f in `ls *.mp4` -o  `ls *.avi`
do 
   
    if [ "$f" == "$tempFile" ]; then
        rm $tempFile
    elif [ "$f" == "-o" ]; then
        echo " "
   
    else
        echo =================================================================
        echo working on $f - $counter of $numMediaFiles
        date +"%Y-%M-%d %H:%M:%S - starting to work $f - $counter of $numMediaFiles" >> chanSplitAll.log.txt
        echo =================================================================

        date +"%Y-%M-%d %H:%M:%S - Starting front channel " >> chanSplitAll.log.txt
        rm -f $outDir/front.$f
        ffmpeg -i $f $sound  -map v:0 -async 1 $outDir/front.$f

        date +"%Y-%M-%d %H:%M:%S - Starting rear channel " >> chanSplitAll.log.txt
        rm -f $outDir/rear.$f
        ffmpeg -i $f $sound  -map v:1 -async 1 $outDir/rear.$f       

        date +"%Y-%M-%d %H:%M:%S - Finished $f - $counter of $numMediaFiles" >> chanSplitAll.log.txt
       
        echo =================================================================
        echo Spliting channels on  $f complete
        echo $counter of $numMediaFiles
        echo =================================================================
        sleep 1
        let counter=counter+1
       
    fi
done
save and close gedit
 
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#31
Using chanSplitAll.sh

Make a directory (folder) and copy (or move) only the multi channel video files you want to split.

Go the folder in a bash terminal, to ensure you got changes to our bath perhaps type bash. Following examples has your video project in “myProject”

Code:
bash.
cd $HOME/Videos/myProject/
This next time you login in start a terminal, you do not need to type “bash” but it will not hurt either.

Code:
chanSplitAll.sh
It will now start to run. I takes a minute per source file on my computer, so you may want to get a coffee or a beer

If you want to see the progress, look at chanSplitAll.log.txt . This can be done in you gui or in bash in another termainal session

Code:
tail -2 chanSplitAll.log.txt



variables

There are few variable in chanSplitAll you can change.

Audio, either yes or no. Yes gives sound, No make turn off sound
examples

Code:
audio=no #exclude dash cam micro channel
audio=yes #turns on sound
outDir – the output directory, the script call it chanSplit, but you could call it anything you want

examples

Code:
outDir=chanSplit
outDir=accidentAtElm

Right now, the script would take the following files
  • 151120_071204_I1.mp4
  • 151120_071305_I1.mp4
  • 151120_071405_I1.mp4

and creates
  • front.151120_071204_I1.mp4
  • front.151120_071305_I1.mp4
  • front.151120_071405_I1.mp4
  • rear.151120_071204_I1.mp4
  • rear.151120_071305_I1.mp4
  • rear.151120_071405_I1.mp4
I plan to change so it would give control over the file name.

151120_071204_I1-front.mp4 and 151120_071204_I1-rear.mp4
 

Gibson99

Well-Known Member
Joined
Jun 20, 2014
Messages
8,298
Likes
5,714
Location
Houston, we've had a problem, Texas
Country
United States
Dash Cam
(2x) A119, (2x) A118c, iTrue X3, Git1, Git2
#32
Funny how these things grow and take on a life of their own. Looks good, esp the part about ffmpeg. but I'm curious why you flip flop the file extensions in the if statements near the beginning. Granted I'm looking at the script in a tiny window on my phone which requires a lot of horizontal scrolling, and I'm tired and have a headache... Yeah I think I'll just stop and look at this tomorrow once I get some caffeine in me. :D
 
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#33
Yes not only did the bash script evolve and with plans to evolve more, just from myself using alone, but I have two other scripts that use ffmpeg. One that takes extracts a certain time, and other that joins several video clips into one.

My dash cam creates file names based on the time and date, which I find not very distinctive from one to the next. At the end of the file, there may be an E, I, or P for Events, non events or Parking respectively. I find it a challenge identifying which Clip I already used in a movie. It would be better if the script made clip001, clip002,clip003 etc. I found this with when I took the 25 clips of the rear ended accident I had. Open shot can only take a clip at a time, and I plan to learn to use a more powerful editor in Linux some time. So the script will eventually give target control over files and have .conf or .cf file that gives the user control over things.

So with just using yes it kind of takes a life of its own.

The script tests for both avi and mp4. It works, but I have not tested with a really multi-channel avi file. It only tests for the presence or lack of presence of an avi file .
Code:
for f in `ls *.mp4` -o  `ls *.avi`
looks for both mp4 and avi files. The f variable will then be used and do further work a file, creating a target file if it is one of the two extensions. I am not sure I am flip flopping, but early on in the script file I need to test for a lack of the file extension , such as no Avi, and then create a dummy file with that extension. I do that for mp4. I had an error message coming up if there was not a dummy file.
 

Gibson99

Well-Known Member
Joined
Jun 20, 2014
Messages
8,298
Likes
5,714
Location
Houston, we've had a problem, Texas
Country
United States
Dash Cam
(2x) A119, (2x) A118c, iTrue X3, Git1, Git2
#34
Ok, dummy file makes sense. Just don't forget to delete the dummy file at the end of the job.

I'll try to post an avi file for you to test with.
 
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#35
How its going with the testing. I made a couple of changes since I posted the script. It now gives control on which channels are extracted in two ways
  1. In the script, there are two variable that you can use frontChan and rearChan. If set to yes, it extracts front and rear channel respectively.
  2. From the file name, if it encounters "-ac.mp4" , "-fc.mp4" and "-rc.mp4" it excludes all channels, front channel only and rear channel only - respectively
On the weekend I will change the target file name to resemble what is in your visual basic program. My bash script is too complicated too list here and will use dropbox or google drive in the future.

I'll try to post an avi file for you to test with.
would it be possible. I would like to experiment with another file format.
 

Gibson99

Well-Known Member
Joined
Jun 20, 2014
Messages
8,298
Likes
5,714
Location
Houston, we've had a problem, Texas
Country
United States
Dash Cam
(2x) A119, (2x) A118c, iTrue X3, Git1, Git2
#36
How its going with the testing.
would it be possible. I would like to experiment with another file format.
honestly haven't had time to mess with much of anything outside travel, car repairs, work, and family...

http://gibson99.com/download/RawVideo/CF-100/

get the one that ends in E2 - E means emergency (i pressed the button or g-sensor locked it) and 2 means it's 2 channel. the other file there is only front video. the rear cam got knocked off the back hatch by accident (which is why the rear channel is hanging/shaking in the first video) so i just unplugged it. yes, front video is a little blurry - gonna have to refocus it or something. but it's good enough for a test clip.
 
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#37
Yes I know what you mean. I had lots more time before I got married, bought an ageing home, got a stepdaughter and acquired three dogs.

Your image quality is better than my Iroad 3300. When I bought it two years ago, I decided to go with the one the lower specs because of money and having more video on my Sd cards.

This is where it gets interesting and the files you provided are very helpful.

There is a substantial image quality loss on the files created compared to the source files. I see that a little on my files but its obvious on the files you provided. I thought ffmpeg was simply extracting channels but it appears to be altering something. I will have to experiment with that line in the script file, its also possible I have a dated version of ffmpeg

Your files convert faster too

In a directory, I placed you two channel file with a file coming from my Iroad.
The script 160104_023909_E1.mp4 is my file, and 20151231_132934_E2.avi is yours

Ffmpeg has a command to determine bit rate and pixels, but I am too lazy to check that

I added a line in the script to get source file size when the log file is produced.

Code:
2016-25-21 22:25:06 - Starting script
2016-25-21 22:25:06 - starting to work on 160104_023909_E1.mp4 - 1 of 2
size of source file 160104_023909_E1.mp4 is 25989526
2016-25-21 22:25:06 - Starting front channel 
2016-26-21 22:26:07 - Starting rear channel 
2016-26-21 22:26:27 - Finished 160104_023909_E1.mp4 - 1 of 2
2016-26-21 22:26:28 - starting to work on 20151231_132934_E2.avi - 2 of 2
size of source file 20151231_132934_E2.avi is 69413212
2016-26-21 22:26:28 - Starting front channel 
2016-26-21 22:26:45 - Starting rear channel 
2016-26-21 22:26:54 - Finished 20151231_132934_E2.avi - 2 of 2
My source file is 25.9 mb and yours is 69.4 mb
My file is 30 seconds, your 29.
My file is 866 kb per second, yours 2.4 mb per second

However your source avi file extracts faster than my source mp4 file
It takes 81 seconds to extract both channels from my files and 26 seconds for your files, despite having a larger file with a better image quality.

Both types of files get "past duration xxxx too large" warnings
 

Gibson99

Well-Known Member
Joined
Jun 20, 2014
Messages
8,298
Likes
5,714
Location
Houston, we've had a problem, Texas
Country
United States
Dash Cam
(2x) A119, (2x) A118c, iTrue X3, Git1, Git2
#39
oh, yeah the scenery in your videos is better than mine too!
Lol well that was from our vacation to Colorado. It was leaving my dad's neighborhood in Buena Vista. Colorado law says yield to pedestrians and wildlife. Who am I to argue? :)
 
OP
OP
GTA Driver

GTA Driver

Well-Known Member
Joined
Apr 5, 2015
Messages
851
Likes
335
Location
Greater Toronto Area
Country
Canada
Dash Cam
Iroad 3300CH, G1W-c, Mobius C, A119, A118-c2
#40
Version 1.09

I made a few changes that I found necessary. Since the last posted version, I have:
  • Made the target file name similar to the established convention used by @Gibson99. Files have gone from text indicating front or rear channel at the front of the file name to the end, ie rear-151119_033807_E1.mp4 to 151119_033807_E1-rear.mp4
  • ability to extract only channels you want. Either by the placement of text in the source file name (151119_033807_E1-ac.mp4) or by using channels frontChan or rearChan. I did this out of necessity for my use. Sixty percent of the time, I only need the front channel and ffmpeg is slow on my mp4 files compared to the avi files that @Gibson99 provided.
The chansplit script files can be acquired from Google drive and box.net
https://drive.google.com/folderview?id=0B-YpHDCsVqFENDd3OWl0em9fNzA&usp=sharing
https://app.box.com/s/dlpz1rud480phgcca5htwp8l1s7fsz2a


New feature - selectively extracting channels to produce files

Aside from deleting files that you don't need, you can specify that you only want one of the two channels by altering variables or placing a string in the file names.

By file name
When I review files produced by the dashcam, I change the file names of my clips to state what the topic is.
ie 150525_174414_E1-distracted-cellphone-MetroParkinglot-fc.mp4
150525_174414_E1 - The date stamp file name my camera produces
distracted-cellphone-MetroParkinglot - Is the topic, in this case A distracted drive in the Metro grocery store parking lot
-fc - front channel only.​

I use the text "-ac", "-fc" and "-rc" to state which channel to extract
  • -ac all channels
  • -fc front channel only
  • -rc rear channel only.

By variables

Code:
#frontChan -extract front channel?
frontChan=yes    #either yes or no

#frontChan -extract rear channel?
rearChan=yes    #either yes or no
If you want to prevent a channel from being extracted and produced as a file, set frontChan or rearChan to "no".

The script, comes with the frontChan and rearChan variables being set to yes, thereby extracting the front and rear channel respectively. If you find you have a multi-file project where only only of the two channels is necessary, you will need to edit the script if you want to exclude a channel

Code:
rearChan=no
If there is text in the file name before the file extention directing which channel(s) to extract, such as -ac, -fc or -rc, it takes presidence over the script file variables. If the files contain the text "-ac.mp4" or "-ac.avi", the script extract all channels even if rearChan variable is set to no.
 
Top