Linux Bash scripts to separate channels

Discussion in 'Multi-Channel Dash Cams' started by GTA Driver, Oct 4, 2015.

  1. Gibson99

    Gibson99 Well-Known Member

    Messages:
    7,770
    Likes Received:
    5,025
    Location:
    Houston, we've had a problem, Texas
    Country:
    United States
    Dash Cam:
    Eh, one or two... ;)
    finally trying to run the script. made the .sh file executable, ran in konsole. no such file or directory. not sure if it can't find ffmpeg or somehow can't find the script, even though i'm running it straight from dolphin so it should already have the path. if i simply type ffmpeg from my home dir, i get command not found. locate finds tons of files with ffmpeg in the name, but none look like they're obviously the right one. i DO have libav-tools installed according to synaptic.

    any suggestions, @GTA Driver ?

    feeling like a total noob right about now. :oops:
     
  2. GTA Driver

    GTA Driver Well-Known Member

    Messages:
    736
    Likes Received:
    257
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    That is strange you have files for ffmpeg, but does not seem to execute.

    In a terminal shell try
    Code:
     whereis ffmpeg  
    Also maybe try installing ffmpeg it again.
     
    Last edited: Dec 21, 2015
  3. GTA Driver

    GTA Driver Well-Known Member

    Messages:
    736
    Likes Received:
    257
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    During the holidays or perhaps during winter, I plan on changing the script so that it gives control on the target file name. Such as where in the target file name do you state front or rear channel (before or after the original (source) name), or making file names in numerical order of the original files names, but removing the original file from the files created in the target folder (01-front, 01-rear, 02-front, 02-rear). The last option would be useful to me as I am struggling associating the front and rear channel from the same clip in 20 file project as the original files are so long.
     
    Last edited: Dec 21, 2015
  4. GTA Driver

    GTA Driver Well-Known Member

    Messages:
    736
    Likes Received:
    257
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    When ffmpeg is in your path, or installed / re-installed, the scripts need to be in your path.

    If the scripts are in /home/yourname/bin, you need to add that directory to your path. Edit .bashrc in your home directory. You can then run the scripts from any where
     
    Last edited: Dec 21, 2015
  5. Gibson99

    Gibson99 Well-Known Member

    Messages:
    7,770
    Likes Received:
    5,025
    Location:
    Houston, we've had a problem, Texas
    Country:
    United States
    Dash Cam:
    Eh, one or two... ;)
    I put the script in the same directory as the videos to eliminate that possibility. I then set outdir to . to make it dump the split files to the same dir. Will try your suggestions after work today. Thanks!
     
  6. GTA Driver

    GTA Driver Well-Known Member

    Messages:
    736
    Likes Received:
    257
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    This evening, or more likely early tomorrow morning, I am going to write organized install, testing and using instructions. Also, I have made some changes since I last posted a change to the script file. I think there was an annoying script file error message that I fixed.

    The newest version produces a log file or perhaps what it really is - a status file -, that shows the status of how far along it is before its complete. ffmpeg gives a lot info and the info regarding file N of NN is lost on the screen, but captured in a text file. I have that on a new version I have NOT posted yet

    If using this on 20 or more files, you will have time to get a beer or coffee after you start it, depending on the time of day. It takes a minute and half per file on my files and system.
     
    Last edited: Dec 21, 2015
  7. Gibson99

    Gibson99 Well-Known Member

    Messages:
    7,770
    Likes Received:
    5,025
    Location:
    Houston, we've had a problem, Texas
    Country:
    United States
    Dash Cam:
    Eh, one or two... ;)
    this machine is an i7 w/ 32gb ram (got it from work). in windows, it splits each file in a few seconds. i'm guessing you can use tail or whatever to watch the "log" file? wonder if you can kick off that process and not wait for it to finish, then come back to the bash script and actually do the splitting. then maybe write one more line to the end of the log file to let the user know it's done. as if the lack of movement in the log wouldn't be enough indication. ;)

    looking forward to your docs. i'm no linux expert by any stretch of imagination, but i'm not totally new either. though sometimes i feel that way. :oops:
     
  8. GTA Driver

    GTA Driver Well-Known Member

    Messages:
    736
    Likes Received:
    257
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    10 to 20 minutes from now and the install and usage instructions will be complete
     
  9. GTA Driver

    GTA Driver Well-Known Member

    Messages:
    736
    Likes Received:
    257
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    Installing / ensuring you have ffmpeg

    Typing the command
    Code:
     ffmpeg  
    should get
    Code:
    ffmpeg version N-76045-g97be5d4 Copyright (c) 2000-2015 the FFmpeg developers
    built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04)
    
    configuration: --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-avresample --disable-debug --enable-nonfree --enable-gpl --enable-version3 --enable-libopencore-amrnb --enable-libopencore-amrwb --disable-decoder=amrnb --disable-decoder=amrwb --enable-libpulse --enable-libdcadec --enable-libfreetype --enable-libx264 --enable-libx265 --enable-libfdk-aac --enable-libvorbis --enable-libmp3lame --enable-libopus --enable-libvpx --enable-libspeex --enable-libass --enable-avisynth --enable-libsoxr --enable-libxvid --enable-libvo-aacenc --enable-libvidstab
    
    libavutil 55. 4.100 / 55. 4.100
    libavcodec 57. 6.100 / 57. 6.100
    libavformat 57. 4.100 / 57. 4.100
    libavdevice 57. 0.100 / 57. 0.100
    libavfilter 6. 11.100 / 6. 11.100
    libavresample 3. 0. 0 / 3. 0. 0
    libswscale 4. 0.100 / 4. 0.100
    libswresample 2. 0.100 / 2. 0.100
    libpostproc 54. 0.100 / 54. 0.100
    Hyper fast Audio and Video encoder
    
    usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...
    

    If you don't get the above then check, try the following for installing ffmpeg


    Creating a script directory
    You should have a directory (folder) to place your own script files. This could be a folder from /usr/local/bin or from your $HOME folder. I use my home folder as I don't want to sudo or change to root. I am the only user on my home folder.

    You could call your script folder, scripts, bin, sh or cmd. I am not aware of any convetion. I perhaps wrongly call mine bin. My examples will use $HOME/bin

    Code:
    mkdir $HOME/bin

    Adding script folder to your path
    You need to edit the script folder created above to your PATH file

    Code:
    gedit .bashrc
    Add the following two lines at the end


    Code:
    PATH=$PATH:/$HOME/bin
    
    export PATH
    Start a new bash shell to inherit the changes you just made. The next time you login, or start a terminal, these settings will be there, but you need to force bashrc to get the changes for now

    Code:
    bash
     
  10. GTA Driver

    GTA Driver Well-Known Member

    Messages:
    736
    Likes Received:
    257
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    Acquiring latest version of the script script

    Go to your script folder and start gedit. The example below using “bin” from your home folder
    Code:
    cd $HOME/bin
    gedit chanSplitAll.sh

    copy the following code and paste into your gedit with wi
    Code:
    # chanSplitAll.sh
    # Script to split streams from ALL files in current directory with more than one video stream inside.  
    # Useful for dashcams such as the Blacksys CF-100, and JaeWonCNC Iroad IOne-3300CH
    # Highly recommended that script be located in a directory in your PATH
    
    # You may need to customize some of the variables.  
    
    # VERSION HISTORY
    # v1.00 - 03 Oct 2015:     Initial Release
    # v1.01 - 05 Oct 2015:     Looks for both MP4 and AVI in directory,
    # v1.02 - 06 Oct 2015:     Sound map and video map improvements.  
    #            In file name, Change from CH# to more clear "front" and rear channel.  
    #                  Fixed bug in or statement
    # v1.03 - 13 Oct 2015: Added counter, fixed OR bug, where there only 1 of 2 media type of files
    # v1.04 - 21 Dec 2015: removes target file if already there
    
    
    
    outDir=chanSplit
    mkdir $outDir
    #audo - use sound from clip or make silent
    audio=no     #either yes or no
    counter=0
    
    wcAvi=$(ls  |grep avi |wc |awk '{print $1'})
    wcMp4=$(ls  |grep mp4 |wc  |awk '{print $1}')
    
    let numMediaFiles=wcAvi+wcMp4
    
    
    if [ $wcAvi -gt 0 ];then
        if [ $wcMp4 -eq 0 ];then
              firstAvi=$(ls *.avi |head -1)
              head $firstAvi|head -2 > tempFile.mp4
              tempFile=tempFile.mp4
          fi   
    fi
    
    if [ $wcMp4 -gt 0 ];then
        if [ $wcAvi -eq 0 ];then
              firstMp4=$(ls *.mp4 |head -1)
              head $firstMp4|head -2 > tempFile.avi
              tempFile=tempFile.avi
          fi   
    fi
    
    
    
    
    if [ $audio = "yes" ];then
          sound='-map a:0'
    fi   
    
    
    date +"%Y-%M-%d %H:%M:%S - Starting script" > chanSplitAll.log.txt
    counter=1
    for f in `ls *.mp4` -o  `ls *.avi`
    do 
       
        if [ "$f" == "$tempFile" ]; then
            rm $tempFile
        elif [ "$f" == "-o" ]; then
            echo " "
       
        else
            echo =================================================================
            echo working on $f - $counter of $numMediaFiles
            date +"%Y-%M-%d %H:%M:%S - starting to work $f - $counter of $numMediaFiles" >> chanSplitAll.log.txt
            echo =================================================================
    
            date +"%Y-%M-%d %H:%M:%S - Starting front channel " >> chanSplitAll.log.txt
            rm -f $outDir/front.$f
            ffmpeg -i $f $sound  -map v:0 -async 1 $outDir/front.$f
    
            date +"%Y-%M-%d %H:%M:%S - Starting rear channel " >> chanSplitAll.log.txt
            rm -f $outDir/rear.$f
            ffmpeg -i $f $sound  -map v:1 -async 1 $outDir/rear.$f       
    
            date +"%Y-%M-%d %H:%M:%S - Finished $f - $counter of $numMediaFiles" >> chanSplitAll.log.txt
           
            echo =================================================================
            echo Spliting channels on  $f complete
            echo $counter of $numMediaFiles
            echo =================================================================
            sleep 1
            let counter=counter+1
           
        fi
    done
    
    save and close gedit
     
    Gibson99 likes this.
  11. GTA Driver

    GTA Driver Well-Known Member

    Messages:
    736
    Likes Received:
    257
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    Using chanSplitAll.sh

    Make a directory (folder) and copy (or move) only the multi channel video files you want to split.

    Go the folder in a bash terminal, to ensure you got changes to our bath perhaps type bash. Following examples has your video project in “myProject”

    Code:
    bash.
    cd $HOME/Videos/myProject/
    
    This next time you login in start a terminal, you do not need to type “bash” but it will not hurt either.

    Code:
    chanSplitAll.sh
    
    It will now start to run. I takes a minute per source file on my computer, so you may want to get a coffee or a beer

    If you want to see the progress, look at chanSplitAll.log.txt . This can be done in you gui or in bash in another termainal session

    Code:
    tail -2 chanSplitAll.log.txt
    



    variables

    There are few variable in chanSplitAll you can change.

    Audio, either yes or no. Yes gives sound, No make turn off sound
    examples

    Code:
    audio=no #exclude dash cam micro channel
    audio=yes #turns on sound
    
    outDir – the output directory, the script call it chanSplit, but you could call it anything you want

    examples

    Code:
    outDir=chanSplit
    outDir=accidentAtElm
    

    Right now, the script would take the following files
    • 151120_071204_I1.mp4
    • 151120_071305_I1.mp4
    • 151120_071405_I1.mp4

    and creates
    • front.151120_071204_I1.mp4
    • front.151120_071305_I1.mp4
    • front.151120_071405_I1.mp4
    • rear.151120_071204_I1.mp4
    • rear.151120_071305_I1.mp4
    • rear.151120_071405_I1.mp4
    I plan to change so it would give control over the file name.

    151120_071204_I1-front.mp4 and 151120_071204_I1-rear.mp4
     
    Gibson99 likes this.
  12. Gibson99

    Gibson99 Well-Known Member

    Messages:
    7,770
    Likes Received:
    5,025
    Location:
    Houston, we've had a problem, Texas
    Country:
    United States
    Dash Cam:
    Eh, one or two... ;)
    Funny how these things grow and take on a life of their own. Looks good, esp the part about ffmpeg. but I'm curious why you flip flop the file extensions in the if statements near the beginning. Granted I'm looking at the script in a tiny window on my phone which requires a lot of horizontal scrolling, and I'm tired and have a headache... Yeah I think I'll just stop and look at this tomorrow once I get some caffeine in me. :D
     
  13. GTA Driver

    GTA Driver Well-Known Member

    Messages:
    736
    Likes Received:
    257
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    Yes not only did the bash script evolve and with plans to evolve more, just from myself using alone, but I have two other scripts that use ffmpeg. One that takes extracts a certain time, and other that joins several video clips into one.

    My dash cam creates file names based on the time and date, which I find not very distinctive from one to the next. At the end of the file, there may be an E, I, or P for Events, non events or Parking respectively. I find it a challenge identifying which Clip I already used in a movie. It would be better if the script made clip001, clip002,clip003 etc. I found this with when I took the 25 clips of the rear ended accident I had. Open shot can only take a clip at a time, and I plan to learn to use a more powerful editor in Linux some time. So the script will eventually give target control over files and have .conf or .cf file that gives the user control over things.

    So with just using yes it kind of takes a life of its own.

    The script tests for both avi and mp4. It works, but I have not tested with a really multi-channel avi file. It only tests for the presence or lack of presence of an avi file .
    Code:
    for f in `ls *.mp4` -o  `ls *.avi` 
    looks for both mp4 and avi files. The f variable will then be used and do further work a file, creating a target file if it is one of the two extensions. I am not sure I am flip flopping, but early on in the script file I need to test for a lack of the file extension , such as no Avi, and then create a dummy file with that extension. I do that for mp4. I had an error message coming up if there was not a dummy file.
     
  14. Gibson99

    Gibson99 Well-Known Member

    Messages:
    7,770
    Likes Received:
    5,025
    Location:
    Houston, we've had a problem, Texas
    Country:
    United States
    Dash Cam:
    Eh, one or two... ;)
    Ok, dummy file makes sense. Just don't forget to delete the dummy file at the end of the job.

    I'll try to post an avi file for you to test with.
     
  15. GTA Driver

    GTA Driver Well-Known Member

    Messages:
    736
    Likes Received:
    257
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    How its going with the testing. I made a couple of changes since I posted the script. It now gives control on which channels are extracted in two ways
    1. In the script, there are two variable that you can use frontChan and rearChan. If set to yes, it extracts front and rear channel respectively.
    2. From the file name, if it encounters "-ac.mp4" , "-fc.mp4" and "-rc.mp4" it excludes all channels, front channel only and rear channel only - respectively
    On the weekend I will change the target file name to resemble what is in your visual basic program. My bash script is too complicated too list here and will use dropbox or google drive in the future.

    would it be possible. I would like to experiment with another file format.
     
  16. Gibson99

    Gibson99 Well-Known Member

    Messages:
    7,770
    Likes Received:
    5,025
    Location:
    Houston, we've had a problem, Texas
    Country:
    United States
    Dash Cam:
    Eh, one or two... ;)
    honestly haven't had time to mess with much of anything outside travel, car repairs, work, and family...

    http://gibson99.com/download/RawVideo/CF-100/

    get the one that ends in E2 - E means emergency (i pressed the button or g-sensor locked it) and 2 means it's 2 channel. the other file there is only front video. the rear cam got knocked off the back hatch by accident (which is why the rear channel is hanging/shaking in the first video) so i just unplugged it. yes, front video is a little blurry - gonna have to refocus it or something. but it's good enough for a test clip.
     
    GTA Driver likes this.
  17. GTA Driver

    GTA Driver Well-Known Member

    Messages:
    736
    Likes Received:
    257
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    Yes I know what you mean. I had lots more time before I got married, bought an ageing home, got a stepdaughter and acquired three dogs.

    Your image quality is better than my Iroad 3300. When I bought it two years ago, I decided to go with the one the lower specs because of money and having more video on my Sd cards.

    This is where it gets interesting and the files you provided are very helpful.

    There is a substantial image quality loss on the files created compared to the source files. I see that a little on my files but its obvious on the files you provided. I thought ffmpeg was simply extracting channels but it appears to be altering something. I will have to experiment with that line in the script file, its also possible I have a dated version of ffmpeg

    Your files convert faster too

    In a directory, I placed you two channel file with a file coming from my Iroad.
    The script 160104_023909_E1.mp4 is my file, and 20151231_132934_E2.avi is yours

    Ffmpeg has a command to determine bit rate and pixels, but I am too lazy to check that

    I added a line in the script to get source file size when the log file is produced.

    Code:
     
    2016-25-21 22:25:06 - Starting script
    2016-25-21 22:25:06 - starting to work on 160104_023909_E1.mp4 - 1 of 2
    size of source file 160104_023909_E1.mp4 is 25989526
    2016-25-21 22:25:06 - Starting front channel 
    2016-26-21 22:26:07 - Starting rear channel 
    2016-26-21 22:26:27 - Finished 160104_023909_E1.mp4 - 1 of 2
    2016-26-21 22:26:28 - starting to work on 20151231_132934_E2.avi - 2 of 2
    size of source file 20151231_132934_E2.avi is 69413212
    2016-26-21 22:26:28 - Starting front channel 
    2016-26-21 22:26:45 - Starting rear channel 
    2016-26-21 22:26:54 - Finished 20151231_132934_E2.avi - 2 of 2
    
    My source file is 25.9 mb and yours is 69.4 mb
    My file is 30 seconds, your 29.
    My file is 866 kb per second, yours 2.4 mb per second

    However your source avi file extracts faster than my source mp4 file
    It takes 81 seconds to extract both channels from my files and 26 seconds for your files, despite having a larger file with a better image quality.

    Both types of files get "past duration xxxx too large" warnings
     
  18. GTA Driver

    GTA Driver Well-Known Member

    Messages:
    736
    Likes Received:
    257
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    oh, yeah the scenery in your videos is better than mine too!
     
  19. Gibson99

    Gibson99 Well-Known Member

    Messages:
    7,770
    Likes Received:
    5,025
    Location:
    Houston, we've had a problem, Texas
    Country:
    United States
    Dash Cam:
    Eh, one or two... ;)
    Lol well that was from our vacation to Colorado. It was leaving my dad's neighborhood in Buena Vista. Colorado law says yield to pedestrians and wildlife. Who am I to argue? :)
     
  20. GTA Driver

    GTA Driver Well-Known Member

    Messages:
    736
    Likes Received:
    257
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    Version 1.09

    I made a few changes that I found necessary. Since the last posted version, I have:
    • Made the target file name similar to the established convention used by @Gibson99. Files have gone from text indicating front or rear channel at the front of the file name to the end, ie rear-151119_033807_E1.mp4 to 151119_033807_E1-rear.mp4
    • ability to extract only channels you want. Either by the placement of text in the source file name (151119_033807_E1-ac.mp4) or by using channels frontChan or rearChan. I did this out of necessity for my use. Sixty percent of the time, I only need the front channel and ffmpeg is slow on my mp4 files compared to the avi files that @Gibson99 provided.
    The chansplit script files can be acquired from Google drive and box.net
    https://drive.google.com/folderview?id=0B-YpHDCsVqFENDd3OWl0em9fNzA&usp=sharing
    https://app.box.com/s/dlpz1rud480phgcca5htwp8l1s7fsz2a


    New feature - selectively extracting channels to produce files

    Aside from deleting files that you don't need, you can specify that you only want one of the two channels by altering variables or placing a string in the file names.

    By file name
    When I review files produced by the dashcam, I change the file names of my clips to state what the topic is.
    ie 150525_174414_E1-distracted-cellphone-MetroParkinglot-fc.mp4
    150525_174414_E1 - The date stamp file name my camera produces
    distracted-cellphone-MetroParkinglot - Is the topic, in this case A distracted drive in the Metro grocery store parking lot
    -fc - front channel only.​

    I use the text "-ac", "-fc" and "-rc" to state which channel to extract
    • -ac all channels
    • -fc front channel only
    • -rc rear channel only.

    By variables

    Code:
    #frontChan -extract front channel?
    frontChan=yes    #either yes or no
    
    #frontChan -extract rear channel?
    rearChan=yes    #either yes or no
    
    If you want to prevent a channel from being extracted and produced as a file, set frontChan or rearChan to "no".

    The script, comes with the frontChan and rearChan variables being set to yes, thereby extracting the front and rear channel respectively. If you find you have a multi-file project where only only of the two channels is necessary, you will need to edit the script if you want to exclude a channel

    Code:
    rearChan=no
    If there is text in the file name before the file extention directing which channel(s) to extract, such as -ac, -fc or -rc, it takes presidence over the script file variables. If the files contain the text "-ac.mp4" or "-ac.avi", the script extract all channels even if rearChan variable is set to no.
     
    Gibson99 likes this.

Share This Page