Linux Bash scripts to separate channels

Discussion in 'Multi-Channel Dash Cams' started by GTA Driver, Oct 4, 2015.

  1. GTA Driver

    GTA Driver Active Member

    Messages:
    706
    Likes Received:
    238
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    Performance Improvement

    A few weeks ago, when looking up how to extract video from a specified time frame using ffmpeg, I encountered an option that greatly improves the performance on using ffmpeg. What used to take a few minutes now takes a few seconds if not shorter.

    if you have already downloaded chansplit all add -c copy -copyts to the ffmpeg lines

    Code:
            if [ $extractFront = "yes" ];then
                date +"%Y-%M-%d %H:%M:%S - Starting front channel " >> chanSplitAll.log.txt
                chan="-front"
                newName=$(echo $outDir/$fname_NoExt$chan.$ext|sed 's/--front/-front/g')
                rm $newName
                ffmpeg -i $f $sound -map v:0 -async 1  -c copy -copyts $newName
            fi
            if [ $extractRear = "yes" ];then
                date +"%Y-%M-%d %H:%M:%S - Starting rear channel " >> chanSplitAll.log.txt
                chan="-rear"   
                newName=$(echo $outDir/$fname_NoExt$chan.$ext|sed 's/--rear/-rear/g')
                rm $newName
                ffmpeg -i $f $sound  -map v:1 -async 1 -c copy -copyts $newName   
            fi
    
    My google drive and box accounts have version 10.

    The chansplit script files can be acquired from Google drive and box.net
    https://drive.google.com/folderview?id=0B-YpHDCsVqFENDd3OWl0em9fNzA&usp=sharing
    https://app.box.com/s/dlpz1rud480phgcca5htwp8l1s7fsz2a


    It appears you may need a recent version of ffmpeg. I needed to update my version I had on Windows to get "-c copy -copyts" to work

    This is the first work I did in over a year on chansplit all as my rear channel camera died on my two channel system and I have bought a few single channel systems since then.
     
    Gibson99 likes this.
  2. Gibson99

    Gibson99 Well-Known Member

    Messages:
    7,439
    Likes Received:
    4,650
    Location:
    Houston, we've had a problem, Texas
    Country:
    United States
    Dash Cam:
    Eh, one or two... ;)
    Sweet! I'll have to add this to my vbscript as well!
     
  3. GTA Driver

    GTA Driver Active Member

    Messages:
    706
    Likes Received:
    238
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    On my windows PC, I was still on the version of ffmpeg from over two years ago, back when i noticed the discussion about your vbscript . Copy and copyts wasn't working on command line on windows box until I downloaded and installed the new version. It doesn't seem to improve the performance in Windows as much as it does in Linux.

    Copy and Copyts works on MOST ffmpeg commands. I am using FFMPeg to extract the time frame I want from a clip and it works really well. It also works on merging\joining\concatenating two or more files. Sadly it doesn't work on altering the speed of clips.
     
  4. Quidagis

    Quidagis Member

    Messages:
    36
    Likes Received:
    12
    Country:
    Colombia
    The script (chanSplitAll.sh) runs smooth on OS X El Capitan using ffmpeg from HomeBrew (https://brew.sh/) (Thinkware F770 and F50 files).
     
    Last edited: Aug 1, 2017
    GTA Driver likes this.
  5. GTA Driver

    GTA Driver Active Member

    Messages:
    706
    Likes Received:
    238
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    That's cool to know about HomeBrew and that you are able to use the script. Do you require homebrew to use ffmpeg, because according to FFMpeg's page it runs OS X intel 64; OS X 10.5 and above?
     
  6. Quidagis

    Quidagis Member

    Messages:
    36
    Likes Received:
    12
    Country:
    Colombia
    Yes, I do require it. This is what I get before installing from Homebrew: ./chanSplitAll.sh: line 122: ffmpeg: command not found

    Maybe it's not installed by default.

    Although I said the script runs smooth, it displays these messages when processing MP4 files:

    [avi @ 0x7fbae1800000] Format avi detected only with low score of 1, misdetection possible! tempFile.avi: Invalid data found when processing input

    Stream map 'v:1' matches no streams. To ignore this, add a trailing '?' to the map.

    [mp4 @ 0x7fed32813c00] Non-monotonous DTS in output stream 0:0; previous: -24024, current: -24024; changing to -24023. This may result in incorrect timestamps in the output file.

    [mp4 @ 0x7fed32813c00] Non-monotonous DTS in output stream 0:0; previous: -24023, current: -24024; changing to -24022. This may result in incorrect timestamps in the output file.

    [mp4 @ 0x7fed32813c00] Non-monotonous DTS in output stream 0:0; previous: -24022, current: -24024; changing to -24021. This may result in incorrect timestamps in the output file.

    [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fdb23800000] unsupported color_parameter_type aclc Guessed Channel Layout for Input Stream #0.1 : mono


    Do I have to customise the script to skip those messages? The video files I get after running the script are just fine, no sound which is what I want, muted videos.

     
    Last edited: Aug 3, 2017
  7. GTA Driver

    GTA Driver Active Member

    Messages:
    706
    Likes Received:
    238
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    Sorry for a late reply, its been one of those weeks.

    You may want to experiment with ffmpeg outside of my script with one file from your camera.

    Code:
    ffmpeg -i YOURFILENAME.mp4 $sound  -map v:0 -async 1 -c copy -copyts  NEWName.mp4
    
    where
    • YOURFILENAME.mp4 is the raw 2 channel file mp4 coming from your camera
    • NEWName.mp4 the filename you specify containing vidwo only the front camera

    It sounds like homebrew installs software and what I would call packages in the Linux world that don't come at the in the standard configuration for Mac OS. For FFmpeg, in Windows and Linux, one has download or acquire the software. I have done a recent Linux install and I had to get FFMpeg. What appears to be cool about Homebrew is that it bundles numerous things such as wget so you don't have to download and install them individually. In Windows, I have to download wget, but it comes default in most Linux distros. I would recommend going to ffmpeg's home page and downloading the newest version in Mac OS as i feel you may have an older version at it may be the source to some of the problems you are having.


    The script makes a dummy AVI or MP4 files if neither exist. It appears Mac OS can not follow something in the statements below as it handles redirection or piping differently.
    Code:
    wcAvi=$(ls  |grep avi |wc     |awk '{print $1}')
    wcMp4=$(ls  |grep mp4 |wc    |awk '{print $1}')
    
    if [ $wcAvi -gt 0 ];then
        if [ $wcMp4 -eq 0 ];then
              firstAvi=$(ls *.avi |head -1)
              head $firstAvi|head -2 > tempFile.mp4
              tempFile=tempFile.mp4
          fi   
    fi
    
    if [ $wcMp4 -gt 0 ];then
        if [ $wcAvi -eq 0 ];then
              firstMp4=$(ls *.mp4 |head -1)
              head $firstMp4|head -2 > tempFile.avi
              tempFile=tempFile.avi
          fi   
    fi
    
    It appears MAC OS does not handle redirection the same way as Linux and Unix. It looks like its breaking on the head statement. You could comment out or remove head $firstMp4|head -2 > tempFile.avi and head $firstAvi|head -2 > tempFile.mp4

    Here is the solution if you only have MP4's
    1. Make a backup/copy of the script in the event the below fails
    2. Remove the above code with, the WC statements and the "IF" statements that use them.
    3. Change the for statement removing the or pointed to AVI files, remove -o `ls *.avi`. That says or List avi files. You aren't look for avi files, so remove it
    Code:
    for f in `ls *.mp4` -o  `ls *.avi`

    I don't know if the Non-montonous DTS is because of an old version of ffmpeg or an issue with your MP4 from your Thinkware camera. FFmpeg can be used for several things aside from just splitting channels and on the earlier versions of ffmpeg, i got a timestamp warning if I tried to cut a segment out of a source file, say 5 to 10 seconds out of a 30 second clip. I don't get that error on the new version of ffmpeg, However, I am getting a similar error when I try to combine two or more files if one file is at fast speed and the other at normal speed with files coming from my A119.

    I would try running ffmpeg on the command line (terminal) on the files in a terminal
     
    Gibson99 likes this.
  8. Quidagis

    Quidagis Member

    Messages:
    36
    Likes Received:
    12
    Country:
    Colombia
    @GTA Driver

    Homebrew ffmpeg version is up to date:
    ffmpeg version 3.3.3 Copyright (c) 2000-2017 the FFmpeg developers built with Apple LLVM version 8.0.0 (clang-800.0.42.1)

    I ran the one line command, NEWName.mp4 file generated fine. Still getting Non-monotonous messages.

    Edited script as per your instructions, got same silent files and these messages displayed:

    [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fcc24001a00] unsupported color_parameter_type aclc Guessed Channel Layout for Input Stream #0.1 : mono


    Stream map 'v:1' matches no streams. To ignore this, add a trailing '?' to the map.
     
  9. GTA Driver

    GTA Driver Active Member

    Messages:
    706
    Likes Received:
    238
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    @Quidagis

    It appears either FFMpeg has different version numbers for the different operating systems or you have a newer version than myself.

    For me
    ffmpeg version 2.8.10-0ubuntu0.16.04.1 Copyright (c) 2000-2016 the FFmpeg developers

    The silent file matter is totally my fault and easy to fix.

    Because I usually have a passenger with me when driving and want to exclude sound to eliminate conversations in the car the audio variable is set to no. So change chanSplitAll.sh to keep sound. In the script, change the sound variable from no to yes. This will be around the 28th line on version 10

    Code:
    #audio         - use sound from clip or make silent, yes is sound, no is silent
    audio=yes    #either yes or no
    
    #frontChan -extract front channel?
    
    I neglected to consider I have a variable for audio when I cut and paste the line from the script when showing how to run in command line
    Code:
    ffmpeg -i YOURFILENAME.mp4 map a:0 -map v:0 -async 1 -c copy -copyts  NEWName.mp4
    ffmpeg -i YOURFILENAME.mp4               -map v:0 -async 1 -c copy -copyts  NEWName.mp4
    
    The first example keeps audio, the second one removes it.

    I have three cameras, two of them being single channel and with all of them I get warnings or message I ignore. I get more messages with my two channel camera than my single channel camera. Most of the time these warnings\mesages have no impact on the files I am creating. The colour parameter warning concerns me a little, but the "This may result in incorrect timestamps in the output file" concerns me a lot. I had the same problem twice and made small video clips that ended up to be 20 or minutes or moving regular speed video to really slow motion. I got the timestamp warning when trying to something with speeding a video clip and then including it with a normal clip which then made the regular speed to be way off. I was planning on joining the ffmpeg forum to get help on this.

    These timestamp warning you are getting could be because
    1. the files produced by your cameras
    2. Operating system matters in Mac OS
    3. a bug in ffmpeg,
     
    Last edited: Aug 6, 2017
  10. GTA Driver

    GTA Driver Active Member

    Messages:
    706
    Likes Received:
    238
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    I copied four old multi channel MP4 files that I am going to provide to @Quidagis to a folder before loading on box.com. It gave me the opportunity to see what impact copy and copyts have on ffmpegs performance and how much of an improvement version 10 has over version 9.

    chanSplitAll.sh withOUT copy and copyts: 4 minutes and 26 seconds
    chanSplitAll.sh WITH copy and copyts: 5 seconds

    @Quidagis
    to determine the source of the problem, I have uploaded four files that convert with minor warnings. You can find the files at
    https://app.box.com/s/67vuugdeez68y08rhgsmd9i2ci556y25

    The one starting with 14 (shot in 2014) has sound, the other three don't. I was able to combine the video from the last three files to produce a video on Youtube. I don't get a time stamp warning on these.

    If you download my files and don't get the errors you are getting on files produced from your camera, we know its not an operating system issue but something with your multi-channel camera.
     
    Last edited: Aug 6, 2017
  11. GTA Driver

    GTA Driver Active Member

    Messages:
    706
    Likes Received:
    238
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    In seems like a new version of ffmpeg has come out since May or June. Here is how to get the latest version if ffmpege is already on your Ubuntu/Mint box
    Code:
    sudo add-apt-repository -y ppa:mc3man/trusty-media; sudo apt-get update; sudo apt-get install --only-upgrade ffmpeg
    
     
  12. Quidagis

    Quidagis Member

    Messages:
    36
    Likes Received:
    12
    Country:
    Colombia
    Hey @GTA Driver

    I executed v1.10 - 27 Jul 2016 of your script (unmodified) on your MP4 video files, and yes, not a single warning/error message. You're right, it's my MP4 video files. But that's fine with me, all I want is to remove audio from my videos, merge them all together and and get one big single silent video file. BTW, how about concatenating the files as an option in your script? Too much coding effort, maybe? Just a thought.

    Thanks for your script and kindly help.
     
    GTA Driver likes this.
  13. GTA Driver

    GTA Driver Active Member

    Messages:
    706
    Likes Received:
    238
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    That's good to know about the Thinkware F770 and F50 files. Just wondering, but do either camera have an option to use another format such as AVI. Some cameras, such as the Mobius have that option but I am not aware of that for a dual channel camera. @Gibson99 's multi-channel camera writes in AVI. Also I have noticed some problems with my MP4 have gone away with a new version of ffmpeg. Hopefully new versions of ffmpeg will fix that.

    Funny, for a month now I have been familiar with using ffmpeg to concatenate several files into one for files coming from my A119 but I never thought about making it an option for chanSplitall.sh. It would take several hours to program that but what I may do is show dash cam users how to this and several other useful things at the command line. The rear camera on my two channel died and I have another front camera and a single channel rear camera. I have even mixed files from Iroad 3300ch (at one time a multi channel) and my A119 using one of ffmpegs concatenation options. ALSO IS SO MUCH FASTER THAN USING A VIDEO EDITOR, AND IF YOU KNOW HOW TO USE REDIRECTION, MORE ACCURATE.

    Here is how to do it at command line for both single and multi-channel cameras.
    1. Put all of the source files in a target folder
    2. Use ls and do redirection to a file
      Code:
      ls *.mp4 *.MP4 *.avi *.AVI > input.txt 
      Ignore the warning about extensions you don't have such AVI or upper or lower case. This for two extensions I am aware of that support multi-channel.
    • Open input.txt, it will look something like this
      Code:
      file001.MP4 
      file002.MP4 
      file003.MP4 
      or this (example files using my Iroad)
      Code:
      141105_072314_E1-front.mp4
      141105_072314_E1-rear.mp4
      151010_020020_I1-front.mp4
      
    • The file input.txt is alphabetical order and has all files from the ls statement in step #2. Remove files you don't want and change the order if you don't want it in alphabetical. If you want to include a clip twice, just copy and paste.
    • Add the single quotes before the file name and the word file. It would look something like
      Code:
      file 'file001.MP4'
      file 'file002.MP4'
      file 'file003.MP4'
      or this (example files using my Iroad)
      Code:
      file '141105_072314_E1-front.mp4'
      file '141105_072314_E1-rear.mp4'
      file '151010_020020_I1-front.mp4'
      
    • save and exit input.txt.
    • Execute either of the commands depending whether you want audio or not
      To remove audio:
      Code:
      ffmpeg -f concat -i input.txt -c copy -an output.mp4 
      to keep audio :
      Code:
      ffmpeg -f concat -i input.txt -c copy  output.mp4 
    Oddly the above works somewhat with RAW NonSplit multi-channel video files ... kind of. If you only want the front channel it works, otherwise if you want the rear, you need to run chanSplitAll.sh and then steps 1 to 7 in the folder chanSplitAll.sh creates, which is called chanSplit.

    By the end of the month, version 11 will include concatenation.

    edit, the numbering for my step to step instructions got messed up.
    Edit, the above works for Linux. I suspect Mac Os handles redirection differently.
     
    Last edited: Aug 6, 2017
    Quidagis likes this.
  14. GTA Driver

    GTA Driver Active Member

    Messages:
    706
    Likes Received:
    238
    Location:
    Greater Toronto Area
    Country:
    Canada
    Dash Cam:
    JaeWonCNC Iroad IOne-3300CH, G1W-c, Mobius C, A119
    A long shot, but perhaps you could convert the files from mp4 to avi and then run my script

    ffmpeg -i source.mp4 -c copy -copyts target.avi
     
    Quidagis likes this.

Share This Page