Tag Archives: bash

Linux crypt methods i’m using.

Some directories on my fileserver are encrypted using ecryptfs.

eCryptfs is a POSIX-compliant enterprise cryptographic stacked filesystem for Linux. eCryptfs stores cryptographic metadata in the header of each file, so that encrypted files can be copied between hosts; the file will be decrypted with the proper key in the Linux kernel keyring. There is no need to keep track of any additional information aside from what is already in the encrypted file itself. You may think of eCryptfs as a sort of “gnupg as a filesystem”.

Example crypted directory using filename encryption

ECRYPTFS_FNEK_ENCRYPTED.FWYQ.y58tWRY7EQqyVPxxMk11BuiLpk8jXCQ8BRz0z5p9C2Pu2HZg-mmv---/ECRYPTFS_FNEK_ENCRYPTED.FWYQ.y58tWRY7EQqyVPxxMk11BuiLpk8jXCQ-Jx6RlQrLhDhdZ9IrcCOAE--
ECRYPTFS_FNEK_ENCRYPTED.FWYQ.y58tWRY7EQqyVPxxMk11BuiLpk8jXCQ8BRz0z5p9C2Pu2HZg-mmv---/ECRYPTFS_FNEK_ENCRYPTED.FWYQ.y58tWRY7EQqyVPxxMk11BuiLpk8jXCQ.cE4XNdvLLui2EamsqU2rE--
ECRYPTFS_FNEK_ENCRYPTED.FWYQ.y58tWRY7EQqyVPxxMk11BuiLpk8jXCQ8BRz0z5p9C2Pu2HZg-mmv---/ECRYPTFS_FNEK_ENCRYPTED.FWYQ.y58tWRY7EQqyVPxxMk11BuiLpk8jXCQ1J..MuVpsw6kaCgwYCwJXk--

Adhoc mounting

You can use the same dir for mounting!
Filenames are encrypted also.
Use umount go back to the crypted state

NOTE: You can write files to the crypted dir, if you are NOT using filename encryption, you can’t see which one is crypted and which is not.
Use the script below to get hints of the readable files!

mount -t ecryptfs securedir securedir
Passphrase:
Select cipher:
 1) aes: blocksize = 16; min keysize = 16; max keysize = 32
 2) blowfish: blocksize = 8; min keysize = 16; max keysize = 56
 3) des3_ede: blocksize = 8; min keysize = 24; max keysize = 24
 4) twofish: blocksize = 16; min keysize = 16; max keysize = 32
 5) cast6: blocksize = 16; min keysize = 16; max keysize = 32
 6) cast5: blocksize = 8; min keysize = 5; max keysize = 16
Selection [aes]:
Select key bytes:
 1) 16
 2) 32
 3) 24
Selection [16]:
Enable plaintext passthrough (y/n) [n]:
Enable filename encryption (y/n) [n]: y
Filename Encryption Key (FNEK) Signature [xxxxxxxxxxxxxxxxxx]:
Attempting to mount with the following options:
  ecryptfs_unlink_sigs
  ecryptfs_fnek_sig=xxxxxxxxxxxxxxxxxx
  ecryptfs_key_bytes=16
  ecryptfs_cipher=aes
  ecryptfs_sig=xxxxxxxxxxxxxxxxxxx
Mounted eCryptfs

Crypting disks for travel/backup

For this i’m using Luks

The Linux Unified Key Setup (LUKS) is a disk encryption specification created by Clemens Fruhwirth in 2004 and was originally intended for Linux.

While most disk encryption software implements different, incompatible, and undocumented formats , LUKS implements a platform-independent standard on-disk format for use in various tools. This not only facilitates compatibility and interoperability among different programs, but also assures that they all implement password management in a secure and documented manner

Formatting a disk and mounting

cryptsetup luksFormat /dev/sdb1

WARNING!
========
This will overwrite data on /dev/sdb1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase for /dev/sdb1:
Verify passphrase:
root@workstation:~# cryptsetup luksOpen /dev/sdb1 crypto
Enter passphrase for /dev/sdb1:
root@workstation:~# mkfs.ext4 /dev/mapper/crypto
mke2fs 1.44.1 (24-Mar-2018)
Creating filesystem with 244188672 4k blocks and 61054976 inodes
Filesystem UUID: 844eb9ee-d4da-4dfd-9d94-b62987e96b93
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

root@workstation:~# mount /dev/mapper/crypto /backup/

Umount and close

umount /backup
cryptsetup luksClose crypto

File crypt checker (checks for data files which COULD be crypted)

#!/bin/bash
find secure -type f  -exec file {} \;| egrep -v "ASCII text|MS Windows shortcut|ISO-8859 text|image data|PDF document|PC bitmap|Composite Document File|WebM|ISO Media|Microsoft Word|HTML| Microsoft Excel|Matroska|vCard|Microsoft ASF|Web/
P|RIFF|MPEG|RealMedia|UTF\-8 Unicode|Zip archive data|Macromedia Flash|RAR archive|EPUB document|Adobe Photoshop Image|AppleDouble|OpenType|empty|gzip compressed data|MS Windows|OpenDocument|Paint Shop Pro|executable|PostScript document|
Rich Text|audio data|SVG Scalable|UDF filesystem|very short file|Web Open Font Format|IFF |TrueType|BeautifulWatches|MTS:" > data-or-not

OR 

File crypt checker ( When the directory is in crypted state )
All files should be raw data

#!/bin/bash
find secure -type f  -exec file {} \;| egrep -v data$  > notcrypted-files

Fast search ‘n sorting on my fileserver

When i’m looking for certain files on my fileserver i sometimes have to fallback to locate.

Most of the times i use the methods mentioned in :
https://www.henriaanstoot.nl/2022/08/04/finding-files-on-my-fileserver/

The script below helps me to copy located files to a temporary directory.
(Which is excluded in al kinds of other find tools)

It will remove the slashes in a path, but keeps the rest.

  • You can pipe to this script
  • All files in 1 directory, but NO overwrite of files (keeps path). This allows for easy browsing with a picture viewer.
  • Want to know original path, look at the filename .. think of where the slashes should go

Example:

Lets find all jpg’s which linux in its name

# this will list all found files
locate -i linux | grep -i jpg$ 

output example (see below) (With slashes and spaces)
/tank/WorkDirectory/TMP/UITZOEKEN/cds uitzoeken/div/141/31/cd3/done/gfx/linux-from-scratch.jpg

cplocatescript

#!/bin/bash
mkdir -p /mnt/private/TEMP/$$
cat - | while read line ; do cp "$line" /mnt/private/TEMP/$$/$(echo "$line" | tr -cd 'A-Za-z0-9._-' ; echo "" ) ;done

When running combined command:

locate -i linux | grep -i jpg$ | cplocatescript

It will create a directory like below

ls /mnt/private/28723/
fileserverHenri__newsort__WorkDirectory2018_ltstufffromlaptopsnew_uitzuitzoekenWERKDIRECTORY-DEC-2010cd6wwwhtdocsworklinux-project14l.jpg
fileserverHenri__newsort__WorkDirectory2018_ltstufffromlaptopsnew_uitzuitzoekenWERKDIRECTORY-DEC-2010cd6wwwhtdocsworklinux-project14s.jpg
tankWorkDirectoryTMPUITZOEKENcds-uitzoekendiv14131cd3donegfxlinux-from-scratch.jpg
tankWorkDirectoryTMPUITZOEKENcds-uitzoekendiv17166dataUNISONBACKUP_TO_HOMEUNISON_DONELINUX_FROM_SCRATCH.JPG
tankWorkDirectoryTMPUITZOEKENcds-uitzoekendiv982datalinux-presentatiespresentatie.hp.nllinux_cd_deel3Imagescdromtitel.jpg
etc ...
etc ...
etc ...

Bash tips part 1

Best Practices

#!/bin/bash     # Set shebang .. interpreter (sh/python/bash)
#set -x            # debug flag, shows all output and variables
set -e             # exit when an error occurs, Dont use this when sourcing a scriptor using bash ./scriptnaam gebruikt. 
set -u             # exit when a variable isn't defined
set -o pipefail # exit when a pipe command fails
# Add comments to your scripts!

PIPEFAIL Example
grep string /nonexistenddir/file | sort # Does NOT give an error, sort works!
So echo $? gives 0
When set -o pipefail is set, above example will print 1 or 2

Bash options example

#!/bin/bash

usage ()
{
    echo >&2 "usage: $0 <list-of-options>"
}

main ()
{
    [ $# -lt 1 ] && usage
    INSTALL_DIR=`dirname $0`
    for i in $@; do
        echo "$INSTALL_DIR $i"
    done
    
}

main $@

Execute output from a script.

Sometimes i write scripts which print the command I want to have executed. Most of the times to check the generated commandline.
simple example

ls | grep 2000 | while read ; do echo rm -f "$REPLY" ;done

Above only prints the lines
rm -f <filename with 2000>
rm -f <other filename with 2000>

Instead of removing the echo command, you can add a | bash , to have it executed.

ls | grep 2000 | while read ; do echo rm -f "$REPLY" ;done | bash

Or even shorter ( Use !! for previous command )

!! | bash 

History

Search with CTRL-R in your command history, and use !<number> to execute this command again.

touch file1 file2 file3 file4
chmod 664 !* # will chmod only the files

When you used typed
systemctl stop httpd
and want to start again use replace
^stop^start
This will take previous command containing stop and places stop with start and executes this.

Toggle history on/off with “set -o history”, depending on your setup you can use “<space>command to be executed”

History log original username per date when sudo is being used
(creates .history.20230103.fash in /root/ )

HISTFILE=~/.history.$(date +%y%b%d).$(who am i | awk '{ print $1 }')

Skip first 2 lines and join 2 lines

#skip 2 lines
tail -n +3

#join 2 lines
sed 'N;s/\n/,/'

#3rd line from a file
sed "3q;d" /etc/hosts

#join on line line using comma's
paste -sd "," -

Find Tricks

#Remove empty directories
find ??? -empty -type d -delete

#Find multiple extentions
find archieven/ \( -name "*.png" -o -name "*.xml" \)
-print0 to handle filenames with spaces

Remove huge directory structures FAST

mkdir /tmp/empty ; rsync -a --delete /tmp/empty /path/blah

Check program installed and in path

which zenity >/dev/null 2>/tmp/err || ( echo "zenity not found, please install" ; exit 1 )

Change directory to location script for relative path usage

cd $(dirname $0)  
ls relativesubdir/

IFS (Internal Field Separator)

IFS=$' '     # internal field separator, strings split here
for f in $(cat /etc/hosts) ; do echo $f ;done
#outputs 127.0.0.1
#        localhost
IFS=$'\n\t'     # internal field separator, split end of line
for f in $(cat /etc/hosts) ; do echo $f ;done
# Outputs 127.0.0.1 localhost 

Difference for and while read example

echo "This will print every word" > text  # Places text in file
echo "second line" >> text                # Append text
for f in $(cat text) ; do echo $f ;done   # for example
This 
will
print 
every
word
second
line

cat text | while read f ; do echo $f ; done # Read example
This will print every word
second line

Correct way to make a tempfile

tempfile=$(mktemp -d -t /tmp/log.$$)    # Temp file, with unique name $$ is the process number

Usage in script :
ls > $tempfile

Cleanup:
rm -f "/tmp/$tempfile"    # Remove

Direcory empty test

[ "$(ls -A /tmp)" ] && echo "Not empty" || echo "Empty" # test directory Empty/filled
test -d /tmp/1 && rmdir /tmp/1           # Removes a directory when it exists, will give an error when NOT empty

Using Expand

On directories archive30_tmp till archive35_tmp setting recursive chmod 2775 

find achief{30..35}_tmp -type d -exec chmod 2775 {} \;
 
or
 
mkdir tmp{1..3} # will create tmp1 tmp2 tmp3

echo pr{ut,utser}s # Outputs "pruts prutsers"

Test root user

if [ $USER == "root" ] ; then … ; fi # execute only when root
if $USER is empty, this will give an error.

if [ w$USER == "wroot" ] works 
qouting $USER also, but qouting a number using less/greater test could be problematic

Mount test

grep -qs /media /proc/mounts && echo "/media is mounted"

Date tricks

datum=$(date +%Y%m%d)                    # datum is yyyymmdd 20230103
today=$(date +%F)
tomorrow=$(date --date="next day" +%F)
p3=$(date --date="$p2" +%F)
dater=$p3
#now=$(date +%s --date="1 days ago")
now=$(date -d $(date --date="1 days ago" +%F) +%s)
p3epoch=$(date --date="$p3" +%s)
dater=$(date -d "$p3 1 year" +%F)

date -d @<UNIX timestamp> # Timestamp to date

Size test in directory

if [ "$(df /tmp |grep -v Available | awk {' print $4 }')" -lt 1000000 ] ; then echo "not enough free in /tmp" ;fi  

Parallel Tasks

4 parallel jobs  
find jpg -type f -name \*.jpg -print0 | xargs -0 -n1 -P4 ./convert.sh

Cluster ssh trick

Using Clusterssh at work
## clusterssh trick
Only needed to do stuff on server having a certain directory

    clusterssh storageservers # ssh to 24 storage servers
    sudo su -
    cd /bricks/*/backup2 # change directory to this if exists
    cd /alternatedir/brick0*/backup2 # change to this alternate directory if it exists 
# directories not found? then you are still in /root
    pwd | grep testdir || logout # no testdir in found subdirs? (there are non in /root so you will be logged out)
    id | grep myuser && logout # if mortal user? then logout again .. you will be disconnected from servers not containing the backup2 directories
    Do your work as root 

Forgot to sudo?

systemctl restart httpd (wont work as user)
sudo !!
(this will do  "sudo systemctl restart httpd")

Top 5 homedir users

du -hsx /home/* | sort -rh | head -5

Removing a huge file takes a long time .. lets truncate it

: > hugefile
rm hugefile

Check memory banks using dmidecode

sudo dmidecode| grep  -i -B1 "Form Factor" (B1 means BEFORE 1 line, A is AFTER)
 
        Size: 8192 MB
        Form Factor: DIMM
--
        Size: 8192 MB
        Form Factor: DIMM
--
        Size: No Module Installed
        Form Factor: DIMM
--
 
sudo dmidecode -t memory | grep -i size
        Size: 4096 MB

--
sudo lshw -short -C memory
H/W path            Device      Class          Description
==========================================================
/0/0                            memory         96KiB BIOS
/0/1000                         memory         4GiB System Memory
/0/1000/0                       memory         4GiB DIMM RAM

Script log replay

    script --timing=/tmp/time.txt /tmp/script.log
start your script
    scriptreplay -t /tmp/time.txt /tmp/script.log
replays

Deleted file takes up disk space (not being released)

lsof

httpd 12209 root 6w REG 253,3 5233639424 2097234 /var/log/httpd/access_log-20200711 (deleted)

cd /proc/12209/fd

find /proc/12209/fd -ls | grep '(deleted)'

121613145 0 l-wx------ 1 root root 64 Jul 10 12:04 /proc/12209/fd/6 -> /var/log/httpd/access_log-20200711\ (deleted)

: > "/proc/12209/fd/6"                 # Truncate

Keyword in line grep

cat file | grep -o 'skip_reason.*' # till end
cat file | grep -o 'skip_reason.*tillhere' 

Remove space filename

mv *\ * a 
rm *\ * 

Upper to lower case

 tr '[:upper:]' '[:lower:]'

Whole directories to lowercase

#!/bin/bash
#print usage 
if [ -z $1 ];then
        echo "Usage :$(basename $0) parent-directory"
        exit 1
fi

#process all subdirectories and files in parent directory
all="$(find $1 -depth)"



for name in ${all}; do
        #set new name in lower case for files and directories
        new_name="$(dirname "${name}")/$(basename "${name}" | tr '[A-Z]' '[a-z]')"

        #check if new name already exists
        if [ "${name}" != "${new_name}" ]; then
                [ ! -e "${new_name}" ] && mv -T "${name}" "${new_name}"; echo "${name} was renamed to ${new_name}" || echo "${name} wasn't renamed!"
        fi
done

echo
echo
#list directories and file new names in lowercase
echo "Directories and files with new names in lowercase letters"
find $(echo $1 | tr 'A-Z' 'a-z') -depth

exit 0

Sort by lenght

awk '{ print length(), $0 | "sort -n" }' 

Order / Delimiter AWK

echo 8.168.192.in-addr.arpa. | awk -F  "." '/1/ { print $3,$2,$1 }'

From – Till using AWK

cat file | awk '/Title1/,/-----/' 

Replace Tabs

sed -e 's/\t/ /g'

Sed Case Insensitive

sed s/a/b/gI 

And using lynx std in

cat index.html | lynx --dump -stdin

Random sleep

sleep .$[ ( $RANDOM % 4 ) + 1 ]s
#remove . for whole seconds instead of 10th of seconds

Copy pasting a lot from terminal , use xclip to copy to clipboard

xclip -sel c < bigsourcefile.c

Cache dir in memory for high IO processes

#!/bin/bash
[ ! -d /ramcache ] && mkdir -p /ramcache
mount -t tmpfs -o size=2G,nr_inodes=5k,mode=777 tmpfs /ramcache

Have a dual boot machine? And want to remotely start the other OS?
reboot2windows.sh

WINDOWS_TITLE=`grep -i "^menuentry 'Windows" /boot/grub/grub.cfg|head -n 1|cut -d"'" -f2`
sudo grub-reboot "$WINDOWS_TITLE"
sudo reboot

Move files in subdirectory per create hour

for f in $(seq -w 0 23) ; do find . -type f -print0 | xargs -0 -r stat -c "%y %N" | grep -E " $f:" | cut -f2 -d\' | while read ; do mkdir -p $f ; mv "$REPLY" $f/ ; done ;d

ABC to Animation for Concertina

I want to take simple ABC Music notation tunes and convert these to animations which shows you which keys to press and to push or pull.

This is a work in progress.
Script is using ImageMagick and ffmpeg

I started this in bash, and i should rewrite this in python.

Bash script which generates the images

#!/bin/bash
convert -size 800x600 canvas:white white.png
x=55
y=55
draw=""
for xx in $(seq 1 6); do
for yy in $(seq 1 5); do
	startx=$(( $x * $xx + 100))
	starty=$(( $y * $yy + 200))
if [ $xx -gt 3 ] ; then startx=$(($startx + 200)) ;fi
draw="$draw  -draw \"circle $startx,$starty $(( $startx - 10)),$(( $starty - 10 ))\""
done
done
echo convert $draw white.png          draw_circle.png | bash

for xx in $(seq 1 6); do
for yy in $(seq 1 5); do
        startx=$(( $x * $xx + 100))
        starty=$(( $y * $yy + 200))
if [ $xx -gt 3 ] ; then startx=$(($startx + 200)) ;fi
convert -fill white -stroke black $draw  -draw "circle $startx,$starty $(( $startx - 10)),$(( $starty - 10 ))" draw_circle.png $xx-$yy.png
convert  -draw "rectangle 300,50 500,100"  $xx-$yy.png push-$xx-$yy.png
convert  -draw "rectangle 150,50 650,100"  $xx-$yy.png pull-$xx-$yy.png
done
done

Example movie generated using random notes/length pause

for f in $(seq 1 50) ; do
pushpull=$(shuf -n1 -e push pull)
row=$(shuf -n1 -e 1 2 2 2 2 2 3 4 4 5 5 5 6)
col=$(shuf -n1 -e 1 1 1 2 2 3)
pauze=$(shuf -n1 -e 0 0 0 0 0 0 0 0 1 )
length=$(shuf -n1 -e 1 1 1 1 2 2 3 4)
file="$pushpull-$row-$col.png"
if [ $pauze == "1" ]; then
file="draw_circle.png"
fi
for f in $(seq 1 $lenght) ; do
echo "file $file"
echo "duration 0.5"
done
done

list to mp4

ffmpeg -f concat -i list  -vf fps=10 -pix_fmt yuv420p test.mp4

Above movie shows left and right hand. Keys to press.
And the bar above represents PULL/PUSH

Todo/In progress:

  • abc parser ( started )
  • Push/Pull as text in above movie also note to play??
  • Rewrite to python

Two minute timelapse of oyster mushrooms growing.

#!/bin/bash
# capture images from Foscam ( 5 minute interval )
while true; do
wget -q "http://IP-WEBCAM:88/cgi-bin/CGIProxy.fcgi?cmd=snapPicture2&usr=adminuser&pwd=adminpass" -O $(date +%Y%m%d%H%M).jpg
sleep 300
done

# from jpg to mp4
ffmpeg -f image2 -r 24 -pattern_type glob -i '*.jpg'   -vcodec libx264 -profile:v high444 -refs 16 -crf 0 -preset ultrafast -vf scale=1920:1080 paddo.mp4

If your ipcam can’t add timestamps, use below part to add text (replace in above script)

...
...
wget -q "http://IP-WEBCAM:88/cgi-bin/CGIProxy.fcgi?cmd=snapPicture2&usr=adminuser&pwd=adminpass" -O /tmp/1.jpg
convert -fill white -pointsize 30 -draw "text 10,30 '$(date)'" /tmp/1.jpg $(date +%Y%m%d%H%M).jpg
rm -f /tmp/1.jpg 
...
...

Locate and delete file copies

A simple script to locate files and select which to delete.
Uses md5sum to compare
(just be sure locatedb is up to date!)

./locatemd5 VID_20130926_211302.mp4
1 : 125a65e830c1f3654714daa0f8a41699  /tank/Backup/Nae Bother Drive/Movies/VID_20130926_211302.mp4
2 : 125a65e830c1f3654714daa0f8a41699  /tank/Backup/rclonegdrive/Nae Bother Drive/Movies/VID_20130926_211302.mp4
3 : 125a65e830c1f3654714daa0f8a41699  /tank/Private/Henri/_All online drives/Google Drive Henri/Nae Bother Drive/Movies/VID_20130926_211302.mp4
4 : 125a65e830c1f3654714daa0f8a41699  /tank/Private/gfx/Gsm/GSM Henri/nexus/2013/20130926/VID_20130926_211302.mp4
5 : 125a65e830c1f3654714daa0f8a41699  /tank/Private/Work Directory/Sorted/movies/own/VID_20130926_211302.mp4
6 : 82cd340b2b54d3ef65a02c8f31b04970  /tank/Private/www/cutshort/VID_20130926_211302.mp4
---------- which to delete ---------------
1 2 3 5
delete 1 /tank/Backup/Nae Bother Drive/Movies/VID_20130926_211302.mp4
rm: remove regular file '/tank/Backup/Nae Bother Drive/Movies/VID_20130926_211302.mp4'? y
delete 2 /tank/Backup/rclonegdrive/Nae Bother Drive/Movies/VID_20130926_211302.mp4
rm: remove regular file '/tank/Backup/rclonegdrive/Nae Bother Drive/Movies/VID_20130926_211302.mp4'? y
delete 3 /tank/Private/Henri/_All online drives/Google Drive Henri/Nae Bother Drive/Movies/VID_20130926_211302.mp4
rm: remove regular file '/tank/Private/Henri/_All online drives/Google Drive Henri/Nae Bother Drive/Movies/VID_20130926_211302.mp4'? y
delete 5 /tank/Private/Work Directory/Sorted/movies/own/VID_20130926_211302.mp4
rm: remove regular file '/tank/Private/Work Directory/Sorted/movies/own/VID_20130926_211302.mp4'? y

Bash Script

#!/bin/bash
# below gets array from subshell
shopt -s lastpipe

nr=1
locate "$1" | while read file ; do
        md5sum=$(md5sum "$file")
        echo  "$nr : $md5sum"
        myArray[$nr]="$file"
        (( nr++ ))
done
echo  "---------- which to delete ---------------"
read numbers
for f in $numbers ; do
 echo "delete $f ${myArray[$f]}"
 rm -i "${myArray[$f]}"
done

Generating PDFs from abc files to include in tunebooks.

see : https://www.henriaanstoot.nl/2022/11/21/tunebook-generator/

Edit or download a ABC music file

(More info about music notation programs https://www.henriaanstoot.nl/2022/08/24/music-notation-programs/ )

I wanted to learn Bella Ciao on my Concertina. I find the sound of the Concerina fitting the tune. So i used vi and conversion tools to create a pdf for my concertina tunebook.

(While being popular nowadays because it was used in La Casa de Papel (Money Heist). It has a interesting history)
https://en.wikipedia.org/wiki/Bella_ciao

X:
T: Bella Ciao
M: 4/4
L: 1/8
R: reel
K:C 
Zz EAB | cA-A z  EAB | cA-A z  EAB | c2 BA c2 BA | 
e2 e2 eede | ff-f2 z fed | | fe-e z edc | B2 e2 c2 B2 | 
A4 z z de | ff-f2 z fed | fe-e2 z edc | B2 e2 c2 B2 | 
w: "repeat\ from\ beginning"
A4 z EAB | cA-A z  EAB | cA-A z  EAB | c2 BA c2 BA |
e2 e2 e2 de | ff-f2 z fed | fe-e z edc | B2 e2 c2 B2 |
A4 z EAB | cA-A z  EAB | cA-A z  EAB | c2 BA c2 BA |
e2 e2 eede | ff-f2 z fed | fe-e2 z edc | B2 e2 c2 B2 | 
A4 z ede | ff-f2 z fed | fe-e2 z edc | B2 e2 ^f2 ^g2 | a4 Zz |

Using below command, I get a nicely formatted PDF

abcm2ps -x -O - bella.abc | ps2pdf  -sPAPERSIZE=a4 - bella.pdf

I’ll probably automate this, for example include this in the tunebook generator. Something like : “If abc file in subdir create pdf to include”

# Reads every file in abcsources and writes filename.pdf in current directory
ls abcsources/*abc | while read abc ; do  abcm2ps -x -O - "$abc" | ps2pdf  -sPAPERSIZE=a4 - "$(echo  $abc | cut -f2 -d/ | sed 's/abc/pdf/g')" ;done

Security cam ftp search major changes in images.

Weird title i know.

I have several ip cameras which monitor movement in and around our home.

I’m using Zoneminder and other home automation systems but i want to scan through a bunch of files uploaded by the security cameras to my secondary fileserver.

So what is interesting?

  • Major movement compared to a base image
  • Movement compared to a previous uploaded image
  • Setting a threshold when to output information (note the 65% mark)
| 76 % | 1438 | SDAlarm_20221129-213515.jpg | 3145 | .......-.......................X.....................................................................
| 76 % | 1439 | SDAlarm_20221129-213517.jpg | 3661 | ..............-.....................X................................................................
| 76 % | 1440 | SDAlarm_20221129-213519.jpg | 3739 | ...........-.........................X...............................................................
| 77 % | 1441 | SDAlarm_20221129-213521.jpg | 3704 | .....-...............................X...............................................................

Looking at the output we see:
At 76% of the captured images (image 1438) the threshold was above 3000 and the minus gives us the information of the difference between this image and the previous, the X marks the difference between current image and the baseline.
| percent | image number | filename | difference | graphbar

Bash script:

#!/bin/bash
threshold=3000
baseline=$( ls  SDAlarm*jpg | head -1)
previous=$( ls  SDAlarm*jpg | head -1)
total=$( ls *.jpg |wc -l)
echo "Number of files : $total"
nr=1
ls *jpg | while read; do
graph="....................................................................................................."
diff=$(compare -verbose -metric MAE $baseline $REPLY /dev/null  2>&1 | grep all | awk '{ print $2 }' | cut -f1 -d. )
prevdiff=$(compare -verbose -metric MAE $previous $REPLY /dev/null  2>&1 | grep all | awk '{ print $2 }' | cut -f1 -d. )
line=$( echo "100 / $total * $nr" | bc -l | cut -f1 -d.)
line=$(( $line + 1))
#echo -n "$line | $nr | $REPLY | "
#echo $diff
draw1=$(( $diff / 100 + 1))
draw2=$(( $prevdiff / 100 + 1))
graph=$(echo $graph | sed "s/./X/$draw1")
graph=$(echo $graph | sed "s/./-/$draw2")
if [ $diff -gt $threshold ] ; then
printf "| %4s %% | %3s | %30s | %5s | %102s \n" $line  $nr $REPLY $diff $graph
fi
nr=$(( $nr +1 ))
previous=$REPLY
done

Want to see only difference with previous image?

change:

if [ $diff -gt $threshold ] ; then
into
if [ $prevdiff -gt $threshold ] ; then

Automatic photo sorting with Gps location lookup

We take a lot of pictures, with our Nikon camera and our mobile phones.

(Apparently in 2019 5544 pictures)

Some stats

757 20190803 - on a single day (Holiday)

Average pictures per month
locate "/2019/" | egrep -i "photoalbum|gsm" | egrep -i "mp4$|jpg$" | grep -Eo '2[[:digit:]]{3}[[:digit:]]{2}[[:digit:]]{2}' | cut -c-6 |sort | uniq -c | sort -n | awk '{ sum += $1; n++ } END { if (n > 0) print sum / n; }'
461

Besides android pictures being automatically uploaded to our nextcloud, I’m using some apps and scripts to get pictures and movies stored on my fileserver. (bash scripts/andftp)

For sorting those media files, i made a sorting script.
(Today I added a location sorting addition using GPS information stored in the exif information.

  • jpg and jpeg (add your own extentions)
  • mp4 and mov (for mobile and nikon)
  • Sorts by camera model/year/date/location
  • tries to extract date from filename when not found in exifinfo
  • Sorts whatsapp media
  • Sorts Raw

INSTALLING

pip3 install reverse_geocoder
You need python3, exiftool, exiftime and mediainfo

copy below python script in ~/bin/reverse2.py

( need more info? change last print entry admin1/admin2)
[{‘lat’: ‘-39.45556’, ‘lon’: ‘173.85833’, ‘name’: ‘Opunake’, ‘admin1’: ‘Taranaki’, ‘admin2’: ‘South Taranaki District’, ‘cc’: ‘NZ’}]

import reverse_geocoder as rg
import sys

lat=sys.argv[1]
lon=sys.argv[2]

coordinates = (lat,lon)

results = rg.search(coordinates) # default mode = 2


#print (results)

for entry in results:
    print(entry['name'] + "(" +  entry['cc'] + ")")

And a bash script /usr/local/bin/exifsort.sh

#!/bin/bash
#set -x

reversepath=/home/henri/projects/reversegeo/reverse2.py

#RAW
rawcnt=`ls | grep -i  nef$ | wc -l`
if [ "$rawcnt" = "0" ] ; then
echo "no raw"
else
mkdir raw 2>/dev/null
ls | grep -i nef$ | while read ; do mv $REPLY raw ; done
fi


ls | egrep -i "jpg$|jpeg" | while read ; do 
 	location=""	
	getmodel=$(exiftool "$REPLY" |grep "Make " | awk '{ print $3 }')
	if [ "$getmodel" != "" ] ; then 
		getmodel=$getmodel/
	fi
	echo "$REPLY" | grep WA0 >/dev/null && getmodel=whatsapp/

	gpsinfo=$(exiftool -c "%+.6f" "$REPLY" |grep "GPS Position" | cut -d":" -f2 | tr -d ' ' | sed s/,/\ /g)
	if [ "$gpsinfo" != "" ] ; then 
		location=$(python3 $reversepath $gpsinfo | grep -vi load | sed s/\(NL\)//g)
	fi
	dater=$(exiftime "$REPLY" 2>/dev/null | egrep "Created|Digitized" | sed s/Digitized/Created/g | tail -1  | cut -c 16-19,21,22,24,25)
	if [ "$dater" = "" ] ; then 
#		echo "Trying from filename"
		dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}-[[:digit:]]{2}-[[:digit:]]{2}')

		if [ "$dater" = "" ] ; then 
#			echo "Trying from filename - maybe without dashes"
			dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}[[:digit:]]{2}[[:digit:]]{2}')

		fi
	fi
	if [ "$dater" != "" ] ; then 
		year=$(echo $dater | cut -c-4)
		mkdir -p "${getmodel}$year/${dater}/$location"
		mv "$REPLY" "${getmodel}${year}/${dater}/$location"
	else
		mkdir -p "${getmodel}unknowndate/$location"
		mv "$REPLY" "${getmodel}unknowndate/$location"
      	fi
done

ls | egrep -i "mov$|mp4$" | while read ; do 
	
 	location=""	
	getmodel=$(exiftool "$REPLY" |grep "Make " | awk '{ print $3 }')
	if [ "$getmodel" != "" ] ; then 
		getmodel=$getmodel/
	fi

	echo "$REPLY" | grep WA0 >/dev/null && getmodel=whatsapp/
	gpsinfo=$(exiftool -c "%+.6f" "$REPLY" |grep "GPS Position" | cut -d":" -f2 | tr -d ' ' | sed s/,/\ /g)
	if [ "$gpsinfo" != "" ] ; then 
		location=$(python3 $reversepath $gpsinfo | grep -vi load | sed s/\(NL\)//g)
	fi
	dater=$(mediainfo "$REPLY" | grep Encode | tail -1 | cut -f2- -d:  | cut -f3 -d" " | sed s/-//g)
	if [ "$dater" = "" ] ; then 
#		echo "Trying from filename"
		dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}-[[:digit:]]{2}-[[:digit:]]{2}')

		if [ "$dater" = "" ] ; then 
#			echo "Trying from filename - maybe without dashes"
			dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}[[:digit:]]{2}[[:digit:]]{2}')

		fi
	fi
	if [ "$dater" != "" ] ; then 
		year=$(echo $dater | cut -c-4)
		mkdir -p "${getmodel}$year/${dater}/$location"
		mv "$REPLY" "${getmodel}${year}/${dater}/$location"
	else
		mkdir -p "${getmodel}unknowndate/$location"
		mv "$REPLY" "${getmodel}unknowndate/$location"
      	fi
done

Example running in a directory with mixed media

# Raw images get moved into a RAW directory
no raw

# Samsung phone detected with date and GPS location
mkdir -p samsung/20220717/Hilversum
mv 20220717_133453.jpg samsung/20220717/Hilversum

# OnePlus phone
mkdir -p OnePlus/20021208/Voorburg
mv IMG_20190109_091825.jpg OnePlus/20021208/Voorburg

# Realme (Added country when not NL)
mkdir -p realme/20220607/Isle of Islay(GB)
mv IMG20220607213630.jpg realme/20220607/Isle of Islay(GB)

# Whatsapp has no date embedded so it gets it from filename
Trying from filename
Trying from filename - maybe without dashes
mkdir -p whatsapp/20221021/
mv IMG-20221021-WA0000.jpg whatsapp/20221021/

# Nikon without GPS
mkdir -p NIKON/20220613/
mv DSC_1423.MOV NIKON/20220613/

# Whatsapp video without exif
mkdir -p whatsapp/20170528/
mv VID-20170528-WA0006.mp4 whatsapp/20170528/

# No camera name detected in exif from mobile movie
mkdir -p 20190114/Maarssen
mv VID_20190114_142455.mp4 20190114/Maarssen

# Location in mp4
mkdir -p 20220607/Lamlash(GB)
mv VID20220607155044.mp4 20220607/Lamlash(GB)

Result

./NIKON/2022/20220613/DSC_1423.MOV
./NIKON/2022/20220610/750_1101.JPG
./realme/2022/20220818/Hilversum/IMG20220818203825.jpg
./realme/2022/20220607/Isle of Islay(GB)/IMG20220607213630.jpg
./2019/20190114/Maarssen/VID_20190114_142455.mp4
./whatsapp/2017/20170528/VID-20170528-WA0006.mp4
./whatsapp/2022/20221021/IMG-20221021-WA0000.jpg
./2022/20220607/Lamlash(GB)/VID20220607155044.mp4
./2022/20220516/Hilversum/VID20220516125913.mp4
./OnePlus/2002/20021208/Voorburg/IMG_20190109_091825.jpg
./samsung/2022/20220717/Hilversum/20220717_133453.jpg

You could automate this using incrond
apt-get install incron
add your user to /etc/incron.allow
incrontab -e
add

/fileserver/mediain/ IN_CREATE /usr/local/bin/sortmymedia.sh
Coping a file in the directory, auto sort and move to correct location

sortmymedia.sh

#!/bin/bash
cd /home/user/media
/usr/local/bin/exifsort.sh

Automating the h*ll out of windows applications using linux

I’m using a windows program for typesetting bagpipe music.
Luckily this runs on Linux using wine.

Sometimes i just want a PDF version of a tune, for example for my tunebook compiler. (Other post)
Or i want to batch convert a lot of bww files.

A long time ago i used a virtual machine with automation software for this.
Why not automate the process on my Laptop?

So i made this script, with a workaround for the xdotool wine problem.
(wine window needs to be active to accept key strokes, other linux xwindows you can use the ID of the window)

#!/bin/bash
# use c for close at the end, without c bgplayer wont be shut down
# bww2pdf path/to/music.bww c
# make tmp file
cat "$1" > /tmp/deze.bww
# start bgplayer if not started .. if not started wait 3 secs
slp=0
pgrep BGPlayer.exe >/dev/null || slp=3
pgrep BGPlayer.exe  >/dev/null|| ( nohup wine ~/.wine/dosdevices/c\:/Program\ Files\ \(x86\)/Bagpipe\ Player/BGPlayer.exe & )
sleep $slp
# get program id
pid=$(pgrep BGPlayer.exe)
# get xwindow id using pid
winid=$(xdotool search --limit 1 --all --pid $pid)
# activate window
xdotool search --desktop 0 --class "BGPlayer.exe" windowactivate
# open file menu and Open file
xdotool key --window $winid "alt+f"
xdotool key --window $winid "o"
# give program time to execute
sleep 1
# open our tmp file
xdotool type '\tmp\deze.bww'
xdotool key KP_Enter
sleep 2
# open file menu select Print and PDF as output
xdotool key "alt+f"
xdotool key "p"
xdotool key "P"
sleep 2
# execute
xdotool key KP_Enter
sleep 1
# File close
xdotool key "alt+f"
xdotool key "c"
sleep 2
# close program when c was added to commandline
mv ~/PDF/deze* "$1.pdf"
if [ "$2" == "c" ] ; then 
	xdotool key "alt+f"
	xdotool key "x"
fi
I’m not touching the keyboard when the program is running, all handled by the script