Tag Archives: bash

Security cam ftp search major changes in images.

2022-12-01 fash

Weird title i know.

I have several ip cameras which monitor movement in and around our home.

I’m using Zoneminder and other home automation systems but i want to scan through a bunch of files uploaded by the security cameras to my secondary fileserver.

So what is interesting?

Major movement compared to a base image
Movement compared to a previous uploaded image
Setting a threshold when to output information (note the 65% mark)

| 76 % | 1438 | SDAlarm_20221129-213515.jpg | 3145 | .......-.......................X.....................................................................

| 76 % | 1439 | SDAlarm_20221129-213517.jpg | 3661 | ..............-.....................X................................................................

| 76 % | 1440 | SDAlarm_20221129-213519.jpg | 3739 | ...........-.........................X...............................................................

| 77 % | 1441 | SDAlarm_20221129-213521.jpg | 3704 | .....-...............................X...............................................................

| 76 % | 1438 | SDAlarm_20221129-213515.jpg | 3145 | .......-.......................X..................................................................... | 76 % | 1439 | SDAlarm_20221129-213517.jpg | 3661 | ..............-.....................X................................................................ | 76 % | 1440 | SDAlarm_20221129-213519.jpg | 3739 | ...........-.........................X............................................................... | 77 % | 1441 | SDAlarm_20221129-213521.jpg | 3704 | .....-...............................X...............................................................

| 76 % | 1438 | SDAlarm_20221129-213515.jpg | 3145 | .......-.......................X.....................................................................
| 76 % | 1439 | SDAlarm_20221129-213517.jpg | 3661 | ..............-.....................X................................................................
| 76 % | 1440 | SDAlarm_20221129-213519.jpg | 3739 | ...........-.........................X...............................................................
| 77 % | 1441 | SDAlarm_20221129-213521.jpg | 3704 | .....-...............................X...............................................................

Looking at the output we see:
At 76% of the captured images (image 1438) the threshold was above 3000 and the minus gives us the information of the difference between this image and the previous, the X marks the difference between current image and the baseline.
| percent | image number | filename | difference | graphbar

Bash script:

#!/bin/bash

threshold=3000

baseline=$( ls SDAlarm*jpg | head -1)

previous=$( ls SDAlarm*jpg | head -1)

total=$( ls *.jpg |wc -l)

echo "Number of files : $total"

nr=1

ls *jpg | while read; do

graph="....................................................................................................."

diff=$(compare -verbose -metric MAE $baseline $REPLY /dev/null 2>&1 | grep all | awk '{ print $2 }' | cut -f1 -d. )

prevdiff=$(compare -verbose -metric MAE $previous $REPLY /dev/null 2>&1 | grep all | awk '{ print $2 }' | cut -f1 -d. )

line=$( echo "100 / $total * $nr" | bc -l | cut -f1 -d.)

line=$(( $line + 1))

#echo -n "$line | $nr | $REPLY | "

#echo $diff

draw1=$(( $diff / 100 + 1))

draw2=$(( $prevdiff / 100 + 1))

graph=$(echo $graph | sed "s/./X/$draw1")

graph=$(echo $graph | sed "s/./-/$draw2")

if [ $diff -gt $threshold ] ; then

printf "| %4s %% | %3s | %30s | %5s | %102s \n" $line $nr $REPLY $diff $graph

nr=$(( $nr +1 ))

previous=$REPLY

done

#!/bin/bash threshold=3000 baseline=$( ls SDAlarm*jpg | head -1) previous=$( ls SDAlarm*jpg | head -1) total=$( ls *.jpg |wc -l) echo "Number of files : $total" nr=1 ls *jpg | while read; do graph="....................................................................................................." diff=$(compare -verbose -metric MAE $baseline $REPLY /dev/null 2>&1 | grep all | awk '{ print $2 }' | cut -f1 -d. ) prevdiff=$(compare -verbose -metric MAE $previous $REPLY /dev/null 2>&1 | grep all | awk '{ print $2 }' | cut -f1 -d. ) line=$( echo "100 / $total * $nr" | bc -l | cut -f1 -d.) line=$(( $line + 1)) #echo -n "$line | $nr | $REPLY | " #echo $diff draw1=$(( $diff / 100 + 1)) draw2=$(( $prevdiff / 100 + 1)) graph=$(echo $graph | sed "s/./X/$draw1") graph=$(echo $graph | sed "s/./-/$draw2") if [ $diff -gt $threshold ] ; then printf "| %4s %% | %3s | %30s | %5s | %102s \n" $line $nr $REPLY $diff $graph fi nr=$(( $nr +1 )) previous=$REPLY done

#!/bin/bash
threshold=3000
baseline=$( ls  SDAlarm*jpg | head -1)
previous=$( ls  SDAlarm*jpg | head -1)
total=$( ls *.jpg |wc -l)
echo "Number of files : $total"
nr=1
ls *jpg | while read; do
graph="....................................................................................................."
diff=$(compare -verbose -metric MAE $baseline $REPLY /dev/null  2>&1 | grep all | awk '{ print $2 }' | cut -f1 -d. )
prevdiff=$(compare -verbose -metric MAE $previous $REPLY /dev/null  2>&1 | grep all | awk '{ print $2 }' | cut -f1 -d. )
line=$( echo "100 / $total * $nr" | bc -l | cut -f1 -d.)
line=$(( $line + 1))
#echo -n "$line | $nr | $REPLY | "
#echo $diff
draw1=$(( $diff / 100 + 1))
draw2=$(( $prevdiff / 100 + 1))
graph=$(echo $graph | sed "s/./X/$draw1")
graph=$(echo $graph | sed "s/./-/$draw2")
if [ $diff -gt $threshold ] ; then
printf "| %4s %% | %3s | %30s | %5s | %102s \n" $line  $nr $REPLY $diff $graph
fi
nr=$(( $nr +1 ))
previous=$REPLY
done

Want to see only difference with previous image?

change:

if [ $diff -gt $threshold ] ; then
into
if [ $prevdiff -gt $threshold ] ; then

Hobby, Photography

Automatic photo sorting with Gps location lookup

2022-11-26 fash

We take a lot of pictures, with our Nikon camera and our mobile phones.

(Apparently in 2019 5544 pictures)

Some stats

757 20190803 - on a single day (Holiday)

Average pictures per month

locate "/2019/" | egrep -i "photoalbum|gsm" | egrep -i "mp4$|jpg$" | grep -Eo '2[[:digit:]]{3}[[:digit:]]{2}[[:digit:]]{2}' | cut -c-6 |sort | uniq -c | sort -n | awk '{ sum += $1; n++ } END { if (n > 0) print sum / n; }'

461

Some stats 757 20190803 - on a single day (Holiday) Average pictures per month locate "/2019/" | egrep -i "photoalbum|gsm" | egrep -i "mp4$|jpg$" | grep -Eo '2[[:digit:]]{3}[[:digit:]]{2}[[:digit:]]{2}' | cut -c-6 |sort | uniq -c | sort -n | awk '{ sum += $1; n++ } END { if (n > 0) print sum / n; }' 461

Some stats

757 20190803 - on a single day (Holiday)

Average pictures per month
locate "/2019/" | egrep -i "photoalbum|gsm" | egrep -i "mp4$|jpg$" | grep -Eo '2[[:digit:]]{3}[[:digit:]]{2}[[:digit:]]{2}' | cut -c-6 |sort | uniq -c | sort -n | awk '{ sum += $1; n++ } END { if (n > 0) print sum / n; }'
461

Besides android pictures being automatically uploaded to our nextcloud, I’m using some apps and scripts to get pictures and movies stored on my fileserver. (bash scripts/andftp)

For sorting those media files, i made a sorting script.
(Today I added a location sorting addition using GPS information stored in the exif information.

jpg and jpeg (add your own extentions)
mp4 and mov (for mobile and nikon)
Sorts by camera model/year/date/location
tries to extract date from filename when not found in exifinfo
Sorts whatsapp media
Sorts Raw

INSTALLING

pip3 install reverse_geocoder
You need python3, exiftool, exiftime and mediainfo

copy below python script in ~/bin/reverse2.py

( need more info? change last print entry admin1/admin2)
[{‘lat’: ‘-39.45556’, ‘lon’: ‘173.85833’, ‘name’: ‘Opunake’, ‘admin1’: ‘Taranaki’, ‘admin2’: ‘South Taranaki District’, ‘cc’: ‘NZ’}]

import reverse_geocoder as rg

import sys

lat=sys.argv[1]

lon=sys.argv[2]

coordinates = (lat,lon)

results = rg.search(coordinates) # default mode = 2

#print (results)

for entry in results:

print(entry['name'] + "(" + entry['cc'] + ")")

import reverse_geocoder as rg import sys lat=sys.argv[1] lon=sys.argv[2] coordinates = (lat,lon) results = rg.search(coordinates) # default mode = 2 #print (results) for entry in results: print(entry['name'] + "(" + entry['cc'] + ")")

import reverse_geocoder as rg
import sys

lat=sys.argv[1]
lon=sys.argv[2]

coordinates = (lat,lon)

results = rg.search(coordinates) # default mode = 2


#print (results)

for entry in results:
    print(entry['name'] + "(" +  entry['cc'] + ")")

And a bash script /usr/local/bin/exifsort.sh

#!/bin/bash

#set -x

reversepath=/home/henri/projects/reversegeo/reverse2.py

#RAW

rawcnt=`ls | grep -i nef$ | wc -l`

if [ "$rawcnt" = "0" ] ; then

echo "no raw"

else

mkdir raw 2>/dev/null

ls | grep -i nef$ | while read ; do mv $REPLY raw ; done

ls | egrep -i "jpg$|jpeg" | while read ; do

location=""

getmodel=$(exiftool "$REPLY" |grep "Make " | awk '{ print $3 }')

if [ "$getmodel" != "" ] ; then

getmodel=$getmodel/

echo "$REPLY" | grep WA0 >/dev/null && getmodel=whatsapp/

gpsinfo=$(exiftool -c "%+.6f" "$REPLY" |grep "GPS Position" | cut -d":" -f2 | tr -d ' ' | sed s/,/\ /g)

if [ "$gpsinfo" != "" ] ; then

location=$(python3 $reversepath $gpsinfo | grep -vi load | sed s/$NL$//g)

if [ "$dater" = "" ] ; then

# echo "Trying from filename"

dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}-[[:digit:]]{2}-[[:digit:]]{2}')

if [ "$dater" = "" ] ; then

# echo "Trying from filename - maybe without dashes"

dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}[[:digit:]]{2}[[:digit:]]{2}')

if [ "$dater" != "" ] ; then

year=$(echo $dater | cut -c-4)

mkdir -p "${getmodel}$year/${dater}/$location"

mv "$REPLY" "${getmodel}${year}/${dater}/$location"

else

mkdir -p "${getmodel}unknowndate/$location"

mv "$REPLY" "${getmodel}unknowndate/$location"

done

ls | egrep -i "mov$|mp4$" | while read ; do

location=""

getmodel=$(exiftool "$REPLY" |grep "Make " | awk '{ print $3 }')

if [ "$getmodel" != "" ] ; then

getmodel=$getmodel/

echo "$REPLY" | grep WA0 >/dev/null && getmodel=whatsapp/

gpsinfo=$(exiftool -c "%+.6f" "$REPLY" |grep "GPS Position" | cut -d":" -f2 | tr -d ' ' | sed s/,/\ /g)

if [ "$gpsinfo" != "" ] ; then

location=$(python3 $reversepath $gpsinfo | grep -vi load | sed s/$NL$//g)

if [ "$dater" = "" ] ; then

# echo "Trying from filename"

dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}-[[:digit:]]{2}-[[:digit:]]{2}')

if [ "$dater" = "" ] ; then

# echo "Trying from filename - maybe without dashes"

dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}[[:digit:]]{2}[[:digit:]]{2}')

if [ "$dater" != "" ] ; then

year=$(echo $dater | cut -c-4)

mkdir -p "${getmodel}$year/${dater}/$location"

mv "$REPLY" "${getmodel}${year}/${dater}/$location"

else

mkdir -p "${getmodel}unknowndate/$location"

mv "$REPLY" "${getmodel}unknowndate/$location"

done

#!/bin/bash #set -x reversepath=/home/henri/projects/reversegeo/reverse2.py #RAW rawcnt=`ls | grep -i nef$ | wc -l` if [ "$rawcnt" = "0" ] ; then echo "no raw" else mkdir raw 2>/dev/null ls | grep -i nef$ | while read ; do mv $REPLY raw ; done fi ls | egrep -i "jpg$|jpeg" | while read ; do location="" getmodel=$(exiftool "$REPLY" |grep "Make " | awk '{ print $3 }') if [ "$getmodel" != "" ] ; then getmodel=$getmodel/ fi echo "$REPLY" | grep WA0 >/dev/null && getmodel=whatsapp/ gpsinfo=$(exiftool -c "%+.6f" "$REPLY" |grep "GPS Position" | cut -d":" -f2 | tr -d ' ' | sed s/,/\ /g) if [ "$gpsinfo" != "" ] ; then location=$(python3 $reversepath $gpsinfo | grep -vi load | sed s/$NL$//g) fi dater=$(exiftime "$REPLY" 2>/dev/null | egrep "Created|Digitized" | sed s/Digitized/Created/g | tail -1 | cut -c 16-19,21,22,24,25) if [ "$dater" = "" ] ; then # echo "Trying from filename" dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}-[[:digit:]]{2}-[[:digit:]]{2}') if [ "$dater" = "" ] ; then # echo "Trying from filename - maybe without dashes" dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}[[:digit:]]{2}[[:digit:]]{2}') fi fi if [ "$dater" != "" ] ; then year=$(echo $dater | cut -c-4) mkdir -p "${getmodel}$year/${dater}/$location" mv "$REPLY" "${getmodel}${year}/${dater}/$location" else mkdir -p "${getmodel}unknowndate/$location" mv "$REPLY" "${getmodel}unknowndate/$location" fi done ls | egrep -i "mov$|mp4$" | while read ; do location="" getmodel=$(exiftool "$REPLY" |grep "Make " | awk '{ print $3 }') if [ "$getmodel" != "" ] ; then getmodel=$getmodel/ fi echo "$REPLY" | grep WA0 >/dev/null && getmodel=whatsapp/ gpsinfo=$(exiftool -c "%+.6f" "$REPLY" |grep "GPS Position" | cut -d":" -f2 | tr -d ' ' | sed s/,/\ /g) if [ "$gpsinfo" != "" ] ; then location=$(python3 $reversepath $gpsinfo | grep -vi load | sed s/$NL$//g) fi dater=$(mediainfo "$REPLY" | grep Encode | tail -1 | cut -f2- -d: | cut -f3 -d" " | sed s/-//g) if [ "$dater" = "" ] ; then # echo "Trying from filename" dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}-[[:digit:]]{2}-[[:digit:]]{2}') if [ "$dater" = "" ] ; then # echo "Trying from filename - maybe without dashes" dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}[[:digit:]]{2}[[:digit:]]{2}') fi fi if [ "$dater" != "" ] ; then year=$(echo $dater | cut -c-4) mkdir -p "${getmodel}$year/${dater}/$location" mv "$REPLY" "${getmodel}${year}/${dater}/$location" else mkdir -p "${getmodel}unknowndate/$location" mv "$REPLY" "${getmodel}unknowndate/$location" fi done

#!/bin/bash
#set -x

reversepath=/home/henri/projects/reversegeo/reverse2.py

#RAW
rawcnt=`ls | grep -i  nef$ | wc -l`
if [ "$rawcnt" = "0" ] ; then
echo "no raw"
else
mkdir raw 2>/dev/null
ls | grep -i nef$ | while read ; do mv $REPLY raw ; done
fi


ls | egrep -i "jpg$|jpeg" | while read ; do 
 	location=""	
	getmodel=$(exiftool "$REPLY" |grep "Make " | awk '{ print $3 }')
	if [ "$getmodel" != "" ] ; then 
		getmodel=$getmodel/
	fi
	echo "$REPLY" | grep WA0 >/dev/null && getmodel=whatsapp/

	gpsinfo=$(exiftool -c "%+.6f" "$REPLY" |grep "GPS Position" | cut -d":" -f2 | tr -d ' ' | sed s/,/\ /g)
	if [ "$gpsinfo" != "" ] ; then 
		location=$(python3 $reversepath $gpsinfo | grep -vi load | sed s/\(NL\)//g)
	fi
	dater=$(exiftime "$REPLY" 2>/dev/null | egrep "Created|Digitized" | sed s/Digitized/Created/g | tail -1  | cut -c 16-19,21,22,24,25)
	if [ "$dater" = "" ] ; then 
#		echo "Trying from filename"
		dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}-[[:digit:]]{2}-[[:digit:]]{2}')

		if [ "$dater" = "" ] ; then 
#			echo "Trying from filename - maybe without dashes"
			dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}[[:digit:]]{2}[[:digit:]]{2}')

		fi
	fi
	if [ "$dater" != "" ] ; then 
		year=$(echo $dater | cut -c-4)
		mkdir -p "${getmodel}$year/${dater}/$location"
		mv "$REPLY" "${getmodel}${year}/${dater}/$location"
	else
		mkdir -p "${getmodel}unknowndate/$location"
		mv "$REPLY" "${getmodel}unknowndate/$location"
      	fi
done

ls | egrep -i "mov$|mp4$" | while read ; do 
	
 	location=""	
	getmodel=$(exiftool "$REPLY" |grep "Make " | awk '{ print $3 }')
	if [ "$getmodel" != "" ] ; then 
		getmodel=$getmodel/
	fi

	echo "$REPLY" | grep WA0 >/dev/null && getmodel=whatsapp/
	gpsinfo=$(exiftool -c "%+.6f" "$REPLY" |grep "GPS Position" | cut -d":" -f2 | tr -d ' ' | sed s/,/\ /g)
	if [ "$gpsinfo" != "" ] ; then 
		location=$(python3 $reversepath $gpsinfo | grep -vi load | sed s/\(NL\)//g)
	fi
	dater=$(mediainfo "$REPLY" | grep Encode | tail -1 | cut -f2- -d:  | cut -f3 -d" " | sed s/-//g)
	if [ "$dater" = "" ] ; then 
#		echo "Trying from filename"
		dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}-[[:digit:]]{2}-[[:digit:]]{2}')

		if [ "$dater" = "" ] ; then 
#			echo "Trying from filename - maybe without dashes"
			dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}[[:digit:]]{2}[[:digit:]]{2}')

		fi
	fi
	if [ "$dater" != "" ] ; then 
		year=$(echo $dater | cut -c-4)
		mkdir -p "${getmodel}$year/${dater}/$location"
		mv "$REPLY" "${getmodel}${year}/${dater}/$location"
	else
		mkdir -p "${getmodel}unknowndate/$location"
		mv "$REPLY" "${getmodel}unknowndate/$location"
      	fi
done

Example running in a directory with mixed media

# Raw images get moved into a RAW directory
no raw

# Samsung phone detected with date and GPS location
mkdir -p samsung/20220717/Hilversum
mv 20220717_133453.jpg samsung/20220717/Hilversum

# OnePlus phone
mkdir -p OnePlus/20021208/Voorburg
mv IMG_20190109_091825.jpg OnePlus/20021208/Voorburg

# Realme (Added country when not NL)
mkdir -p realme/20220607/Isle of Islay(GB)
mv IMG20220607213630.jpg realme/20220607/Isle of Islay(GB)

# Whatsapp has no date embedded so it gets it from filename
Trying from filename
Trying from filename - maybe without dashes
mkdir -p whatsapp/20221021/
mv IMG-20221021-WA0000.jpg whatsapp/20221021/

# Nikon without GPS
mkdir -p NIKON/20220613/
mv DSC_1423.MOV NIKON/20220613/

# Whatsapp video without exif
mkdir -p whatsapp/20170528/
mv VID-20170528-WA0006.mp4 whatsapp/20170528/

# No camera name detected in exif from mobile movie
mkdir -p 20190114/Maarssen
mv VID_20190114_142455.mp4 20190114/Maarssen

# Location in mp4
mkdir -p 20220607/Lamlash(GB)
mv VID20220607155044.mp4 20220607/Lamlash(GB)

Result

./NIKON/2022/20220613/DSC_1423.MOV

./NIKON/2022/20220610/750_1101.JPG

./realme/2022/20220818/Hilversum/IMG20220818203825.jpg

./realme/2022/20220607/Isle of Islay(GB)/IMG20220607213630.jpg

./2019/20190114/Maarssen/VID_20190114_142455.mp4

./whatsapp/2017/20170528/VID-20170528-WA0006.mp4

./whatsapp/2022/20221021/IMG-20221021-WA0000.jpg

./2022/20220607/Lamlash(GB)/VID20220607155044.mp4

./2022/20220516/Hilversum/VID20220516125913.mp4

./OnePlus/2002/20021208/Voorburg/IMG_20190109_091825.jpg

./samsung/2022/20220717/Hilversum/20220717_133453.jpg

./NIKON/2022/20220613/DSC_1423.MOV ./NIKON/2022/20220610/750_1101.JPG ./realme/2022/20220818/Hilversum/IMG20220818203825.jpg ./realme/2022/20220607/Isle of Islay(GB)/IMG20220607213630.jpg ./2019/20190114/Maarssen/VID_20190114_142455.mp4 ./whatsapp/2017/20170528/VID-20170528-WA0006.mp4 ./whatsapp/2022/20221021/IMG-20221021-WA0000.jpg ./2022/20220607/Lamlash(GB)/VID20220607155044.mp4 ./2022/20220516/Hilversum/VID20220516125913.mp4 ./OnePlus/2002/20021208/Voorburg/IMG_20190109_091825.jpg ./samsung/2022/20220717/Hilversum/20220717_133453.jpg

./NIKON/2022/20220613/DSC_1423.MOV
./NIKON/2022/20220610/750_1101.JPG
./realme/2022/20220818/Hilversum/IMG20220818203825.jpg
./realme/2022/20220607/Isle of Islay(GB)/IMG20220607213630.jpg
./2019/20190114/Maarssen/VID_20190114_142455.mp4
./whatsapp/2017/20170528/VID-20170528-WA0006.mp4
./whatsapp/2022/20221021/IMG-20221021-WA0000.jpg
./2022/20220607/Lamlash(GB)/VID20220607155044.mp4
./2022/20220516/Hilversum/VID20220516125913.mp4
./OnePlus/2002/20021208/Voorburg/IMG_20190109_091825.jpg
./samsung/2022/20220717/Hilversum/20220717_133453.jpg

You could automate this using incrond
apt-get install incron
add your user to /etc/incron.allow
incrontab -e
add

/fileserver/mediain/ IN_CREATE /usr/local/bin/sortmymedia.sh

Coping a file in the directory, auto sort and move to correct location

sortmymedia.sh

#!/bin/bash

cd /home/user/media

/usr/local/bin/exifsort.sh

#!/bin/bash cd /home/user/media /usr/local/bin/exifsort.sh

#!/bin/bash
cd /home/user/media
/usr/local/bin/exifsort.sh

Computer, Music

Automating the h*ll out of windows applications using linux

2022-11-05 fash

I’m using a windows program for typesetting bagpipe music.
Luckily this runs on Linux using wine.

Sometimes i just want a PDF version of a tune, for example for my tunebook compiler. (Other post)
Or i want to batch convert a lot of bww files.

A long time ago i used a virtual machine with automation software for this.
Why not automate the process on my Laptop?

So i made this script, with a workaround for the xdotool wine problem.
(wine window needs to be active to accept key strokes, other linux xwindows you can use the ID of the window)

#!/bin/bash

# use c for close at the end, without c bgplayer wont be shut down

# bww2pdf path/to/music.bww c

# make tmp file

cat "$1" > /tmp/deze.bww

# start bgplayer if not started .. if not started wait 3 secs

slp=0

pgrep BGPlayer.exe >/dev/null || slp=3

pgrep BGPlayer.exe >/dev/null|| ( nohup wine ~/.wine/dosdevices/c\:/Program\ Files\ $x86$/Bagpipe\ Player/BGPlayer.exe & )

sleep $slp

# get program id

pid=$(pgrep BGPlayer.exe)

# get xwindow id using pid

winid=$(xdotool search --limit 1 --all --pid $pid)

# activate window

xdotool search --desktop 0 --class "BGPlayer.exe" windowactivate

# open file menu and Open file

xdotool key --window $winid "alt+f"

xdotool key --window $winid "o"

# give program time to execute

sleep 1

# open our tmp file

xdotool type '\tmp\deze.bww'

xdotool key KP_Enter

sleep 2

# open file menu select Print and PDF as output

xdotool key "alt+f"

xdotool key "p"

xdotool key "P"

sleep 2

# execute

xdotool key KP_Enter

sleep 1

# File close

xdotool key "alt+f"

xdotool key "c"

sleep 2

# close program when c was added to commandline

mv ~/PDF/deze* "$1.pdf"

if [ "$2" == "c" ] ; then

xdotool key "alt+f"

xdotool key "x"

#!/bin/bash # use c for close at the end, without c bgplayer wont be shut down # bww2pdf path/to/music.bww c # make tmp file cat "$1" > /tmp/deze.bww # start bgplayer if not started .. if not started wait 3 secs slp=0 pgrep BGPlayer.exe >/dev/null || slp=3 pgrep BGPlayer.exe >/dev/null|| ( nohup wine ~/.wine/dosdevices/c\:/Program\ Files\ $x86$/Bagpipe\ Player/BGPlayer.exe & ) sleep $slp # get program id pid=$(pgrep BGPlayer.exe) # get xwindow id using pid winid=$(xdotool search --limit 1 --all --pid $pid) # activate window xdotool search --desktop 0 --class "BGPlayer.exe" windowactivate # open file menu and Open file xdotool key --window $winid "alt+f" xdotool key --window $winid "o" # give program time to execute sleep 1 # open our tmp file xdotool type '\tmp\deze.bww' xdotool key KP_Enter sleep 2 # open file menu select Print and PDF as output xdotool key "alt+f" xdotool key "p" xdotool key "P" sleep 2 # execute xdotool key KP_Enter sleep 1 # File close xdotool key "alt+f" xdotool key "c" sleep 2 # close program when c was added to commandline mv ~/PDF/deze* "$1.pdf" if [ "$2" == "c" ] ; then xdotool key "alt+f" xdotool key "x" fi

#!/bin/bash
# use c for close at the end, without c bgplayer wont be shut down
# bww2pdf path/to/music.bww c
# make tmp file
cat "$1" > /tmp/deze.bww
# start bgplayer if not started .. if not started wait 3 secs
slp=0
pgrep BGPlayer.exe >/dev/null || slp=3
pgrep BGPlayer.exe  >/dev/null|| ( nohup wine ~/.wine/dosdevices/c\:/Program\ Files\ \(x86\)/Bagpipe\ Player/BGPlayer.exe & )
sleep $slp
# get program id
pid=$(pgrep BGPlayer.exe)
# get xwindow id using pid
winid=$(xdotool search --limit 1 --all --pid $pid)
# activate window
xdotool search --desktop 0 --class "BGPlayer.exe" windowactivate
# open file menu and Open file
xdotool key --window $winid "alt+f"
xdotool key --window $winid "o"
# give program time to execute
sleep 1
# open our tmp file
xdotool type '\tmp\deze.bww'
xdotool key KP_Enter
sleep 2
# open file menu select Print and PDF as output
xdotool key "alt+f"
xdotool key "p"
xdotool key "P"
sleep 2
# execute
xdotool key KP_Enter
sleep 1
# File close
xdotool key "alt+f"
xdotool key "c"
sleep 2
# close program when c was added to commandline
mv ~/PDF/deze* "$1.pdf"
if [ "$2" == "c" ] ; then 
	xdotool key "alt+f"
	xdotool key "x"
fi

I’m not touching the keyboard when the program is running, all handled by the script

Computer

Motion detection for stupid webcams

2022-10-01 fash

Last year i made a script for a friend who wanted to detect visually if his garden sprinkler was on or off. A few days ago i saw someone who wanted to see if things where moving in his house. (didn’t trust his landlord i think)
But he only had a dumb/simple/cheap camera .. so it had no motion detection.

I was thinking of my script, and could easily adapt it for this usage.

Most ipcams have somekind of URL/API you can use to capture a image.
Some examples below

# Reolink

wget "http://ipnumber/cgi-bin/api.cgi?cmd=Snap&channel=0&rs=randomstring&user=user&password=password" -O image.jpg

# Foscam

wget "http://ipnumber:88/CGIProxy.fcgi?cmd=snapPicture2&usr=user&pwd=password&cnt=randomstring" -O image.jpg

# Edimax

wget "http://user:password@ipnumber/jpg/image.jpg" -O image.jpg

# Hikvision

wget "http://user:password@ipnumber/ISAPI/Streaming/channels/101/picture" -O image.jpg

# Reolink wget "http://ipnumber/cgi-bin/api.cgi?cmd=Snap&channel=0&rs=randomstring&user=user&password=password" -O image.jpg # Foscam wget "http://ipnumber:88/CGIProxy.fcgi?cmd=snapPicture2&usr=user&pwd=password&cnt=randomstring" -O image.jpg # Edimax wget "http://user:password@ipnumber/jpg/image.jpg" -O image.jpg # Hikvision wget "http://user:password@ipnumber/ISAPI/Streaming/channels/101/picture" -O image.jpg

# Reolink
wget "http://ipnumber/cgi-bin/api.cgi?cmd=Snap&channel=0&rs=randomstring&user=user&password=password" -O image.jpg

# Foscam
wget "http://ipnumber:88/CGIProxy.fcgi?cmd=snapPicture2&usr=user&pwd=password&cnt=randomstring" -O image.jpg

# Edimax
wget "http://user:password@ipnumber/jpg/image.jpg" -O image.jpg

# Hikvision
wget "http://user:password@ipnumber/ISAPI/Streaming/channels/101/picture" -O image.jpg

So using below script i can capture a image, compare it to the previous, and when it’s above a certain threshold sends a email.

#!/bin/bash

# Only uses wget and image-magick

treshhold=500

fuzzyness=20%

# CHANGE WEBCAM THINGY TO OWN URL AND CREDENTIALS

wget -q "http://webcamip/cgi-bin/api.cgi?cmd=Snap&channel=0&user=user&password=password" -O previous.jpg

while true; do

wget -q "http://webcamip/cgi-bin/api.cgi?cmd=Snap&channel=0&user=user&password=password" -O current.jpg

value=$(compare -fuzz $fuzzyness previous.jpg current.jpg -metric mae diff.jpg 2>&1 | cut -f1 -d.)

if [ $value -gt $treshhold ] ; then

echo "ping $treshhold"

echo "Something moved" | mail -s "Movement" user@example.com -A diff.jpg

# Comment below if you want to compare against a base line .. not previous image

cat current.jpg > previous.jpg

sleep 60

done

#!/bin/bash # Only uses wget and image-magick treshhold=500 fuzzyness=20% # CHANGE WEBCAM THINGY TO OWN URL AND CREDENTIALS wget -q "http://webcamip/cgi-bin/api.cgi?cmd=Snap&channel=0&user=user&password=password" -O previous.jpg while true; do wget -q "http://webcamip/cgi-bin/api.cgi?cmd=Snap&channel=0&user=user&password=password" -O current.jpg value=$(compare -fuzz $fuzzyness previous.jpg current.jpg -metric mae diff.jpg 2>&1 | cut -f1 -d.) if [ $value -gt $treshhold ] ; then echo "ping $treshhold" echo "Something moved" | mail -s "Movement" user@example.com -A diff.jpg fi # Comment below if you want to compare against a base line .. not previous image cat current.jpg > previous.jpg sleep 60 done

#!/bin/bash
# Only uses wget and image-magick
treshhold=500
fuzzyness=20%
# CHANGE WEBCAM THINGY TO OWN URL AND CREDENTIALS
wget -q "http://webcamip/cgi-bin/api.cgi?cmd=Snap&channel=0&user=user&password=password" -O previous.jpg
while true;  do
wget -q "http://webcamip/cgi-bin/api.cgi?cmd=Snap&channel=0&user=user&password=password" -O current.jpg
value=$(compare -fuzz $fuzzyness previous.jpg current.jpg -metric mae diff.jpg 2>&1 | cut -f1 -d.)
if [ $value -gt $treshhold ] ; then
echo "ping $treshhold"
echo "Something moved" | mail -s "Movement" user@example.com -A diff.jpg
fi
# Comment below if you want to compare against a base line .. not previous image
cat current.jpg > previous.jpg
sleep 60
done

Example previous picture

Example current picture

I got mailed with result

Hints tips:

Use crop to detect only a part.

copy current.jpg to a second file
Use painting black a part and compair with different treshhold fuzzyness to get different hotspots.

Below detects RED, use above ide with crop to detect red/green/blue leds

compare -verbose -metric mae 1.jpg 2.jpg /tmp/1.diff
1.jpg JPEG 2560x1920 2560x1920+0+0 8-bit sRGB 248819B 0.050u 0:00.057
2.jpg JPEG 2560x1920 2560x1920+0+0 8-bit sRGB 248949B 0.030u 0:00.137
Image: 1.jpg
  Channel distortion: MAE
  Channel distortion: MAE
    red: 12517.5 (0.191005)
    green: 11967.1 (0.182607)
    blue: 12492.8 (0.190628)
    all: 12325.8 (0.18808)
1.jpg=>/tmp/1.diff JPEG 2560x1920 2560x1920+0+0 8-bit sRGB 1.19495MiB 1.470u 0:00.197

Computer, IOT / Domoticz

Energy bill and ledserver

2022-09-26 fash

Having a lot of devices and running a Lab wil use a lot of energy. Now with the energy crisis in Europe, i had to take a closer look at whats using power in my house.

I notished some weird usage patterns while measuring.

I’m using a few shelly power plugs, to measure devices and powerstrips.

With these devices you can control devices connected to it.
On/Off/Timer etcetera.
It wil measure the power usage in watts, and it even got a temperature sensor.
I like the fact that it perfectly integrates into your home automation using an extensive API.
curl commands to controll, and even MQTT messaging. Intergrating in Home Assistant is a breeze.

So i was monitoring a bunch of stuff using Nodered/Grafana/Homeassistant and saw some recurring usage.
But being always late to check things, i made use of my ledserver i’ve build a long time ago.

This ledserver consists of a Raspberry Pi Zero, with a led string and a API written in python.

Below is autostarted on the Raspberry

( I made this ledserver for work, it showed the status of servers and services. Beside that every colleage had a range which he could use for his own scripts. I made some little bash script templates to have led funtions standard in your bash profile.

#!/usr/bin/python

# apt-get install python-flask

import Adafruit_WS2801

import Adafruit_GPIO.SPI as SPI

import struct

from flask import Flask, render_template, request

app = Flask(__name__)

PIXEL_COUNT = 32

SPI_PORT = 0

SPI_DEVICE = 0

pixels = Adafruit_WS2801.WS2801Pixels(PIXEL_COUNT, spi=SPI.SpiDev(SPI_PORT, SPI_DEVICE))

pixels.clear()

pixels.show()

@app.route("/led/<deviceName>/<color>")

def action(deviceName, color):

if deviceName == 'reset':

print ("reset")

pixels.clear()

print (deviceName)

led = int(deviceName)

s = color

r = int(s[ :2], 16)

b = int(s[2:4], 16)

g = int(s[4: ], 16)

pixels.set_pixel_rgb(led, r,g,b)

pixels.show()

templateData = {

'rled' : r,

'bled' : b,

'gled' : g,

'deviceName' : deviceName,

}

return render_template('index.html', **templateData)

@app.route("/control/<controlcommand>")

def actioncommand(controlcommand):

if controlcommand == 'clear':

print("clear")

pixels.clear()

pixels.show()

templateData = {

'controlcommand' : controlcommand,

}

return render_template('index.html', **templateData)

@app.route("/range/<start>/<stop>/<color>")

def rangecommand(start,stop,color):

s = color

r = int(s[ :2], 16)

b = int(s[2:4], 16)

g = int(s[4: ], 16)

startled = int(start)

stopled = int(stop)

while (startled < stopled):

pixels.set_pixel_rgb(startled, r,g,b)

startled=startled + 1

pixels.show()

templateData = {

'rangecommand' : rangecommand,

}

return render_template('index.html', **templateData)

if __name__ == "__main__":

app.run(host='0.0.0.0', port=8080, debug=True)

#!/usr/bin/python # apt-get install python-flask # import Adafruit_WS2801 import Adafruit_GPIO.SPI as SPI import struct from flask import Flask, render_template, request app = Flask(__name__) PIXEL_COUNT = 32 SPI_PORT = 0 SPI_DEVICE = 0 pixels = Adafruit_WS2801.WS2801Pixels(PIXEL_COUNT, spi=SPI.SpiDev(SPI_PORT, SPI_DEVICE)) pixels.clear() pixels.show() @app.route("/led/<deviceName>/<color>") def action(deviceName, color): if deviceName == 'reset': print ("reset") pixels.clear() print (deviceName) led = int(deviceName) s = color r = int(s[ :2], 16) b = int(s[2:4], 16) g = int(s[4: ], 16) pixels.set_pixel_rgb(led, r,g,b) pixels.show() templateData = { 'rled' : r, 'bled' : b, 'gled' : g, 'deviceName' : deviceName, } return render_template('index.html', **templateData) @app.route("/control/<controlcommand>") def actioncommand(controlcommand): if controlcommand == 'clear': print("clear") pixels.clear() pixels.show() templateData = { 'controlcommand' : controlcommand, } return render_template('index.html', **templateData) @app.route("/range/<start>/<stop>/<color>") def rangecommand(start,stop,color): s = color r = int(s[ :2], 16) b = int(s[2:4], 16) g = int(s[4: ], 16) startled = int(start) stopled = int(stop) while (startled < stopled): pixels.set_pixel_rgb(startled, r,g,b) startled=startled + 1 pixels.show() templateData = { 'rangecommand' : rangecommand, } return render_template('index.html', **templateData) if __name__ == "__main__": app.run(host='0.0.0.0', port=8080, debug=True)

#!/usr/bin/python


# apt-get install python-flask
#

import Adafruit_WS2801
import Adafruit_GPIO.SPI as SPI
import struct

from flask import Flask, render_template, request
app = Flask(__name__)

PIXEL_COUNT = 32
SPI_PORT   = 0
SPI_DEVICE = 0

pixels = Adafruit_WS2801.WS2801Pixels(PIXEL_COUNT, spi=SPI.SpiDev(SPI_PORT, SPI_DEVICE))

pixels.clear()
pixels.show()

@app.route("/led/<deviceName>/<color>")
def action(deviceName, color):
        if deviceName == 'reset':
                print ("reset")
                pixels.clear()
        print (deviceName)
        led = int(deviceName)
        s = color
        r = int(s[ :2], 16)
        b = int(s[2:4], 16)
        g = int(s[4: ], 16)

        pixels.set_pixel_rgb(led, r,g,b)
        pixels.show()

        templateData = {
                'rled'  : r,
                'bled'  : b,
                'gled'  : g,
                'deviceName'  : deviceName,
        }
        return render_template('index.html', **templateData)

@app.route("/control/<controlcommand>")
def actioncommand(controlcommand):
        if controlcommand == 'clear':

                print("clear")
                pixels.clear()
                pixels.show()

        templateData = {
                'controlcommand'  : controlcommand,
        }
        return render_template('index.html', **templateData)

@app.route("/range/<start>/<stop>/<color>")
def rangecommand(start,stop,color):
        s = color
        r = int(s[ :2], 16)
        b = int(s[2:4], 16)
        g = int(s[4: ], 16)
        startled = int(start)
        stopled = int(stop)
        while (startled < stopled):
          pixels.set_pixel_rgb(startled, r,g,b)
          startled=startled + 1
        pixels.show()

        templateData = {
                'rangecommand'  : rangecommand,
        }


        return render_template('index.html', **templateData)



if __name__ == "__main__":
   app.run(host='0.0.0.0', port=8080, debug=True)

Now you can control the leds with a simple curl command:

curl http://ledserver:8080/range/startled/endled/colorinrgb
curl http://ledserver:8080/led/lednumber/colorinrgb
curl http://ledserver:8080/control/clear

So today i made a little script to show power usage.

I’m reading the current power usage from a LS120 Youless

Youless LS120 device, which you can connect to your P1 connector.

With below bash script i’m reading the webinterface and update the ledstring.
I was using this ledserver for general notification usage. Below a 2 minute hack ..

#!/bin/bash

while true; do

number=$(echo $(curl -s http://youlessip/a | grep Watt | head -1 | awk '{ print $1 }') / 100 | bc)

curl -s http://ledserver:8080/control/clear

curl -s http://ledserver:8080/range/00/$number/010101

sleep 10

done

#!/bin/bash while true; do number=$(echo $(curl -s http://youlessip/a | grep Watt | head -1 | awk '{ print $1 }') / 100 | bc) curl -s http://ledserver:8080/control/clear curl -s http://ledserver:8080/range/00/$number/010101 sleep 10 done

#!/bin/bash
while true; do
number=$(echo $(curl -s http://youlessip/a | grep Watt | head -1 | awk '{ print $1 }') / 100  | bc)
curl -s http://ledserver:8080/control/clear
curl -s http://ledserver:8080/range/00/$number/010101
sleep 10
done

Computer, Photography

Challenged myself into programming a photo quiz

2022-09-22 fash

Got bored, and challenged myself …

How about a quess the picture from our photo collection??
(123580 photos .. )
So i show a random picture, and when i press ESC it will show some information about the picture …
Quess the year and the event

Well i gave myself 15 minutes to program something ..

I was watching a tv show meanwhile .. but i managed to come up with this …

This script is showing a picture, when you press ESC it wil show some details.After that it will select another random picture.

Improvements : reading tags and other metadata from my photo database, to give more information.

#!/bin/bash

while true ; do

shuf -n 1 allpix > /tmp/randompix

pix=$(cat /tmp/randompix | cut -f2 -d\> | cut -f1 -d\<)

dir=$(cat /tmp/randompix | cut -f4 -d\> | cut -f1 -d\< |cut -f3 -d\; | sed 's#\\#/#g')

displaypath=/mnt/${dir}/${pix}

info=$(cat /tmp/randompix | rev | cut -f-4 -d\\ | rev | cut -f1 -d \<)

convert -background black -fill white -size 1920x1080 -pointsize 24 label:"$info" info.jpg

pqiv -i --fullscreen "$displaypath"

pqiv -i --fullscreen info.jpg

done

#!/bin/bash while true ; do shuf -n 1 allpix > /tmp/randompix pix=$(cat /tmp/randompix | cut -f2 -d\> | cut -f1 -d\<) dir=$(cat /tmp/randompix | cut -f4 -d\> | cut -f1 -d\< |cut -f3 -d\; | sed 's#\\#/#g') displaypath=/mnt/${dir}/${pix} info=$(cat /tmp/randompix | rev | cut -f-4 -d\\ | rev | cut -f1 -d \<) convert -background black -fill white -size 1920x1080 -pointsize 24 label:"$info" info.jpg pqiv -i --fullscreen "$displaypath" pqiv -i --fullscreen info.jpg done

#!/bin/bash
while true ; do
shuf -n 1 allpix > /tmp/randompix
pix=$(cat /tmp/randompix | cut -f2 -d\> | cut -f1 -d\<)
dir=$(cat /tmp/randompix | cut -f4 -d\> | cut -f1 -d\< |cut -f3 -d\; | sed 's#\\#/#g')
displaypath=/mnt/${dir}/${pix}
info=$(cat /tmp/randompix | rev | cut -f-4 -d\\ | rev | cut -f1 -d \<)
convert -background black -fill white  -size 1920x1080 -pointsize 24 label:"$info" info.jpg
pqiv -i --fullscreen "$displaypath"
pqiv -i --fullscreen info.jpg
done

Which gave me … this

Welllll .. it toke 20 minutes .. so i lost
(Must have been the wine)

IOT / Domoticz

Homewizard Watermeter

2022-09-21 fash

Today i got my watermeter in.

It’s a little device you can place on your watermeter.

You get a plastic clip/holder which you can use to place the device on your watermeter. You can easily remove the device to read the values.

The device measures the little round gauge, so it has no idea what the current values are. Add the current values of your meter to the output of the device.

The device works by measuring the rotation of the red part.

By default it sends the information over the internet to a server. Your phone connects to this server and the app wil give you the graphs.

If you want your own intergration, you have to enable “local api”
See image on the right.
When you want realtime data, you have to connect a usb-c power supply.
When using batteries, the device wil only connect to wifi once per 5 minutes, and you can’t use the API.

I wrote a little test script in bash to draw graphs using mqtt and Nodered.

#!/bin/bash

calib=29.621

data=$(curl -s http://10.1.1.176/api/v1/data)

totalwater=$(echo $data | cut -f9 -d\" | tr -d ':,')

currentwater=$(echo $data | cut -f11 -d\" | tr -d ':,}')

totalwater=$( echo $totalwater + $calib | bc)

mosquitto_pub -h 127.0.0.1 -m "$totalwater" -t watermeter/total

mosquitto_pub -h 127.0.0.1 -m "$currentwater" -t watermeter/current

#!/bin/bash calib=29.621 data=$(curl -s http://10.1.1.176/api/v1/data) totalwater=$(echo $data | cut -f9 -d\" | tr -d ':,') currentwater=$(echo $data | cut -f11 -d\" | tr -d ':,}') totalwater=$( echo $totalwater + $calib | bc) mosquitto_pub -h 127.0.0.1 -m "$totalwater" -t watermeter/total mosquitto_pub -h 127.0.0.1 -m "$currentwater" -t watermeter/current

#!/bin/bash
calib=29.621
data=$(curl -s http://10.1.1.176/api/v1/data)
totalwater=$(echo $data | cut -f9 -d\" | tr -d ':,')
currentwater=$(echo $data | cut -f11 -d\" | tr -d ':,}')
totalwater=$( echo $totalwater + $calib | bc)
mosquitto_pub -h 127.0.0.1 -m "$totalwater" -t watermeter/total
mosquitto_pub -h 127.0.0.1 -m "$currentwater" -t watermeter/current

Variable calib is what my watermeter was when i attached the device.
It wil add this to the newly measured values.

Home Assistant found the device using autodiscovery

I have to look into adjusting for offset in HA

Music

Spotify export

2022-09-13 fash

Nice .. you can request a data export from spotify, much like google’s takeout.

I wil export your streaming history, playlists, library and more.

Nice to generate some statistics from this data.

For now the top 50 played artists (nr of times, artist)

790 "Rammstein"

507 "Fred Morrison"

478 "Lincoln Hilton"

437 "Avicii"

420 "Shooglenifty"

347 "Treacherous Orchestra"

323 "Battlefield Band"

321 "Breabach"

295 "Nightwish"

254 "The Fraser Shaw Trust"

236 "Michael McGoldrick"

207 "Peatbog Faeries"

207 "Calum Stewart"

203 "Lúnasa"

197 "Martyn Bennett"

194 "Jeff Wayne"

193 "The Prodigy"

173 "Kíla"

172 "Ross Ainslie and Jarlath Henderson"

170 "Buena Vista Social Club"

153 "Ross Ainslie"

150 "Dàimh"

138 "Bliss"

125 "Hans Zimmer"

124 "Rare Air"

118 "Michael McCann"

107 "Kyle Tuttle"

107 "Beinn Lee"

105 "Bourne & Macleod"

89 "Wolfstone"

88 "Ímar"

83 "Afro Celt Sound System"

81 "Gordon Duncan"

81 "Armin van Buuren"

80 "Face The West"

79 "Tyurgen Kam"

79 "Aly Bain"

72 "Keltik Elektrik"

71 "Duncan Chisholm"

66 "Liz Carroll"

63 "Project Smok"

63 "Blazin' Fiddles"

62 "Dagda"

61 "Trail West"

59 "Julian Winding"

57 "Solar Fields"

57 "Dougie McCance"

56 "John McSherry"

56 "AES DANA"

52 "Gaelic Storm"

790 "Rammstein" 507 "Fred Morrison" 478 "Lincoln Hilton" 437 "Avicii" 420 "Shooglenifty" 347 "Treacherous Orchestra" 323 "Battlefield Band" 321 "Breabach" 295 "Nightwish" 254 "The Fraser Shaw Trust" 236 "Michael McGoldrick" 207 "Peatbog Faeries" 207 "Calum Stewart" 203 "Lúnasa" 197 "Martyn Bennett" 194 "Jeff Wayne" 193 "The Prodigy" 173 "Kíla" 172 "Ross Ainslie and Jarlath Henderson" 170 "Buena Vista Social Club" 153 "Ross Ainslie" 150 "Dàimh" 138 "Bliss" 125 "Hans Zimmer" 124 "Rare Air" 118 "Michael McCann" 107 "Kyle Tuttle" 107 "Beinn Lee" 105 "Bourne & Macleod" 89 "Wolfstone" 88 "Ímar" 83 "Afro Celt Sound System" 81 "Gordon Duncan" 81 "Armin van Buuren" 80 "Face The West" 79 "Tyurgen Kam" 79 "Aly Bain" 72 "Keltik Elektrik" 71 "Duncan Chisholm" 66 "Liz Carroll" 63 "Project Smok" 63 "Blazin' Fiddles" 62 "Dagda" 61 "Trail West" 59 "Julian Winding" 57 "Solar Fields" 57 "Dougie McCance" 56 "John McSherry" 56 "AES DANA" 52 "Gaelic Storm"

790     "Rammstein"
507     "Fred Morrison"
478     "Lincoln Hilton"
437     "Avicii"
420     "Shooglenifty"
347     "Treacherous Orchestra"
323     "Battlefield Band"
321     "Breabach"
295     "Nightwish"
254     "The Fraser Shaw Trust"
236     "Michael McGoldrick"
207     "Peatbog Faeries"
207     "Calum Stewart"
203     "Lúnasa"
197     "Martyn Bennett"
194     "Jeff Wayne"
193     "The Prodigy"
173     "Kíla"
172     "Ross Ainslie and Jarlath Henderson"
170     "Buena Vista Social Club"
153     "Ross Ainslie"
150     "Dàimh"
138     "Bliss"
125     "Hans Zimmer"
124     "Rare Air"
118     "Michael McCann"
107     "Kyle Tuttle"
107     "Beinn Lee"
105     "Bourne & Macleod"
 89     "Wolfstone"
 88     "Ímar"
 83     "Afro Celt Sound System"
 81     "Gordon Duncan"
 81     "Armin van Buuren"
 80     "Face The West"
 79     "Tyurgen Kam"
 79     "Aly Bain"
 72     "Keltik Elektrik"
 71     "Duncan Chisholm"
 66     "Liz Carroll"
 63     "Project Smok"
 63     "Blazin' Fiddles"
 62     "Dagda"
 61     "Trail West"
 59     "Julian Winding"
 57     "Solar Fields"
 57     "Dougie McCance"
 56     "John McSherry"
 56     "AES DANA"
 52     "Gaelic Storm"

Get top 50 script

 cat  *json | grep artistName  | sort | uniq -c | sort -nr | head -50 | cut -f1,4,5 -d\" | sed s/,$//g

Computer

Collection of useful Linux scripts/commands

2022-08-05 fash

Bash tips part 1

Sorting my fileserver i found a lot of Bash/Linux scripts maybe useful.

Some are tool usage only, maybe I’ll remove this page.
But this being my own log/notebook, who knows

Port knocker .. opens port 22 when you send a tcp packet to port 1600 first. Note: when portscanning it wil close again when accessing 1601 or 1599. Note2: NOT active on my server .. duh!

(Open with a browser or telnet/netcat)

iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -m recent --rcheck --name SSH -j ACCEPT

iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 1599 -m recent --name SSH --remove -j DROP

iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 1600 -m recent --name SSH --set -j DROP

iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 1601 -m recent --name SSH --remove -j DROP

iptables -A INPUT -m tcp -p tcp --dport 22 -j DROP

iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -m recent --rcheck --name SSH -j ACCEPT iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 1599 -m recent --name SSH --remove -j DROP iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 1600 -m recent --name SSH --set -j DROP iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 1601 -m recent --name SSH --remove -j DROP iptables -A INPUT -m tcp -p tcp --dport 22 -j DROP

iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -m recent --rcheck --name SSH -j ACCEPT
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 1599 -m recent --name SSH --remove -j DROP
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 1600 -m recent --name SSH --set -j DROP
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 1601 -m recent --name SSH --remove -j DROP
iptables -A INPUT -m tcp -p tcp --dport 22 -j DROP

Dump and share your log, you get a short link to share

cat /var/log/Xorg.0.log | nc termbin.com 9999

cat /var/log/Xorg.0.log | nc termbin.com 9999

Multiboot? This will reboot into another OS

#!/bin/bash

WINDOWS_TITLE=`grep -i "^menuentry 'Windows" /boot/grub/grub.

grub-reboot "$WINDOWS_TITLE"

reboot

#!/bin/bash WINDOWS_TITLE=`grep -i "^menuentry 'Windows" /boot/grub/grub. grub-reboot "$WINDOWS_TITLE" reboot

#!/bin/bash
WINDOWS_TITLE=`grep -i "^menuentry 'Windows" /boot/grub/grub.
grub-reboot "$WINDOWS_TITLE"
reboot

Unreadable json?

cat file | python3 -m json.tool

cat file | python3 -m json.tool

screen stuff

Start in screen

Start screen in "detached" mode. This creates a new session but doesn't attach to it. This is useful for system startup scripts.

/usr/bin/screen -d -m -S backup /usr/local/bin/backup.sh

screen -r backup

Screensize

ctrl-a :fix # Screensize fix

Start in screen Start screen in "detached" mode. This creates a new session but doesn't attach to it. This is useful for system startup scripts. /usr/bin/screen -d -m -S backup /usr/local/bin/backup.sh screen -r backup Screensize ctrl-a :fix # Screensize fix

Start in screen
Start screen in "detached" mode. This creates a new session but doesn't attach to it. This is useful for system startup scripts.

/usr/bin/screen -d -m -S backup /usr/local/bin/backup.sh
screen -r backup
Screensize
ctrl-a :fix # Screensize fix

Memory leak check

pmap <pid> | tail -1

Memory leak .. pmap opslaan en vergelijken met een pmap later.

pmap <pid> | tail -1 Memory leak .. pmap opslaan en vergelijken met een pmap later.

pmap <pid> | tail -1
Memory leak .. pmap opslaan en vergelijken met een pmap later.

Listen to remote microphone local playback

ssh -C monitor@remoteserver arecord -f dat | aplay -f dat

ssh -C monitor@remoteserver arecord -f dat | aplay -f dat

Root login to named account

ssh-copy-id root@remote-server (one time only)

then:

ssh root@remoteserver

Lookup named account

cat /var/log/secure | grep publickey | cut -f6 -d: | while read ; do ssh-keygen -lf ~/.ssh/authorized_keys | grep $REPLY ;done

2048 SHA256:l85g/NvPnEy85UVfJ5LJw3NvPnEy85UVfJ5LJw3NvPnEy85UVfJ5LJw3 haanstoot@minka (RSA)

ssh-copy-id root@remote-server (one time only) then: ssh root@remoteserver Lookup named account cat /var/log/secure | grep publickey | cut -f6 -d: | while read ; do ssh-keygen -lf ~/.ssh/authorized_keys | grep $REPLY ;done 2048 SHA256:l85g/NvPnEy85UVfJ5LJw3NvPnEy85UVfJ5LJw3NvPnEy85UVfJ5LJw3 haanstoot@minka (RSA)

ssh-copy-id root@remote-server (one time only)

then:

ssh root@remoteserver

Lookup named account

cat /var/log/secure | grep publickey | cut -f6 -d: | while read ; do ssh-keygen -lf ~/.ssh/authorized_keys | grep $REPLY ;done
2048 SHA256:l85g/NvPnEy85UVfJ5LJw3NvPnEy85UVfJ5LJw3NvPnEy85UVfJ5LJw3 haanstoot@minka (RSA)

Get all VMs from RHEV/Ovirt

curl -X GET -H "Accept: application/xml" -u username@domain:PaSSwoRd --cacert rhevm.cer https://rhvserver:443/api/vms | grep -i "<name>" | grep -v " " | cut -f2 -d\> | cut -f1 -d\<

curl -X GET -H "Accept: application/xml" -u username@domain:PaSSwoRd --cacert rhevm.cer https://rhvserver:443/api/vms  | grep -i "<name>" | grep -v "           " | cut -f2 -d\> | cut -f1 -d\<

Get all disks from RHV

log into ovirt engine and postgres

psql -d engine -U postgres -c 'select vm_names,disk_profile_name,storage_id,disk_id,image_guid,disk_alias from public.all_disks;'

log into ovirt engine and postgres psql -d engine -U postgres -c 'select vm_names,disk_profile_name,storage_id,disk_id,image_guid,disk_alias from public.all_disks;'

log into ovirt engine and postgres
psql -d engine -U postgres -c 'select vm_names,disk_profile_name,storage_id,disk_id,image_guid,disk_alias from public.all_disks;'

Gluster heal info

ls /tmp/*out | while read ;do cat $REPLY | grep entries | grep -v "Number of entries: 0" >/dev/null && echo $REPLY ;done | while read heal ; do cat $heal ;done | tac |sed -e '/Number of entries: 0/I,+1 d' | tac | uniq

gluster volume info | egrep "Volume Name|Type" | grep -B1 Replicate | grep "Volume Name" | cut -c14- | while read ;do gluster volume heal $REPLY info > /tmp/$REPLY.out ; done ls /tmp/*out | while read ;do cat $REPLY | grep entries | grep -v "Number of entries: 0" >/dev/null && echo $REPLY ;done | while read heal ; do cat $heal ;done | tac |sed -e '/Number of entries: 0/I,+1 d' | tac | uniq

gluster volume info | egrep "Volume Name|Type"  | grep -B1 Replicate | grep "Volume Name" | cut -c14- | while read ;do gluster volume heal $REPLY info > /tmp/$REPLY.out ; done
ls /tmp/*out  | while read ;do cat $REPLY | grep entries | grep -v "Number of entries: 0" >/dev/null && echo $REPLY ;done | while read heal ; do cat $heal ;done | tac |sed -e '/Number of entries: 0/I,+1 d' | tac | uniq

Some work tips i’ve posted

Vim comments

toevoegen

ga op de eerste regel van de te commenten regel staan

ctrl-v (visual)

naar beneden tot eind regel (met cursor of pagedown)

ctrl-i

esc (kan even duren)

verwijderen

ga op de eerste regel van de te uncommenten regel staan

ctrl-v (visual)

naar beneden tot eind regel (met cursor of pagedown)

esc

Als user config geedit en geen schrijfrechten?

:!bash

chmod 666 bestand

:wq

chmod bestand naar wat hij was

Input / verwerk door bash vanuit vi

:0,5 !sort # eerste 5 regels door sort heen halen

:r !date # datum in je tekst document

Vim tabs / multiple files

vi /etc/hosts /etc/services

:prev

VIM7 tabs

vim -p /etc/hosts /etc/profile

:tabe /etc/services

:tabn

:tabp

Key mappen

:map <F7> :tabp <CR>

:map <F8> :tabn <CR>

zet deze in je $HOME/.vimrc ... zonder de eerste ":"

Vi redirect

:r date # datum in text

:1,5!sort -n # sorteren van eerste 5 regels met een extern commando (kan ook met shift v een visual gedeelte selecteren)

:0,$!cut -f1 -d: # alleen field 1 van text overhouden (delimiter :)

Vi inspringen bij loop code

selecteer met shift-v de regels die moeten inspringen.

daarna > of < gebruiken om te tabben

Speciale charakters met vi bekijken

:set list

een tab zier er dan uit als ^I

Bash

sudo vergeten in commando

service apache restart

sudo !!

Bash truck vorige commando

doet vorig commando met aangepast keyword

systemctl status ovirt-ha-broker.service

^status^start

{} expand

directory's archief30_tmp t/m archief35_tmp recursive directories chmod 2775 zetten

find achief{30..35}_tmp -type d -exec chmod 2775 {} \;

mkdir tmp{1..3}

echo pr{ut,utser}s

1 regel uit een script starten zonder copy-paste (bijvoorbeeld in een remote-console sessie)

grep ipa command uit cobber.ks en voor deze uit

( /usr/sbin/ipa-client-install --domain=mgtdomain --enable-dns-updates -w password --realm=domain --server=server.domain --hostname=hypervisor.domain --unattended --force-ntpd )

cat cobbler.ks | grep ipa | bash

Bash karakters omdraaien

Linkerhand rechterhand coordinatie probleem smile

upadte

ga op de d staan en druk ctrl-t

Bash laatste woord vorige regel

ls -latrd /var/data/extra/backup

chmod 775 <ESC(punt)>

Ssh forward bij een running sessie

[haanstoot@xxx202 ~]$

~C (tilde C)

-L8080:localhost:80 (redirect localhost port 80 naar eigen machine poort 8080

-D9999 (Dynamic socks forward, zie andere post in deze wiki)

CTRL-C werkt niet?

ctrl\

SSH .ssh/config

KeepAlive yes <---------- dont die on me

ServerAliveInterval 60 <---------- dont die on me

Host *.domain <-- voor alle hosts in radlan fast login en geen vraag over key

StrictHostKeyChecking no

UserKnownHostsFile=/dev/null

host *.domain <-- domain is altijd mijn user

user haanstoot

GSSAPIAuthentication no <-- speedup

host pruts*.domain

user pi

Parallel tasks

4 parallel jobs

find jpg -type f -name \*.jpg -print0 | xargs -0 -n1 -P4 ./convert.sh

clusterssh truck

clusterssh alle svgs <--- naar alle svgs

sudo su -

cd /bricks/*/store <--- cd naar deze als bestaat

cd /rhgs/brick0*/store <-- cd naar deze als bestaat (nu sta je in atelier directories als deze bestaat als root in de svgs)

pwd | grep store || logout <--- geen atelier dir waar je staat? dan logout

id | grep haanstoot && logout <-- net logout? dus eigen user ... dan logout

eindresultaat ... als root in atelier volumes op svgs waar ze bestaan

Start in screen

/usr/bin/screen -d -m -S backup /usr/local/bin/backup.sh

screen -r backup

Start screen in "detached" mode. This creates a new session but doesn't attach to it. This is useful for system startup scripts.

Sudo vergeten

systemctl restart httpd (wil niet als user)

sudo !!

(doet "sudo systemctl restart httpd")

Reuse arguments

Bijvoorbeeld:

# touch file1 file2 file3 file4

# chmod 777 !*

Voert uit: chmod 777 file1 file2 file3 file4

5 meeste disk gebruikers human readable

du -hsx * | sort -rh | head -5

rm groot bestand duurt lang

: > /mappen2/archieven/test.img

Check memory bankjes

sudo dmidecode| grep -i -B1 "Form Factor" (B1 betekend BEFORE 1 regel meenemen, A is AFTER)

Size: 8192 MB

Form Factor: DIMM

Size: 8192 MB

Form Factor: DIMM

Size: No Module Installed

Form Factor: DIMM

Set time and restart NTP service

sudo service ntpd stop && sudo ntpdate ntp1.domain && sudo ntpdate ntp2.domain && sudo service ntpd start

Aliases en terminal kleuren

In een terminal f*ckedup colors?

komt door de alias

alias ls='ls --color'

deze even uitzetten / niet gebruiken?

\ls

Werkt voor alle aliassen

Snelle delete

Niet onderstaande gebruiken voor grote dirs

rm -rf /path/bla

Maar

mkdir /tmp/leeg ; rsync -a --delete /tmp/leeg /path/bla

Bash best practices

BashBestPractices

Script log replay

script --timing=/tmp/time.txt /tmp/script.log

scriptreplay -t /tmp/time.txt /tmp/script.log

VI Vim comments toevoegen ga op de eerste regel van de te commenten regel staan ctrl-v (visual) naar beneden tot eind regel (met cursor of pagedown) ctrl-i # esc (kan even duren) verwijderen ga op de eerste regel van de te uncommenten regel staan ctrl-v (visual) naar beneden tot eind regel (met cursor of pagedown) x esc Als user config geedit en geen schrijfrechten? :!bash chmod 666 bestand :wq chmod bestand naar wat hij was Input / verwerk door bash vanuit vi :0,5 !sort # eerste 5 regels door sort heen halen :r !date # datum in je tekst document Vim tabs / multiple files vi /etc/hosts /etc/services :n :prev VIM7 tabs vim -p /etc/hosts /etc/profile :tabe /etc/services :tabn :tabp Key mappen :map <F7> :tabp <CR> :map <F8> :tabn <CR> zet deze in je $HOME/.vimrc ... zonder de eerste ":" Vi redirect :r date # datum in text :1,5!sort -n # sorteren van eerste 5 regels met een extern commando (kan ook met shift v een visual gedeelte selecteren) :0,$!cut -f1 -d: # alleen field 1 van text overhouden (delimiter :) Vi inspringen bij loop code selecteer met shift-v de regels die moeten inspringen. daarna > of < gebruiken om te tabben Speciale charakters met vi bekijken :set list een tab zier er dan uit als ^I Bash sudo vergeten in commando service apache restart sudo !! Bash truck vorige commando doet vorig commando met aangepast keyword systemctl status ovirt-ha-broker.service ^status^start {} expand directory's archief30_tmp t/m archief35_tmp recursive directories chmod 2775 zetten find achief{30..35}_tmp -type d -exec chmod 2775 {} \; of mkdir tmp{1..3} echo pr{ut,utser}s 1 regel uit een script starten zonder copy-paste (bijvoorbeeld in een remote-console sessie) grep ipa command uit cobber.ks en voor deze uit ( /usr/sbin/ipa-client-install --domain=mgtdomain --enable-dns-updates -w password --realm=domain --server=server.domain --hostname=hypervisor.domain --unattended --force-ntpd ) cat cobbler.ks | grep ipa | bash Bash karakters omdraaien Linkerhand rechterhand coordinatie probleem smile upadte ga op de d staan en druk ctrl-t Bash laatste woord vorige regel ls -latrd /var/data/extra/backup chmod 775 <ESC(punt)> Ssh forward bij een running sessie [haanstoot@xxx202 ~]$ ~C (tilde C) -L8080:localhost:80 (redirect localhost port 80 naar eigen machine poort 8080 -D9999 (Dynamic socks forward, zie andere post in deze wiki) CTRL-C werkt niet? ctrl\ SSH .ssh/config KeepAlive yes <---------- dont die on me ServerAliveInterval 60 <---------- dont die on me Host *.domain <-- voor alle hosts in radlan fast login en geen vraag over key StrictHostKeyChecking no UserKnownHostsFile=/dev/null host *.domain <-- domain is altijd mijn user user haanstoot GSSAPIAuthentication no <-- speedup host pruts*.domain user pi Parallel tasks 4 parallel jobs find jpg -type f -name \*.jpg -print0 | xargs -0 -n1 -P4 ./convert.sh clusterssh truck clusterssh alle svgs <--- naar alle svgs sudo su - cd /bricks/*/store <--- cd naar deze als bestaat cd /rhgs/brick0*/store <-- cd naar deze als bestaat (nu sta je in atelier directories als deze bestaat als root in de svgs) pwd | grep store || logout <--- geen atelier dir waar je staat? dan logout id | grep haanstoot && logout <-- net logout? dus eigen user ... dan logout eindresultaat ... als root in atelier volumes op svgs waar ze bestaan Start in screen /usr/bin/screen -d -m -S backup /usr/local/bin/backup.sh screen -r backup Start screen in "detached" mode. This creates a new session but doesn't attach to it. This is useful for system startup scripts. Sudo vergeten systemctl restart httpd (wil niet als user) sudo !! (doet "sudo systemctl restart httpd") Reuse arguments Bijvoorbeeld: # touch file1 file2 file3 file4 # chmod 777 !* Voert uit: chmod 777 file1 file2 file3 file4 5 meeste disk gebruikers human readable du -hsx * | sort -rh | head -5 rm groot bestand duurt lang : > /mappen2/archieven/test.img Check memory bankjes sudo dmidecode| grep -i -B1 "Form Factor" (B1 betekend BEFORE 1 regel meenemen, A is AFTER) Size: 8192 MB Form Factor: DIMM -- Size: 8192 MB Form Factor: DIMM -- Size: No Module Installed Form Factor: DIMM -- Set time and restart NTP service sudo service ntpd stop && sudo ntpdate ntp1.domain && sudo ntpdate ntp2.domain && sudo service ntpd start Aliases en terminal kleuren In een terminal f*ckedup colors? komt door de alias alias ls='ls --color' deze even uitzetten / niet gebruiken? \ls Werkt voor alle aliassen Snelle delete Niet onderstaande gebruiken voor grote dirs rm -rf /path/bla Maar mkdir /tmp/leeg ; rsync -a --delete /tmp/leeg /path/bla Bash best practices BashBestPractices Script log replay script --timing=/tmp/time.txt /tmp/script.log scriptreplay -t /tmp/time.txt /tmp/script.log

VI
Vim comments
toevoegen
ga op de eerste regel van de te commenten regel staan
ctrl-v (visual)
naar beneden tot eind regel (met cursor of pagedown)
ctrl-i
#
esc (kan even duren)
verwijderen
ga op de eerste regel van de te uncommenten regel staan
ctrl-v (visual)
naar beneden tot eind regel (met cursor of pagedown)
x
esc
Als user config geedit en geen schrijfrechten?
:!bash
chmod 666 bestand
:wq
chmod bestand naar wat hij was
Input / verwerk door bash vanuit vi
:0,5 !sort # eerste 5 regels door sort heen halen

:r !date # datum in je tekst document
Vim tabs / multiple files
vi /etc/hosts /etc/services
:n 
:prev
VIM7 tabs
vim -p /etc/hosts /etc/profile
:tabe /etc/services
:tabn
:tabp
Key mappen
:map <F7> :tabp <CR>
:map <F8> :tabn <CR>
zet deze in je $HOME/.vimrc ... zonder de eerste ":"
Vi redirect
:r date               # datum in text 
:1,5!sort -n          # sorteren van eerste 5 regels met een extern commando (kan ook met shift v een visual gedeelte selecteren)
:0,$!cut -f1 -d:      # alleen field 1 van text overhouden (delimiter :)
Vi inspringen bij loop code
selecteer met shift-v de regels die moeten inspringen.
daarna > of < gebruiken om te tabben
Speciale charakters met vi bekijken
:set list
een tab zier er dan uit als ^I
Bash
sudo vergeten in commando
service apache restart
sudo !!
Bash truck vorige commando
doet vorig commando met aangepast keyword
systemctl status ovirt-ha-broker.service
^status^start
{} expand
directory's archief30_tmp t/m archief35_tmp recursive directories chmod 2775 zetten
find achief{30..35}_tmp -type d -exec chmod 2775 {} \;

of

mkdir tmp{1..3}
echo pr{ut,utser}s
1 regel uit een script starten zonder copy-paste (bijvoorbeeld in een remote-console sessie)
grep ipa command uit cobber.ks en voor deze uit
( /usr/sbin/ipa-client-install --domain=mgtdomain --enable-dns-updates -w password --realm=domain --server=server.domain --hostname=hypervisor.domain --unattended --force-ntpd )
cat cobbler.ks | grep ipa | bash
Bash karakters omdraaien
Linkerhand rechterhand coordinatie probleem smile
upadte
ga op de d staan en druk ctrl-t
Bash laatste woord vorige regel
ls -latrd /var/data/extra/backup
chmod 775 <ESC(punt)>
Ssh forward bij een running sessie
[haanstoot@xxx202 ~]$ 
~C (tilde C)
-L8080:localhost:80 (redirect localhost port 80 naar eigen machine poort 8080
-D9999 (Dynamic socks forward, zie andere post in deze wiki)
CTRL-C werkt niet?
ctrl\
SSH .ssh/config
KeepAlive yes <---------- dont die on me
ServerAliveInterval 60 <---------- dont die on me
Host *.domain <-- voor alle hosts in radlan fast login en geen vraag over key
   StrictHostKeyChecking no 
   UserKnownHostsFile=/dev/null

host *.domain <-- domain is altijd mijn user
        user haanstoot
        GSSAPIAuthentication no <-- speedup

host pruts*.domain
        user pi
Parallel tasks
4 parallel jobs
find jpg -type f -name \*.jpg -print0 | xargs -0 -n1 -P4 ./convert.sh
clusterssh truck
clusterssh alle svgs <--- naar alle svgs
sudo su -
cd /bricks/*/store <--- cd naar deze als bestaat
cd /rhgs/brick0*/store <-- cd naar deze als bestaat (nu sta je in atelier directories als deze bestaat als root in de svgs)
pwd | grep store || logout <--- geen atelier dir waar je staat? dan logout
id | grep haanstoot && logout <-- net logout? dus eigen user ... dan logout
eindresultaat ... als root in atelier volumes op svgs waar ze bestaan
Start in screen
/usr/bin/screen -d -m -S backup /usr/local/bin/backup.sh
screen -r backup
Start screen in "detached" mode. This creates a new session but doesn't attach to it. This is useful for system startup scripts.
Sudo vergeten
systemctl restart httpd (wil niet als user)
sudo !!
(doet "sudo systemctl restart httpd")
Reuse arguments

Bijvoorbeeld:
# touch file1 file2 file3 file4
# chmod 777 !*

Voert uit: chmod 777 file1 file2 file3 file4
5 meeste disk gebruikers human readable
du -hsx * | sort -rh | head -5
rm groot bestand duurt lang
: > /mappen2/archieven/test.img
Check memory bankjes
sudo dmidecode| grep  -i -B1 "Form Factor" (B1 betekend BEFORE 1 regel meenemen, A is AFTER)

        Size: 8192 MB
        Form Factor: DIMM
--
        Size: 8192 MB
        Form Factor: DIMM
--
        Size: No Module Installed
        Form Factor: DIMM
--
Set time and restart NTP service
sudo service ntpd stop && sudo ntpdate ntp1.domain && sudo ntpdate ntp2.domain && sudo service ntpd start
Aliases en terminal kleuren
In een terminal f*ckedup colors?
komt door de alias
alias ls='ls --color'
deze even uitzetten / niet gebruiken?
\ls
Werkt voor alle aliassen
Snelle delete
Niet onderstaande gebruiken voor grote dirs
rm -rf /path/bla
Maar
mkdir /tmp/leeg ; rsync -a --delete /tmp/leeg /path/bla
Bash best practices
BashBestPractices
Script log replay
script --timing=/tmp/time.txt /tmp/script.log
scriptreplay -t /tmp/time.txt /tmp/script.log

There are toooo many cool sniplets .. only adding when i’m bored.

Computer

Finding files on my fileserver

2022-08-04 fash

I use several tools to find files on my server.

Loads of stuff on my main fileserver.
(Graph is a great tool called DUC) https://duc.zevv.nl/

Besides a search engine, i have a file finder.
Due to the massive amount of data, i like to find things by other means than knowing the directory structure.

I can find files by filename, but also by contents.

I’ll talk about find by contents first.

I’ve got loads of documents in Pdf, HTML, txt, doc, sheets , wordperfect etcetera.
Those documents i can find using a tool named Namazu.
This is quite a old tool, but i’m using it for a long time and it still works great.
I didn’t find a better replacement yet.
(But i’ve been looking into : elasticsearch, Solr, Lucene)

http://www.namazu.org/ is easy to install, but if you want the tool to scrape different kinds of documents you have to add some additional software.

My multipurpose printer can scan pages in pdf.
Those are only embedded jpg’s in a pdf container.
I will talk about how i handle these later.

My current start page :
This index contains 267,763 documents and 14,036,762 keywords.

Some things to consider when implementing namazu:

tweak the file types to scrape, it makes no sense to scrape binaries
tweak the directories to scrape (example below)
- 0 1 * * 1 fash /usr/bin/mknmz -f /etc/namazu/mknmzrc –output-dir=/namazu/ /mnt/private/paperwork/ /mnt/private/information/ /mnt/private/Art\ en\ hobby\ Projects/ /mnt/private/Music\ Projects/ /mnt/private/bagpipe-music-writer/ –exclude=XXX –exclude=/mnt/binaries > /tmp/namazu.log 2>&1
you can set a parameter in the config for search only, this disables downloading the found link in the results!

Before Namazu i used HtDig.

HtDIg also can scrape remote websites, Namazu can’t.

Preparing PDF for indexing:

I’ve written some scripts to make PDFs containing scanned text scrape-able.
( https://gitlab.com/fash/inotify-scanner-parser )
What it does:

My scanner puts a scanned pdf on my fileserver in a certain directory
Inotify detects a written file
it will copy the file, run OCR on it (tesseract) and writes a txt file (scapeable)
After that the text will be embedded (overlay) on the PDF, so now it becomes searchable/scrapeable
When certain keywords are found, it will sort documents in subdirs

Example from a scanned jpg, i can find OCR words!
(note .. the overlay is exact on the found words)

Finding files by name:

For finding files a made a little webpage like this:

It is a simple webpage grabbing through a list of files.
It takes the first keyword and does a grep, it takes a second keyword to match also.
I can select different file databases to search. (This case is private)
Between search and private i can give the number of entries to print.
So i can do
Search “ansible” NOT “tower” 50 entries from the public fileset

Crontab:

20 5 * * * /usr/bin/find /mnt/shark* > /var/www/html/findfiles/sharkoon

10 4 * * * /usr/bin/find /tank/populair > /var/www/html/findfiles/populair

20 4 * * * /usr/bin/find /tank/celtic > /var/www/html/findfiles/celtic

etc

20 5 * * * /usr/bin/find /mnt/shark* > /var/www/html/findfiles/sharkoon 10 4 * * * /usr/bin/find /tank/populair > /var/www/html/findfiles/populair 20 4 * * * /usr/bin/find /tank/celtic > /var/www/html/findfiles/celtic etc

20 5 * * * /usr/bin/find /mnt/shark*  > /var/www/html/findfiles/sharkoon
10 4 * * * /usr/bin/find /tank/populair > /var/www/html/findfiles/populair
20 4 * * * /usr/bin/find /tank/celtic > /var/www/html/findfiles/celtic
etc

And a php script (dirty fast hack, never came around it to make it a final version)

Search: <form method="post" action="/findfiles/?"><input type="Text" name="words" size=10 value=""><input type="Text" name="words2" size=10 value=""><input type="Submit" name="submit" value="search"><input type="Text" name="nrlines" size=3 value=""><input type="checkbox" name="not" unchecked>

<OPTION VALUE=private>private

<OPTION VALUE=celtic>celtic

<OPTION VALUE=populair>populair

<OPTION VALUE=dump>public

<OPTION VALUE=sharkoon>sharkoon

</SELECT>

</form>

<?php

$words2=$_POST['words2'];

$words=$_POST['words'];

$filefile=$_POST['filefile'];

$findfile=$_POST['findfile'];

$nrlines=$_POST['nrlines'];

$not=$_POST['not'];

if ($words2=="xsearch") { $findfile="other"; $words2=""; }

if ($nrlines) { } else { $nrlines=100; }

if ($words && $words2=="") {

$words = preg_replace("(\r\n|\n|\r)", "", $words);

$words = preg_replace("/[^0-9a-z]/i",'', $words);

$command = "/bin/cat $findfile |/bin/grep -i $words |head -$nrlines";

$blah=shell_exec($command);

$blah=str_replace($words, "<b><font color=red>$words</font></b>",$blah);

print $blah;

}

if (($words) and ($words2)) {

$words = preg_replace("(\r\n|\n|\r)", "", $words);

$words = preg_replace("/[^0-9a-z.]/i",'', $words);

$words2 = preg_replace("(\r\n|\n|\r)", "", $words2);

$words2 = preg_replace("/[^0-9a-z.]/i",'', $words2);

if ($not=="on") {

$command = "/bin/cat $findfile |/bin/grep -i $words | /bin/grep -iv $words2 |head -$nrlines";

} else {

$command = "/bin/cat $findfile |/bin/grep -i $words | /bin/grep -i $words2 |head -$nrlines";

}

$blah=shell_exec($command);

$blah=str_replace($words, "<b><font color=red>$words</font></b>",$blah);

$blah=str_replace($words2, "<b><font color=red>$words2</font></b>",$blah);

print $blah;

}

</PRE>

</body></html>

<html><head><title></title></body> <font face="Tahoma"><small>|keyword|(keyword)|search|(nr results)|NOT SECOND KEYWORD|share|</small><BR> Search: <form method="post" action="/findfiles/?"><input type="Text" name="words" size=10 value=""><input type="Text" name="words2" size=10 value=""><input type="Submit" name="submit" value="search"><input type="Text" name="nrlines" size=3 value=""><input type="checkbox" name="not" unchecked> <SELECT NAME=findfile> <OPTION VALUE=private>private <OPTION VALUE=celtic>celtic <OPTION VALUE=populair>populair <OPTION VALUE=dump>public <OPTION VALUE=sharkoon>sharkoon </SELECT> </form> <P><PRE> <?php $words2=$_POST['words2']; $words=$_POST['words']; $filefile=$_POST['filefile']; $findfile=$_POST['findfile']; $nrlines=$_POST['nrlines']; $not=$_POST['not']; if ($words2=="xsearch") { $findfile="other"; $words2=""; } if ($nrlines) { } else { $nrlines=100; } if ($words && $words2=="") { $words = preg_replace("(\r\n|\n|\r)", "", $words); $words = preg_replace("/[^0-9a-z]/i",'', $words); $command = "/bin/cat $findfile |/bin/grep -i $words |head -$nrlines"; $blah=shell_exec($command); $blah=str_replace($words, "<b><font color=red>$words</font></b>",$blah); print $blah; } if (($words) and ($words2)) { $words = preg_replace("(\r\n|\n|\r)", "", $words); $words = preg_replace("/[^0-9a-z.]/i",'', $words); $words2 = preg_replace("(\r\n|\n|\r)", "", $words2); $words2 = preg_replace("/[^0-9a-z.]/i",'', $words2); if ($not=="on") { $command = "/bin/cat $findfile |/bin/grep -i $words | /bin/grep -iv $words2 |head -$nrlines"; } else { $command = "/bin/cat $findfile |/bin/grep -i $words | /bin/grep -i $words2 |head -$nrlines"; } $blah=shell_exec($command); $blah=str_replace($words, "<b><font color=red>$words</font></b>",$blah); $blah=str_replace($words2, "<b><font color=red>$words2</font></b>",$blah); print $blah; } ?> </PRE> </body></html>

<html><head><title></title></body>
<font face="Tahoma"><small>|keyword|(keyword)|search|(nr results)|NOT SECOND KEYWORD|share|</small><BR>
Search: <form method="post" action="/findfiles/?"><input type="Text" name="words" size=10 value=""><input type="Text" name="words2" size=10 value=""><input type="Submit" name="submit" value="search"><input type="Text" name="nrlines" size=3 value=""><input type="checkbox" name="not" unchecked>
<SELECT NAME=findfile>
<OPTION VALUE=private>private
<OPTION VALUE=celtic>celtic
<OPTION VALUE=populair>populair
<OPTION VALUE=dump>public
<OPTION VALUE=sharkoon>sharkoon
</SELECT>
</form>
<P><PRE>
<?php
$words2=$_POST['words2'];
$words=$_POST['words'];
$filefile=$_POST['filefile'];
$findfile=$_POST['findfile'];
$nrlines=$_POST['nrlines'];
$not=$_POST['not'];


if ($words2=="xsearch") { $findfile="other"; $words2=""; }
if ($nrlines) {  } else { $nrlines=100; }
if ($words && $words2=="") {
$words = preg_replace("(\r\n|\n|\r)", "", $words);
$words = preg_replace("/[^0-9a-z]/i",'', $words);
$command = "/bin/cat $findfile |/bin/grep -i $words |head -$nrlines";
$blah=shell_exec($command);
$blah=str_replace($words, "<b><font color=red>$words</font></b>",$blah);
print $blah;
}
if (($words) and ($words2)) {
$words = preg_replace("(\r\n|\n|\r)", "", $words);
$words = preg_replace("/[^0-9a-z.]/i",'', $words);
$words2 = preg_replace("(\r\n|\n|\r)", "", $words2);
$words2 = preg_replace("/[^0-9a-z.]/i",'', $words2);
if ($not=="on") {
$command = "/bin/cat $findfile |/bin/grep -i $words | /bin/grep -iv $words2 |head -$nrlines";
} else {
$command = "/bin/cat $findfile |/bin/grep -i $words | /bin/grep -i $words2 |head -$nrlines";
}
$blah=shell_exec($command);
$blah=str_replace($words, "<b><font color=red>$words</font></b>",$blah);
$blah=str_replace($words2, "<b><font color=red>$words2</font></b>",$blah);
print $blah;
}
?>
</PRE>
</body></html>

Prutser ..

Tag Archives: bash

Security cam ftp search major changes in images.

Automatic photo sorting with Gps location lookup

Automating the h*ll out of windows applications using linux

Motion detection for stupid webcams

Energy bill and ledserver

Challenged myself into programming a photo quiz

Homewizard Watermeter

Spotify export

Collection of useful Linux scripts/commands

Finding files on my fileserver

"If something is worth doing, it's worth overdoing."