Tag Archives: bash

Scraping podcast which uses a javascript to obfuscate mp3 links

wget-ting the page only gave me flat html, but no readable links.

We need the rendered version, phantomjs wil help

wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2

printsource.js

var system = require('system');
var page   = require('webpage').create();
var url    = system.args[1];
page.open(url, function () {
  console.log(page.content);
  phantom.exit();
});

Run phantomjs

phantomjs-2.1.1-linux-x86_64/bin/phantomjs printsource.js  https://xxxxxxxx/show/xxxx > out

So now i got the rendered page, get mp3’s and titles, for this example

cat out | sed 'N;s/\n/,/' | cut -f2,7 -d\" | while read line ; do
mp3=$( echo $line | cut -f1 -d\")
title=$( echo $line | cut -f3 -d\> | tr -d '/<>[]]\!,;' | tr -d "'" | sed s/CDATA//g | sed s#title##g | sed s/:/-/g )
echo "$mp3 $title"
wget $mp3 -O "$title.mp3"
done

bash downloadscript
done

ZFS replace disk

I’m using ZFS for my main fileserver, this pool was created over 10 years ago.
Meanwhile i’ve: Swapped broken disks, switched disks for bigger ones and effectively resized my storage 2 or 3 times. Never had any corruption.

Yesterday i say a warning that one of the disks in the pool was OFFLINE.
Today i replaced it using below command’s

  • Put the disk in OFFLINE mode (if needed, mine was already offline)
    • zpool offline tank sdb
  • Remove disk from system
    • echo 1 | sudo tee /sys/block/sdb/device/delete
  • Remove the disk physically
  • Insert the replacement disk. And copy headers/structure from another disk
    • sgdisk –replicate=/dev/sdb /dev/sda
    • sgdisk –randomize-guids /dev/sdb
  • Run the zpool replace command. 
    • zpool replace tank /dev/sdb
  • Use online command to activate disk (no needed in my case, it already did that)
Tips:

# My labels with serials fell off :(
dd if=/dev/sdb of=/dev/null -> blinky led .. 

# What is the serial?
sudo hdparm -i /dev/sdb | grep Serial

Output

root@latex:~# sgdisk --replicate=/dev/sdb /dev/sda
The operation has completed successfully.
root@latex:~# sgdisk --randomize-guids /dev/sdb
The operation has completed successfully.
root@latex:~# zpool replace tank /dev/sdb
root@latex:~# zpool status
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed May 18 11:31:21 2022
    5.64T scanned out of 14.4T at 331M/s, 7h42m to go
    1.88T resilvered, 39.16% done
config:

        NAME             STATE     READ WRITE CKSUM
        tank             DEGRADED     0     0     0
          raidz1-0       DEGRADED     0     0     0
            sda          ONLINE       0     0     0
            replacing-1  REMOVED      0     0     0
              old        REMOVED      0     0     0
              sdb        ONLINE       0     0     0  (resilvering)
            sdc          ONLINE       0     0     0

errors: No known data errors

New NFO generator using yad

old post : https://www.henriaanstoot.nl/2021/02/08/kodi-movies-and-metadata/

Added poster art generator at the bottom of this page

Made a new NFO generator to get private movies properly scraped in kodi.

What does it do?

  • Genererates NFO for Kodi
  • Goes recursive though given directories
  • Skips movies with existing nfo
  • Previews the movie using VLC (So you can see what it is you are adding information for)
  • Tries to make a nice title from filename and year
  • Gets year from metadata
  • Fills tags from yad forms
  • next version i will add a poster generator from another nfo script i made. (It allready works, but i want to add a frame selector.
  • Adds new genre/actors/places if they dont exists
#!/bin/bash
# Needs mediainfo and yad
#find /media/geisha/private/Media/video/ -type f | egrep -i "mkv$|avi$|mpg$|mp4$|ogv$" | while read ; do
find $1 -type f | egrep -i "mkv$|avi$|mpg$|mp4$|ogv$" | while read ; do
# check if nfo exists
short=$(echo  "$REPLY" | rev | cut -f-2 -d/ | rev)
file=$( echo "$REPLY" | rev | cut -f2 -d. | rev).nfo
filename=$( basename "$REPLY" | cut -f1 -d. )
if [ -f "$file" ] ; then
	echo "$file found .. so skipping"
else
	echo "Not found $file"
nohup vlc "$REPLY" &
# check if find date
year=$(mediainfo "$REPLY" | grep Encoded |head -1 2>/dev/null | awk '{ print $5 }' | cut -c-4)
yeartitle="$year - "
if [ "$year" == "" ] ; then 
	year=$(echo $file | tr -cd '2[0-9][0-9][0-9]')
	if [ ! "$year" == "" ] ; then 
		#no year in filename
		yeartitle=""
	else
		yeartitle="$year - "
	fi
	
fi
# create nfo
forminfo=$(yad --title="nfo form $short" --text="Please enter:" --form --field="Title" --field="Year" --field="Placenew" --field="Genrenew" "$yeartitle $filename" "$year" "" "" --form --columns=2 --item-
separator="," --field="Place":CB --field="Genre":CB "$(paste -s -d"," < places)" "$(paste -s -d"," < genres)")
#echo $forminfo
#2020 - |2020|placenem|genrene|Hilversum|Vacation|
title=$(echo $forminfo | cut -f1 -d\|)
year=$(echo $forminfo | cut -f2 -d\|)
placenew=$(echo $forminfo | cut -f3 -d\|)
genrenew=$(echo $forminfo | cut -f4 -d\|)
place=$(echo $forminfo | cut -f5 -d\|)
genre=$(echo $forminfo | cut -f6 -d\|)
# Plot
plot=$(yad --form --field="Text::TXT" --geometry="600x200")
#plot=$( echo $plot | cut -f1 -d\\)
#plot=$( echo $plot | cut -f1 -d|)
# Actors
actors=$(yad --list  --geometry="200x480"  --print-all --column= --column=:chk --column= --column=:chk $( cat actors | cut -f1 -d" "  |while read user; do echo -n "$user false " ;done ) )
echo $actors | grep "NEW|TRUE" >/dev/null 
if [ $? -eq 0 ] ; then
vi actors
actors=$(yad --list  --geometry="200x480"  --print-all --column= --column=:chk --column= --column=:chk $( cat actors  | cut -f1 -d" " |while read user; do echo -n "$user false " ;done ) )
fi
if [ ! "$placenew" == "" ] ; then
	place="$placenew"
	echo "$placenew" >> places
fi
if [ ! "$genrenew" == "" ] ; then
	genre="$genrenew"
	echo "$genrenew" >> genres
fi
(
echo '<?xml version="1.0" encoding="utf-8" standalone="yes"?>'
echo "<movie>"
echo "  <plot>$plot</plot>"
echo "  <outline />"
echo "  <title>$title</title>"
echo "  <year>$year</year>"
echo "  <country>$place</country>"
cat actors | cut -f1 -d" " | while read actorname ; do
	echo $actors | grep "$actorname|TRUE"  >/dev/null
		if [ $? -eq 0 ] ; then
		name=$(grep $actorname actors | cut -f2 -d\" )
		echo "<actor>"
		echo "  <name>$actorname</name>"
		echo "  <role>$name</role>"
		echo "</actor>"
	fi
done
echo "  <genre>$genre</genre>"
echo "  <art>"
echo "  </art>"
echo "</movie>" 
) > "$file"  
fi
killall vlc
done

And some files

> actors
Firstname "Firstname Lastname"
> genres
Vacation
Pruts
> places
Netherlands

Poster generator (select poster thumbnail to use)

Generates 3 poster images and 3 thumbs to select.
Thumbs are 10 seconds into the movieclip, midway and on 2/3

short=$(basename $1 | cut -f1 -d.)

three=$(( $(mediainfo --Inform="Video;%Duration%" $1) / 3000 ))
rm -f thumb*.jpg poster*.jpg
first=10
#ffmpeg -ss $first -i $1 -vf scale=320:-1 -frames:v 1 -q:v 2 thumb1.jpg  -hide_banner -loglevel error
ffmpeg -ss $first -i $1 -frames:v 1 -q:v 2 thumb1.jpg  -hide_banner -loglevel error
second=$(( $three  ))
ffmpeg -ss $second -i $1 -frames:v 1 -q:v 2 thumb2.jpg  -hide_banner -loglevel error
third=$(( $three * 2 ))
ffmpeg -ss $third -i $1 -frames:v 1 -q:v 2 thumb3.jpg  -hide_banner -loglevel error
convert thumb1.jpg -resize x1000 -gravity center -crop 666x1000 poster1.jpg
convert thumb1.jpg -resize x320 -gravity center -crop 212x320 thumb1s.jpg
convert thumb2.jpg -resize x1000 -gravity center -crop 666x1000  poster2.jpg
convert thumb2.jpg -resize x320 -gravity center -crop 212x320 thumb2s.jpg
convert thumb3.jpg -resize x1000 -gravity center -crop 666x1000  poster3.jpg
convert thumb3.jpg -resize x320 -gravity center -crop 212x320 thumb3s.jpg


out=$(yad --form --title="Select poster thumbnail"  --geometry="+0+0" \
       --columns="3" \
       --field="!thumb1s-1.jpg! :fbtn" "echo poster1-1.jpg"  \
       --field="!thumb2s-1.jpg! :fbtn" "echo poster2-1.jpg"  \
       --field="!thumb3s-1.jpg! :fbtn" "echo poster3-1.jpg"
)
if [ ! "$out" == "|||" ] ; then
choice=$(echo $out | sed s/\ \|\|\|//g | rev | cut -f1 -d" " | rev)
echo cp $choice ${short}-poster.jpg
fi

Add effect to movie using bash

mkdir -p videoframessource videoframes
ffmpeg -r 60 -i sweden.mp4 videoframessource/%05d.png
cd videoframessource
ls *png | while read file ; do ../convert2comic $file ; echo -n "." ;done ; echo ""
ffmpeg -f image2 -r 60  -i videoframes/%05d.png  -vcodec libx264 final.mp4
#!/bin/bash
# convert script .. call me convert2comic
convert -quiet $1 +repage -depth 8 -selective-blur 0x5+10% ./tmp1
convert ./tmp1 -level 0x80% -colorspace gray -posterize 6 -depth 8 -colorspace sRGB ./tmp2
convert ./tmp1 '(' ./tmp2 -blur 0x1 ')' '(' -clone 0 -clone 1 -compose over -compose multiply -composite -modulate 100,150,100 ')' '(' -clone 0 -colorspace gray ')' '(' -clone 3 -negate -blur 0x2 ')' '(' -c
lone 3 -clone 4 -compose over -compose colordodge -composite -evaluate pow 4 -threshold 90% -median 1 ')' -delete 0,1,3,4 -compose over -compose multiply -composite ../videoframes/$1
# Another nice effect
convert frame.png \( +clone -negate -blur 0x6 \) -compose ColorDodge -composite -modulate 100,0,100 out.png

Mqtt Bash Nodered Notify

When running scripts which take a long time, i don’t want to wait for things to finish before i can start the next one.

For example, using my dedup script or compiling stuff. I wanna know when it is finished.

So i made some scripts

Maybe you can hear the spoken text in the background playing downstairs

I’ve put a function in .bashrc, so i can use a command like
notify “Compiling is ready”
A command like this i can put at the end of a command or in a script file at the end.
make && make install && notify “compile ready”

What does it do when executed?

  • Send a mqtt message to the broker
  • Node-red will read this message and:
    • Send a message to my display on my desk – Sound and message notification. (See another post how i made this )
    • Send a message to a script on my Domoticz instance downstairs.
      • This will use a script to get a speech file from google, and play this on some small speakers in my livingroom
    • Send a pushover message to my phone
    • Display a message on my TV ( not in code below )

How?

At the end of your .bashrc

function notify() {
    if [ -z "$1" ]; then
        echo "Usage: $0 \"message\"";
        exit 1;
    fi
    mosquitto_pub -h 10.1.0.17 -t notify/bashscript -m "$1"
}

Scripts on my Domoticz instance

Python script

#!/usr/bin/python
import paho.mqtt.client as mqttClient
import time
import os
import subprocess
import shlex
Connected = False

def on_connect(client, userdata, flags, rc):
    if rc == 0:
        print("Connected to broker")
        global Connected
        Connected =True
    else:
        print("Connection failed")

def on_message(client, userdata, message):
    print "Message received: " + message.topic + " : " + message.payload
    fileName = "/home/pi/domoticz/scripts/speech" + " \"" + message.payload + "\""
    print fileName
    args = shlex.split(fileName)
    time.sleep(1)
    p = subprocess.Popen(args)


broker_address = "10.1.0.17"
port = 1883
#user = "user"
#password = "password"
client = mqttClient.Client("speechcmd")
#client.username_pw_set(user, password=password)
client.on_connect = on_connect
client.on_message = on_message

client.connect(broker_address, port=port)
client.loop_start()

while Connected != True:
    time.sleep(0.1)

client.subscribe('speech/cmd')

try:
    while True:
        time.sleep(1)

except KeyboardInterrupt:
    print "exiting"
    client.disconnect()
    client.loop_stop()

Caching speech script

This script will look for a cached audiofile with requested text, and uses that. Else it wil request a audio file from google, caches it and plays it though the speakers.

#!/bin/bash
INPUT=$*
input2=$(echo $INPUT | base64)
echo "$input2 = $INPUT" >> /home/pi/cache/files-text-relation
if [ -f /home/pi/cache/$input2.mp3 ] ; then
mpg123 -q /home/pi/cache/$input2.mp3 1>/dev/null 2>/dev/null
else
echo not cached
STRINGNUM=0

ary=($INPUT)
for key in "${!ary[@]}"
  do
    SHORTTMP[$STRINGNUM]="${SHORTTMP[$STRINGNUM]} ${ary[$key]}"
    LENGTH=$(echo ${#SHORTTMP[$STRINGNUM]})
    #echo "word:$key, ${ary[$key]}"
    #echo "adding to: $STRINGNUM"
    if [[ "$LENGTH" -lt "100" ]]; then
      #echo starting new line
      SHORT[$STRINGNUM]=${SHORTTMP[$STRINGNUM]}
    else
      STRINGNUM=$(($STRINGNUM+1))
      SHORTTMP[$STRINGNUM]="${ary[$key]}"
      SHORT[$STRINGNUM]="${ary[$key]}"
    fi
done

for key in "${!SHORT[@]}"
  do
    echo "Playing line: $(($key+1)) of $(($STRINGNUM+1))"
    NEXTURL=$(echo ${SHORT[$key]} | xxd -plain | tr -d '\n' | sed 's/\(..\)/%\1/g')
echo $NEXTURL
    mpg123 -w $input2 -q "http://translate.google.com/translate_tts?ie=UTF-8&client=tw-ob&q=$NEXTURL&tl=En-us"
    ffmpeg -i $input2 -codec:a libmp3lame -qscale:a 2 /home/pi/cache/$input2.mp3
mpg123 /home/pi/cache/$input2.mp3
done
fi

Node-red flow

function notify() {
    if [ -z "$1" ]; then
        echo "Usage: $0 \"message\"";
        exit 1;
    fi
    mosquitto_pub -h 10.1.0.17 -t notify/bashscript -m [
    {
        "id": "1442fca698589679",
        "type": "mqtt in",
        "z": "cb6f001b.721c3",
        "name": "",
        "topic": "notify/bashscript",
        "qos": "2",
        "datatype": "auto",
        "broker": "8c74c5f6.9a7a48",
        "nl": false,
        "rap": false,
        "inputs": 0,
        "x": 180,
        "y": 580,
        "wires": [
            [
                "ddf5744bb5b73d4d",
                "faa5c794652d7a57",
                "b4e0107399248fea",
                "443f960b5d1cf40e"
            ]
        ]
    },
    {
        "id": "ddf5744bb5b73d4d",
        "type": "mqtt out",
        "z": "cb6f001b.721c3",
        "name": "",
        "topic": "speech/cmd",
        "qos": "",
        "retain": "",
        "broker": "8c74c5f6.9a7a48",
        "x": 590,
        "y": 560,
        "wires": []
    },
    {
        "id": "e95e828451d83158",
        "type": "comment",
        "z": "cb6f001b.721c3",
        "name": "bash notify",
        "info": "",
        "x": 170,
        "y": 540,
        "wires": []
    },
    {
        "id": "faa5c794652d7a57",
        "type": "mqtt out",
        "z": "cb6f001b.721c3",
        "name": "",
        "topic": "mqttlcd/message",
        "qos": "",
        "retain": "",
        "broker": "8c74c5f6.9a7a48",
        "x": 570,
        "y": 720,
        "wires": []
    },
    {
        "id": "b4e0107399248fea",
        "type": "delay",
        "z": "cb6f001b.721c3",
        "name": "",
        "pauseType": "delay",
        "timeout": "5",
        "timeoutUnits": "seconds",
        "rate": "1",
        "nbRateUnits": "1",
        "rateUnits": "second",
        "randomFirst": "1",
        "randomLast": "5",
        "randomUnits": "seconds",
        "drop": false,
        "outputs": 1,
        "x": 480,
        "y": 640,
        "wires": [
            [
                "ac4faf30b8adbe3f"
            ]
        ]
    },
    {
        "id": "ac4faf30b8adbe3f",
        "type": "function",
        "z": "cb6f001b.721c3",
        "name": "Empty payload",
        "func": "msg.payload = \"\";\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 640,
        "y": 640,
        "wires": [
            [
                "faa5c794652d7a57"
            ]
        ]
    },
    {
        "id": "dfbe26c12fc5e742",
        "type": "pushover",
        "z": "cb6f001b.721c3",
        "name": "Alleen Henri",
        "device": "rmx1931",
        "title": "Node-Red-Pushover",
        "priority": "1",
        "sound": "pushover",
        "url": "",
        "url_title": "",
        "html": false,
        "x": 850,
        "y": 500,
        "wires": []
    },
    {
        "id": "443f960b5d1cf40e",
        "type": "function",
        "z": "cb6f001b.721c3",
        "name": "Set pushover payload",
        "func": "\nmsg.topic = \"Bash Notify\";\nmsg.priority = 1;\nmsg.sound = \"cosmic\";\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 660,
        "y": 500,
        "wires": [
            [
                "dfbe26c12fc5e742"
            ]
        ]
    },
    {
        "id": "8c74c5f6.9a7a48",
        "type": "mqtt-broker",
        "name": "10.1.0.17",
        "broker": "10.1.0.17",
        "port": "1883",
        "clientid": "",
        "usetls": false,
        "compatmode": true,
        "keepalive": "15",
        "cleansession": true,
        "birthTopic": "",
        "birthQos": "0",
        "birthPayload": "",
        "closeTopic": "",
        "closePayload": "",
        "willTopic": "",
        "willQos": "0",
        "willPayload": ""
    }
]


Dedup script v0.2

Update 20220510

Sorting out my fileserver, i had the need for a deduplication script.
Many files i’ve been copying from backup, clouds mobile devices and workstation. Inevitable to get many copies.

Below script walks a directory, using locate it tries to find files with same name. Using a md5sum it wil check if it is the same file, when found a simular file it stops searching, removes the one from the check-directory and checks the next one.

#!/bin/bash
# Copy this script to your to clean directory, 
# when you got a copy on your fileserver from this script
# then the copy in your clean dir will be removed also.
# Dont want that? change
# find -type f | 
# into
# find -type f | grep -v &lt;nameofthisscript&gt; |

# dont is current directory, skip these from locations
dont=$(pwd)
# Never start in /mnt ? uncomment below
# echo "$dont" | grep "^/mnt" &amp;&amp; ( echo "start in tank" ; exit )

find -type f | while read file ; do
        filemd5=$(md5sum "$file" | cut -f1 -d" ")
        basenamefile=$(basename "$file")
        echo "searching $basenamefile"
        locate -i "/$basenamefile" | grep -v "$dont" | while read location ; do
        if [ -f "$location" ] ; then
                locatedfilemd5sum=$(md5sum "$location" | cut -f1 -d" ")
                if [ "$filemd5" == "$locatedfilemd5sum" ] ; then
                        echo "found same md5sum at $location"
                        rm "$file"
                        break
                fi
        fi
        done
done
# Remove empty dirs?
# find . -type d -empty -delete

Locate can be slow, sometimes it is better to put the locate DB in memory of on another fast storage system.

mkdir /ramdisk
mount ramfs -t ramfs /ramdisk/
cp /var/lib/mlocate/mlocate.db /ramdisk/ 

# change above script locate command
locate -d /var/lib/mlocate/mlocate.db -i IMG20191123.jpg

And remove empty directories?
Add below at the end of the script

find . -type d -empty -delete

Gluster and Rhev scripts

List Ovirt/RHV VM’s

( curl -s -u admin@internal:secretpassword -k https://ovirt-engine.lab/ovirt-engine/api/vms/ | grep "^        <name>" | cut -f2 -d\> | cut -f1 -d \< | while read;  do
echo "$REPLY"
done ) > list
echo "NOT IN BACKUP"
cat list | while read ; do ls vmbackup/$REPLY.ova 2>/dev/null >/dev/null || echo $REPLY ;done
echo ""
ls vmbackup/*ova | while read ; do
host=$( echo $REPLY | cut -f2 -d/ | cut -f1 -d.)
age=$(stat -c %y $REPLY  | cut -f1 -d" ")
echo "$age $host"
done

Backup: (UPDATE)
While i made my own backup script for Ovirt, i found this ansible based one.
https://github.com/silverorange/ovirt_ansible_backup

Backup Ovirt Engine using ansible

Usage : ansible-playbook -i ovirt-engine.lab, playbook.yml -u root (-k)

## Playbook
---
- hosts: all
  vars:
    ovirt_backup_mode: 'backup'
    ovirt_backup_archive: '/tmp/engine-backup.gzip'
    ovirt_backup_log_file: '/tmp/engine-backup.log'
    ovirt_backup_scope: 'all'
    ansible_python_interpreter: /usr/bin/python3
  roles:
      - ovirt-engine-backup

## roles/ovirt-engine-backup/tasks/main.yml
---
- name: run engine-backup
  shell: 'engine-backup --mode={{ ovirt_backup_mode }} --file={{ ovirt_backup_archive }} --log={{ ovirt_backup_log_file }} --scope={{ ovirt_backup_scope }}'
  tags:
    - skip_ansible_lint
  register: ovirt_backup_status

- name: download engine backup file
  fetch:
    src: "{{ ovirt_backup_archive }}"
    dest: "{{ ovirt_backup_archive }}"
  when: ovirt_backup_status is succeeded

- name: download log file
  fetch:
    src: "{{ ovirt_log_file }}"
    dest: "{{ ovirt_log_file }}"
  when: ovirt_backup_status is failed

Gluster overview

#!/bin/bash
ansible -i inventory all  -m shell -a "/sbin/gluster v info" --become  -K > output
cat output  | grep "^Volume Name"  | sort | cut -c14- | sort | uniq
cat output  | egrep "^Volume Name|^Brick|^Type"   | grep -v Bricks

Gluster to grafana using influx

#!/bin/bash
export PATH=$PATH:/sbin
(
# Get short shotname
host=$(echo $HOSTNAME | cut -f1 -d.)

echo "-- vg info --"
# Get vg info - pool nog doen
vgs --units k | egrep "brick|data0" | while read brickinfo ; do
name=$(echo $brickinfo | awk '{ print $1}'  )
size=$(echo $brickinfo | awk '{ print $6}' | cut -f1 -d.)
free=$(echo $brickinfo | awk '{ print $7}' | cut -f1 -d. | sed s/k//g)
usage=$(echo $(( $size - $free )))
curl -i -XPOST 'http://grafanaserver:8086/write?db=gluster' --data-binary "$host-$name size=$size,free=$free,usage=$usage"
done
echo "-- vg info done --"
echo "-- lv info --"
# Get lv info - these are mountpoints (not bricks)
#lvs --units k | egrep "brick|data0" | grep -v pool | while read volinfo ; do
df  | egrep "brick|data0" | while read volinfo ; do
name=$(echo $volinfo | awk '{ print $1}' | rev | cut -f1 -d/ | rev  )
size=$(df | grep -v svg |grep "$name" | awk '{ print $2}')
usage=$(df | grep -v svg |grep "$name" | awk '{ print $3}')
free=$(df | grep -v svg |grep "$name" | awk '{ print $4}')
curl -i -XPOST 'http://grafanaserver:8086/write?db=gluster' --data-binary "$host-$name size=$size,free=$free,usage=$usage"
done

echo "-- lv info done --"

# Get gluster volumes without gluster command, due to issues running multiple instances
# One volume on one mountpoint ? Then df == volume metrics
: > /tmp/vollist
find /var/lib/glusterd/| grep $HOSTNAME | grep vol$ | while read brickpath ; do
grep brick-path $brickpath | awk '{print $3 }' >> /tmp/vollist
done
mount | egrep "brick|data0" | awk '{print $3 }' | while read mountpoint ; do
count=$(grep $mountpoint /tmp/vollist | wc -l)
if [ $count -lt 2 ] ; then
echo "lv = brick .. df info"
#brick=$(echo $mountpoint)
#volume=$(echo $mountpoint |rev | cut -f1 -d/ | rev)
#size=$(df $mountpoint | grep -v Filesystem | awk '{ print $2 }')
#free=$(df $mountpoint | grep -v Filesystem | awk '{ print $4 }')
#usage=$(df $mountpoint | grep -v Filesystem | awk '{ print $5 }' | sed s/%//g )


dfinfo=$(df $mountpoint | grep -v ^Filesystem)
name=$(echo $dfinfo | awk '{print $1}' | rev | cut -f1 -d/ | rev)
size=$(echo $dfinfo | awk '{print $2}')
usage=$(echo $dfinfo | awk '{print $3}')
free=$(echo $dfinfo | awk '{print $4}')
perc=$(echo $dfinfo | awk '{print $5}' | sed s/%//g)
curl -i -XPOST 'http://grafanaserver:8086/write?db=gluster' --data-binary "$host-$name size=$size,free=$free,usage=$usage"
#curl -i -XPOST 'http://grafanaserver:8086/write?db=gluster' --data-binary "$host-$brick-$volume size=$size,free=$free,usage=$usage"
else
echo "du script draaien voor $mountpoint"
fi

done

echo "-- data% --"
lvs --units k -a -oname,vg_name,size,vgfree,data_percent,pool_lv | grep -v \\[ | egrep  "brick|data" | grep -v sysdata | grep -v "^  thin" | while read ; do
vg=$(echo $REPLY | awk '{print $2}')
name=$(echo $REPLY | awk '{print $6"-"$1}')
size=$(echo $REPLY | awk '{ print $3 }' | cut -f1 -d.)
free=$(echo $REPLY | awk '{ print $4 }' | cut -f1 -d. | sed s/k//g)
dataperc=$(echo $REPLY | awk '{ print $5 }' | cut -f1 -d. )
if [ "w$dataperc" == "w" ] ; then dataperc=0 ;fi
curl -i -XPOST 'http://grafanaserver:8086/write?db=gluster' --data-binary "$host-$name-$vg size=$size,free=$free,dataperc=$dataperc"
done


) 2> /dev/null >/dev/null

Running Ovirt and want a overview of glustermounts in VMs?

run makeinventory.sh (this creats an list of linux hosts which are UP in RHV/Ovirt)

#!/bin/bash
#ik haal een lijst op van virtual machines die aanstaan en linux draaien
#set -x
mkdir -p tmp
rm -f tmp/linuxup
read -p "Username: " x2
read -s -p "Password: " x1
echo " "
curl -s -X GET -H "Accept: application/xml"  --insecure -u $x2@radlan:$x1 https://ovirtserver/ovirt-engine/api/vms | grep "vm href=" | cut -c37-72 > tmp/vms
cat tmp/vms | while read vm ; do
echo -n "."
curl -s -X GET -H "Accept: application/xml"  --insecure -u $x2@radlan:$x1 https://ovirtserver/ovirt-engine/api/vms/$vm > tmp/$vm
done
echo " "
: > tmp/linuxup
find tmp -type f -mtime 0 | egrep -v "vms$|linuxup" | while read ; do

name=$(cat $REPLY | grep -i "^    <name>"| cut -f2 -d\> | cut -f1 -d \<)
os=$(cat $REPLY | egrep "^        <family>" | egrep -v "svw|SVNT|debian|ubuntu|server|spice|vnc|windows_|native" | cut -f2 -d\> | cut -f1 -d \<)
up=$(cat $REPLY |  grep -A1 start_time | tail -1| cut -f2 -d\> | cut -f1 -d \<)


echo $name,$os,$up | grep -v ",," | grep ",up" >> tmp/linuxup

cat tmp/linuxup
echo ""
echo "staat in tmp/linuxup"

done
echo "[all]" > inventory
cat tmp/linuxup | grep "."| grep -i linux | cut -f1 -d, >> inventory

Run get.sh
* ansible on all hosts (inventory) and fetches gluster mounts
* reorders this info
* generates a dot file (graphviz)
* running dot fot a svg/png output for every gluster host

#!/bin/bash
echo "draai eerst makeinventory.sh voor een nieuwe host lijst"
read d
echo "Kijk ook of ansible alle servers kan bereiken"
read d
ansible -i inventory all -m shell -a cat /etc/mtab | grep gluster | while read; do source=$(echo $REPLY | cut -f1 -d" ");dest=$(echo $REPLY | cut -f2 -d" ") ; echo " " ;echo "$source $HOSTNAME:$dest" ;done > gluster
cat gluster  | grep ^s | grep : > gluster.grep

cat gluster.grep  | cut -f1 -d: | sort | uniq | while read gluster ; do

cat header > $gluster.dot
cat gluster.grep | grep $gluster | cut -f1 -d" "| sort | uniq | while read ; do echo "    \"$REPLY\";" ;done >> $gluster.dot
cat middel >> $gluster.dot
cat gluster.grep | grep $gluster| sed s/^/\"/ | sed s/\ /\"\ \-\>\ \"/ | sed s/$/\"\;/ >> $gluster.dot
cat footer >> $gluster.dot
dot $gluster.dot -Tsvg >$gluster.svg
dot -Gsize=9,15\! -Gdpi=100  $gluster.dot -Tpng >$gluster.png
done

A php select page to few them

<form method="post">
<select name="volume">
    <?PHP
    foreach (glob("*.svg") as $filename) {
        echo "<option value=" . $filename .">" . $filename . "</option>";
    }
    ?>
    </select>
 <input type="submit" value="Submit">
</form>
<?PHP
    if (isset($_POST['volume'])){
            $image=$_POST['volume'];
            echo "<img src=\"$image\">";
    }
?>



example below:

Get a list of RHV snapshots and descriptions

#!/bin/bash
: > outlist
read -p  "Username: " user
read -s -p "Password: " pass

curl -X GET -H "Accept: application/xml" -u $user@internal:$pass -k  https://ovirt-engine:443/ovirt-engine/api/vms   | egrep "<name>|snapshots" | grep -v "           " | sed '$!N;s/\n/ /' > templist
cat templist  | cut -f2,3 -d\> | cut -f1,3 -d\< | sed 's/<link href=\"/ /g' | cut -f1 -d\" > templist2
cat templist2 | while read ; do
host=$(echo $REPLY | awk '{ print $1 }')
link=$(echo $REPLY | awk '{ print $2 }')
echo $host >> outlist
curl -X GET -H "Accept: application/xml" -u $user@radlan:$pass -k  https://ovirt-engine:443/$link | egrep "description|date" | grep -v '            <'  >> outlist ; echo "----" >> outlist
done
cat outlist | awk '/Active VM/ {getline; next} { print }' > /tmp/outlist
grep -B3 date /tmp/outlist  |sed -e 's/<[^>]*>//g' > /tmp/snapshots
cat /tmp/snapshots | awk '/----/ {printf "\n%s\n",$0;next} {printf "%s ",$0}' | sed 's/ --*//g' | grep -v '\-\-\-\-' | grep "  " > snapshots

Output:
svr04.lab         Backup              2019-06-28T07:55:28.909+02:00
svr05.lab         upgrade             2019-06-18T14:18:49.745+02:00
svr33.lab         pre_ansible         2019-06-18T14:04:43.569+02:00

Somewhere i've got a snapshot age warn script.
This information was posted in Mattermost when sending the "/snapshots" command

Ansible Playbook to add virtual nics

---
- name: adnic
  hosts: localhost
  gather_facts: false
  become: no


  vars_prompt:
    - name: "username"
      prompt: "IPA username"
      private: no
    - name: "password"
      prompt: "IPA password"
      private: yes
    - name: "server"
      prompt: "server"
      private: no
    - name: "interface"
      prompt: "interface"
      private: no
      default: 'eth1'
    - name: "nicprofile"
      prompt: "dmz or other"
      private: no
      default: 'dmz'

  tasks:
  - name: Obtain SSO token with using username/password credentials
    ovirt_auth:
      url: https://ovirt-engine.lab/ovirt-engine/api
      username: "{{ username }}@internal"
      insecure: yes
      password: "{{ password }}"
    
  - ovirt_nics:
      auth: "{{ ovirt_auth }}"
      vm: "{{ server }}"
      state: "present"
      name: "{{ interface }}"
      network: "{{ nicprofile }}"
      profile: "{{ nicprofile }}"

Ansible playbook to create snapshots

- name: makesnap
  hosts: localhost
  gather_facts: false

  vars_prompt:
    - name: "username"
      prompt: "IPA username"
      private: no
    - name: "password"
      prompt: "IPA password"
      private: yes
    - name: "hostname"
      prompt: "hostname"
      private: no
    - name: "snapshotname"
      prompt: "snapshotname"
      private: no

  tasks:
  - name: Make snapshot
    ovirt_snapshots:
      auth:
        username: "{{ username }}@internal"
        password: "{{ password }}"
        insecure: True
        url: https://ovirt-engine.lab/ovirt-engine/api
      description: "{{ snapshotname }}"
      use_memory: true
      state: present
      timeout: 600
      vm_name: "{{ hostname }}"

Scripts on my domoticz instance

Script NO1:
Below is a script which checks all sensors and switches available on my domoticz instance, and gives me information about last updates.
For example when a device is out of reach or battery empty.

Usage:

Outputs to console, or you can use it in check_mk monitoring.
(For the latter, create a script

vi /usr/lib/check_mk_agent/local/checkfrontdoor
chmod +x /usr/lib/check_mk_agent/local/checkfrontdoor
put in the script 
#!/bin/bash
cd /path/to/script/
./belowscript "Frontdoor" 300 checkmk

Code:

#!/bin/bash
#set -x
# after running once check stateseconds for names
if [ $# -eq 0 ]; then
echo "$(basename $0) - without options .. getting states"
echo "use $(basename $0) "Sensorname" seconds" for check
echo "Getting all states"
: > stateseconds
curl -s -i -H "Accept: application/json" "http://127.0.0.1:8080/json.htm?type=devices&filter=all&used=true&order=Name" | egrep "Name|Last" | grep -v HardwareName |grep -vi levelnames> states
sed -i 'N;s/\n/,/' states
now=$(date +%s)
cat states  | awk '{ print $3" "$4$7$8$9$10$11$12 }' | sed s/,,/,/g |rev | cut -c2- | rev | while read ; do
name="$(echo $REPLY | cut -f2 -d, )"
dater="$(echo $REPLY | cut -f1 -d, | sed s/\"//g)"
#echo $dater
seen=$(date -d "$dater" +%s)
#echo $seen
echo "$name $(( $now - $seen))" >> stateseconds
echo -n "."
done
fi
echo ""
if [[ ! -z "$1" && ! -z $2 ]]; then
checkold=$(cat stateseconds | grep "$1\"" | head -1 | awk '{ print $2 }')
total=$(( checkold - $2 ))
if [ -z $3 ] ; then
if [ $checkold -gt $2 ] ; then echo "$name lastseen longer than $2 seconds ago ($total sec)" ; exit 1 ;fi
else
if [ $checkold -gt $2 ] ; then echo "2 \"$1\" - Sensor older than $2 seconds" ; exit 1
else
echo "0 \"$1\" - Sensor age ok" ; exit 0
fi
fi
fi

Above to your Mqtt broker?

#!/bin/bash
# change 300 ( age below 5 minutes )
# and $2>=900 ( age above 15 minutes )
./agecheck.sh # Updates statefile
cat stateseconds | tr -d '"-/:"_' | sort -n -k2 | awk '$2<=300' | while read; do
sensorname=$(echo $REPLY | awk '{ print $1 }' )
age=$(echo $REPLY | awk '{ print $2 }' )
mosquitto_pub -h 10.1.0.17 -t sensorsage/$sensorname -m "$age"
done

Script NO:2
This is a mqtt speech script with caching sounds, speech generated for me by google

MQTT Part

#!/usr/bin/python
import paho.mqtt.client as mqttClient
import time
import os
import subprocess
import shlex
Connected = False

def on_connect(client, userdata, flags, rc):
    if rc == 0:
        print("Connected to broker")
        global Connected
        Connected =True
    else:
        print("Connection failed")

def on_message(client, userdata, message):
    print "Message received: " + message.topic + " : " + message.payload
    fileName = "/home/pi/domoticz/scripts/speech" + " \"" + message.payload + "\""
    print fileName
    args = shlex.split(fileName)
    time.sleep(1)
    p = subprocess.Popen(args)


broker_address = "mqttbroker"
port = 1883
user = "myuser"
password = "mysecretpass"
client = mqttClient.Client("speechcmd")
client.username_pw_set(user, password=password)
client.on_connect = on_connect
client.on_message = on_message

client.connect(broker_address, port=port)
client.loop_start()

while Connected != True:
    time.sleep(0.1)

client.subscribe('speech/cmd')

try:
    while True:
        time.sleep(1)

except KeyboardInterrupt:
    print "exiting"
    client.disconnect()
    client.loop_stop()

Speech part (uses mpg123 to play sound though speaker)

#!/bin/bash
#set -x

INPUT=$*
input2=$(echo $INPUT | base64)
echo "$input2 = $INPUT" >> /home/pi/cache/files-text-relation
if [ -f /home/pi/cache/$input2.mp3 ] ; then
mpg123 -q /home/pi/cache/$input2.mp3 1>/dev/null 2>/dev/null
else
echo not cached
STRINGNUM=0

ary=($INPUT)
for key in "${!ary[@]}"
  do
    SHORTTMP[$STRINGNUM]="${SHORTTMP[$STRINGNUM]} ${ary[$key]}"
    LENGTH=$(echo ${#SHORTTMP[$STRINGNUM]})
    #echo "word:$key, ${ary[$key]}"
    #echo "adding to: $STRINGNUM"
    if [[ "$LENGTH" -lt "100" ]]; then
      #echo starting new line
      SHORT[$STRINGNUM]=${SHORTTMP[$STRINGNUM]}
    else
      STRINGNUM=$(($STRINGNUM+1))
      SHORTTMP[$STRINGNUM]="${ary[$key]}"
      SHORT[$STRINGNUM]="${ary[$key]}"
    fi
done

for key in "${!SHORT[@]}"
  do
    #echo "line: $key is: ${SHORT[$key]}"

    echo "Playing line: $(($key+1)) of $(($STRINGNUM+1))"
    NEXTURL=$(echo ${SHORT[$key]} | xxd -plain | tr -d '\n' | sed 's/\(..\)/%\1/g')
echo $NEXTURL
    mpg123 -w $input2 -q "http://translate.google.com/translate_tts?ie=UTF-8&client=tw-ob&q=$NEXTURL&tl=En-us"
    ffmpeg -i $input2 -codec:a libmp3lame -qscale:a 2 /home/pi/cache/$input2.mp3
mpg123 /home/pi/cache/$input2.mp3
done
fi

set msg.payload to “Doorbell”
and Mqtt topic to “speech/cmd”

To be continued !

FFmpeg tricks

Mp4 to gif (resize, cut and convert)

startsecond=0
duration=3
ffmpeg -ss $startsecond -t $duration -i 'input.mp4' -vf "fps=10,scale=320:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" -loop 0 output.gif

Gif to MP4

ffmpeg -i kanban.gif -movflags faststart -pix_fmt yuv420p -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" gifkanban.mp4

Converting MP4 for Davinci MOV

ffmpeg -i input.mp4 -c:v dnxhd -profile:v dnxhr_hq -pix_fmt yuv422p -c:a pcm_s16le -f mov output.mov

And back from Davinci to MP4

ffmpeg -i render.mov -c:v libx264 -pix_fmt yuv420p -crf 16 -force_key_frames 'expr:gte(t,n_forced/2)' -bf 2 -vf yadif -use_editlist 0 -movflags +faststart -c:a aac -q:a 1 -ac 2 -ar 48000 -f mp4 out.mp4

QTVR to png (note number of frames)

ffmpeg -i ../test.mov %02d.png
ffmpeg -i %02d.png -vf "tile=1x24,transpose=1" mh.png

Nikon MOV to mp4 (no reencoding)

ffmpeg -i input.mov -c copy -movflags +faststart  output.mp4

Nikon MOV to mp4 (problems with audio?
“Could not find tag for codec pcm_s16le in stream #1, codec not currently supported in container”

ffmpeg -i input.mov -c copy -c:a aac -b:a 128k -movflags +faststart  output.mp4
Maybe add -strict experimental 

MPG to MP4 (Not happy with this one)

ffmpeg -i vhs-rip.mpg"  -c:v libx264 -crf 10 -strict -2 1993-luv.mp4

FLV to MP4

ffmpeg -i filename.flv -c:v libx264 -crf 19 -strict experimental filename.mp4

Lossless cut with ffmpeg

ffmpeg  -ss 00:$2 -t 00:$3 -i $1 -vcodec copy -an cut_$1  
# an = no sound
# use -acodec copy for same codec as original audio

Timelapse JPG to MP4

ffmpeg -f image2 -r 24 -start_number 8296 -i "750_%04d.JPG"  -vcodec libx264 -profile:v high444 -refs 16 -crf 0 -preset ultrafast -vf scale=1920:1080 b.mp4

png to gif from a list

#cat png.list
file off.png
duration 1
file 20.png
duration 1
file 40.png
duration 1
file 60.png
duration 1
file 80.png
duration 1
file 90.png
duration 1
file 100.png
duration 1

# ffmpeg -f concat -i png.list -loop 0 hexlight.gif


Rotate mp4 clockwise/anticlockwise

ffmpeg -i VID20230110132531.mp4 -filter:v transpose=2 -c:v libx264 -preset veryfast -crf 22 -c:a copy -metadata:s:v rotate="" test.mp4

Concat video together

ffmpeg -f concat -i inputs.txt -c copy merge.mp4
inputs.txt
file start.mp4
file end.mp4


Error “width/height not divisible by 2 (1301×933)”

ffmpeg -i input.mp4 -vf "crop=trunc(iw/2)*2:trunc(ih/2)*2" output.mp4

Dvd/DV/Mpg etcetera overviews

I made two scripts, which take a movie, find out its lenght divide by 9. And generates a image montage of snapshots at certain times in the movie. ( A movie of 90 minutes, gets a snapshot every 10 minutes )

Example of snapshots of a movie

Below takes images as above and convert them into animated gifs. All movies in a path or directory structure gets a index page containing all animated gifs.

Example of generated directory index

I’ll post something about my media sorter also.
A ajax website using drag drop viewing/sorting images (png/gifs/jpgs) and movies.

#!/bin/bash
# Uses montage, ffmpeg, convert and mediainfo
rm thumbs/*
rm montage/*
(
set -x
numberofframes=9
find  -type f | egrep -i "flv$|mov$|mkv$|avi$|mpg$|mp4$|VTS_01_1.VOB$|wmv$" |  while read filez; do
        crunchpath=$(echo $filez | sed "s,/,,g" |  sed "s, ,,g" |tr -cd '[:alnum:]' ).gif
        crunchback=$crunchpath
        echo $crunchpath
        if [ -f thumbs/$crunchpath ] ; then
                echo "$file exists"
                else
	mkdir /tmp/thumbs/ 2>/dev/null
	mkdir thumbs 2>/dev/null
	mkdir montage 2>/dev/null
        rm -f /tmp/thumbs/*png
sec=0
                totaltime1=$( mediainfo -f "$filez" | grep ^Dura  | awk '{print $3}' |grep -v "\:" | tail -1 )
                totaltime2=$( mediainfo -f "$filez" | grep ^Dura  | awk '{print $4}' |grep -v "\:" | tail -1 )
totaltime1a=$(echo $totaltime1| tr -d 'a-z')
echo $totaltime1 | grep h && sec=$((3600 * $totaltime1a))
echo $totaltime1 | grep mn && sec=$((60 * $totaltime1a))
echo $totaltime1 | grep s && sec=$totaltime1a
echo "sec : $sec"
echo $totaltime2 | grep -v ms | grep  s && sec=$[ $sec + $(echo $totaltime2 | tr -d 'a-z')]
echo "sec : $sec"
                width=$( mediainfo -f "$filez" | grep "Width"| grep pixels | awk '{ print $3 }' )
                height=$( mediainfo -f "$filez" | grep "Height"| grep pixels | awk '{ print $3 }' )
        if [ "$sec" == "" ] ; then sec=1 ; fi
                timer=$[ $sec / $numberofframes ]
                        echo  "converting step "
                        echo  "timer is $timer"
mark=0
                for f in `seq -w 1 $numberofframes` ; do
                        echo  "$f,$timer"
			echo $[$f*$timer]
                        ffmpeg -ss $[$f*$timer] -i "$filez" -vframes 1 /tmp/thumbs/$f.png 1>/dev/null 2>/dev/null </dev/null
if [ $f -eq 8 ] ; then cp /tmp/thumbs/8.png /tmp/thumbs9.png ;fi
newsizer=$(echo -n $widther ; echo -n "x" ; echo -n $sizeh)
if [ $sizeh -gt $sizew ] ; then mv /tmp/thumbs/$f.png /tmp/ff.png ; convert -resize $newsizer\! /tmp/ff.png /tmp/thumbs/$f.png ; echo aangepast ; echo $newsizer ; mark=1; fi
                done
                        echo " "
                        echo "generating $crunchpath"
	newwidth=$[ $width / 2 ]
	newheight=$[ $height / 2 ]
	newsize=$(echo -n $newwidth ; echo -n "x" ; echo -n $newheight)
	newsize2=$(echo -n $width ; echo -n "x" ; echo -n $newheight)
	echo $newsize
if [ $mark == 0 ] ; then        convert -delay 100 -resize $newsize\! /tmp/thumbs/*.png thumbs/$crunchpath </dev/null ; else
convert -delay 100 -resize $newsize2\! /tmp/thumbs/*.png thumbs/$crunchpath </dev/null 
fi
montage /tmp/thumbs/*png  -geometry +3+3 -tile 3x3 montage/$crunchpath.jpg
	echo "<img src=\"$crunchpath\">" >> thumbs/_index.html
        fi
done
) > /tmp/thumb.log

Audio switcher

Doing my work from home via Jitsi (Jabra headset) and Listening to music on my speakers. Sometimes i want to switch output. For example spotify on my headset.

Using below code, i can easily switch between output sinks.

#!/bin/bash

if [ -z "$1" ]; then
    echo "DEVICE"
    pactl list sinks | egrep "^Sink|Description" | sed 'N;s/\n/,/' |sed -e 's/\t/ /g'  | cut -f2,5- -d" "
        echo "APPLICATIONS"
    pactl list sink-inputs | egrep -ie "^Sink|application.name" | grep -v "\-application-name" | sed 'N;s/\n/,/' | sed -e 's/\t/ /g' | sed -e 's/  / /g' | cut -f3,6- -d" "
    echo "Usage: $0 APPID DEVICEID" >&2
    exit 1
fi

pactl move-sink-input $1 $2

It shows output devices, and applications which are using audio sinks.
Just match them up.