Tag Archives: bash

Finding files on my fileserver

I use several tools to find files on my server.

Loads of stuff on my main fileserver.
(Graph is a great tool called DUC) https://duc.zevv.nl/

Besides a search engine, i have a file finder.
Due to the massive amount of data, i like to find things by other means than knowing the directory structure.

I can find files by filename, but also by contents.

I’ll talk about find by contents first.

I’ve got loads of documents in Pdf, HTML, txt, doc, sheets , wordperfect etcetera.
Those documents i can find using a tool named Namazu.
This is quite a old tool, but i’m using it for a long time and it still works great.
I didn’t find a better replacement yet.
(But i’ve been looking into : elasticsearch, Solr, Lucene)

http://www.namazu.org/ is easy to install, but if you want the tool to scrape different kinds of documents you have to add some additional software.

My multipurpose printer can scan pages in pdf.
Those are only embedded jpg’s in a pdf container.
I will talk about how i handle these later.

My current start page :
This index contains 267,763 documents and 14,036,762 keywords.
Search example of namazu

Some things to consider when implementing namazu:

  • tweak the file types to scrape, it makes no sense to scrape binaries
  • tweak the directories to scrape (example below)
    • 0 1 * * 1 fash /usr/bin/mknmz -f /etc/namazu/mknmzrc –output-dir=/namazu/ /mnt/private/paperwork/ /mnt/private/information/ /mnt/private/Art\ en\ hobby\ Projects/ /mnt/private/Music\ Projects/ /mnt/private/bagpipe-music-writer/ –exclude=XXX –exclude=/mnt/binaries > /tmp/namazu.log 2>&1
  • you can set a parameter in the config for search only, this disables downloading the found link in the results!

Before Namazu i used HtDig.

Screenshot htdig

HtDIg also can scrape remote websites, Namazu can’t.

Preparing PDF for indexing:

I’ve written some scripts to make PDFs containing scanned text scrape-able.
( https://gitlab.com/fash/inotify-scanner-parser )
What it does:

  • My scanner puts a scanned pdf on my fileserver in a certain directory
  • Inotify detects a written file
  • it will copy the file, run OCR on it (tesseract) and writes a txt file (scapeable)
  • After that the text will be embedded (overlay) on the PDF, so now it becomes searchable/scrapeable
  • When certain keywords are found, it will sort documents in subdirs
Example from a scanned jpg, i can find OCR words!
(note .. the overlay is exact on the found words)

Finding files by name:

For finding files a made a little webpage like this:

It is a simple webpage grabbing through a list of files.
It takes the first keyword and does a grep, it takes a second keyword to match also.
I can select different file databases to search. (This case is private)
Between search and private i can give the number of entries to print.
So i can do
Search “ansible” NOT “tower” 50 entries from the public fileset

Crontab:

20 5 * * * /usr/bin/find /mnt/shark*  > /var/www/html/findfiles/sharkoon
10 4 * * * /usr/bin/find /tank/populair > /var/www/html/findfiles/populair
20 4 * * * /usr/bin/find /tank/celtic > /var/www/html/findfiles/celtic
etc

And a php script (dirty fast hack, never came around it to make it a final version)

<html><head><title></title></body>
<font face="Tahoma"><small>|keyword|(keyword)|search|(nr results)|NOT SECOND KEYWORD|share|</small><BR>
Search: <form method="post" action="/findfiles/?"><input type="Text" name="words" size=10 value=""><input type="Text" name="words2" size=10 value=""><input type="Submit" name="submit" value="search"><input type="Text" name="nrlines" size=3 value=""><input type="checkbox" name="not" unchecked>
<SELECT NAME=findfile>
<OPTION VALUE=private>private
<OPTION VALUE=celtic>celtic
<OPTION VALUE=populair>populair
<OPTION VALUE=dump>public
<OPTION VALUE=sharkoon>sharkoon
</SELECT>
</form>
<P><PRE>
<?php
$words2=$_POST['words2'];
$words=$_POST['words'];
$filefile=$_POST['filefile'];
$findfile=$_POST['findfile'];
$nrlines=$_POST['nrlines'];
$not=$_POST['not'];


if ($words2=="xsearch") { $findfile="other"; $words2=""; }
if ($nrlines) {  } else { $nrlines=100; }
if ($words && $words2=="") {
$words = preg_replace("(\r\n|\n|\r)", "", $words);
$words = preg_replace("/[^0-9a-z]/i",'', $words);
$command = "/bin/cat $findfile |/bin/grep -i $words |head -$nrlines";
$blah=shell_exec($command);
$blah=str_replace($words, "<b><font color=red>$words</font></b>",$blah);
print $blah;
}
if (($words) and ($words2)) {
$words = preg_replace("(\r\n|\n|\r)", "", $words);
$words = preg_replace("/[^0-9a-z.]/i",'', $words);
$words2 = preg_replace("(\r\n|\n|\r)", "", $words2);
$words2 = preg_replace("/[^0-9a-z.]/i",'', $words2);
if ($not=="on") {
$command = "/bin/cat $findfile |/bin/grep -i $words | /bin/grep -iv $words2 |head -$nrlines";
} else {
$command = "/bin/cat $findfile |/bin/grep -i $words | /bin/grep -i $words2 |head -$nrlines";
}
$blah=shell_exec($command);
$blah=str_replace($words, "<b><font color=red>$words</font></b>",$blah);
$blah=str_replace($words2, "<b><font color=red>$words2</font></b>",$blah);
print $blah;
}
?>
</PRE>
</body></html>

Scraping podcast which uses a javascript to obfuscate mp3 links

wget-ting the page only gave me flat html, but no readable links.

We need the rendered version, phantomjs wil help

wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2

printsource.js

var system = require('system');
var page   = require('webpage').create();
var url    = system.args[1];
page.open(url, function () {
  console.log(page.content);
  phantom.exit();
});

Run phantomjs

phantomjs-2.1.1-linux-x86_64/bin/phantomjs printsource.js  https://xxxxxxxx/show/xxxx > out

So now i got the rendered page, get mp3’s and titles, for this example

cat out | sed 'N;s/\n/,/' | cut -f2,7 -d\" | while read line ; do
mp3=$( echo $line | cut -f1 -d\")
title=$( echo $line | cut -f3 -d\&gt; | tr -d '/&lt;&gt;[]]\!,;' | tr -d "'" | sed s/CDATA//g | sed s#title##g | sed s/:/-/g )
echo "$mp3 $title"
wget $mp3 -O "$title.mp3"
done

bash downloadscript
done

ZFS replace disk

I’m using ZFS for my main fileserver, this pool was created over 10 years ago.
Meanwhile i’ve: Swapped broken disks, switched disks for bigger ones and effectively resized my storage 2 or 3 times. Never had any corruption.

Yesterday i say a warning that one of the disks in the pool was OFFLINE.
Today i replaced it using below command’s

  • Put the disk in OFFLINE mode (if needed, mine was already offline)
    • zpool offline tank sdb
  • Remove disk from system
    • echo 1 | sudo tee /sys/block/sdb/device/delete
  • Remove the disk physically
  • Insert the replacement disk. And copy headers/structure from another disk
    • sgdisk –replicate=/dev/sdb /dev/sda
    • sgdisk –randomize-guids /dev/sdb
  • Run the zpool replace command. 
    • zpool replace tank /dev/sdb
  • Use online command to activate disk (no needed in my case, it already did that)
Tips:

# My labels with serials fell off :(
dd if=/dev/sdb of=/dev/null -> blinky led .. 

# What is the serial?
sudo hdparm -i /dev/sdb | grep Serial

Output

root@latex:~# sgdisk --replicate=/dev/sdb /dev/sda
The operation has completed successfully.
root@latex:~# sgdisk --randomize-guids /dev/sdb
The operation has completed successfully.
root@latex:~# zpool replace tank /dev/sdb
root@latex:~# zpool status
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed May 18 11:31:21 2022
    5.64T scanned out of 14.4T at 331M/s, 7h42m to go
    1.88T resilvered, 39.16% done
config:

        NAME             STATE     READ WRITE CKSUM
        tank             DEGRADED     0     0     0
          raidz1-0       DEGRADED     0     0     0
            sda          ONLINE       0     0     0
            replacing-1  REMOVED      0     0     0
              old        REMOVED      0     0     0
              sdb        ONLINE       0     0     0  (resilvering)
            sdc          ONLINE       0     0     0

errors: No known data errors

New NFO generator using yad

old post : https://www.henriaanstoot.nl/2021/02/08/kodi-movies-and-metadata/

Added poster art generator at the bottom of this page

Made a new NFO generator to get private movies properly scraped in kodi.

What does it do?

  • Genererates NFO for Kodi
  • Goes recursive though given directories
  • Skips movies with existing nfo
  • Previews the movie using VLC (So you can see what it is you are adding information for)
  • Tries to make a nice title from filename and year
  • Gets year from metadata
  • Fills tags from yad forms
  • next version i will add a poster generator from another nfo script i made. (It allready works, but i want to add a frame selector.
  • Adds new genre/actors/places if they dont exists
#!/bin/bash
# Needs mediainfo and yad
#find /media/geisha/private/Media/video/ -type f | egrep -i "mkv$|avi$|mpg$|mp4$|ogv$" | while read ; do
find $1 -type f | egrep -i "mkv$|avi$|mpg$|mp4$|ogv$" | while read ; do
# check if nfo exists
short=$(echo  "$REPLY" | rev | cut -f-2 -d/ | rev)
file=$( echo "$REPLY" | rev | cut -f2 -d. | rev).nfo
filename=$( basename "$REPLY" | cut -f1 -d. )
if [ -f "$file" ] ; then
	echo "$file found .. so skipping"
else
	echo "Not found $file"
nohup vlc "$REPLY" &
# check if find date
year=$(mediainfo "$REPLY" | grep Encoded |head -1 2>/dev/null | awk '{ print $5 }' | cut -c-4)
yeartitle="$year - "
if [ "$year" == "" ] ; then 
	year=$(echo $file | tr -cd '2[0-9][0-9][0-9]')
	if [ ! "$year" == "" ] ; then 
		#no year in filename
		yeartitle=""
	else
		yeartitle="$year - "
	fi
	
fi
# create nfo
forminfo=$(yad --title="nfo form $short" --text="Please enter:" --form --field="Title" --field="Year" --field="Placenew" --field="Genrenew" "$yeartitle $filename" "$year" "" "" --form --columns=2 --item-
separator="," --field="Place":CB --field="Genre":CB "$(paste -s -d"," < places)" "$(paste -s -d"," < genres)")
#echo $forminfo
#2020 - |2020|placenem|genrene|Hilversum|Vacation|
title=$(echo $forminfo | cut -f1 -d\|)
year=$(echo $forminfo | cut -f2 -d\|)
placenew=$(echo $forminfo | cut -f3 -d\|)
genrenew=$(echo $forminfo | cut -f4 -d\|)
place=$(echo $forminfo | cut -f5 -d\|)
genre=$(echo $forminfo | cut -f6 -d\|)
# Plot
plot=$(yad --form --field="Text::TXT" --geometry="600x200")
#plot=$( echo $plot | cut -f1 -d\\)
#plot=$( echo $plot | cut -f1 -d|)
# Actors
actors=$(yad --list  --geometry="200x480"  --print-all --column= --column=:chk --column= --column=:chk $( cat actors | cut -f1 -d" "  |while read user; do echo -n "$user false " ;done ) )
echo $actors | grep "NEW|TRUE" >/dev/null 
if [ $? -eq 0 ] ; then
vi actors
actors=$(yad --list  --geometry="200x480"  --print-all --column= --column=:chk --column= --column=:chk $( cat actors  | cut -f1 -d" " |while read user; do echo -n "$user false " ;done ) )
fi
if [ ! "$placenew" == "" ] ; then
	place="$placenew"
	echo "$placenew" >> places
fi
if [ ! "$genrenew" == "" ] ; then
	genre="$genrenew"
	echo "$genrenew" >> genres
fi
(
echo '<?xml version="1.0" encoding="utf-8" standalone="yes"?>'
echo "<movie>"
echo "  <plot>$plot</plot>"
echo "  <outline />"
echo "  <title>$title</title>"
echo "  <year>$year</year>"
echo "  <country>$place</country>"
cat actors | cut -f1 -d" " | while read actorname ; do
	echo $actors | grep "$actorname|TRUE"  >/dev/null
		if [ $? -eq 0 ] ; then
		name=$(grep $actorname actors | cut -f2 -d\" )
		echo "<actor>"
		echo "  <name>$actorname</name>"
		echo "  <role>$name</role>"
		echo "</actor>"
	fi
done
echo "  <genre>$genre</genre>"
echo "  <art>"
echo "  </art>"
echo "</movie>" 
) > "$file"  
fi
killall vlc
done

And some files

> actors
Firstname "Firstname Lastname"
> genres
Vacation
Pruts
> places
Netherlands

Poster generator (select poster thumbnail to use)

Generates 3 poster images and 3 thumbs to select.
Thumbs are 10 seconds into the movieclip, midway and on 2/3

short=$(basename $1 | cut -f1 -d.)

three=$(( $(mediainfo --Inform="Video;%Duration%" $1) / 3000 ))
rm -f thumb*.jpg poster*.jpg
first=10
#ffmpeg -ss $first -i $1 -vf scale=320:-1 -frames:v 1 -q:v 2 thumb1.jpg  -hide_banner -loglevel error
ffmpeg -ss $first -i $1 -frames:v 1 -q:v 2 thumb1.jpg  -hide_banner -loglevel error
second=$(( $three  ))
ffmpeg -ss $second -i $1 -frames:v 1 -q:v 2 thumb2.jpg  -hide_banner -loglevel error
third=$(( $three * 2 ))
ffmpeg -ss $third -i $1 -frames:v 1 -q:v 2 thumb3.jpg  -hide_banner -loglevel error
convert thumb1.jpg -resize x1000 -gravity center -crop 666x1000 poster1.jpg
convert thumb1.jpg -resize x320 -gravity center -crop 212x320 thumb1s.jpg
convert thumb2.jpg -resize x1000 -gravity center -crop 666x1000  poster2.jpg
convert thumb2.jpg -resize x320 -gravity center -crop 212x320 thumb2s.jpg
convert thumb3.jpg -resize x1000 -gravity center -crop 666x1000  poster3.jpg
convert thumb3.jpg -resize x320 -gravity center -crop 212x320 thumb3s.jpg


out=$(yad --form --title="Select poster thumbnail"  --geometry="+0+0" \
       --columns="3" \
       --field="!thumb1s-1.jpg! :fbtn" "echo poster1-1.jpg"  \
       --field="!thumb2s-1.jpg! :fbtn" "echo poster2-1.jpg"  \
       --field="!thumb3s-1.jpg! :fbtn" "echo poster3-1.jpg"
)
if [ ! "$out" == "|||" ] ; then
choice=$(echo $out | sed s/\ \|\|\|//g | rev | cut -f1 -d" " | rev)
echo cp $choice ${short}-poster.jpg
fi

Add effect to movie using bash

mkdir -p videoframessource videoframes
ffmpeg -r 60 -i sweden.mp4 videoframessource/%05d.png
cd videoframessource
ls *png | while read file ; do ../convert2comic $file ; echo -n "." ;done ; echo ""
ffmpeg -f image2 -r 60  -i videoframes/%05d.png  -vcodec libx264 final.mp4
#!/bin/bash
# convert script .. call me convert2comic
convert -quiet $1 +repage -depth 8 -selective-blur 0x5+10% ./tmp1
convert ./tmp1 -level 0x80% -colorspace gray -posterize 6 -depth 8 -colorspace sRGB ./tmp2
convert ./tmp1 '(' ./tmp2 -blur 0x1 ')' '(' -clone 0 -clone 1 -compose over -compose multiply -composite -modulate 100,150,100 ')' '(' -clone 0 -colorspace gray ')' '(' -clone 3 -negate -blur 0x2 ')' '(' -c
lone 3 -clone 4 -compose over -compose colordodge -composite -evaluate pow 4 -threshold 90% -median 1 ')' -delete 0,1,3,4 -compose over -compose multiply -composite ../videoframes/$1
# Another nice effect
convert frame.png \( +clone -negate -blur 0x6 \) -compose ColorDodge -composite -modulate 100,0,100 out.png

Mqtt Bash Nodered Notify

When running scripts which take a long time, i don’t want to wait for things to finish before i can start the next one.

For example, using my dedup script or compiling stuff. I wanna know when it is finished.

So i made some scripts

Maybe you can hear the spoken text in the background playing downstairs

I’ve put a function in .bashrc, so i can use a command like
notify “Compiling is ready”
A command like this i can put at the end of a command or in a script file at the end.
make && make install && notify “compile ready”

What does it do when executed?

  • Send a mqtt message to the broker
  • Node-red will read this message and:
    • Send a message to my display on my desk – Sound and message notification. (See another post how i made this )
    • Send a message to a script on my Domoticz instance downstairs.
      • This will use a script to get a speech file from google, and play this on some small speakers in my livingroom
    • Send a pushover message to my phone
    • Display a message on my TV ( not in code below )

How?

At the end of your .bashrc

function notify() {
    if [ -z "$1" ]; then
        echo "Usage: $0 \"message\"";
        exit 1;
    fi
    mosquitto_pub -h 10.1.0.17 -t notify/bashscript -m "$1"
}

Scripts on my Domoticz instance

Python script

#!/usr/bin/python
import paho.mqtt.client as mqttClient
import time
import os
import subprocess
import shlex
Connected = False

def on_connect(client, userdata, flags, rc):
    if rc == 0:
        print("Connected to broker")
        global Connected
        Connected =True
    else:
        print("Connection failed")

def on_message(client, userdata, message):
    print "Message received: " + message.topic + " : " + message.payload
    fileName = "/home/pi/domoticz/scripts/speech" + " \"" + message.payload + "\""
    print fileName
    args = shlex.split(fileName)
    time.sleep(1)
    p = subprocess.Popen(args)


broker_address = "10.1.0.17"
port = 1883
#user = "user"
#password = "password"
client = mqttClient.Client("speechcmd")
#client.username_pw_set(user, password=password)
client.on_connect = on_connect
client.on_message = on_message

client.connect(broker_address, port=port)
client.loop_start()

while Connected != True:
    time.sleep(0.1)

client.subscribe('speech/cmd')

try:
    while True:
        time.sleep(1)

except KeyboardInterrupt:
    print "exiting"
    client.disconnect()
    client.loop_stop()

Caching speech script

This script will look for a cached audiofile with requested text, and uses that. Else it wil request a audio file from google, caches it and plays it though the speakers.

#!/bin/bash
INPUT=$*
input2=$(echo $INPUT | base64)
echo "$input2 = $INPUT" >> /home/pi/cache/files-text-relation
if [ -f /home/pi/cache/$input2.mp3 ] ; then
mpg123 -q /home/pi/cache/$input2.mp3 1>/dev/null 2>/dev/null
else
echo not cached
STRINGNUM=0

ary=($INPUT)
for key in "${!ary[@]}"
  do
    SHORTTMP[$STRINGNUM]="${SHORTTMP[$STRINGNUM]} ${ary[$key]}"
    LENGTH=$(echo ${#SHORTTMP[$STRINGNUM]})
    #echo "word:$key, ${ary[$key]}"
    #echo "adding to: $STRINGNUM"
    if [[ "$LENGTH" -lt "100" ]]; then
      #echo starting new line
      SHORT[$STRINGNUM]=${SHORTTMP[$STRINGNUM]}
    else
      STRINGNUM=$(($STRINGNUM+1))
      SHORTTMP[$STRINGNUM]="${ary[$key]}"
      SHORT[$STRINGNUM]="${ary[$key]}"
    fi
done

for key in "${!SHORT[@]}"
  do
    echo "Playing line: $(($key+1)) of $(($STRINGNUM+1))"
    NEXTURL=$(echo ${SHORT[$key]} | xxd -plain | tr -d '\n' | sed 's/\(..\)/%\1/g')
echo $NEXTURL
    mpg123 -w $input2 -q "http://translate.google.com/translate_tts?ie=UTF-8&client=tw-ob&q=$NEXTURL&tl=En-us"
    ffmpeg -i $input2 -codec:a libmp3lame -qscale:a 2 /home/pi/cache/$input2.mp3
mpg123 /home/pi/cache/$input2.mp3
done
fi

Node-red flow

function notify() {
    if [ -z "$1" ]; then
        echo "Usage: $0 \"message\"";
        exit 1;
    fi
    mosquitto_pub -h 10.1.0.17 -t notify/bashscript -m [
    {
        "id": "1442fca698589679",
        "type": "mqtt in",
        "z": "cb6f001b.721c3",
        "name": "",
        "topic": "notify/bashscript",
        "qos": "2",
        "datatype": "auto",
        "broker": "8c74c5f6.9a7a48",
        "nl": false,
        "rap": false,
        "inputs": 0,
        "x": 180,
        "y": 580,
        "wires": [
            [
                "ddf5744bb5b73d4d",
                "faa5c794652d7a57",
                "b4e0107399248fea",
                "443f960b5d1cf40e"
            ]
        ]
    },
    {
        "id": "ddf5744bb5b73d4d",
        "type": "mqtt out",
        "z": "cb6f001b.721c3",
        "name": "",
        "topic": "speech/cmd",
        "qos": "",
        "retain": "",
        "broker": "8c74c5f6.9a7a48",
        "x": 590,
        "y": 560,
        "wires": []
    },
    {
        "id": "e95e828451d83158",
        "type": "comment",
        "z": "cb6f001b.721c3",
        "name": "bash notify",
        "info": "",
        "x": 170,
        "y": 540,
        "wires": []
    },
    {
        "id": "faa5c794652d7a57",
        "type": "mqtt out",
        "z": "cb6f001b.721c3",
        "name": "",
        "topic": "mqttlcd/message",
        "qos": "",
        "retain": "",
        "broker": "8c74c5f6.9a7a48",
        "x": 570,
        "y": 720,
        "wires": []
    },
    {
        "id": "b4e0107399248fea",
        "type": "delay",
        "z": "cb6f001b.721c3",
        "name": "",
        "pauseType": "delay",
        "timeout": "5",
        "timeoutUnits": "seconds",
        "rate": "1",
        "nbRateUnits": "1",
        "rateUnits": "second",
        "randomFirst": "1",
        "randomLast": "5",
        "randomUnits": "seconds",
        "drop": false,
        "outputs": 1,
        "x": 480,
        "y": 640,
        "wires": [
            [
                "ac4faf30b8adbe3f"
            ]
        ]
    },
    {
        "id": "ac4faf30b8adbe3f",
        "type": "function",
        "z": "cb6f001b.721c3",
        "name": "Empty payload",
        "func": "msg.payload = \"\";\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 640,
        "y": 640,
        "wires": [
            [
                "faa5c794652d7a57"
            ]
        ]
    },
    {
        "id": "dfbe26c12fc5e742",
        "type": "pushover",
        "z": "cb6f001b.721c3",
        "name": "Alleen Henri",
        "device": "rmx1931",
        "title": "Node-Red-Pushover",
        "priority": "1",
        "sound": "pushover",
        "url": "",
        "url_title": "",
        "html": false,
        "x": 850,
        "y": 500,
        "wires": []
    },
    {
        "id": "443f960b5d1cf40e",
        "type": "function",
        "z": "cb6f001b.721c3",
        "name": "Set pushover payload",
        "func": "\nmsg.topic = \"Bash Notify\";\nmsg.priority = 1;\nmsg.sound = \"cosmic\";\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 660,
        "y": 500,
        "wires": [
            [
                "dfbe26c12fc5e742"
            ]
        ]
    },
    {
        "id": "8c74c5f6.9a7a48",
        "type": "mqtt-broker",
        "name": "10.1.0.17",
        "broker": "10.1.0.17",
        "port": "1883",
        "clientid": "",
        "usetls": false,
        "compatmode": true,
        "keepalive": "15",
        "cleansession": true,
        "birthTopic": "",
        "birthQos": "0",
        "birthPayload": "",
        "closeTopic": "",
        "closePayload": "",
        "willTopic": "",
        "willQos": "0",
        "willPayload": ""
    }
]


Dedup script v0.2

Update 20220510

Sorting out my fileserver, i had the need for a deduplication script.
Many files i’ve been copying from backup, clouds mobile devices and workstation. Inevitable to get many copies.

Below script walks a directory, using locate it tries to find files with same name. Using a md5sum it wil check if it is the same file, when found a simular file it stops searching, removes the one from the check-directory and checks the next one.

#!/bin/bash
# Copy this script to your to clean directory, 
# when you got a copy on your fileserver from this script
# then the copy in your clean dir will be removed also.
# Dont want that? change
# find -type f | 
# into
# find -type f | grep -v &lt;nameofthisscript&gt; |

# dont is current directory, skip these from locations
dont=$(pwd)
# Never start in /mnt ? uncomment below
# echo "$dont" | grep "^/mnt" &amp;&amp; ( echo "start in tank" ; exit )

find -type f | while read file ; do
        filemd5=$(md5sum "$file" | cut -f1 -d" ")
        basenamefile=$(basename "$file")
        echo "searching $basenamefile"
        locate -i "/$basenamefile" | grep -v "$dont" | while read location ; do
        if [ -f "$location" ] ; then
                locatedfilemd5sum=$(md5sum "$location" | cut -f1 -d" ")
                if [ "$filemd5" == "$locatedfilemd5sum" ] ; then
                        echo "found same md5sum at $location"
                        rm "$file"
                        break
                fi
        fi
        done
done
# Remove empty dirs?
# find . -type d -empty -delete

Locate can be slow, sometimes it is better to put the locate DB in memory of on another fast storage system.

mkdir /ramdisk
mount ramfs -t ramfs /ramdisk/
cp /var/lib/mlocate/mlocate.db /ramdisk/ 

# change above script locate command
locate -d /var/lib/mlocate/mlocate.db -i IMG20191123.jpg

And remove empty directories?
Add below at the end of the script

find . -type d -empty -delete

Gluster and Rhev scripts

List Ovirt/RHV VM’s

( curl -s -u admin@internal:secretpassword -k https://ovirt-engine.lab/ovirt-engine/api/vms/ | grep "^        <name>" | cut -f2 -d\> | cut -f1 -d \< | while read;  do
echo "$REPLY"
done ) > list
echo "NOT IN BACKUP"
cat list | while read ; do ls vmbackup/$REPLY.ova 2>/dev/null >/dev/null || echo $REPLY ;done
echo ""
ls vmbackup/*ova | while read ; do
host=$( echo $REPLY | cut -f2 -d/ | cut -f1 -d.)
age=$(stat -c %y $REPLY  | cut -f1 -d" ")
echo "$age $host"
done

Backup: (UPDATE)
While i made my own backup script for Ovirt, i found this ansible based one.
https://github.com/silverorange/ovirt_ansible_backup

Backup Ovirt Engine using ansible

Usage : ansible-playbook -i ovirt-engine.lab, playbook.yml -u root (-k)

## Playbook
---
- hosts: all
  vars:
    ovirt_backup_mode: 'backup'
    ovirt_backup_archive: '/tmp/engine-backup.gzip'
    ovirt_backup_log_file: '/tmp/engine-backup.log'
    ovirt_backup_scope: 'all'
    ansible_python_interpreter: /usr/bin/python3
  roles:
      - ovirt-engine-backup

## roles/ovirt-engine-backup/tasks/main.yml
---
- name: run engine-backup
  shell: 'engine-backup --mode={{ ovirt_backup_mode }} --file={{ ovirt_backup_archive }} --log={{ ovirt_backup_log_file }} --scope={{ ovirt_backup_scope }}'
  tags:
    - skip_ansible_lint
  register: ovirt_backup_status

- name: download engine backup file
  fetch:
    src: "{{ ovirt_backup_archive }}"
    dest: "{{ ovirt_backup_archive }}"
  when: ovirt_backup_status is succeeded

- name: download log file
  fetch:
    src: "{{ ovirt_log_file }}"
    dest: "{{ ovirt_log_file }}"
  when: ovirt_backup_status is failed

Gluster overview

#!/bin/bash
ansible -i inventory all  -m shell -a "/sbin/gluster v info" --become  -K > output
cat output  | grep "^Volume Name"  | sort | cut -c14- | sort | uniq
cat output  | egrep "^Volume Name|^Brick|^Type"   | grep -v Bricks

Gluster to grafana using influx

#!/bin/bash
export PATH=$PATH:/sbin
(
# Get short shotname
host=$(echo $HOSTNAME | cut -f1 -d.)

echo "-- vg info --"
# Get vg info - pool nog doen
vgs --units k | egrep "brick|data0" | while read brickinfo ; do
name=$(echo $brickinfo | awk '{ print $1}'  )
size=$(echo $brickinfo | awk '{ print $6}' | cut -f1 -d.)
free=$(echo $brickinfo | awk '{ print $7}' | cut -f1 -d. | sed s/k//g)
usage=$(echo $(( $size - $free )))
curl -i -XPOST 'http://grafanaserver:8086/write?db=gluster' --data-binary "$host-$name size=$size,free=$free,usage=$usage"
done
echo "-- vg info done --"
echo "-- lv info --"
# Get lv info - these are mountpoints (not bricks)
#lvs --units k | egrep "brick|data0" | grep -v pool | while read volinfo ; do
df  | egrep "brick|data0" | while read volinfo ; do
name=$(echo $volinfo | awk '{ print $1}' | rev | cut -f1 -d/ | rev  )
size=$(df | grep -v svg |grep "$name" | awk '{ print $2}')
usage=$(df | grep -v svg |grep "$name" | awk '{ print $3}')
free=$(df | grep -v svg |grep "$name" | awk '{ print $4}')
curl -i -XPOST 'http://grafanaserver:8086/write?db=gluster' --data-binary "$host-$name size=$size,free=$free,usage=$usage"
done

echo "-- lv info done --"

# Get gluster volumes without gluster command, due to issues running multiple instances
# One volume on one mountpoint ? Then df == volume metrics
: > /tmp/vollist
find /var/lib/glusterd/| grep $HOSTNAME | grep vol$ | while read brickpath ; do
grep brick-path $brickpath | awk '{print $3 }' >> /tmp/vollist
done
mount | egrep "brick|data0" | awk '{print $3 }' | while read mountpoint ; do
count=$(grep $mountpoint /tmp/vollist | wc -l)
if [ $count -lt 2 ] ; then
echo "lv = brick .. df info"
#brick=$(echo $mountpoint)
#volume=$(echo $mountpoint |rev | cut -f1 -d/ | rev)
#size=$(df $mountpoint | grep -v Filesystem | awk '{ print $2 }')
#free=$(df $mountpoint | grep -v Filesystem | awk '{ print $4 }')
#usage=$(df $mountpoint | grep -v Filesystem | awk '{ print $5 }' | sed s/%//g )


dfinfo=$(df $mountpoint | grep -v ^Filesystem)
name=$(echo $dfinfo | awk '{print $1}' | rev | cut -f1 -d/ | rev)
size=$(echo $dfinfo | awk '{print $2}')
usage=$(echo $dfinfo | awk '{print $3}')
free=$(echo $dfinfo | awk '{print $4}')
perc=$(echo $dfinfo | awk '{print $5}' | sed s/%//g)
curl -i -XPOST 'http://grafanaserver:8086/write?db=gluster' --data-binary "$host-$name size=$size,free=$free,usage=$usage"
#curl -i -XPOST 'http://grafanaserver:8086/write?db=gluster' --data-binary "$host-$brick-$volume size=$size,free=$free,usage=$usage"
else
echo "du script draaien voor $mountpoint"
fi

done

echo "-- data% --"
lvs --units k -a -oname,vg_name,size,vgfree,data_percent,pool_lv | grep -v \\[ | egrep  "brick|data" | grep -v sysdata | grep -v "^  thin" | while read ; do
vg=$(echo $REPLY | awk '{print $2}')
name=$(echo $REPLY | awk '{print $6"-"$1}')
size=$(echo $REPLY | awk '{ print $3 }' | cut -f1 -d.)
free=$(echo $REPLY | awk '{ print $4 }' | cut -f1 -d. | sed s/k//g)
dataperc=$(echo $REPLY | awk '{ print $5 }' | cut -f1 -d. )
if [ "w$dataperc" == "w" ] ; then dataperc=0 ;fi
curl -i -XPOST 'http://grafanaserver:8086/write?db=gluster' --data-binary "$host-$name-$vg size=$size,free=$free,dataperc=$dataperc"
done


) 2> /dev/null >/dev/null

Running Ovirt and want a overview of glustermounts in VMs?

run makeinventory.sh (this creats an list of linux hosts which are UP in RHV/Ovirt)

#!/bin/bash
#ik haal een lijst op van virtual machines die aanstaan en linux draaien
#set -x
mkdir -p tmp
rm -f tmp/linuxup
read -p "Username: " x2
read -s -p "Password: " x1
echo " "
curl -s -X GET -H "Accept: application/xml"  --insecure -u $x2@radlan:$x1 https://ovirtserver/ovirt-engine/api/vms | grep "vm href=" | cut -c37-72 > tmp/vms
cat tmp/vms | while read vm ; do
echo -n "."
curl -s -X GET -H "Accept: application/xml"  --insecure -u $x2@radlan:$x1 https://ovirtserver/ovirt-engine/api/vms/$vm > tmp/$vm
done
echo " "
: > tmp/linuxup
find tmp -type f -mtime 0 | egrep -v "vms$|linuxup" | while read ; do

name=$(cat $REPLY | grep -i "^    <name>"| cut -f2 -d\> | cut -f1 -d \<)
os=$(cat $REPLY | egrep "^        <family>" | egrep -v "svw|SVNT|debian|ubuntu|server|spice|vnc|windows_|native" | cut -f2 -d\> | cut -f1 -d \<)
up=$(cat $REPLY |  grep -A1 start_time | tail -1| cut -f2 -d\> | cut -f1 -d \<)


echo $name,$os,$up | grep -v ",," | grep ",up" >> tmp/linuxup

cat tmp/linuxup
echo ""
echo "staat in tmp/linuxup"

done
echo "[all]" > inventory
cat tmp/linuxup | grep "."| grep -i linux | cut -f1 -d, >> inventory

Run get.sh
* ansible on all hosts (inventory) and fetches gluster mounts
* reorders this info
* generates a dot file (graphviz)
* running dot fot a svg/png output for every gluster host

#!/bin/bash
echo "draai eerst makeinventory.sh voor een nieuwe host lijst"
read d
echo "Kijk ook of ansible alle servers kan bereiken"
read d
ansible -i inventory all -m shell -a cat /etc/mtab | grep gluster | while read; do source=$(echo $REPLY | cut -f1 -d" ");dest=$(echo $REPLY | cut -f2 -d" ") ; echo " " ;echo "$source $HOSTNAME:$dest" ;done > gluster
cat gluster  | grep ^s | grep : > gluster.grep

cat gluster.grep  | cut -f1 -d: | sort | uniq | while read gluster ; do

cat header > $gluster.dot
cat gluster.grep | grep $gluster | cut -f1 -d" "| sort | uniq | while read ; do echo "    \"$REPLY\";" ;done >> $gluster.dot
cat middel >> $gluster.dot
cat gluster.grep | grep $gluster| sed s/^/\"/ | sed s/\ /\"\ \-\>\ \"/ | sed s/$/\"\;/ >> $gluster.dot
cat footer >> $gluster.dot
dot $gluster.dot -Tsvg >$gluster.svg
dot -Gsize=9,15\! -Gdpi=100  $gluster.dot -Tpng >$gluster.png
done

A php select page to few them

<form method="post">
<select name="volume">
    <?PHP
    foreach (glob("*.svg") as $filename) {
        echo "<option value=" . $filename .">" . $filename . "</option>";
    }
    ?>
    </select>
 <input type="submit" value="Submit">
</form>
<?PHP
    if (isset($_POST['volume'])){
            $image=$_POST['volume'];
            echo "<img src=\"$image\">";
    }
?>



example below:

Get a list of RHV snapshots and descriptions

#!/bin/bash
: > outlist
read -p  "Username: " user
read -s -p "Password: " pass

curl -X GET -H "Accept: application/xml" -u $user@internal:$pass -k  https://ovirt-engine:443/ovirt-engine/api/vms   | egrep "<name>|snapshots" | grep -v "           " | sed '$!N;s/\n/ /' > templist
cat templist  | cut -f2,3 -d\> | cut -f1,3 -d\< | sed 's/<link href=\"/ /g' | cut -f1 -d\" > templist2
cat templist2 | while read ; do
host=$(echo $REPLY | awk '{ print $1 }')
link=$(echo $REPLY | awk '{ print $2 }')
echo $host >> outlist
curl -X GET -H "Accept: application/xml" -u $user@radlan:$pass -k  https://ovirt-engine:443/$link | egrep "description|date" | grep -v '            <'  >> outlist ; echo "----" >> outlist
done
cat outlist | awk '/Active VM/ {getline; next} { print }' > /tmp/outlist
grep -B3 date /tmp/outlist  |sed -e 's/<[^>]*>//g' > /tmp/snapshots
cat /tmp/snapshots | awk '/----/ {printf "\n%s\n",$0;next} {printf "%s ",$0}' | sed 's/ --*//g' | grep -v '\-\-\-\-' | grep "  " > snapshots

Output:
svr04.lab         Backup              2019-06-28T07:55:28.909+02:00
svr05.lab         upgrade             2019-06-18T14:18:49.745+02:00
svr33.lab         pre_ansible         2019-06-18T14:04:43.569+02:00

Somewhere i've got a snapshot age warn script.
This information was posted in Mattermost when sending the "/snapshots" command

Ansible Playbook to add virtual nics

---
- name: adnic
  hosts: localhost
  gather_facts: false
  become: no


  vars_prompt:
    - name: "username"
      prompt: "IPA username"
      private: no
    - name: "password"
      prompt: "IPA password"
      private: yes
    - name: "server"
      prompt: "server"
      private: no
    - name: "interface"
      prompt: "interface"
      private: no
      default: 'eth1'
    - name: "nicprofile"
      prompt: "dmz or other"
      private: no
      default: 'dmz'

  tasks:
  - name: Obtain SSO token with using username/password credentials
    ovirt_auth:
      url: https://ovirt-engine.lab/ovirt-engine/api
      username: "{{ username }}@internal"
      insecure: yes
      password: "{{ password }}"
    
  - ovirt_nics:
      auth: "{{ ovirt_auth }}"
      vm: "{{ server }}"
      state: "present"
      name: "{{ interface }}"
      network: "{{ nicprofile }}"
      profile: "{{ nicprofile }}"

Ansible playbook to create snapshots

- name: makesnap
  hosts: localhost
  gather_facts: false

  vars_prompt:
    - name: "username"
      prompt: "IPA username"
      private: no
    - name: "password"
      prompt: "IPA password"
      private: yes
    - name: "hostname"
      prompt: "hostname"
      private: no
    - name: "snapshotname"
      prompt: "snapshotname"
      private: no

  tasks:
  - name: Make snapshot
    ovirt_snapshots:
      auth:
        username: "{{ username }}@internal"
        password: "{{ password }}"
        insecure: True
        url: https://ovirt-engine.lab/ovirt-engine/api
      description: "{{ snapshotname }}"
      use_memory: true
      state: present
      timeout: 600
      vm_name: "{{ hostname }}"

Scripts on my domoticz instance

Script NO1:
Below is a script which checks all sensors and switches available on my domoticz instance, and gives me information about last updates.
For example when a device is out of reach or battery empty.

Usage:

Outputs to console, or you can use it in check_mk monitoring.
(For the latter, create a script

vi /usr/lib/check_mk_agent/local/checkfrontdoor
chmod +x /usr/lib/check_mk_agent/local/checkfrontdoor
put in the script 
#!/bin/bash
cd /path/to/script/
./belowscript "Frontdoor" 300 checkmk

Code:

#!/bin/bash
#set -x
# after running once check stateseconds for names
if [ $# -eq 0 ]; then
echo "$(basename $0) - without options .. getting states"
echo "use $(basename $0) "Sensorname" seconds" for check
echo "Getting all states"
: > stateseconds
curl -s -i -H "Accept: application/json" "http://127.0.0.1:8080/json.htm?type=devices&filter=all&used=true&order=Name" | egrep "Name|Last" | grep -v HardwareName |grep -vi levelnames> states
sed -i 'N;s/\n/,/' states
now=$(date +%s)
cat states  | awk '{ print $3" "$4$7$8$9$10$11$12 }' | sed s/,,/,/g |rev | cut -c2- | rev | while read ; do
name="$(echo $REPLY | cut -f2 -d, )"
dater="$(echo $REPLY | cut -f1 -d, | sed s/\"//g)"
#echo $dater
seen=$(date -d "$dater" +%s)
#echo $seen
echo "$name $(( $now - $seen))" >> stateseconds
echo -n "."
done
fi
echo ""
if [[ ! -z "$1" && ! -z $2 ]]; then
checkold=$(cat stateseconds | grep "$1\"" | head -1 | awk '{ print $2 }')
total=$(( checkold - $2 ))
if [ -z $3 ] ; then
if [ $checkold -gt $2 ] ; then echo "$name lastseen longer than $2 seconds ago ($total sec)" ; exit 1 ;fi
else
if [ $checkold -gt $2 ] ; then echo "2 \"$1\" - Sensor older than $2 seconds" ; exit 1
else
echo "0 \"$1\" - Sensor age ok" ; exit 0
fi
fi
fi

Above to your Mqtt broker?

#!/bin/bash
# change 300 ( age below 5 minutes )
# and $2>=900 ( age above 15 minutes )
./agecheck.sh # Updates statefile
cat stateseconds | tr -d '"-/:"_' | sort -n -k2 | awk '$2<=300' | while read; do
sensorname=$(echo $REPLY | awk '{ print $1 }' )
age=$(echo $REPLY | awk '{ print $2 }' )
mosquitto_pub -h 10.1.0.17 -t sensorsage/$sensorname -m "$age"
done

Script NO:2
This is a mqtt speech script with caching sounds, speech generated for me by google

MQTT Part

#!/usr/bin/python
import paho.mqtt.client as mqttClient
import time
import os
import subprocess
import shlex
Connected = False

def on_connect(client, userdata, flags, rc):
    if rc == 0:
        print("Connected to broker")
        global Connected
        Connected =True
    else:
        print("Connection failed")

def on_message(client, userdata, message):
    print "Message received: " + message.topic + " : " + message.payload
    fileName = "/home/pi/domoticz/scripts/speech" + " \"" + message.payload + "\""
    print fileName
    args = shlex.split(fileName)
    time.sleep(1)
    p = subprocess.Popen(args)


broker_address = "mqttbroker"
port = 1883
user = "myuser"
password = "mysecretpass"
client = mqttClient.Client("speechcmd")
client.username_pw_set(user, password=password)
client.on_connect = on_connect
client.on_message = on_message

client.connect(broker_address, port=port)
client.loop_start()

while Connected != True:
    time.sleep(0.1)

client.subscribe('speech/cmd')

try:
    while True:
        time.sleep(1)

except KeyboardInterrupt:
    print "exiting"
    client.disconnect()
    client.loop_stop()

Speech part (uses mpg123 to play sound though speaker)

#!/bin/bash
#set -x

INPUT=$*
input2=$(echo $INPUT | base64)
echo "$input2 = $INPUT" >> /home/pi/cache/files-text-relation
if [ -f /home/pi/cache/$input2.mp3 ] ; then
mpg123 -q /home/pi/cache/$input2.mp3 1>/dev/null 2>/dev/null
else
echo not cached
STRINGNUM=0

ary=($INPUT)
for key in "${!ary[@]}"
  do
    SHORTTMP[$STRINGNUM]="${SHORTTMP[$STRINGNUM]} ${ary[$key]}"
    LENGTH=$(echo ${#SHORTTMP[$STRINGNUM]})
    #echo "word:$key, ${ary[$key]}"
    #echo "adding to: $STRINGNUM"
    if [[ "$LENGTH" -lt "100" ]]; then
      #echo starting new line
      SHORT[$STRINGNUM]=${SHORTTMP[$STRINGNUM]}
    else
      STRINGNUM=$(($STRINGNUM+1))
      SHORTTMP[$STRINGNUM]="${ary[$key]}"
      SHORT[$STRINGNUM]="${ary[$key]}"
    fi
done

for key in "${!SHORT[@]}"
  do
    #echo "line: $key is: ${SHORT[$key]}"

    echo "Playing line: $(($key+1)) of $(($STRINGNUM+1))"
    NEXTURL=$(echo ${SHORT[$key]} | xxd -plain | tr -d '\n' | sed 's/\(..\)/%\1/g')
echo $NEXTURL
    mpg123 -w $input2 -q "http://translate.google.com/translate_tts?ie=UTF-8&client=tw-ob&q=$NEXTURL&tl=En-us"
    ffmpeg -i $input2 -codec:a libmp3lame -qscale:a 2 /home/pi/cache/$input2.mp3
mpg123 /home/pi/cache/$input2.mp3
done
fi

set msg.payload to “Doorbell”
and Mqtt topic to “speech/cmd”

To be continued !

FFmpeg tricks

Mp4 to gif (resize, cut and convert)

startsecond=0
duration=3
ffmpeg -ss $startsecond -t $duration -i 'input.mp4' -vf "fps=10,scale=320:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" -loop 0 output.gif

Gif to MP4

ffmpeg -i kanban.gif -movflags faststart -pix_fmt yuv420p -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" gifkanban.mp4

Converting MP4 for Davinci MOV

ffmpeg -i input.mp4 -c:v dnxhd -profile:v dnxhr_hq -pix_fmt yuv422p -c:a pcm_s16le -f mov output.mov

And back from Davinci to MP4

ffmpeg -i render.mov -c:v libx264 -pix_fmt yuv420p -crf 16 -force_key_frames 'expr:gte(t,n_forced/2)' -bf 2 -vf yadif -use_editlist 0 -movflags +faststart -c:a aac -q:a 1 -ac 2 -ar 48000 -f mp4 out.mp4

QTVR to png (note number of frames)

ffmpeg -i ../test.mov %02d.png
ffmpeg -i %02d.png -vf "tile=1x24,transpose=1" mh.png

Nikon MOV to mp4 (no reencoding)

ffmpeg -i input.mov -c copy -movflags +faststart  output.mp4

Nikon MOV to mp4 (problems with audio?
“Could not find tag for codec pcm_s16le in stream #1, codec not currently supported in container”

ffmpeg -i input.mov -c copy -c:a aac -b:a 128k -movflags +faststart  output.mp4
Maybe add -strict experimental 

MPG to MP4 (Not happy with this one)

ffmpeg -i vhs-rip.mpg"  -c:v libx264 -crf 10 -strict -2 1993-luv.mp4

FLV to MP4

ffmpeg -i filename.flv -c:v libx264 -crf 19 -strict experimental filename.mp4

Lossless cut with ffmpeg

ffmpeg  -ss 00:$2 -t 00:$3 -i $1 -vcodec copy -an cut_$1  
# an = no sound
# use -acodec copy for same codec as original audio

Timelapse JPG to MP4

ffmpeg -f image2 -r 24 -start_number 8296 -i "750_%04d.JPG"  -vcodec libx264 -profile:v high444 -refs 16 -crf 0 -preset ultrafast -vf scale=1920:1080 b.mp4

png to gif from a list

#cat png.list
file off.png
duration 1
file 20.png
duration 1
file 40.png
duration 1
file 60.png
duration 1
file 80.png
duration 1
file 90.png
duration 1
file 100.png
duration 1

# ffmpeg -f concat -i png.list -loop 0 hexlight.gif


Rotate mp4 clockwise/anticlockwise

ffmpeg -i VID20230110132531.mp4 -filter:v transpose=2 -c:v libx264 -preset veryfast -crf 22 -c:a copy -metadata:s:v rotate="" test.mp4

Concat video together

ffmpeg -f concat -i inputs.txt -c copy merge.mp4
inputs.txt
file start.mp4
file end.mp4


Error “width/height not divisible by 2 (1301×933)”

ffmpeg -i input.mp4 -vf "crop=trunc(iw/2)*2:trunc(ih/2)*2" output.mp4

Dvd/DV/Mpg etcetera overviews

I made two scripts, which take a movie, find out its lenght divide by 9. And generates a image montage of snapshots at certain times in the movie. ( A movie of 90 minutes, gets a snapshot every 10 minutes )

Example of snapshots of a movie

Below takes images as above and convert them into animated gifs. All movies in a path or directory structure gets a index page containing all animated gifs.

Example of generated directory index

I’ll post something about my media sorter also.
A ajax website using drag drop viewing/sorting images (png/gifs/jpgs) and movies.

#!/bin/bash
# Uses montage, ffmpeg, convert and mediainfo
rm thumbs/*
rm montage/*
(
set -x
numberofframes=9
find  -type f | egrep -i "flv$|mov$|mkv$|avi$|mpg$|mp4$|VTS_01_1.VOB$|wmv$" |  while read filez; do
        crunchpath=$(echo $filez | sed "s,/,,g" |  sed "s, ,,g" |tr -cd '[:alnum:]' ).gif
        crunchback=$crunchpath
        echo $crunchpath
        if [ -f thumbs/$crunchpath ] ; then
                echo "$file exists"
                else
	mkdir /tmp/thumbs/ 2>/dev/null
	mkdir thumbs 2>/dev/null
	mkdir montage 2>/dev/null
        rm -f /tmp/thumbs/*png
sec=0
                totaltime1=$( mediainfo -f "$filez" | grep ^Dura  | awk '{print $3}' |grep -v "\:" | tail -1 )
                totaltime2=$( mediainfo -f "$filez" | grep ^Dura  | awk '{print $4}' |grep -v "\:" | tail -1 )
totaltime1a=$(echo $totaltime1| tr -d 'a-z')
echo $totaltime1 | grep h && sec=$((3600 * $totaltime1a))
echo $totaltime1 | grep mn && sec=$((60 * $totaltime1a))
echo $totaltime1 | grep s && sec=$totaltime1a
echo "sec : $sec"
echo $totaltime2 | grep -v ms | grep  s && sec=$[ $sec + $(echo $totaltime2 | tr -d 'a-z')]
echo "sec : $sec"
                width=$( mediainfo -f "$filez" | grep "Width"| grep pixels | awk '{ print $3 }' )
                height=$( mediainfo -f "$filez" | grep "Height"| grep pixels | awk '{ print $3 }' )
        if [ "$sec" == "" ] ; then sec=1 ; fi
                timer=$[ $sec / $numberofframes ]
                        echo  "converting step "
                        echo  "timer is $timer"
mark=0
                for f in `seq -w 1 $numberofframes` ; do
                        echo  "$f,$timer"
			echo $[$f*$timer]
                        ffmpeg -ss $[$f*$timer] -i "$filez" -vframes 1 /tmp/thumbs/$f.png 1>/dev/null 2>/dev/null </dev/null
if [ $f -eq 8 ] ; then cp /tmp/thumbs/8.png /tmp/thumbs9.png ;fi
newsizer=$(echo -n $widther ; echo -n "x" ; echo -n $sizeh)
if [ $sizeh -gt $sizew ] ; then mv /tmp/thumbs/$f.png /tmp/ff.png ; convert -resize $newsizer\! /tmp/ff.png /tmp/thumbs/$f.png ; echo aangepast ; echo $newsizer ; mark=1; fi
                done
                        echo " "
                        echo "generating $crunchpath"
	newwidth=$[ $width / 2 ]
	newheight=$[ $height / 2 ]
	newsize=$(echo -n $newwidth ; echo -n "x" ; echo -n $newheight)
	newsize2=$(echo -n $width ; echo -n "x" ; echo -n $newheight)
	echo $newsize
if [ $mark == 0 ] ; then        convert -delay 100 -resize $newsize\! /tmp/thumbs/*.png thumbs/$crunchpath </dev/null ; else
convert -delay 100 -resize $newsize2\! /tmp/thumbs/*.png thumbs/$crunchpath </dev/null 
fi
montage /tmp/thumbs/*png  -geometry +3+3 -tile 3x3 montage/$crunchpath.jpg
	echo "<img src=\"$crunchpath\">" >> thumbs/_index.html
        fi
done
) > /tmp/thumb.log