ZFS replace disk

I’m using ZFS for my main fileserver, this pool was created over 10 years ago.
Meanwhile i’ve: Swapped broken disks, switched disks for bigger ones and effectively resized my storage 2 or 3 times. Never had any corruption.

Yesterday i say a warning that one of the disks in the pool was OFFLINE.
Today i replaced it using below command’s

  • Put the disk in OFFLINE mode (if needed, mine was already offline)
    • zpool offline tank sdb
  • Remove disk from system
    • echo 1 | sudo tee /sys/block/sdb/device/delete
  • Remove the disk physically
  • Insert the replacement disk. And copy headers/structure from another disk
    • sgdisk –replicate=/dev/sdb /dev/sda
    • sgdisk –randomize-guids /dev/sdb
  • Run the zpool replace command. 
    • zpool replace tank /dev/sdb
  • Use online command to activate disk (no needed in my case, it already did that)
Tips:

# My labels with serials fell off :(
dd if=/dev/sdb of=/dev/null -> blinky led .. 

# What is the serial?
sudo hdparm -i /dev/sdb | grep Serial

Output

root@latex:~# sgdisk --replicate=/dev/sdb /dev/sda
The operation has completed successfully.
root@latex:~# sgdisk --randomize-guids /dev/sdb
The operation has completed successfully.
root@latex:~# zpool replace tank /dev/sdb
root@latex:~# zpool status
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed May 18 11:31:21 2022
    5.64T scanned out of 14.4T at 331M/s, 7h42m to go
    1.88T resilvered, 39.16% done
config:

        NAME             STATE     READ WRITE CKSUM
        tank             DEGRADED     0     0     0
          raidz1-0       DEGRADED     0     0     0
            sda          ONLINE       0     0     0
            replacing-1  REMOVED      0     0     0
              old        REMOVED      0     0     0
              sdb        ONLINE       0     0     0  (resilvering)
            sdc          ONLINE       0     0     0

errors: No known data errors

New NFO generator using yad

old post : https://www.henriaanstoot.nl/2021/02/08/kodi-movies-and-metadata/

Added poster art generator at the bottom of this page

Made a new NFO generator to get private movies properly scraped in kodi.

What does it do?

  • Genererates NFO for Kodi
  • Goes recursive though given directories
  • Skips movies with existing nfo
  • Previews the movie using VLC (So you can see what it is you are adding information for)
  • Tries to make a nice title from filename and year
  • Gets year from metadata
  • Fills tags from yad forms
  • next version i will add a poster generator from another nfo script i made. (It allready works, but i want to add a frame selector.
  • Adds new genre/actors/places if they dont exists
#!/bin/bash
# Needs mediainfo and yad
#find /media/geisha/private/Media/video/ -type f | egrep -i "mkv$|avi$|mpg$|mp4$|ogv$" | while read ; do
find $1 -type f | egrep -i "mkv$|avi$|mpg$|mp4$|ogv$" | while read ; do
# check if nfo exists
short=$(echo  "$REPLY" | rev | cut -f-2 -d/ | rev)
file=$( echo "$REPLY" | rev | cut -f2 -d. | rev).nfo
filename=$( basename "$REPLY" | cut -f1 -d. )
if [ -f "$file" ] ; then
	echo "$file found .. so skipping"
else
	echo "Not found $file"
nohup vlc "$REPLY" &
# check if find date
year=$(mediainfo "$REPLY" | grep Encoded |head -1 2>/dev/null | awk '{ print $5 }' | cut -c-4)
yeartitle="$year - "
if [ "$year" == "" ] ; then 
	year=$(echo $file | tr -cd '2[0-9][0-9][0-9]')
	if [ ! "$year" == "" ] ; then 
		#no year in filename
		yeartitle=""
	else
		yeartitle="$year - "
	fi
	
fi
# create nfo
forminfo=$(yad --title="nfo form $short" --text="Please enter:" --form --field="Title" --field="Year" --field="Placenew" --field="Genrenew" "$yeartitle $filename" "$year" "" "" --form --columns=2 --item-
separator="," --field="Place":CB --field="Genre":CB "$(paste -s -d"," < places)" "$(paste -s -d"," < genres)")
#echo $forminfo
#2020 - |2020|placenem|genrene|Hilversum|Vacation|
title=$(echo $forminfo | cut -f1 -d\|)
year=$(echo $forminfo | cut -f2 -d\|)
placenew=$(echo $forminfo | cut -f3 -d\|)
genrenew=$(echo $forminfo | cut -f4 -d\|)
place=$(echo $forminfo | cut -f5 -d\|)
genre=$(echo $forminfo | cut -f6 -d\|)
# Plot
plot=$(yad --form --field="Text::TXT" --geometry="600x200")
#plot=$( echo $plot | cut -f1 -d\\)
#plot=$( echo $plot | cut -f1 -d|)
# Actors
actors=$(yad --list  --geometry="200x480"  --print-all --column= --column=:chk --column= --column=:chk $( cat actors | cut -f1 -d" "  |while read user; do echo -n "$user false " ;done ) )
echo $actors | grep "NEW|TRUE" >/dev/null 
if [ $? -eq 0 ] ; then
vi actors
actors=$(yad --list  --geometry="200x480"  --print-all --column= --column=:chk --column= --column=:chk $( cat actors  | cut -f1 -d" " |while read user; do echo -n "$user false " ;done ) )
fi
if [ ! "$placenew" == "" ] ; then
	place="$placenew"
	echo "$placenew" >> places
fi
if [ ! "$genrenew" == "" ] ; then
	genre="$genrenew"
	echo "$genrenew" >> genres
fi
(
echo '<?xml version="1.0" encoding="utf-8" standalone="yes"?>'
echo "<movie>"
echo "  <plot>$plot</plot>"
echo "  <outline />"
echo "  <title>$title</title>"
echo "  <year>$year</year>"
echo "  <country>$place</country>"
cat actors | cut -f1 -d" " | while read actorname ; do
	echo $actors | grep "$actorname|TRUE"  >/dev/null
		if [ $? -eq 0 ] ; then
		name=$(grep $actorname actors | cut -f2 -d\" )
		echo "<actor>"
		echo "  <name>$actorname</name>"
		echo "  <role>$name</role>"
		echo "</actor>"
	fi
done
echo "  <genre>$genre</genre>"
echo "  <art>"
echo "  </art>"
echo "</movie>" 
) > "$file"  
fi
killall vlc
done

And some files

> actors
Firstname "Firstname Lastname"
> genres
Vacation
Pruts
> places
Netherlands

Poster generator (select poster thumbnail to use)

Generates 3 poster images and 3 thumbs to select.
Thumbs are 10 seconds into the movieclip, midway and on 2/3

short=$(basename $1 | cut -f1 -d.)

three=$(( $(mediainfo --Inform="Video;%Duration%" $1) / 3000 ))
rm -f thumb*.jpg poster*.jpg
first=10
#ffmpeg -ss $first -i $1 -vf scale=320:-1 -frames:v 1 -q:v 2 thumb1.jpg  -hide_banner -loglevel error
ffmpeg -ss $first -i $1 -frames:v 1 -q:v 2 thumb1.jpg  -hide_banner -loglevel error
second=$(( $three  ))
ffmpeg -ss $second -i $1 -frames:v 1 -q:v 2 thumb2.jpg  -hide_banner -loglevel error
third=$(( $three * 2 ))
ffmpeg -ss $third -i $1 -frames:v 1 -q:v 2 thumb3.jpg  -hide_banner -loglevel error
convert thumb1.jpg -resize x1000 -gravity center -crop 666x1000 poster1.jpg
convert thumb1.jpg -resize x320 -gravity center -crop 212x320 thumb1s.jpg
convert thumb2.jpg -resize x1000 -gravity center -crop 666x1000  poster2.jpg
convert thumb2.jpg -resize x320 -gravity center -crop 212x320 thumb2s.jpg
convert thumb3.jpg -resize x1000 -gravity center -crop 666x1000  poster3.jpg
convert thumb3.jpg -resize x320 -gravity center -crop 212x320 thumb3s.jpg


out=$(yad --form --title="Select poster thumbnail"  --geometry="+0+0" \
       --columns="3" \
       --field="!thumb1s-1.jpg! :fbtn" "echo poster1-1.jpg"  \
       --field="!thumb2s-1.jpg! :fbtn" "echo poster2-1.jpg"  \
       --field="!thumb3s-1.jpg! :fbtn" "echo poster3-1.jpg"
)
if [ ! "$out" == "|||" ] ; then
choice=$(echo $out | sed s/\ \|\|\|//g | rev | cut -f1 -d" " | rev)
echo cp $choice ${short}-poster.jpg
fi

Add effect to movie using bash

mkdir -p videoframessource videoframes
ffmpeg -r 60 -i sweden.mp4 videoframessource/%05d.png
cd videoframessource
ls *png | while read file ; do ../convert2comic $file ; echo -n "." ;done ; echo ""
ffmpeg -f image2 -r 60  -i videoframes/%05d.png  -vcodec libx264 final.mp4
#!/bin/bash
# convert script .. call me convert2comic
convert -quiet $1 +repage -depth 8 -selective-blur 0x5+10% ./tmp1
convert ./tmp1 -level 0x80% -colorspace gray -posterize 6 -depth 8 -colorspace sRGB ./tmp2
convert ./tmp1 '(' ./tmp2 -blur 0x1 ')' '(' -clone 0 -clone 1 -compose over -compose multiply -composite -modulate 100,150,100 ')' '(' -clone 0 -colorspace gray ')' '(' -clone 3 -negate -blur 0x2 ')' '(' -c
lone 3 -clone 4 -compose over -compose colordodge -composite -evaluate pow 4 -threshold 90% -median 1 ')' -delete 0,1,3,4 -compose over -compose multiply -composite ../videoframes/$1
# Another nice effect
convert frame.png \( +clone -negate -blur 0x6 \) -compose ColorDodge -composite -modulate 100,0,100 out.png

Gotek stuff

Recently back to old skool retro!

As posted https://www.henriaanstoot.nl/2022/05/03/c64-and-sd2iec/

Wellll, i bought some goodies from gotek.nl

A new SUB-D(23) to scart video cable for amiga.

A 9pindin mouse to PS/2 convertor

My tank mouse died .. beyond repair. Blue 9pins to ps2 works great!

A switchless (using key combination) diskdrive switcher df0<>df1

I used to make a floppy switch myself in the past, i used a wirewrap socket, and a cross switch. Made a hole in the back, with the little switch.
This one does not need a hole in your case for the switch.

Own floppy switch

Gotek Floppydrive Emulator with Rotary Encoder

This drive supports a lot of disk formats and systems, i will have to look into that ..
https://github.com/keirf/FlashFloppy/wiki/Host-Platforms

Commodore day

Update: 20220514 – Vic Graf cartridge and more

One Vic-20 working ( switched some keyboards and chips around )
Something i made in 1984? .. then the fuse in my vic-20 power blew (250v 160mA)
Another Vic-20 – with a Bad U31 (Oscillator)? or Vic video chip?
Vic Graf Cartridge, graph a function with annoying sound

Manual : https://archive.org/details/VIC_Graf_1982_Commodore/page/n11/mode/2up

I’ve got a load of cartridges, some of them i tested:

  • Vic-20 – Super expander plus 3K ram ( also some draw and sound functionality in this one )
  • Vic-20 – 32K Ram expander (switchable)
  • Vic-20 – 3K expander
  • C64 – KCS power cardridge
  • C64 – Final Cartridge III
  • C64 – Data Manager 2 – Data Base, hard to find information on this one, will post later on this one.

Mqtt Bash Nodered Notify

When running scripts which take a long time, i don’t want to wait for things to finish before i can start the next one.

For example, using my dedup script or compiling stuff. I wanna know when it is finished.

So i made some scripts

Maybe you can hear the spoken text in the background playing downstairs

I’ve put a function in .bashrc, so i can use a command like
notify “Compiling is ready”
A command like this i can put at the end of a command or in a script file at the end.
make && make install && notify “compile ready”

What does it do when executed?

  • Send a mqtt message to the broker
  • Node-red will read this message and:
    • Send a message to my display on my desk – Sound and message notification. (See another post how i made this )
    • Send a message to a script on my Domoticz instance downstairs.
      • This will use a script to get a speech file from google, and play this on some small speakers in my livingroom
    • Send a pushover message to my phone
    • Display a message on my TV ( not in code below )

How?

At the end of your .bashrc

function notify() {
    if [ -z "$1" ]; then
        echo "Usage: $0 \"message\"";
        exit 1;
    fi
    mosquitto_pub -h 10.1.0.17 -t notify/bashscript -m "$1"
}

Scripts on my Domoticz instance

Python script

#!/usr/bin/python
import paho.mqtt.client as mqttClient
import time
import os
import subprocess
import shlex
Connected = False

def on_connect(client, userdata, flags, rc):
    if rc == 0:
        print("Connected to broker")
        global Connected
        Connected =True
    else:
        print("Connection failed")

def on_message(client, userdata, message):
    print "Message received: " + message.topic + " : " + message.payload
    fileName = "/home/pi/domoticz/scripts/speech" + " \"" + message.payload + "\""
    print fileName
    args = shlex.split(fileName)
    time.sleep(1)
    p = subprocess.Popen(args)


broker_address = "10.1.0.17"
port = 1883
#user = "user"
#password = "password"
client = mqttClient.Client("speechcmd")
#client.username_pw_set(user, password=password)
client.on_connect = on_connect
client.on_message = on_message

client.connect(broker_address, port=port)
client.loop_start()

while Connected != True:
    time.sleep(0.1)

client.subscribe('speech/cmd')

try:
    while True:
        time.sleep(1)

except KeyboardInterrupt:
    print "exiting"
    client.disconnect()
    client.loop_stop()

Caching speech script

This script will look for a cached audiofile with requested text, and uses that. Else it wil request a audio file from google, caches it and plays it though the speakers.

#!/bin/bash
INPUT=$*
input2=$(echo $INPUT | base64)
echo "$input2 = $INPUT" >> /home/pi/cache/files-text-relation
if [ -f /home/pi/cache/$input2.mp3 ] ; then
mpg123 -q /home/pi/cache/$input2.mp3 1>/dev/null 2>/dev/null
else
echo not cached
STRINGNUM=0

ary=($INPUT)
for key in "${!ary[@]}"
  do
    SHORTTMP[$STRINGNUM]="${SHORTTMP[$STRINGNUM]} ${ary[$key]}"
    LENGTH=$(echo ${#SHORTTMP[$STRINGNUM]})
    #echo "word:$key, ${ary[$key]}"
    #echo "adding to: $STRINGNUM"
    if [[ "$LENGTH" -lt "100" ]]; then
      #echo starting new line
      SHORT[$STRINGNUM]=${SHORTTMP[$STRINGNUM]}
    else
      STRINGNUM=$(($STRINGNUM+1))
      SHORTTMP[$STRINGNUM]="${ary[$key]}"
      SHORT[$STRINGNUM]="${ary[$key]}"
    fi
done

for key in "${!SHORT[@]}"
  do
    echo "Playing line: $(($key+1)) of $(($STRINGNUM+1))"
    NEXTURL=$(echo ${SHORT[$key]} | xxd -plain | tr -d '\n' | sed 's/\(..\)/%\1/g')
echo $NEXTURL
    mpg123 -w $input2 -q "http://translate.google.com/translate_tts?ie=UTF-8&client=tw-ob&q=$NEXTURL&tl=En-us"
    ffmpeg -i $input2 -codec:a libmp3lame -qscale:a 2 /home/pi/cache/$input2.mp3
mpg123 /home/pi/cache/$input2.mp3
done
fi

Node-red flow

function notify() {
    if [ -z "$1" ]; then
        echo "Usage: $0 \"message\"";
        exit 1;
    fi
    mosquitto_pub -h 10.1.0.17 -t notify/bashscript -m [
    {
        "id": "1442fca698589679",
        "type": "mqtt in",
        "z": "cb6f001b.721c3",
        "name": "",
        "topic": "notify/bashscript",
        "qos": "2",
        "datatype": "auto",
        "broker": "8c74c5f6.9a7a48",
        "nl": false,
        "rap": false,
        "inputs": 0,
        "x": 180,
        "y": 580,
        "wires": [
            [
                "ddf5744bb5b73d4d",
                "faa5c794652d7a57",
                "b4e0107399248fea",
                "443f960b5d1cf40e"
            ]
        ]
    },
    {
        "id": "ddf5744bb5b73d4d",
        "type": "mqtt out",
        "z": "cb6f001b.721c3",
        "name": "",
        "topic": "speech/cmd",
        "qos": "",
        "retain": "",
        "broker": "8c74c5f6.9a7a48",
        "x": 590,
        "y": 560,
        "wires": []
    },
    {
        "id": "e95e828451d83158",
        "type": "comment",
        "z": "cb6f001b.721c3",
        "name": "bash notify",
        "info": "",
        "x": 170,
        "y": 540,
        "wires": []
    },
    {
        "id": "faa5c794652d7a57",
        "type": "mqtt out",
        "z": "cb6f001b.721c3",
        "name": "",
        "topic": "mqttlcd/message",
        "qos": "",
        "retain": "",
        "broker": "8c74c5f6.9a7a48",
        "x": 570,
        "y": 720,
        "wires": []
    },
    {
        "id": "b4e0107399248fea",
        "type": "delay",
        "z": "cb6f001b.721c3",
        "name": "",
        "pauseType": "delay",
        "timeout": "5",
        "timeoutUnits": "seconds",
        "rate": "1",
        "nbRateUnits": "1",
        "rateUnits": "second",
        "randomFirst": "1",
        "randomLast": "5",
        "randomUnits": "seconds",
        "drop": false,
        "outputs": 1,
        "x": 480,
        "y": 640,
        "wires": [
            [
                "ac4faf30b8adbe3f"
            ]
        ]
    },
    {
        "id": "ac4faf30b8adbe3f",
        "type": "function",
        "z": "cb6f001b.721c3",
        "name": "Empty payload",
        "func": "msg.payload = \"\";\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 640,
        "y": 640,
        "wires": [
            [
                "faa5c794652d7a57"
            ]
        ]
    },
    {
        "id": "dfbe26c12fc5e742",
        "type": "pushover",
        "z": "cb6f001b.721c3",
        "name": "Alleen Henri",
        "device": "rmx1931",
        "title": "Node-Red-Pushover",
        "priority": "1",
        "sound": "pushover",
        "url": "",
        "url_title": "",
        "html": false,
        "x": 850,
        "y": 500,
        "wires": []
    },
    {
        "id": "443f960b5d1cf40e",
        "type": "function",
        "z": "cb6f001b.721c3",
        "name": "Set pushover payload",
        "func": "\nmsg.topic = \"Bash Notify\";\nmsg.priority = 1;\nmsg.sound = \"cosmic\";\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 660,
        "y": 500,
        "wires": [
            [
                "dfbe26c12fc5e742"
            ]
        ]
    },
    {
        "id": "8c74c5f6.9a7a48",
        "type": "mqtt-broker",
        "name": "10.1.0.17",
        "broker": "10.1.0.17",
        "port": "1883",
        "clientid": "",
        "usetls": false,
        "compatmode": true,
        "keepalive": "15",
        "cleansession": true,
        "birthTopic": "",
        "birthQos": "0",
        "birthPayload": "",
        "closeTopic": "",
        "closePayload": "",
        "willTopic": "",
        "willQos": "0",
        "willPayload": ""
    }
]


Phpipam – Added Switch port Custom Field

Added custom fields to Phpipam, so i can remove my port sheet and administer everything in Phpipam. And to prepare for a Augmented reality project i want to create.

Howto add fields:
Go to the administrator menu on the right.
Select IP related management > Custom fields
Custom IP addresses fields press the plus sign
Give it a name and description.
I’m using varchar(10) at the moment
so i can enter switchname:port (media:4)

See below for a example: Using a QR code and the information, i want to display port information using a app.

Example Augmented switch ports

Dedup script v0.2

Update 20220510

Sorting out my fileserver, i had the need for a deduplication script.
Many files i’ve been copying from backup, clouds mobile devices and workstation. Inevitable to get many copies.

Below script walks a directory, using locate it tries to find files with same name. Using a md5sum it wil check if it is the same file, when found a simular file it stops searching, removes the one from the check-directory and checks the next one.

#!/bin/bash
# Copy this script to your to clean directory, 
# when you got a copy on your fileserver from this script
# then the copy in your clean dir will be removed also.
# Dont want that? change
# find -type f | 
# into
# find -type f | grep -v &lt;nameofthisscript&gt; |

# dont is current directory, skip these from locations
dont=$(pwd)
# Never start in /mnt ? uncomment below
# echo "$dont" | grep "^/mnt" &amp;&amp; ( echo "start in tank" ; exit )

find -type f | while read file ; do
        filemd5=$(md5sum "$file" | cut -f1 -d" ")
        basenamefile=$(basename "$file")
        echo "searching $basenamefile"
        locate -i "/$basenamefile" | grep -v "$dont" | while read location ; do
        if [ -f "$location" ] ; then
                locatedfilemd5sum=$(md5sum "$location" | cut -f1 -d" ")
                if [ "$filemd5" == "$locatedfilemd5sum" ] ; then
                        echo "found same md5sum at $location"
                        rm "$file"
                        break
                fi
        fi
        done
done
# Remove empty dirs?
# find . -type d -empty -delete

Locate can be slow, sometimes it is better to put the locate DB in memory of on another fast storage system.

mkdir /ramdisk
mount ramfs -t ramfs /ramdisk/
cp /var/lib/mlocate/mlocate.db /ramdisk/ 

# change above script locate command
locate -d /var/lib/mlocate/mlocate.db -i IMG20191123.jpg

And remove empty directories?
Add below at the end of the script

find . -type d -empty -delete

"If something is worth doing, it's worth overdoing."