Photo manager addition using ML!

Last Updated or created 2022-05-29

A few years ago i wrote a photo manager .. again .. ( see post about my first previous photo manager )
It is a web gui to find photos in my huge photo archive.
I manually added 190k tags to 120k photos in 20+ years.

I thought wouldn’t it be nice if i can generate additional metadata using Machine Learning. A few years ago i did some testing and followed a podcast and free course about machine learning.

So today i started to implement a addition to my gui. Machine recognition tags!

It already kinda works.

Things to do :

  • Make it a background job, my fileserver doesn’t run Tensorflow on a GPU, so it is slooow
  • Embed in existing GUI and stats
  • Design a editor to remove wrong tags

Below a part of ML images

Command to get a thumbnail sheet with only directory names:

montage -verbose -units PixelsPerInch -density 300 -tile 7x6 -label "%d" -font Arial -pointsize 6 -background "#FFFFFF" -fill "black" -define jpeg:size=253x154 -geometry 253x154+2+2 -auto-orient */*.JPG -title "ML Thumbs" thumbsheet.jpg

Maybe, i can use debug output like below.

['lakeside, lakeshore (score = 0.47934)', 'seashore, coast, seacoast, sea-coast (score = 0.11385)', 'sandbar, sand bar (score = 0.08822)', 'breakwater, groin, groyne, mole, bulwark, seawall, jetty (score = 0.06281)', 'valley, vale (score = 0.01790)', '']

Scraping podcast which uses a javascript to obfuscate mp3 links

Last Updated or created 2022-06-29

wget-ting the page only gave me flat html, but no readable links.

We need the rendered version, phantomjs wil help

wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2

printsource.js

var system = require('system');
var page   = require('webpage').create();
var url    = system.args[1];
page.open(url, function () {
  console.log(page.content);
  phantom.exit();
});

Run phantomjs

phantomjs-2.1.1-linux-x86_64/bin/phantomjs printsource.js  https://xxxxxxxx/show/xxxx > out

So now i got the rendered page, get mp3’s and titles, for this example

cat out | sed 'N;s/\n/,/' | cut -f2,7 -d\" | while read line ; do
mp3=$( echo $line | cut -f1 -d\")
title=$( echo $line | cut -f3 -d\> | tr -d '/<>[]]\!,;' | tr -d "'" | sed s/CDATA//g | sed s#title##g | sed s/:/-/g )
echo "$mp3 $title"
wget $mp3 -O "$title.mp3"
done

bash downloadscript
done

Portable Logitech Media Server again

Last Updated or created 2022-05-28

See post: https://www.henriaanstoot.nl/2014/04/10/portable-squeeze-server/

In the past i’ve used a home build Logitech Squeezebox server (as it was called then), Picore player and tried volumio.
Picore player has been sitting in my livingroom for ages, but was converted to a Node-Red Dashboard and recently Home Assistant Dashboard. (Has been a dasticz daskboard also)

Today i build another version, smaller and with a screen.
Why? .. because of being ‘offline’ or ‘offgrid’ on our holidays.
The car we are driving only has a Aux input.

Most of the installation is as mentioned on:
https://docs.picoreplayer.org/projects/add-a-display/

I edited  /opt/bootsync.sh
to get /dev/sda1 mounted persistent
use pcp br after editing.

Default user/pass : tc piCore

Controlling the thing is via touch or a app on my phone using wifi hotspot.

  • Audio cable 3.5mm
  • Raspberry 3
  • Large usb thumbdrive
  • 3.5inch RPi Display – 480×320 Pixel – XPT2046 Touch Controller
  • car cigarette lighter adapter for power

PiCore uses below alliases

ceChange directory to /mnt/mmcblk0p2/tce
ceoChange directory to /mnt/mmcblk0p2/tce/optional
m1Mount the boot partition /mnt/mmcblk0p1
m2Mount the second partition /mnt/mmcblk0p2
c1Change directory to /mnt/mmcblk0p1
c2Change directory to /mnt/mmcblk0p2
vicfgEdit configuration file config.txt using vi
vicmdEdit boot file cmdline.txt using vi
u1Unmount the boot partition /mnt/mmcblk0p1
u2Unmount the second partition /mnt/mmcblk0p2

Shutting down piCore is done by cutting the power, due to everything being mounted readonly. EXEPT
When you are using LMS server installation, which uses a database.
But there is a tweak for a shutdown button.

I’m using GPIO 16 because i’ve got a screen connected.
Active LOW, means you have to connect a pushbutton/switch between GND and GPIO pin. (nearest Vcc OR Gnd)

Table lamp hack

Last Updated or created 2022-05-25

Added: ino file 20220525

Bought a cheap table lamp a few weeks ago.
Runs on batteries and when you flip it over, it turns on or off.

I thought, when i strip this thing of its internals. I can make a wifi/mqtt enabled one.


Opening it up today, i saw a minimalistic print and a battery holder. There was a tilt switch like

Which i wanted to replace by a mercury one i bought in a bunch of sensors a few years ago.

So why go though all the trouble stripping and replacing .. so i didnt

GND and 5v to the batteries, and D4 to the tilt switch. (Measure which side you have to take!) .. I used a pull down of 3k3 ohms

Esp was flashed in the past with Easy ESP .. well lets keep that one for now.

INO file

#include <ESP8266WiFi.h>
#include <PubSubClient.h>
#include <Wire.h>
#include <Ethernet.h>

const char* ssid = "SSID";
const char* password = "PASSWORD";
const char* mqtt_server = "10.1.0.17";
const char* mqtt_username = "";
const char* mqtt_password = "";
const char* clientID = "wankel";

const int tiltPin = 4;
int tiltState = 0;    
int previousState = 0;    
WiFiClient espClient;

PubSubClient client(espClient);

void reconnect() {
  while (!client.connected()) {
    if (client.connect(clientID, mqtt_username, mqtt_password)) {
    } else {
      delay(2000);
    }
  }
}

void setup()
{
  {
    client.setServer(mqtt_server, 1883);
    pinMode(tiltPin, INPUT);
  }
}
void loop() {
  tiltState = digitalRead(tiltPin);
  if (tiltState != previousState) {
    if (tiltState == HIGH) {
      client.publish("onoff-wankel/wankel/State", "0"); //
    } else {
      client.publish("onoff-wankel/wankel/State", "1"); //
    }
    delay(100);
  }
  previousState = tiltState;

  {
    if (!client.connected()) {
      reconnect();
    }
    client.loop();
  }
}
Node red + led server

Example is using my ledserver, see other post, but i intent to made a easy to configure node red panel where the to be controlled devices are preconfigured.

[
    {
        "id": "9ec21acaec91aecc",
        "type": "mqtt in",
        "z": "54f3b5b461471f2c",
        "name": "",
        "topic": "onoff-wankel/wankel/State",
        "qos": "2",
        "datatype": "auto",
        "broker": "8c74c5f6.9a7a48",
        "nl": false,
        "rap": true,
        "rh": 0,
        "inputs": 0,
        "x": 400,
        "y": 260,
        "wires": [
            [
                "0fe77b535517f818"
            ]
        ]
    },
    {
        "id": "159f65f444a0d7c2",
        "type": "http request",
        "z": "54f3b5b461471f2c",
        "name": "1 - 30 red",
        "method": "GET",
        "ret": "txt",
        "paytoqs": "ignore",
        "url": "http://10.1.0.42:8080/range/01/30/ff0000",
        "tls": "",
        "persist": false,
        "proxy": "",
        "authType": "",
        "senderr": false,
        "credentials": {},
        "x": 900,
        "y": 280,
        "wires": [
            []
        ]
    },
    {
        "id": "5806fbfd0e99daab",
        "type": "http request",
        "z": "54f3b5b461471f2c",
        "name": "1 - 30 black",
        "method": "GET",
        "ret": "txt",
        "paytoqs": "ignore",
        "url": "http://10.1.0.42:8080/range/01/30/000000",
        "tls": "",
        "persist": false,
        "proxy": "",
        "authType": "",
        "senderr": false,
        "credentials": {
            "user": "",
            "password": ""
        },
        "x": 910,
        "y": 220,
        "wires": [
            []
        ]
    },
    {
        "id": "0fe77b535517f818",
        "type": "switch",
        "z": "54f3b5b461471f2c",
        "name": "",
        "property": "payload",
        "propertyType": "msg",
        "rules": [
            {
                "t": "eq",
                "v": "0",
                "vt": "str"
            },
            {
                "t": "eq",
                "v": "1",
                "vt": "str"
            }
        ],
        "checkall": "true",
        "repair": false,
        "outputs": 2,
        "x": 660,
        "y": 260,
        "wires": [
            [
                "5806fbfd0e99daab"
            ],
            [
                "159f65f444a0d7c2"
            ]
        ]
    },
    {
        "id": "8c74c5f6.9a7a48",
        "type": "mqtt-broker",
        "name": "10.1.0.17",
        "broker": "10.1.0.17",
        "port": "1883",
        "clientid": "",
        "usetls": false,
        "compatmode": true,
        "keepalive": "15",
        "cleansession": true,
        "birthTopic": "",
        "birthQos": "0",
        "birthPayload": "",
        "closeTopic": "",
        "closePayload": "",
        "willTopic": "",
        "willQos": "0",
        "willPayload": ""
    }
]

Arcade buttons retro games

Last Updated or created 2023-01-13

Connecting to my little handheld or on the big TV

See post https://www.henriaanstoot.nl/2021/11/21/retropi-handheld/

My good friend Vincent gave me his Arcade setup.

Button set

Games for two players:

Games played:

  • Metal slug
  • 1942
  • Galaga – First time i saw this was in Germany with the Concord Pipe Band
  • Track & Field, also known as Hyper Olympic – played a lot with a friend from the LTS

ZFS replace disk

Last Updated or created 2022-05-18

I’m using ZFS for my main fileserver, this pool was created over 10 years ago.
Meanwhile i’ve: Swapped broken disks, switched disks for bigger ones and effectively resized my storage 2 or 3 times. Never had any corruption.

Yesterday i say a warning that one of the disks in the pool was OFFLINE.
Today i replaced it using below command’s

  • Put the disk in OFFLINE mode (if needed, mine was already offline)
    • zpool offline tank sdb
  • Remove disk from system
    • echo 1 | sudo tee /sys/block/sdb/device/delete
  • Remove the disk physically
  • Insert the replacement disk. And copy headers/structure from another disk
    • sgdisk –replicate=/dev/sdb /dev/sda
    • sgdisk –randomize-guids /dev/sdb
  • Run the zpool replace command. 
    • zpool replace tank /dev/sdb
  • Use online command to activate disk (no needed in my case, it already did that)
Tips:

# My labels with serials fell off :(
dd if=/dev/sdb of=/dev/null -> blinky led .. 

# What is the serial?
sudo hdparm -i /dev/sdb | grep Serial

Output

root@latex:~# sgdisk --replicate=/dev/sdb /dev/sda
The operation has completed successfully.
root@latex:~# sgdisk --randomize-guids /dev/sdb
The operation has completed successfully.
root@latex:~# zpool replace tank /dev/sdb
root@latex:~# zpool status
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed May 18 11:31:21 2022
    5.64T scanned out of 14.4T at 331M/s, 7h42m to go
    1.88T resilvered, 39.16% done
config:

        NAME             STATE     READ WRITE CKSUM
        tank             DEGRADED     0     0     0
          raidz1-0       DEGRADED     0     0     0
            sda          ONLINE       0     0     0
            replacing-1  REMOVED      0     0     0
              old        REMOVED      0     0     0
              sdb        ONLINE       0     0     0  (resilvering)
            sdc          ONLINE       0     0     0

errors: No known data errors

New NFO generator using yad

Last Updated or created 2022-05-18

old post : https://www.henriaanstoot.nl/2021/02/08/kodi-movies-and-metadata/

Added poster art generator at the bottom of this page

Made a new NFO generator to get private movies properly scraped in kodi.

What does it do?

  • Genererates NFO for Kodi
  • Goes recursive though given directories
  • Skips movies with existing nfo
  • Previews the movie using VLC (So you can see what it is you are adding information for)
  • Tries to make a nice title from filename and year
  • Gets year from metadata
  • Fills tags from yad forms
  • next version i will add a poster generator from another nfo script i made. (It allready works, but i want to add a frame selector.
  • Adds new genre/actors/places if they dont exists
#!/bin/bash
# Needs mediainfo and yad
#find /media/geisha/private/Media/video/ -type f | egrep -i "mkv$|avi$|mpg$|mp4$|ogv$" | while read ; do
find $1 -type f | egrep -i "mkv$|avi$|mpg$|mp4$|ogv$" | while read ; do
# check if nfo exists
short=$(echo  "$REPLY" | rev | cut -f-2 -d/ | rev)
file=$( echo "$REPLY" | rev | cut -f2 -d. | rev).nfo
filename=$( basename "$REPLY" | cut -f1 -d. )
if [ -f "$file" ] ; then
	echo "$file found .. so skipping"
else
	echo "Not found $file"
nohup vlc "$REPLY" &
# check if find date
year=$(mediainfo "$REPLY" | grep Encoded |head -1 2>/dev/null | awk '{ print $5 }' | cut -c-4)
yeartitle="$year - "
if [ "$year" == "" ] ; then 
	year=$(echo $file | tr -cd '2[0-9][0-9][0-9]')
	if [ ! "$year" == "" ] ; then 
		#no year in filename
		yeartitle=""
	else
		yeartitle="$year - "
	fi
	
fi
# create nfo
forminfo=$(yad --title="nfo form $short" --text="Please enter:" --form --field="Title" --field="Year" --field="Placenew" --field="Genrenew" "$yeartitle $filename" "$year" "" "" --form --columns=2 --item-
separator="," --field="Place":CB --field="Genre":CB "$(paste -s -d"," < places)" "$(paste -s -d"," < genres)")
#echo $forminfo
#2020 - |2020|placenem|genrene|Hilversum|Vacation|
title=$(echo $forminfo | cut -f1 -d\|)
year=$(echo $forminfo | cut -f2 -d\|)
placenew=$(echo $forminfo | cut -f3 -d\|)
genrenew=$(echo $forminfo | cut -f4 -d\|)
place=$(echo $forminfo | cut -f5 -d\|)
genre=$(echo $forminfo | cut -f6 -d\|)
# Plot
plot=$(yad --form --field="Text::TXT" --geometry="600x200")
#plot=$( echo $plot | cut -f1 -d\\)
#plot=$( echo $plot | cut -f1 -d|)
# Actors
actors=$(yad --list  --geometry="200x480"  --print-all --column= --column=:chk --column= --column=:chk $( cat actors | cut -f1 -d" "  |while read user; do echo -n "$user false " ;done ) )
echo $actors | grep "NEW|TRUE" >/dev/null 
if [ $? -eq 0 ] ; then
vi actors
actors=$(yad --list  --geometry="200x480"  --print-all --column= --column=:chk --column= --column=:chk $( cat actors  | cut -f1 -d" " |while read user; do echo -n "$user false " ;done ) )
fi
if [ ! "$placenew" == "" ] ; then
	place="$placenew"
	echo "$placenew" >> places
fi
if [ ! "$genrenew" == "" ] ; then
	genre="$genrenew"
	echo "$genrenew" >> genres
fi
(
echo '<?xml version="1.0" encoding="utf-8" standalone="yes"?>'
echo "<movie>"
echo "  <plot>$plot</plot>"
echo "  <outline />"
echo "  <title>$title</title>"
echo "  <year>$year</year>"
echo "  <country>$place</country>"
cat actors | cut -f1 -d" " | while read actorname ; do
	echo $actors | grep "$actorname|TRUE"  >/dev/null
		if [ $? -eq 0 ] ; then
		name=$(grep $actorname actors | cut -f2 -d\" )
		echo "<actor>"
		echo "  <name>$actorname</name>"
		echo "  <role>$name</role>"
		echo "</actor>"
	fi
done
echo "  <genre>$genre</genre>"
echo "  <art>"
echo "  </art>"
echo "</movie>" 
) > "$file"  
fi
killall vlc
done

And some files

> actors
Firstname "Firstname Lastname"
> genres
Vacation
Pruts
> places
Netherlands

Poster generator (select poster thumbnail to use)

Generates 3 poster images and 3 thumbs to select.
Thumbs are 10 seconds into the movieclip, midway and on 2/3

short=$(basename $1 | cut -f1 -d.)

three=$(( $(mediainfo --Inform="Video;%Duration%" $1) / 3000 ))
rm -f thumb*.jpg poster*.jpg
first=10
#ffmpeg -ss $first -i $1 -vf scale=320:-1 -frames:v 1 -q:v 2 thumb1.jpg  -hide_banner -loglevel error
ffmpeg -ss $first -i $1 -frames:v 1 -q:v 2 thumb1.jpg  -hide_banner -loglevel error
second=$(( $three  ))
ffmpeg -ss $second -i $1 -frames:v 1 -q:v 2 thumb2.jpg  -hide_banner -loglevel error
third=$(( $three * 2 ))
ffmpeg -ss $third -i $1 -frames:v 1 -q:v 2 thumb3.jpg  -hide_banner -loglevel error
convert thumb1.jpg -resize x1000 -gravity center -crop 666x1000 poster1.jpg
convert thumb1.jpg -resize x320 -gravity center -crop 212x320 thumb1s.jpg
convert thumb2.jpg -resize x1000 -gravity center -crop 666x1000  poster2.jpg
convert thumb2.jpg -resize x320 -gravity center -crop 212x320 thumb2s.jpg
convert thumb3.jpg -resize x1000 -gravity center -crop 666x1000  poster3.jpg
convert thumb3.jpg -resize x320 -gravity center -crop 212x320 thumb3s.jpg


out=$(yad --form --title="Select poster thumbnail"  --geometry="+0+0" \
       --columns="3" \
       --field="!thumb1s-1.jpg! :fbtn" "echo poster1-1.jpg"  \
       --field="!thumb2s-1.jpg! :fbtn" "echo poster2-1.jpg"  \
       --field="!thumb3s-1.jpg! :fbtn" "echo poster3-1.jpg"
)
if [ ! "$out" == "|||" ] ; then
choice=$(echo $out | sed s/\ \|\|\|//g | rev | cut -f1 -d" " | rev)
echo cp $choice ${short}-poster.jpg
fi

Add effect to movie using bash

Last Updated or created 2022-05-17

mkdir -p videoframessource videoframes
ffmpeg -r 60 -i sweden.mp4 videoframessource/%05d.png
cd videoframessource
ls *png | while read file ; do ../convert2comic $file ; echo -n "." ;done ; echo ""
ffmpeg -f image2 -r 60  -i videoframes/%05d.png  -vcodec libx264 final.mp4
#!/bin/bash
# convert script .. call me convert2comic
convert -quiet $1 +repage -depth 8 -selective-blur 0x5+10% ./tmp1
convert ./tmp1 -level 0x80% -colorspace gray -posterize 6 -depth 8 -colorspace sRGB ./tmp2
convert ./tmp1 '(' ./tmp2 -blur 0x1 ')' '(' -clone 0 -clone 1 -compose over -compose multiply -composite -modulate 100,150,100 ')' '(' -clone 0 -colorspace gray ')' '(' -clone 3 -negate -blur 0x2 ')' '(' -c
lone 3 -clone 4 -compose over -compose colordodge -composite -evaluate pow 4 -threshold 90% -median 1 ')' -delete 0,1,3,4 -compose over -compose multiply -composite ../videoframes/$1
# Another nice effect
convert frame.png \( +clone -negate -blur 0x6 \) -compose ColorDodge -composite -modulate 100,0,100 out.png

Gotek stuff

Last Updated or created 2022-10-06

Recently back to old skool retro!

As posted https://www.henriaanstoot.nl/2022/05/03/c64-and-sd2iec/

Wellll, i bought some goodies from gotek.nl

A new SUB-D(23) to scart video cable for amiga.

A 9pindin mouse to PS/2 convertor

My tank mouse died .. beyond repair. Blue 9pins to ps2 works great!

A switchless (using key combination) diskdrive switcher df0<>df1

I used to make a floppy switch myself in the past, i used a wirewrap socket, and a cross switch. Made a hole in the back, with the little switch.
This one does not need a hole in your case for the switch.

Own floppy switch

Gotek Floppydrive Emulator with Rotary Encoder

This drive supports a lot of disk formats and systems, i will have to look into that ..
https://github.com/keirf/FlashFloppy/wiki/Host-Platforms