Tag Archives: bash

Automatic photo sorting with Gps location lookup

We take a lot of pictures, with our Nikon camera and our mobile phones.

(Apparently in 2019 5544 pictures)

Some stats

757 20190803 - on a single day (Holiday)

Average pictures per month
locate "/2019/" | egrep -i "photoalbum|gsm" | egrep -i "mp4$|jpg$" | grep -Eo '2[[:digit:]]{3}[[:digit:]]{2}[[:digit:]]{2}' | cut -c-6 |sort | uniq -c | sort -n | awk '{ sum += $1; n++ } END { if (n > 0) print sum / n; }'
461

Besides android pictures being automatically uploaded to our nextcloud, I’m using some apps and scripts to get pictures and movies stored on my fileserver. (bash scripts/andftp)

For sorting those media files, i made a sorting script.
(Today I added a location sorting addition using GPS information stored in the exif information.

  • jpg and jpeg (add your own extentions)
  • mp4 and mov (for mobile and nikon)
  • Sorts by camera model/year/date/location
  • tries to extract date from filename when not found in exifinfo
  • Sorts whatsapp media
  • Sorts Raw

INSTALLING

pip3 install reverse_geocoder
You need python3, exiftool, exiftime and mediainfo

copy below python script in ~/bin/reverse2.py

( need more info? change last print entry admin1/admin2)
[{‘lat’: ‘-39.45556’, ‘lon’: ‘173.85833’, ‘name’: ‘Opunake’, ‘admin1’: ‘Taranaki’, ‘admin2’: ‘South Taranaki District’, ‘cc’: ‘NZ’}]

import reverse_geocoder as rg
import sys

lat=sys.argv[1]
lon=sys.argv[2]

coordinates = (lat,lon)

results = rg.search(coordinates) # default mode = 2


#print (results)

for entry in results:
    print(entry['name'] + "(" +  entry['cc'] + ")")

And a bash script /usr/local/bin/exifsort.sh

#!/bin/bash
#set -x

reversepath=/home/henri/projects/reversegeo/reverse2.py

#RAW
rawcnt=`ls | grep -i  nef$ | wc -l`
if [ "$rawcnt" = "0" ] ; then
echo "no raw"
else
mkdir raw 2>/dev/null
ls | grep -i nef$ | while read ; do mv $REPLY raw ; done
fi


ls | egrep -i "jpg$|jpeg" | while read ; do 
 	location=""	
	getmodel=$(exiftool "$REPLY" |grep "Make " | awk '{ print $3 }')
	if [ "$getmodel" != "" ] ; then 
		getmodel=$getmodel/
	fi
	echo "$REPLY" | grep WA0 >/dev/null && getmodel=whatsapp/

	gpsinfo=$(exiftool -c "%+.6f" "$REPLY" |grep "GPS Position" | cut -d":" -f2 | tr -d ' ' | sed s/,/\ /g)
	if [ "$gpsinfo" != "" ] ; then 
		location=$(python3 $reversepath $gpsinfo | grep -vi load | sed s/\(NL\)//g)
	fi
	dater=$(exiftime "$REPLY" 2>/dev/null | egrep "Created|Digitized" | sed s/Digitized/Created/g | tail -1  | cut -c 16-19,21,22,24,25)
	if [ "$dater" = "" ] ; then 
#		echo "Trying from filename"
		dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}-[[:digit:]]{2}-[[:digit:]]{2}')

		if [ "$dater" = "" ] ; then 
#			echo "Trying from filename - maybe without dashes"
			dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}[[:digit:]]{2}[[:digit:]]{2}')

		fi
	fi
	if [ "$dater" != "" ] ; then 
		year=$(echo $dater | cut -c-4)
		mkdir -p "${getmodel}$year/${dater}/$location"
		mv "$REPLY" "${getmodel}${year}/${dater}/$location"
	else
		mkdir -p "${getmodel}unknowndate/$location"
		mv "$REPLY" "${getmodel}unknowndate/$location"
      	fi
done

ls | egrep -i "mov$|mp4$" | while read ; do 
	
 	location=""	
	getmodel=$(exiftool "$REPLY" |grep "Make " | awk '{ print $3 }')
	if [ "$getmodel" != "" ] ; then 
		getmodel=$getmodel/
	fi

	echo "$REPLY" | grep WA0 >/dev/null && getmodel=whatsapp/
	gpsinfo=$(exiftool -c "%+.6f" "$REPLY" |grep "GPS Position" | cut -d":" -f2 | tr -d ' ' | sed s/,/\ /g)
	if [ "$gpsinfo" != "" ] ; then 
		location=$(python3 $reversepath $gpsinfo | grep -vi load | sed s/\(NL\)//g)
	fi
	dater=$(mediainfo "$REPLY" | grep Encode | tail -1 | cut -f2- -d:  | cut -f3 -d" " | sed s/-//g)
	if [ "$dater" = "" ] ; then 
#		echo "Trying from filename"
		dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}-[[:digit:]]{2}-[[:digit:]]{2}')

		if [ "$dater" = "" ] ; then 
#			echo "Trying from filename - maybe without dashes"
			dater=$(echo $REPLY | grep -Eo '2[[:digit:]]{3}[[:digit:]]{2}[[:digit:]]{2}')

		fi
	fi
	if [ "$dater" != "" ] ; then 
		year=$(echo $dater | cut -c-4)
		mkdir -p "${getmodel}$year/${dater}/$location"
		mv "$REPLY" "${getmodel}${year}/${dater}/$location"
	else
		mkdir -p "${getmodel}unknowndate/$location"
		mv "$REPLY" "${getmodel}unknowndate/$location"
      	fi
done

Example running in a directory with mixed media

# Raw images get moved into a RAW directory
no raw

# Samsung phone detected with date and GPS location
mkdir -p samsung/20220717/Hilversum
mv 20220717_133453.jpg samsung/20220717/Hilversum

# OnePlus phone
mkdir -p OnePlus/20021208/Voorburg
mv IMG_20190109_091825.jpg OnePlus/20021208/Voorburg

# Realme (Added country when not NL)
mkdir -p realme/20220607/Isle of Islay(GB)
mv IMG20220607213630.jpg realme/20220607/Isle of Islay(GB)

# Whatsapp has no date embedded so it get it from filename
Trying from filename
Trying from filename - maybe without dashes
mkdir -p whatsapp/20221021/
mv IMG-20221021-WA0000.jpg whatsapp/20221021/

# Nikon without GPS
mkdir -p NIKON/20220613/
mv DSC_1423.MOV NIKON/20220613/

# Whatsapp video without exif
mkdir -p whatsapp/20170528/
mv VID-20170528-WA0006.mp4 whatsapp/20170528/

# No camera name detected in exif from mobile movie
mkdir -p 20190114/Maarssen
mv VID_20190114_142455.mp4 20190114/Maarssen

# Location in mp4
mkdir -p 20220607/Lamlash(GB)
mv VID20220607155044.mp4 20220607/Lamlash(GB)

Result

./NIKON/2022/20220613/DSC_1423.MOV
./NIKON/2022/20220610/750_1101.JPG
./realme/2022/20220818/Hilversum/IMG20220818203825.jpg
./realme/2022/20220607/Isle of Islay(GB)/IMG20220607213630.jpg
./2019/20190114/Maarssen/VID_20190114_142455.mp4
./whatsapp/2017/20170528/VID-20170528-WA0006.mp4
./whatsapp/2022/20221021/IMG-20221021-WA0000.jpg
./2022/20220607/Lamlash(GB)/VID20220607155044.mp4
./2022/20220516/Hilversum/VID20220516125913.mp4
./OnePlus/2002/20021208/Voorburg/IMG_20190109_091825.jpg
./samsung/2022/20220717/Hilversum/20220717_133453.jpg

You could automate this using incrond
apt-get install incron
add your user to /etc/incron.allow
incrontab -e
add

/fileserver/mediain/ IN_CREATE /usr/local/bin/sortmymedia.sh
Coping a file in the directory, auto sort and move to correct location

sortmymedia.sh

#!/bin/bash
cd /home/user/media
/usr/local/bin/exifsort.sh

Automating the h*ll out of windows applications using linux

I’m using a windows program for typesetting bagpipe music.
Luckily this runs on Linux using wine.

Sometimes i just want a PDF version of a tune, for example for my tunebook compiler. (Other post)
Or i want to batch convert a lot of bww files.

A long time ago i used a virtual machine with automation software for this.
Why not automate the process on my Laptop?

So i made this script, with a workaround for the xdotool wine problem.
(wine window needs to be active to accept key strokes, other linux xwindows you can use the ID of the window)

#!/bin/bash
# use c for close at the end, without c bgplayer wont be shut down
# bww2pdf path/to/music.bww c
# make tmp file
cat "$1" > /tmp/deze.bww
# start bgplayer if not started .. if not started wait 3 secs
slp=0
pgrep BGPlayer.exe >/dev/null || slp=3
pgrep BGPlayer.exe  >/dev/null|| ( nohup wine ~/.wine/dosdevices/c\:/Program\ Files\ \(x86\)/Bagpipe\ Player/BGPlayer.exe & )
sleep $slp
# get program id
pid=$(pgrep BGPlayer.exe)
# get xwindow id using pid
winid=$(xdotool search --limit 1 --all --pid $pid)
# activate window
xdotool search --desktop 0 --class "BGPlayer.exe" windowactivate
# open file menu and Open file
xdotool key --window $winid "alt+f"
xdotool key --window $winid "o"
# give program time to execute
sleep 1
# open our tmp file
xdotool type '\tmp\deze.bww'
xdotool key KP_Enter
sleep 2
# open file menu select Print and PDF as output
xdotool key "alt+f"
xdotool key "p"
xdotool key "P"
sleep 2
# execute
xdotool key KP_Enter
sleep 1
# File close
xdotool key "alt+f"
xdotool key "c"
sleep 2
# close program when c was added to commandline
mv ~/PDF/deze* "$1.pdf"
if [ "$2" == "c" ] ; then 
	xdotool key "alt+f"
	xdotool key "x"
fi
I’m not touching the keyboard when the program is running, all handled by the script

Motion detection for stupid webcams

Last year i made a script for a friend who wanted to detect visually if his garden sprinkler was on or off. A few days ago i saw someone who wanted to see if things where moving in his house. (didn’t trust his landlord i think)
But he only had a dumb/simple/cheap camera .. so it had no motion detection.

I was thinking of my script, and could easily adapt it for this usage.

Most ipcams have somekind of URL/API you can use to capture a image.
Some examples below

# Reolink
wget "http://ipnumber/cgi-bin/api.cgi?cmd=Snap&channel=0&rs=randomstring&user=user&password=password" -O image.jpg

# Foscam
wget "http://ipnumber:88/CGIProxy.fcgi?cmd=snapPicture2&usr=user&pwd=password&cnt=randomstring" -O image.jpg

# Edimax
wget "http://user:password@ipnumber/jpg/image.jpg" -O image.jpg

# Hikvision
wget "http://user:password@ipnumber/ISAPI/Streaming/channels/101/picture" -O image.jpg

So using below script i can capture a image, compare it to the previous, and when it’s above a certain threshold sends a email.

#!/bin/bash
# Only uses wget and image-magick
treshhold=500
fuzzyness=20%
# CHANGE WEBCAM THINGY TO OWN URL AND CREDENTIALS
wget -q "http://webcamip/cgi-bin/api.cgi?cmd=Snap&channel=0&user=user&password=password" -O previous.jpg
while true;  do
wget -q "http://webcamip/cgi-bin/api.cgi?cmd=Snap&channel=0&user=user&password=password" -O current.jpg
value=$(compare -fuzz $fuzzyness previous.jpg current.jpg -metric mae diff.jpg 2>&1 | cut -f1 -d.)
if [ $value -gt $treshhold ] ; then
echo "ping $treshhold"
echo "Something moved" | mail -s "Movement" user@example.com -A diff.jpg
fi
# Comment below if you want to compare against a base line .. not previous image
cat current.jpg > previous.jpg
sleep 60
done

Example previous picture

Example current picture

I got mailed with result

Hints tips:

Use crop to detect only a part.

copy current.jpg to a second file
Use painting black a part and compair with different treshhold fuzzyness to get different hotspots.

Below detects RED, use above ide with crop to detect red/green/blue leds

compare -verbose -metric mae 1.jpg 2.jpg /tmp/1.diff
1.jpg JPEG 2560x1920 2560x1920+0+0 8-bit sRGB 248819B 0.050u 0:00.057
2.jpg JPEG 2560x1920 2560x1920+0+0 8-bit sRGB 248949B 0.030u 0:00.137
Image: 1.jpg
  Channel distortion: MAE
  Channel distortion: MAE
    red: 12517.5 (0.191005)
    green: 11967.1 (0.182607)
    blue: 12492.8 (0.190628)
    all: 12325.8 (0.18808)
1.jpg=>/tmp/1.diff JPEG 2560x1920 2560x1920+0+0 8-bit sRGB 1.19495MiB 1.470u 0:00.197

Energy bill and ledserver

Having a lot of devices and running a Lab wil use a lot of energy. Now with the energy crisis in Europe, i had to take a closer look at whats using power in my house.

I notished some weird usage patterns while measuring.

I’m using a few shelly power plugs, to measure devices and powerstrips.

With these devices you can control devices connected to it.
On/Off/Timer etcetera.
It wil measure the power usage in watts, and it even got a temperature sensor.
I like the fact that it perfectly integrates into your home automation using an extensive API.
curl commands to controll, and even MQTT messaging. Intergrating in Home Assistant is a breeze.

So i was monitoring a bunch of stuff using Nodered/Grafana/Homeassistant and saw some recurring usage.
But being always late to check things, i made use of my ledserver i’ve build a long time ago.

This ledserver consists of a Raspberry Pi Zero, with a led string and a API written in python.

Below is autostarted on the Raspberry

( I made this ledserver for work, it showed the status of servers and services. Beside that every colleage had a range which he could use for his own scripts. I made some little bash script templates to have led funtions standard in your bash profile.

#!/usr/bin/python


# apt-get install python-flask
#

import Adafruit_WS2801
import Adafruit_GPIO.SPI as SPI
import struct

from flask import Flask, render_template, request
app = Flask(__name__)

PIXEL_COUNT = 32
SPI_PORT   = 0
SPI_DEVICE = 0

pixels = Adafruit_WS2801.WS2801Pixels(PIXEL_COUNT, spi=SPI.SpiDev(SPI_PORT, SPI_DEVICE))

pixels.clear()
pixels.show()

@app.route("/led/<deviceName>/<color>")
def action(deviceName, color):
        if deviceName == 'reset':
                print ("reset")
                pixels.clear()
        print (deviceName)
        led = int(deviceName)
        s = color
        r = int(s[ :2], 16)
        b = int(s[2:4], 16)
        g = int(s[4: ], 16)

        pixels.set_pixel_rgb(led, r,g,b)
        pixels.show()

        templateData = {
                'rled'  : r,
                'bled'  : b,
                'gled'  : g,
                'deviceName'  : deviceName,
        }
        return render_template('index.html', **templateData)

@app.route("/control/<controlcommand>")
def actioncommand(controlcommand):
        if controlcommand == 'clear':

                print("clear")
                pixels.clear()
                pixels.show()

        templateData = {
                'controlcommand'  : controlcommand,
        }
        return render_template('index.html', **templateData)

@app.route("/range/<start>/<stop>/<color>")
def rangecommand(start,stop,color):
        s = color
        r = int(s[ :2], 16)
        b = int(s[2:4], 16)
        g = int(s[4: ], 16)
        startled = int(start)
        stopled = int(stop)
        while (startled < stopled):
          pixels.set_pixel_rgb(startled, r,g,b)
          startled=startled + 1
        pixels.show()

        templateData = {
                'rangecommand'  : rangecommand,
        }


        return render_template('index.html', **templateData)



if __name__ == "__main__":
   app.run(host='0.0.0.0', port=8080, debug=True)

Now you can control the leds with a simple curl command:

curl http://ledserver:8080/range/startled/endled/colorinrgb
curl http://ledserver:8080/led/lednumber/colorinrgb
curl http://ledserver:8080/control/clear

So today i made a little script to show power usage.

I’m reading the current power usage from a LS120 Youless

Youless LS120 device, which you can connect to your P1 connector.

With below bash script i’m reading the webinterface and update the ledstring.
I was using this ledserver for general notification usage. Below a 2 minute hack ..

#!/bin/bash
while true; do
number=$(echo $(curl -s http://youlessip/a | grep Watt | head -1 | awk '{ print $1 }') / 100  | bc)
curl -s http://ledserver:8080/control/clear
curl -s http://ledserver:8080/range/00/$number/010101
sleep 10
done
Using 9 leds = 9xx watt

Challenged myself into programming a photo quiz

Got bored, and challenged myself …

How about a quess the picture from our photo collection??
(123580 photos .. )
So i show a random picture, and when i press ESC it will show some information about the picture …
Quess the year and the event

Well i gave myself 15 minutes to program something ..

I was watching a tv show meanwhile .. but i managed to come up with this …

This script is showing a picture, when you press ESC it wil show some details.After that it will select another random picture.

Improvements : reading tags and other metadata from my photo database, to give more information.

#!/bin/bash
while true ; do
shuf -n 1 allpix > /tmp/randompix
pix=$(cat /tmp/randompix | cut -f2 -d\> | cut -f1 -d\<)
dir=$(cat /tmp/randompix | cut -f4 -d\> | cut -f1 -d\< |cut -f3 -d\; | sed 's#\\#/#g')
displaypath=/mnt/${dir}/${pix}
info=$(cat /tmp/randompix | rev | cut -f-4 -d\\ | rev | cut -f1 -d \<)
convert -background black -fill white  -size 1920x1080 -pointsize 24 label:"$info" info.jpg
pqiv -i --fullscreen "$displaypath"
pqiv -i --fullscreen info.jpg
done

Which gave me … this

Welllll .. it toke 20 minutes .. so i lost
(Must have been the wine)

Homewizard Watermeter

Today i got my watermeter in.

It’s a little device you can place on your watermeter.

You get a plastic clip/holder which you can use to place the device on your watermeter. You can easily remove the device to read the values.

The device measures the little round gauge, so it has no idea what the current values are. Add the current values of your meter to the output of the device.

The device works by measuring the rotation of the red part.

By default it sends the information over the internet to a server. Your phone connects to this server and the app wil give you the graphs.

If you want your own intergration, you have to enable “local api”
See image on the right.
When you want realtime data, you have to connect a usb-c power supply.
When using batteries, the device wil only connect to wifi once per 5 minutes, and you can’t use the API.

I wrote a little test script in bash to draw graphs using mqtt and Nodered.

#!/bin/bash
calib=29.621
data=$(curl -s http://10.1.1.176/api/v1/data)
totalwater=$(echo $data | cut -f9 -d\" | tr -d ':,')
currentwater=$(echo $data | cut -f11 -d\" | tr -d ':,}')
totalwater=$( echo $totalwater + $calib | bc)
mosquitto_pub -h 127.0.0.1 -m "$totalwater" -t watermeter/total
mosquitto_pub -h 127.0.0.1 -m "$currentwater" -t watermeter/current

Variable calib is what my watermeter was when i attached the device.
It wil add this to the newly measured values.

Home Assistant found the device using autodiscovery

I have to look into adjusting for offset in HA

Spotify export

Nice .. you can request a data export from spotify, much like google’s takeout.

I wil export your streaming history, playlists, library and more.

Login spotify on a desktop, goto privacy settings and request data.

Nice to generate some statistics from this data.

For now the top 50 played artists (nr of times, artist)

790     "Rammstein"
507     "Fred Morrison"
478     "Lincoln Hilton"
437     "Avicii"
420     "Shooglenifty"
347     "Treacherous Orchestra"
323     "Battlefield Band"
321     "Breabach"
295     "Nightwish"
254     "The Fraser Shaw Trust"
236     "Michael McGoldrick"
207     "Peatbog Faeries"
207     "Calum Stewart"
203     "Lúnasa"
197     "Martyn Bennett"
194     "Jeff Wayne"
193     "The Prodigy"
173     "Kíla"
172     "Ross Ainslie and Jarlath Henderson"
170     "Buena Vista Social Club"
153     "Ross Ainslie"
150     "Dàimh"
138     "Bliss"
125     "Hans Zimmer"
124     "Rare Air"
118     "Michael McCann"
107     "Kyle Tuttle"
107     "Beinn Lee"
105     "Bourne & Macleod"
 89     "Wolfstone"
 88     "Ímar"
 83     "Afro Celt Sound System"
 81     "Gordon Duncan"
 81     "Armin van Buuren"
 80     "Face The West"
 79     "Tyurgen Kam"
 79     "Aly Bain"
 72     "Keltik Elektrik"
 71     "Duncan Chisholm"
 66     "Liz Carroll"
 63     "Project Smok"
 63     "Blazin' Fiddles"
 62     "Dagda"
 61     "Trail West"
 59     "Julian Winding"
 57     "Solar Fields"
 57     "Dougie McCance"
 56     "John McSherry"
 56     "AES DANA"
 52     "Gaelic Storm"

Get top 50 script

 cat  *json | grep artistName  | sort | uniq -c | sort -nr | head -50 | cut -f1,4,5 -d\" | sed s/,$//g

Finding files on my fileserver

I use several tools to find files on my server.

Loads of stuff on my main fileserver.
(Graph is a great tool called DUC) https://duc.zevv.nl/

Besides a search engine, i have a file finder.
Due to the massive amount of data, i like to find things by other means than knowing the directory structure.

I can find files by filename, but also by contents.

I’ll talk about find by contents first.

I’ve got loads of documents in Pdf, HTML, txt, doc, sheets , wordperfect etcetera.
Those documents i can find using a tool named Namazu.
This is quite a old tool, but i’m using it for a long time and it still works great.
I didn’t find a better replacement yet.
(But i’ve been looking into : elasticsearch, Solr, Lucene)

http://www.namazu.org/ is easy to install, but if you want the tool to scrape different kinds of documents you have to add some additional software.

My multipurpose printer can scan pages in pdf.
Those are only embedded jpg’s in a pdf container.
I will talk about how i handle these later.

My current start page :
This index contains 267,763 documents and 14,036,762 keywords.
Search example of namazu

Some things to consider when implementing namazu:

  • tweak the file types to scrape, it makes no sense to scrape binaries
  • tweak the directories to scrape (example below)
    • 0 1 * * 1 fash /usr/bin/mknmz -f /etc/namazu/mknmzrc –output-dir=/namazu/ /mnt/private/paperwork/ /mnt/private/information/ /mnt/private/Art\ en\ hobby\ Projects/ /mnt/private/Music\ Projects/ /mnt/private/bagpipe-music-writer/ –exclude=XXX –exclude=/mnt/binaries > /tmp/namazu.log 2>&1
  • you can set a parameter in the config for search only, this disables downloading the found link in the results!

Before Namazu i used HtDig.

Screenshot htdig

HtDIg also can scrape remote websites, Namazu can’t.

Preparing PDF for indexing:

I’ve written some scripts to make PDFs containing scanned text scrape-able.
( https://gitlab.com/fash/inotify-scanner-parser )
What it does:

  • My scanner puts a scanned pdf on my fileserver in a certain directory
  • Inotify detects a written file
  • it will copy the file, run OCR on it (tesseract) and writes a txt file (scapeable)
  • After that the text will be embedded (overlay) on the PDF, so now it becomes searchable/scrapeable
  • When certain keywords are found, it will sort documents in subdirs
Example from a scanned jpg, i can find OCR words!
(note .. the overlay is exact on the found words)

Finding files by name:

For finding files a made a little webpage like this:

It is a simple webpage grabbing through a list of files.
It takes the first keyword and does a grep, it takes a second keyword to match also.
I can select different file databases to search. (This case is private)
Between search and private i can give the number of entries to print.
So i can do
Search “ansible” NOT “tower” 50 entries from the public fileset

Crontab:

20 5 * * * /usr/bin/find /mnt/shark*  > /var/www/html/findfiles/sharkoon
10 4 * * * /usr/bin/find /tank/populair > /var/www/html/findfiles/populair
20 4 * * * /usr/bin/find /tank/celtic > /var/www/html/findfiles/celtic
etc

And a php script (dirty fast hack, never came around it to make it a final version)

<html><head><title></title></body>
<font face="Tahoma"><small>|keyword|(keyword)|search|(nr results)|NOT SECOND KEYWORD|share|</small><BR>
Search: <form method="post" action="/findfiles/?"><input type="Text" name="words" size=10 value=""><input type="Text" name="words2" size=10 value=""><input type="Submit" name="submit" value="search"><input type="Text" name="nrlines" size=3 value=""><input type="checkbox" name="not" unchecked>
<SELECT NAME=findfile>
<OPTION VALUE=private>private
<OPTION VALUE=celtic>celtic
<OPTION VALUE=populair>populair
<OPTION VALUE=dump>public
<OPTION VALUE=sharkoon>sharkoon
</SELECT>
</form>
<P><PRE>
<?php
$words2=$_POST['words2'];
$words=$_POST['words'];
$filefile=$_POST['filefile'];
$findfile=$_POST['findfile'];
$nrlines=$_POST['nrlines'];
$not=$_POST['not'];


if ($words2=="xsearch") { $findfile="other"; $words2=""; }
if ($nrlines) {  } else { $nrlines=100; }
if ($words && $words2=="") {
$words = preg_replace("(\r\n|\n|\r)", "", $words);
$words = preg_replace("/[^0-9a-z]/i",'', $words);
$command = "/bin/cat $findfile |/bin/grep -i $words |head -$nrlines";
$blah=shell_exec($command);
$blah=str_replace($words, "<b><font color=red>$words</font></b>",$blah);
print $blah;
}
if (($words) and ($words2)) {
$words = preg_replace("(\r\n|\n|\r)", "", $words);
$words = preg_replace("/[^0-9a-z.]/i",'', $words);
$words2 = preg_replace("(\r\n|\n|\r)", "", $words2);
$words2 = preg_replace("/[^0-9a-z.]/i",'', $words2);
if ($not=="on") {
$command = "/bin/cat $findfile |/bin/grep -i $words | /bin/grep -iv $words2 |head -$nrlines";
} else {
$command = "/bin/cat $findfile |/bin/grep -i $words | /bin/grep -i $words2 |head -$nrlines";
}
$blah=shell_exec($command);
$blah=str_replace($words, "<b><font color=red>$words</font></b>",$blah);
$blah=str_replace($words2, "<b><font color=red>$words2</font></b>",$blah);
print $blah;
}
?>
</PRE>
</body></html>

Scraping podcast which uses a javascript to obfuscate mp3 links

wget-ting the page only gave me flat html, but no readable links.

We need the rendered version, phantomjs wil help

wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2

printsource.js

var system = require('system');
var page   = require('webpage').create();
var url    = system.args[1];
page.open(url, function () {
  console.log(page.content);
  phantom.exit();
});

Run phantomjs

phantomjs-2.1.1-linux-x86_64/bin/phantomjs printsource.js  https://xxxxxxxx/show/xxxx > out

So now i got the rendered page, get mp3’s and titles, for this example

cat out | sed 'N;s/\n/,/' | cut -f2,7 -d\" | while read line ; do
mp3=$( echo $line | cut -f1 -d\")
title=$( echo $line | cut -f3 -d\&gt; | tr -d '/&lt;&gt;[]]\!,;' | tr -d "'" | sed s/CDATA//g | sed s#title##g | sed s/:/-/g )
echo "$mp3 $title"
wget $mp3 -O "$title.mp3"
done

bash downloadscript
done

ZFS replace disk

I’m using ZFS for my main fileserver, this pool was created over 10 years ago.
Meanwhile i’ve: Swapped broken disks, switched disks for bigger ones and effectively resized my storage 2 or 3 times. Never had any corruption.

Yesterday i say a warning that one of the disks in the pool was OFFLINE.
Today i replaced it using below command’s

  • Put the disk in OFFLINE mode (if needed, mine was already offline)
    • zpool offline tank sdb
  • Remove disk from system
    • echo 1 | sudo tee /sys/block/sdb/device/delete
  • Remove the disk physically
  • Insert the replacement disk. And copy headers/structure from another disk
    • sgdisk –replicate=/dev/sdb /dev/sda
    • sgdisk –randomize-guids /dev/sdb
  • Run the zpool replace command. 
    • zpool replace tank /dev/sdb
  • Use online command to activate disk (no needed in my case, it already did that)
Tips:

# My labels with serials fell off :(
dd if=/dev/sdb of=/dev/null -> blinky led .. 

# What is the serial?
sudo hdparm -i /dev/sdb | grep Serial

Output

root@latex:~# sgdisk --replicate=/dev/sdb /dev/sda
The operation has completed successfully.
root@latex:~# sgdisk --randomize-guids /dev/sdb
The operation has completed successfully.
root@latex:~# zpool replace tank /dev/sdb
root@latex:~# zpool status
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed May 18 11:31:21 2022
    5.64T scanned out of 14.4T at 331M/s, 7h42m to go
    1.88T resilvered, 39.16% done
config:

        NAME             STATE     READ WRITE CKSUM
        tank             DEGRADED     0     0     0
          raidz1-0       DEGRADED     0     0     0
            sda          ONLINE       0     0     0
            replacing-1  REMOVED      0     0     0
              old        REMOVED      0     0     0
              sdb        ONLINE       0     0     0  (resilvering)
            sdc          ONLINE       0     0     0

errors: No known data errors