Category Archives: Nerd stuff

Reprocess mbox file

Please keep in mind that this post is about 4 years old.
Technology may have changed in the meantime.

Say you’ve found an mbox file somewhere, and you’d like Postfix to reprocess the messages it contains, to have them imported into your regular mailbox.

Then you may want to copy this script:

#!/usr/bin/env python3

# Script to reprocess an mbox file.

################################################################################
#
# Copyright (c) 2021 Rob LA LAU <https://www.ohreally.nl/>
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
# this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its contributors
# may be used to endorse or promote products derived from this software
# without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
################################################################################

import mailbox, os, subprocess, sys

if len(sys.argv) < 3:
	print('Usage:')
	print('  %s recipient@example.com /path/to/mbox/file' % sys.argv[0])
	sys.exit(1)

recipient = sys.argv[1]
mbox = sys.argv[2]
if not os.access(mbox, os.R_OK):
	print('mbox file is not accessible')
	sys.exit(1)

for message in mailbox.mbox(mbox):
	proc = subprocess.Popen(
		['sendmail', '-i', recipient],
		stdin = subprocess.PIPE,
		text = True
	)
	proc.communicate(message.as_string())

Save that script as ~/bin/resend_mbox.py, make it executable, and execute it as follows:

# ~/bin/resend_mbox.py george@example.com /var/mail/root

Obviously, you replace george@example.com with your own email address (don’t spam poor George, please), and /var/mail/root with the path to your mbox file.

JWPlayer errors

Please keep in mind that this post is about 4 years old.
Technology may have changed in the meantime.
Error Code 224002: This video file cannot be played
Error Code 224003: This video file cannot be played
Error Code 232011: This video file cannot be played

A few of the more common web video player errors, but they tell you nothing.

If you search the internet for these errors, all you find are generic bullshit ‘solutions’: clear cache, disable all extensions, disable hardware acceleration, etc. So, here’s the actual list of JWPlayer error codes: Player errors reference. Get you some real answers.

By the way, the meaning of the above errors:

  • Failed to decode the associated resource (224002),
  • Failed to play the associated resource because it is not supported by this browser (224003), and
  • A manifest request was made without proper crossdomain credentials (232011)
    (The manifest is a file that contains the URLs for the various segments that make up the entire video.)

What it comes down to, is that JWPlayer just sucks a lot, especially if you use a proxy. Let’s hope that with the growing acceptance of HTML5, with built-in video support, projects like this will go the same way Flash players did.

Counting annotations in a PDF

Please keep in mind that this post is about 4 years old.
Technology may have changed in the meantime.

Suppose that you wrote a book, and your publisher sent you a PDF of the final proof to review and correct, before the book is printed. Then you may want to know how many notes you’ve added to the document when you’re done.

$ env LC_CTYPE=C tr -d '\000-\011\013\014\016-\037' < FILENAME.pdf | grep -E '^<</Type /Annot /Rect \[[0-9\. ]+\] /Subtype /Text' | wc -l

It’s that simple…

And if you’d like to know how many terms you highlighted, all you have to do is replace /Text with /Highlight.

$ env LC_CTYPE=C tr -d '\000-\011\013\014\016-\037' < FILENAME.pdf | grep -E '^<</Type /Annot /Rect \[[0-9\. ]+\] /Subtype /Highlight' | wc -l

Inline notes are of subtype /FreeText.

And to count all your annotations, regardless of type, just delete the subtype altogether.

$ env LC_CTYPE=C tr -d '\000-\011\013\014\016-\037' < FILENAME.pdf | grep -E '^<</Type /Annot' | wc -l

Open your PDF in less to see what other interesting things you could do with grep; pipe the file through tr to get rid of the control characters.

$ env LC_CTYPE=C tr -d '\000-\011\013\014\016-\037' < FILENAME.pdf | less

FreeBSD jails: a complete example

Please keep in mind that this post is about 4 years old.
Technology may have changed in the meantime.

FreeBSD jails are a great way to separate and compartmentalize processes, which enhances the security of your system. A jail is an enhanced chroot: it prevents an attacker who manages to compromise a service from gaining access to the rest of the system.

This post documents the setup of 2 jails that serve data to the outside world, and communicate between each other (through a Unix Domain Socket, not a TCP socket).

Read More

FreeBSD: which ports/packages did I update today?

Please keep in mind that this post is about 4 years old.
Technology may have changed in the meantime.

A FreeBSD oneliner this time.
This won’t work on Linux!

Tools like portmaster and portupgrade allow you to update all installed ports/packages in a single run, which is great.
But these tools do not list the ports and packages that have been upgraded, leaving you to guess which daemons and services must be restarted…

Luckily, all installed packages are registered in an SQLite database, together with a timestamp for their last upgrade (or installation). Add the following alias to your ~/.bashrc:

alias puptd='sqlite3 /var/db/pkg/local.sqlite "select origin from packages where time > $(date -v-2d +%s) order by origin" | less'

Clearly, if you don’t use the Bash shell, you should figure out how to add aliases in your shell. The alias will be active after you re-login; invoke it like any other command.

$ puptd

This alias will list all ports/packages that were updated (or installed) in the last 2 days (an update of all ports can run for quite some time). Obviously you should feel free to change the -v-2d to any other period (-v-3H for the last 3 hours, -v-1w for the last week, …). You can then check the list to see if any services must be restarted (use service -e to list enabled services), configurations must be verified, etc.

Remember that some services may depend on other packages. E.g. if you updated Python, you may have to restart Radicale and Fail2ban, and if you updated PHP and you use mod_php, you may want to restart Apache.

If you’re going to play around with that database to see what other info you can extract from it, you should probably make a copy of it, to make sure you don’t accidentally write to the original; you don’t want to mess up your package database.

P.s.: the name for the alias comes from ‘Ports UPdated ToDay’; change it to anything you like.

Related: pupl

Multigrep

Please keep in mind that this post is about 4 years old.
Technology may have changed in the meantime.

I am an author. And even though the actual books are well-organized, the writing process isn’t always. For a single book I have many mega-bytes of PDFs, ODTs and text files full of notes, drafts, documentation, etc.
So I needed a simple tool to find that one note I once wrote, in that huge pile of data.
Now, if those files were all text-based, I could use grep. But they aren’t. So I wrote a wrapper around grep, that allows me to also search PDFs and OpenDocument files.

Read More

HEAD requests

Please keep in mind that this post is about 4 years old.
Technology may have changed in the meantime.

A quicky this time.

I needed a simple script that would make a HEAD request for a certain URL, and then make a new HEAD request if the result was a redirect, etc.

Read More

Installation et configuration d’un serveur internet

Please keep in mind that this post is about 4 years old.
Technology may have changed in the meantime.

Et voilà mon nouveau livre !

Ce livre guide le lecteur ou la lectrice à travers l’installation et la configuration d’un serveur internet.

Ce livre s’adresse aux administrateurs système, débutants comme plus expérimentés, qui souhaitent, à partir d’un serveur sur lequel seul le système d’exploitation est installé, configurer un serveur internet d’entreprise fonctionnel, prêt à être mis en production.

Pour bien appréhender la lecture, un minimum de connaissances sur Unix/Linux, sur le fonctionnement de l’interface en ligne de commande et la configuration à l’aide des fichiers texte est conseillée. La connaissance de commandes de base telles que cd, ls, cat, less, tar et gzip est également un plus.

Après un chapitre sur les bases d’un système Unix/Linux, l’auteur amène rapidement le lecteur au cœur de la mission d’administration système avec la mise en œuvre de la configuration d’un serveur, illustrée avec plusieurs systèmes d’exploitation tels que FreeBSD, Debian et CentOS.

A l’aide d’exemples de configuration et de commandes, l’auteur explique étape par étape l’installation et la configuration d’un pare-feu, d’un serveur DNS, d’un serveur web (Apache ou Nginx) et d’un serveur mail.

Il détaille également le chiffrement par SSL/TLS des connexions (web et courriels) ainsi que la gestion dans un annuaire LDAP des utilisateurs n’ayant pas besoin de l’accès shell. Ce livre propose également des pistes pour l’analyse de problèmes éventuels, pour la maintenance quotidienne et les sauvegardes, ainsi que pour donner la possibilité à l’administrateur système de faire évoluer le serveur.

Des éléments complémentaires sont en téléchargement sur le site de l’éditeur et sur le site de l’auteur.

→ Plus d’informations sur www.librobert.net, ou commandez-le directement dans la boutique en ligne de mon éditeur Éditions ENI.

Ce livre est également disponible en néerlandais.
La version anglaise sortira début 2021.

Cleaning up the web server logs

Please keep in mind that this post is about 4 years old.
Technology may have changed in the meantime.

I am between 2 books currently, author-wise. So I’m using this time to do the server-housekeeping I’ve been putting of in the past months. Today, I had a look at my web server logs. And I almost regretted it instantly.

The logs were full of requests clearly made by script kiddies. Half of the error-triggering requests were for the file wp-login.php, even for my non-WordPress sites, and what’s worse: even for non-PHP sites. And then there were requests for the usual 1337w0rm, indoxploit, adminer, etcetera. I could have left it, of course, since it doesn’t really hurt to have all these idle requests, but all this noise makes it difficult to distil the log messages that really do need my attention.

In this post, I’ll share what I did to have cleaner Apache log files in the future.
And at the same time, these instructions help protect the web server against known vulnerabilities and exploits.

Read More

Configuratie van een internetserver

Please keep in mind that this post is about 4 years old.
Technology may have changed in the meantime.


Vanaf deze week is mijn boek uit!
En dan bedoel ik niet een boek dat ik bezit, maar het boek dat ik geschreven heb.
En ik bedoel ook niet uitgelezen, maar uitgegeven.

Configuratie van een internetserver beschrijft de volledige installatie en configuratie van een bedrijfsinternetserver (die uiteraard ook gebruikt mag worden voor een familie of een vriendengroep). Na een paar paragrafen over de selectie van de provider en de server, en een hoofdstuk met wat Unix-basiskennis, wordt de beheerder bij de hand genomen, en in rap tempo naar het ‘echte’ systeembeheer begeleid.

Het uiteindelijke resultaat is een volledig functionele bedrijfsinternetserver die klaar is om in productie genomen te worden. Deze server handelt DNS-, web-, en e-mailverkeer af voor meerdere domeinen, en bedient tientallen of zelfs honderden gebruikers; gebruikers die geen shell-toegang nodig hebben worden opgenomen in een LDAP directoryservice. Gebruikers kunnen bestanden delen en synchroniseren op al hun apparaten, en de e-mailgebruikers beschikken over gedeelde kalenders en adresboeken. De server wordt beveiligd middels een firewall, en de web- en e-mailverbindingen worden versleuteld middels SSL/TLS. Met dit boek in de hand heeft de systeembeheerder bovendien de nodige handvatten om de server te onderhouden, en om verder te groeien.

Dit boek vereist geen diepgaande kennis van Unix; een eerste geslaagde kennismaking zou genoeg moeten zijn. De lezer of lezeres wordt geacht bekend te zijn met het concept opdrachtregel (command line), en met de configuratie middels tekstbestanden. Ook wordt hij of zij geacht basiscommando’s als cd, ls, cat, less, tar en gzip te kennen.
Dit boek is echter niet specifiek gericht op de beginnende beheerder: het probeert deze beginnende beheerder zo snel mogelijk te begeleiden naar ‘echt’ systeembeheer.

Het boek is rijkelijk voorzien van configuratievoorbeelden, en alle onderwerpen worden beschreven en geïllustreerd voor FreeBSD, Debian en CentOS, zodat de systeembeheerder volledig vrij is in de keuze van het besturingssysteem.

→ Meer informatie op www.librobert.net, of bestel het direct in de webshop van mijn uitgever Boom Beroepsonderwijs.

De Franse versie is inmiddels ook zo goed als af, en komt eind september of begin oktober uit.
En ik ben in gesprek met een Amerikaanse uitgever voor de publicatie van de Engelstalige versie.

En voor wie zich afvraagt waarom ik op de kaft Robert heet: Rob is een lastige naam voor Fransen, dus in Frankrijk noem ik me Robert (de naam die overigens ook in mijn paspoort staat). En omdat het boek ook in het Frans uitkomt, en het me een goed idee lijkt om op elke uitgave dezelfde naam te vermelden, is het Robert geworden. In de omgang blijf ik gewoon Rob.