raspberry pi Archives | Stubborn Fanatic

If your Pi-hole admin page stopped loading after the v6 upgrade: try http://<pi-address>:8080/admin, or https://<pi-address>/admin and accept the certificate warning. That was my whole problem. If neither loads, Step 5 shows how to find the port in a minute. The rest of this post is how I found it the long way round, and what I changed while I was in there.

On 25 October 2026 the Pi-hole admin page would not load. The browser said the site could not be reached and that the Pi had refused to connect. It had said the same when I last tried, in 2025, and I had let it go.

This time I also found I had forgotten both passwords: the one for the Pi-hole web page and the one for the Pi’s own login. SSH answered, but pi / raspberry was refused. That is expected. Raspberry Pi OS images have shipped without a default user since 2022.

One thing did not fit. The internet in the house still worked, and ads were still blocked.

I did the work over two days with Claude. I made the decisions and ran every command on the Pi; the diagnosis plan, the scripts and some of the wrong turns came from the model. The wrong turns are in here as well, because they were the useful part.

New to the command line? How to read the boxes

PS> means: type this in PowerShell on Windows (Start menu → type powershell → Enter). $ means: type this in a terminal on Linux or a Mac, or on the Pi after you have logged in with ssh. Do not type the $ or PS> itself.

Each line that starts with $ or PS> is one command. Click that $ to copy just that line, paste it, press Enter, and wait for it to finish before the next. A box with a Copy button at the top holds one command (or one file), so copy it whole. A dashed box shows what you should see back; you do not type it.

The setup

Raspberry Pi 4 Model B, on the official Raspberry Pi 15 W USB-C supply (5.1 V, 3 A).
SanDisk 128 GB microSD card, made in February 2024.
Wired by Ethernet to a mesh Wi-Fi router. The ISP’s fibre router sits in front of it.
Raspberry Pi OS Bookworm, desktop edition, installed in May 2024. Pi-hole v5 installed the same month, upgraded to v6 in July 2025.
The router hands out the Pi as the only DNS server. That is what makes ad-blocking cover every device in the house, and it is also why the Pi matters more than its size suggests.

In this article the Pi is 192.168.x.51 and the router is 192.168.x.1.

Test DNS before you decide Pi-hole is down

A dead web page and a dead Pi-hole are different things. From a Windows laptop:

nslookup doubleclick.net 192.168.x.51

The answer was 0.0.0.0. The Pi was answering DNS queries and blocking an ad domain. Pi-hole was working. Only its web page looked dead.

That changed the job. I was not repairing an outage. I was getting back into a box that worked, without breaking it on the way in.

Step 1: protect the house before touching the Pi

With the Pi as the only DNS server, pulling its card stops name resolution for every phone, TV and laptop in the house.

Changing the router’s DNS a few minutes before the work is not enough. Devices keep the DNS server they were given until their DHCP lease renews, which can take hours. So on the night of 25 October I set the router’s primary DNS to 1.1.1.1. By morning every device had picked it up and the Pi could go offline without anyone noticing.

I kept one rollback rule through the whole job: the original card, untouched, goes back in and the house is as it was.

When the work was done, I set the router’s primary DNS back to the Pi. The lease delay works the same way in reverse: each device returns to the Pi when its lease renews. To hurry a Windows laptop, run ipconfig /renew; on a phone, turn Wi-Fi off and on. Then nslookup doubleclick.net with no server named should answer 0.0.0.0.

Step 2: image the card first

I had no spare card, so any change would be made on the original. A full image came first.

On Windows I used Win32 Disk Imager:

Choose a file name for the image on a drive with 130 GB free.
Select the card’s drive letter.
Tick Read Only Allocated Partitions.
Press Read. Not Write. Write copies the image file onto the card and destroys what is on it.

128 GB at about 28 MB/s took roughly 75 minutes.

I then checked the image’s structure:

MBR signature 55aa present.
Partition 1: FAT, labelled bootfs.
Partition 2: ext4, labelled rootfs, ext4 magic number 0xEF53 in the right place, and its end matches the end of the image file, so the copy was not cut short.

That proves the layout, not every byte. Win32 Disk Imager’s Verify Only button compares the whole image with the card; it takes about as long again.

One side lesson. The image was the full 128 GB though the card turned out to be 6% used. “Read Only Allocated Partitions” stops at the end of the last partition, and on a Pi the root partition expands to fill the card at first boot. The size of the image says nothing about how full the card is.

I also copied the files from the boot partition, which Windows shows as a drive called bootfs. Those are the files I was about to edit.

Step 3: getting in without a password

By then I had a screen and a keyboard on the Pi, and it booted to its desktop. I did not try them. On Raspberry Pi OS the desktop user can often run sudo without a password, so a terminal there may have been enough to reset the password. I wanted SSH from the laptop, by key, for everything that followed: commands I could paste, files I could copy, and no password to remember. So I took the longer route.

Raspberry Pi Imager has the mechanism built in, for Bookworm images. When you set a user name or Wi-Fi in Imager, it writes a script to the boot partition and asks systemd to run it once at first boot, from the kernel command line. I used the same mechanism with my own script.

The reset needs three files on the boot partition: the script, pi_reset.sh; the laptop’s SSH public key, saved as pi_key.pub; and an edited cmdline.txt. cmdline.txt is a single line. The edit appends this to its end, with one space before it and no line break:

systemd.run=/boot/firmware/pi_reset.sh systemd.run_success_action=reboot systemd.unit=kernel-command-line.target

Bookworm mounts the boot partition at /boot/firmware; older images use /boot. I made the edit with a short Python helper that refuses to touch a line that already carries such an entry.

At boot the script adds the key for root and for every login user, enables SSH, writes pi_reset.log to the boot partition, removes its own entry from cmdline.txt and deletes itself. systemd then reboots the Pi. The script, the helper and the copy steps are in The kit at the end.

Three cautions:

Public key only. The private key never leaves the laptop.
Line endings. A script saved with Windows (CRLF) line endings does not run at all. Then nothing reboots the Pi and nothing removes the entry from cmdline.txt, so the Pi stops at the same point on every boot, with no network. The helper converts the script to Unix (LF) endings for you. If a Pi does hang this way, put the card in the laptop and copy your original cmdline.txt back.
Test the cleanup. If the cleanup damages cmdline.txt, the Pi will not boot. Before I put the card back, I ran the script’s cleanup against the edited cmdline.txt and compared the result with my copy of the original. It matched byte for byte. The kit has the one-line test.

The Pi booted, ran the script, rebooted, and this worked with no password:

ssh root@192.168.x.51

Debian’s default SSH setting, PermitRootLogin prohibit-password, refuses root passwords but allows root login by key. The log on the boot partition named my old user. The kit’s version of the script also writes the cleaned cmdline.txt line into the log, so the cleanup can be confirmed without opening the file.

One distraction: on the desktop edition a Wi-Fi password prompt appeared on the screen after the reboot. On a wired Pi it does not matter. Cancel it.

Step 4: diagnosis from the logs

Before changing anything I ran one script, pi_diag.sh (in The kit), that changes no settings. If the page had stopped answering, I wanted to know whether power, the card or the software was failing. The script gathers everything below into one report and removes hardware addresses from it.

Power. vcgencmd get_throttled returns a bit field of power and heat events since boot. It read 0x0. I then loaded all four cores for 60 seconds:

for i in 1 2 3 4; do timeout 60 sh -c 'while :; do :; done' & done; wait
vcgencmd get_throttled
vcgencmd measure_temp

Still 0x0, and no “Undervoltage” lines in the journal, which held entries back to May 2026. The supply passes. The temperature reached 73 °C; a GPIO fan overlay in config.txt switches the fan on at 70 °C, so that is the fan doing its job.

Card. The kernel reports the card’s identity:

cat /sys/block/mmcblk0/device/{name,manfid,oemid,date}

SN128, SanDisk, made 02/2024. No mmc or ext4 errors in the logs. df -h showed 6.5 GB used of 117 GB. The card passes.

Where the space went. /var/log held 785 MB. Pi-hole’s query database held 325 MB, a year of history. gravity.db and gravity_old.db were about 145 MB each. The rest was the desktop: Chromium, Firefox, VLC, CUPS, VNC. A DNS server needs none of them.

Restarts. About 20 boots since May. The times are unreliable, because the Pi has no clock battery and restores the last saved time at boot. Power cuts are the likely cause, and I found no damage from them. A plug that sits loose in its socket is worth checking if your count is higher than your power cuts.

Pi-hole. pihole -v showed v6 (Core 6.1.4, Web 6.2.1, FTL 6.2.3). The apt history dates the upgrade to July 2025. lighttpd was inactive and disabled. In v5 that would be the fault. In v6 it is correct, because FTL now serves the web page itself.

Before I changed anything I took a settings backup:

pihole-FTL --teleporter

That writes a Teleporter zip in the current folder. The v5 command, pihole -a -t, no longer exists in v6.

Step 5: why the Pi-hole admin page was not loading

pihole-FTL --config webserver.port

returned:

8080o,443os,[::]:8080o,[::]:443os

and ss -tlnp | grep -i ftl showed FTL listening on 8080, 443 and 53.

When Pi-hole upgraded from v5 to v6 in July 2025, its new web server ended up on port 8080. That is a known outcome of v5-to-v6 upgrades; the cause is not settled. http://192.168.x.51:8080/admin had worked the whole time.

Nothing was broken. The page had not died twice. It had been on 8080 since July 2025, and I had looked at port 80 twice. My bookmark pointed at port 80, and nothing listens there any more.

What would have found this in a minute. I did not need the card image or the reset to see it. From the laptop, in PowerShell:

Test-NetConnection 192.168.x.51 -Port 80
Test-NetConnection 192.168.x.51 -Port 8080

TcpTestSucceeded : False on 80 and True on 8080 tells the whole story. On Linux or a Mac, nc -zv 192.168.x.51 8080 does the same. I would still have needed a way in for the forgotten passwords. But I would have gone in knowing the box was healthy, and the morning would have been shorter.

I kept 8080 and changed the bookmark. To move it back to 80, first make sure nothing else holds port 80, then change the setting and check:

sudo systemctl disable --now lighttpd
sudo pihole-FTL --config webserver.port '80o,443os,[::]:80o,[::]:443os'
sudo systemctl restart pihole-FTL
sudo ss -tlnp | grep ':80 '

The last line should show pihole-FTL. The o in 80o makes the port optional: if something else holds it, FTL skips it without an error. If you use the health check in the kit, set its WEB_URL to the new port.

Step 6: modernise in place, no reflash

The power, the card and the software were all healthy. A clean reinstall would have fixed nothing and cost a morning. Bookworm is still supported: Debian handed security support for it to the Debian LTS team in July 2026, and LTS runs to 30 June 2028.

Boot to the console, not the desktop.

sudo systemctl set-default multi-user.target

Update the OS and keep security updates coming.

sudo apt-get update && sudo apt-get -y full-upgrade
sudo apt-get install -y unattended-upgrades

In /etc/apt/apt.conf.d/20auto-upgrades:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

I left automatic reboots off. I would rather reboot a DNS server when I choose.

This Pi runs 64-bit Raspberry Pi OS. Its Debian packages match the default rule in unattended-upgrades, so Debian security fixes install themselves. The kernel and firmware come from Raspberry Pi’s own archive and still need sudo apt full-upgrade by hand; the monthly report below says when a reboot is due. To see what would install on your own Pi: sudo unattended-upgrade --dry-run -d.

Cap the journal at 64 MB. My Pi already kept its logs on disk; the setting makes that explicit. The folder may not exist yet:

sudo mkdir -p /etc/systemd/journald.conf.d

Then, in /etc/systemd/journald.conf.d/size.conf:

[Journal]
Storage=persistent
SystemMaxUse=64M

then sudo systemctl restart systemd-journald.

Update Pi-hole.

sudo pihole -up

This took it to the current v6 release, and then printed “Everything is up to date!”, which reads as if nothing had happened. Confirm with pihole -v and cat /etc/pihole/versions rather than trusting the last line.

Keep 30 days of query history, not a year.

sudo pihole-FTL --config database.maxDBdays 30

Old rows are deleted daily. SQLite reuses the space rather than shrinking the file, so the database stops growing instead of getting smaller.

Set a new web password.

sudo pihole setpassword

Save the password in a password manager before you type it. I did not, forgot it within minutes, and reset it the same way.

Step 7: make it look after itself

The page had been “dead” for months without my knowing what was wrong. I wanted the Pi to tell me, and to fix the one failure a restart can fix, without making anything worse.

A health check every 15 minutes, run by a systemd timer. In order:

Pause during updates. If Pi-hole is updating itself or rebuilding its block lists, it may stop its own DNS on purpose, so the check stands aside. If that lasts over 30 minutes, it alerts and checks anyway.
Is DNS answering at all? It asks Pi-hole for pi.hole. Any reply counts, even “no such name”. Only no reply at all is a DNS failure. A failed check is repeated once after 5 seconds, so one dropped packet changes nothing.
If DNS is not answering: restart pihole-FTL, wait up to 60 seconds for it, and report whether it came back. At most two restarts in six hours; after that it only alerts, so a fault a restart cannot fix does not keep cutting the house’s DNS.
If DNS answers, three more checks, each of which only alerts: the test domain (added once with pihole deny health.blocked.test) must come back 0.0.0.0, or ad-blocking may have stopped; the admin page at WEB_URL must answer, where a redirect or a login prompt counts as alive; and an outside name must resolve, or the internet is down. None of these is fixed by a restart, and a restart would cut DNS for the whole house.
The card and the power. An alert when the card passes 80%. An alert when the under-voltage or throttling flags first appear (get_throttled bits 16 and 18, mask 0x50000), not again every day.
Limits. Every run has a five-minute ceiling, so a stuck command cannot quietly end all monitoring. State lives in /run, in RAM; the only writes to the card are systemd’s log lines for each run.

Notifications go through ntfy, a free phone app that needs no account. You subscribe to a topic, and anything posted to that topic arrives on the phone:

curl -s -d "Pi-hole DNS failed after restart" https://ntfy.sh/<your-long-random-topic>

Anyone who knows the topic name can read it and post to it, so make it long and random and treat it like a password. Repeat alerts are suppressed: once an hour for DNS down, every six hours for the web page and the internet, once a day for the card.

The worst-case message says how to restore the house without me: open the router, go to DHCP settings, set the primary DNS to 1.1.1.1. Whoever reads it does not need to know what a Pi-hole is. The check cannot see what the router hands out, so it will not notice a house left on 1.1.1.1; ipconfig /all on a laptop shows which DNS server it was given.

One trap, and it was real on my Pi. That worst-case alert is sent when DNS is down, and curl has to look up ntfy.sh before it can send anything. My Pi uses Pi-hole as its own resolver, so in that moment the lookup would fail and the alert would never leave. To see whether your Pi is in the same position:

cat /etc/resolv.conf

A nameserver line of 127.0.0.1, ::1 or the Pi’s own address means it is. The kit closes it twice. The alert helper looks up ntfy.sh through 1.1.1.1, or 9.9.9.9 if that fails, and hands the address to curl. And, optionally, the Pi’s own lookups get a fallback: Pi-hole first, then 1.1.1.1. That keeps updates, Tailscale and alerts working on the Pi when Pi-hole is down. The house still uses only the Pi.

Why every 15 minutes? The check takes well under a second of CPU, so cost is not the limit. The interval sets how long the house can be without DNS before the Pi repairs itself. I started at 5 minutes and settled on 15. Choose 5 if you want faster repair.

A monthly report to the phone on the 1st: Pi-hole versions and whether an update waits, OS packages waiting, whether automatic security updates are switched on, whether a reboot is due (the running kernel against the installed one), the Pi’s own DNS servers, the date of the last backup, card use, database size, power flags and uptime. Pi-hole itself I update by hand on that prompt. Pi-hole’s developers advise against unattended Pi-hole updates, and I agree with them for a box everything depends on.

A weekly settings backup every Sunday with pihole-FTL --teleporter, keeping the last four, readable by root only because they hold the web password hash. One warning that cost nothing to learn: with no argument, --teleporter exports a backup; given a file name, it imports that file and overwrites your settings. The backups sit on the same card, so I copy them to the laptop now and then.

How it was tested. Every failure path ran against mock versions of the Pi’s commands: healthy, web page down, test domain not blocked, internet down, a single brief failure, DNS dead and recovering, DNS dead for good, a third failure within six hours, an update in progress, and the power flags. The tests passed and still missed the trap above: a mocked curl never looks anything up. A review also found that an earlier version restarted Pi-hole when only the web page failed, which would have cut the house’s DNS to fix a page. That is why only “no reply at all” now leads to a restart.

The scripts, their timers and the install steps are in The kit.

Step 8: remote administration with Tailscale

The last piece was reaching the Pi to administer it from outside the house. I used Tailscale:

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --accept-dns=false
sudo tailscale set --auto-update

--accept-dns=false matters on a Pi-hole. Without it, Tailscale rewrites the Pi’s own DNS settings.

In the Tailscale admin console I disabled key expiry for the Pi. By default a device’s key expires after about 180 days and the device drops off the network. That will happen when you are furthest from it.

SSH now works from anywhere through the Pi’s Tailscale address. Port forwarding looked like a dead end: the ISP appears to use carrier-grade NAT, because its public hostname contains “cgn”. The firmer test is the internet-side address on the ISP router; one in 100.64.0.0/10 means carrier-grade NAT, and there is no public address to forward. A mesh VPN is the practical route.

Tailscale warned that “UDP GRO forwarding is suboptimally configured on eth0”. That matters only if the Pi forwards heavy UDP traffic as a subnet router or exit node. For SSH, ignore it.

Note, 27 October 2026. On 25 October a researcher published “Tailworm”, a report of 11 flaws in Tailscale 1.104.2, the current version at the time. The serious ones are in Tailscale SSH, Tailscale’s own SSH server, and need an attacker who already controls a device that is allowed to log in. This Pi does not run Tailscale SSH. I reach it with ordinary OpenSSH over the tailnet, which the report does not touch. Auto-update is on, so the fix will arrive when Tailscale ships one. At the time of writing Tailscale had issued no bulletin, CVE or fixed version.

To check your own devices, run tailscale debug prefs on each one. "RunSSH": false means Tailscale SSH is off.

Two network surprises

1. A second, unfiltered Wi-Fi. At one point the laptop could not reach the Pi by its LAN address though everything looked right. ipconfig showed an address in a different range from the mesh, with the ISP’s DNS servers. The laptop had joined the ISP fibre router’s own Wi-Fi, not the mesh. Every device on that network bypasses Pi-hole. The fix: move anything that still uses that Wi-Fi to the mesh or to a cable, then switch off the ISP router’s Wi-Fi. If you run a mesh behind an ISP router, check that the ISP router’s Wi-Fi is off.

2. The wrong window. Test-NetConnection is a PowerShell command. In Command Prompt it fails with “not recognized”. Several of my “commands not working” moments were the right command in the wrong window.

What I thought, and what it was

I thought the Pi-hole had died twice. It had moved its web page once, during an upgrade, and I had been knocking on the old door.

The morning of 26 October 2026 went on imaging the card, getting back in and reading logs. By noon the Pi was updated, watching itself, and serving the house again. By the afternoon I could reach it from anywhere.

What I would tell anyone with a Pi-hole whose page has vanished:

Test DNS first. nslookup an ad domain against the Pi.
Check which ports answer (Test-NetConnection <pi> -Port 8080), then try :8080/admin.
Move the house off the Pi’s DNS the night before you touch it.
Image the card and check the image before you edit anything.
Save the new password before you type it.
Point the router back at the Pi when you are done.

The kit

Every script in this post, ready to copy. The names carry a pi_ prefix; rename them if you like, but keep the names the same across scripts, units and commands. The scripts on the Pi run as root.

1. Reset: back in without a password

You need a laptop that can read the card’s boot partition (Windows can), a card reader, an SSH key pair on the laptop, and Python 3 for the helper.

Check for an SSH key on the laptop. In PowerShell: Test-Path $env:USERPROFILE\.ssh\id_ed25519.pub. If it says False, create one with ssh-keygen -t ed25519, accept the default file and set a passphrase. If it says True, do not create another: a new key replaces the old one and breaks anything that uses it.
Make a folder for the kit. Save pi_reset.sh and pi_make_cmdline.py (below) in it. Check that Windows has not added .txt to either name. Copy the public key in beside them: Copy-Item $env:USERPROFILE\.ssh\id_ed25519.pub .\pi_key.pub
Power the Pi off and put its card in the reader. If Windows offers to format a drive, click Cancel: that is the Linux partition, which Windows cannot read.
Copy the whole boot drive (labelled bootfs, or boot on older images) to a backup folder. Copy its cmdline.txt into the kit folder as cmdline_old.txt.
In the kit folder run python pi_make_cmdline.py cmdline_old.txt bootfs (use boot for an older image). It prints the old and new line, writes a new cmdline.txt, and converts pi_reset.sh to Unix (LF) line endings. Your cmdline_old.txt is not changed.
Test the cleanup before you trust it. This applies the script’s cleanup to the new line and compares the result with the original:

python -c "import re;a=open('cmdline_old.txt',newline='').read();b=re.sub(r' systemd\.run.*','',open('cmdline.txt',newline='').read());print(a==b)"

It must print True. If it prints False, stop and compare the two files by eye.

Copy pi_reset.sh, pi_key.pub and the new cmdline.txt to the boot drive. Let it replace cmdline.txt; the original is in your backup.
Eject the drive, put the card back and power on. Wait about four minutes: boot, script, reboot, boot. Then ssh root@192.168.x.51.

If the key is refused, power off, put the card in the laptop and read pi_reset.log on the boot drive. It lists the users the key was added for.
If the Pi does not answer at all after ten minutes, the script did not run and the Pi stops at the same point on every boot. Power off, put the card in the laptop, and copy cmdline_old.txt from your kit folder onto the boot drive as cmdline.txt. That returns the card to where it started.

Once in, give your own user a password and save it in a password manager, so access no longer depends on one laptop’s key: passwd <your-user>. If you would rather not leave root login open, first open a second window and check that ssh <your-user>@192.168.x.51 and sudo -v both work. Then:

echo 'PermitRootLogin no' > /etc/ssh/sshd_config.d/no-root.conf
systemctl restart ssh

Keep the first window open until a new login as your user works. With root login off, the later commands in this kit change: log in as <your-user>@, run the diagnosis with sudo bash, and copy backups with sudo cp into your home folder before scp.

pi_reset.sh:

#!/bin/bash
# pi_reset.sh - one-shot, headless access fix for the Pi.
# Run once by systemd at boot via cmdline.txt "systemd.run=...". It:
#   - appends the laptop's PUBLIC key (pi_key.pub on the boot partition) to
#     root's and every login user's authorized_keys (no password is read or set);
#   - enables ssh; writes pi_reset.log on the boot partition (user names, OS);
#   - removes its own cmdline entry and itself. systemd then reboots the Pi.
set +e
B=/boot/firmware
[ -f "$B/cmdline.txt" ] || B=/boot
LOG="$B/pi_reset.log"
addkey() {  # append the key on its own line, even if the file lacks a final newline
  [ -s "$1" ] && [ -n "$(tail -c1 "$1")" ] && echo >> "$1"
  cat "$B/pi_key.pub" >> "$1"
}
{
  echo "pi_reset start $(date -Is)"
  echo "boot dir: $B"
  if [ -s "$B/pi_key.pub" ]; then
    install -d -m 700 /root/.ssh
    addkey /root/.ssh/authorized_keys
    chmod 600 /root/.ssh/authorized_keys
    echo "key added for root"
    getent passwd | awk -F: '$3>=1000 && $3<60000 {print $1 ":" $6}' |
    while IFS=: read -r u h; do
      [ -d "$h" ] || continue
      g=$(id -gn "$u")
      install -d -m 700 -o "$u" -g "$g" "$h/.ssh"
      addkey "$h/.ssh/authorized_keys"
      chown "$u:$g" "$h/.ssh/authorized_keys"
      chmod 600 "$h/.ssh/authorized_keys"
      echo "key added for user: $u"
    done
  else
    echo "ERROR: $B/pi_key.pub missing or empty - nothing changed"
  fi
  systemctl enable ssh
  echo "sshd PermitRootLogin lines (none = Debian default prohibit-password, key login allowed):"
  grep -hiE '^[[:space:]]*PermitRootLogin' /etc/ssh/sshd_config /etc/ssh/sshd_config.d/*.conf 2>/dev/null
  # shellcheck disable=SC1091
  echo "os: $(. /etc/os-release; echo "$PRETTY_NAME")"
  echo "pi_reset end $(date -Is)"
} >> "$LOG" 2>&1
sed -i 's| systemd.run.*||g' "$B/cmdline.txt"
echo "cmdline.txt after cleanup: $(cat "$B/cmdline.txt")" >> "$LOG"
rm -f "$B/pi_reset.sh"
sync
exit 0

pi_make_cmdline.py:

#!/usr/bin/env python3
# pi_make_cmdline.py - build the reset cmdline.txt.
# usage: python3 pi_make_cmdline.py cmdline_old.txt bootfs|boot
# Reads the old card's cmdline.txt copy, appends the one-shot systemd.run entry,
# writes cmdline.txt next to it (LF ending). The original copy is not changed.
# If pi_reset.sh sits in the same folder, it is converted to Unix (LF) line endings.
import sys, os, hashlib
src, label = sys.argv[1], sys.argv[2]
base = {"bootfs": "/boot/firmware", "boot": "/boot"}[label]
raw = open(src, "rb").read()
line = raw.decode("ascii").replace("\r", "").strip()
if "\n" in line:
    sys.exit("STOP: cmdline has more than one line - check it by hand")
if "systemd.run" in line or "init=/bin/sh" in line:
    sys.exit("STOP: cmdline already carries a run/init entry - check it by hand")
new = line + f" systemd.run={base}/pi_reset.sh systemd.run_success_action=reboot systemd.unit=kernel-command-line.target\n"
out = os.path.join(os.path.dirname(os.path.abspath(src)), "cmdline.txt")
open(out, "wb").write(new.encode("ascii"))
print("old :", line)
print("new :", new.rstrip())
rs = os.path.join(os.path.dirname(os.path.abspath(src)), "pi_reset.sh")
if os.path.exists(rs):
    b = open(rs, "rb").read()
    if b"\r" in b:
        open(rs, "wb").write(b.replace(b"\r\n", b"\n"))
        print("pi_reset.sh: converted to LF line endings")
    else:
        print("pi_reset.sh: line endings already LF")
else:
    print("WARNING: pi_reset.sh not found next to", src)
print("sha256 old copy", hashlib.sha256(raw).hexdigest()[:16], "new", hashlib.sha256(new.encode()).hexdigest()[:16])

2. Diagnosis (changes no settings)

One run collects power, card, log, disk and Pi-hole facts into a report in /root, with hardware addresses removed. It also writes a Teleporter backup and a copy of /etc/pihole without the query database, both to the card itself; on my Pi that copy was about 130 MB, because it includes the blocklist databases. It takes about 70 seconds, most of it the 60-second load test. From the laptop:

scp pi_diag.sh root@192.168.x.51:/root/
ssh root@192.168.x.51 "bash /root/pi_diag.sh"

pi_diag.sh:

#!/bin/bash
# pi_diag.sh - diagnosis of a Pi-hole card. Changes no settings.
# Run as root:  bash /root/pi_diag.sh
# Writes /root/pi_diag_<date>.txt (hardware addresses scrubbed), a Pi-hole
# Teleporter backup and a copy of /etc/pihole (without the query database) in /root.
# Changes no settings. Includes a 60-second full-CPU load test to check the power supply.
set +e
TS=$(date +%Y%m%d_%H%M)
OUT=/root/pi_diag_$TS.txt
scrub() { sed -E 's/([0-9A-Fa-f]{2}[:-]){5}[0-9A-Fa-f]{2}/[hw-addr removed]/g'; }
sec() { echo; echo "===== $* ====="; }
logs() { dmesg 2>/dev/null; journalctl -k --no-pager 2>/dev/null; zcat -f /var/log/kern.log* /var/log/syslog* /var/log/messages* 2>/dev/null; }
{
  sec "when"; date -Is; uptime
  sec "os"; cat /etc/os-release; uname -a
  sec "model"; tr -d '\0' < /proc/device-tree/model; echo
  sec "power: throttled flags since this boot (0x0 = never)"; vcgencmd get_throttled; vcgencmd measure_volts core; vcgencmd measure_temp
  sec "power: undervoltage lines in kept logs"; logs | grep -iE 'under-?voltage|voltage normali[sz]ed' | sort | uniq -c | tail -40
  sec "card: identity"; for f in name manfid oemid date type; do printf '%s: ' "$f"; cat "/sys/block/mmcblk0/device/$f" 2>/dev/null; done
  sec "card: errors in kept logs"; logs | grep -iE 'mmc0|mmcblk|I/O error|EXT4-fs (error|warning)|Buffer I/O|remount.*read-only' | sort | uniq -c | sort -rn | head -60
  sec "journal boots kept"; journalctl --list-boots --no-pager 2>/dev/null | tail -20
  sec "reboots and shutdowns"; last -x reboot shutdown 2>/dev/null | head -20
  sec "disk"; df -h; du -sh /var/log /etc/pihole /var/log/pihole 2>/dev/null; ls -la /etc/pihole
  sec "pihole version and status"; pihole -v 2>&1; pihole status 2>&1
  sec "failed services"; systemctl --no-pager --failed
  for s in pihole-FTL lighttpd; do sec "service $s"; systemctl --no-pager -l status "$s" 2>&1 | head -30; done
  sec "lighttpd error logs"; tail -60 /var/log/lighttpd/error*.log 2>/dev/null
  sec "php"; php -v 2>&1 | head -2; ls /etc/php 2>/dev/null; grep -rh php /etc/lighttpd/conf-enabled/ 2>/dev/null | head -20
  sec "apt history (dates, commands, php, lighttpd, pihole)"; zcat -f /var/log/apt/history.log* 2>/dev/null | grep -E '^(Start-Date|Commandline)|php|lighttpd|pihole' | tail -150
  sec "FTL log tail"; tail -40 /var/log/pihole/FTL.log /var/log/pihole-FTL.log 2>/dev/null
} 2>&1 | scrub > "$OUT"
cd /root || exit 1
{
  sec "teleporter backup"
  if pihole -v 2>/dev/null | grep -q 'version is v6'; then pihole-FTL --teleporter; else pihole -a -t; fi
  tar czf "/root/pihole_etc_$TS.tgz" --exclude='pihole-FTL.db*' --exclude='macvendor.db' /etc/pihole /etc/dnsmasq.d 2>&1
  sec "power: 60-second full-CPU load test"
  echo "before: $(vcgencmd get_throttled)"
  for i in 1 2 3 4; do timeout 60 sh -c 'while :; do :; done' & done; wait
  echo "after:  $(vcgencmd get_throttled)  $(vcgencmd measure_volts core)  $(vcgencmd measure_temp)"
  sec "files in /root"; ls -la /root
} 2>&1 | scrub >> "$OUT"
echo "REPORT WRITTEN: $OUT"

3. Health check, monthly report and weekly backup

All commands run on the Pi over SSH.

Tools. The scripts use dig and curl:

sudo apt-get install -y dnsutils curl

Settings file. This reads the port your admin page uses (the first plain-HTTP entry, skipping the HTTPS ones), makes a random ntfy topic, and writes both, with the test domain, to /etc/pi-health.conf. The scripts read it from there, so they hold no secret.

P=$(sudo pihole-FTL --config webserver.port | tr ',' '\n' | grep -v 's' | grep -oE '^[0-9]+' | head -1)
T="pihole-$(tr -dc a-z0-9 < /dev/urandom | head -c 20)"
printf 'TOPIC=%s\nHC_DOMAIN=health.blocked.test\nWEB_URL=http://127.0.0.1:%s/admin/\n' "$T" "$P" | sudo tee /etc/pi-health.conf
sudo chmod 600 /etc/pi-health.conf

tee prints the three lines. Check that WEB_URL shows your port (80 on a fresh v6 install, 8080 on many upgraded ones). Subscribe to the topic in the ntfy app. Running these lines again makes a new topic and replaces the old one, so your phone would need to subscribe again. If you later move the admin page to another port, change WEB_URL to match.

Optional: an outside “dead man’s switch” service gives you a ping address and alerts you when the pings stop, which covers a Pi that loses power or its internet. Add PING_URL=<that address> to the same file.

Test domain. Block it and check the answer:

sudo pihole deny health.blocked.test
dig +short @127.0.0.1 health.blocked.test

The second line must print 0.0.0.0. If it prints nothing, the health check will report that blocking has stopped on every run; fix this first.

Scripts. Save each one with sudo nano /usr/local/bin/<name> (paste, then Ctrl + O, Enter, Ctrl + X), then make them executable:

sudo chmod 755 /usr/local/bin/pi_ntfy.sh /usr/local/bin/pi_health.sh /usr/local/bin/pi_report.sh /usr/local/bin/pi_backup.sh

pi_ntfy.sh, the alert helper the other scripts call:

#!/bin/bash
# pi_ntfy.sh - send one phone notification through ntfy.sh.
# Usage: pi_ntfy.sh "Title" "message"     Exit 0 only if ntfy accepted it.
# ntfy.sh is looked up through 1.1.1.1, then 9.9.9.9 - not through Pi-hole - so alerts
# still go out when Pi-hole's DNS has failed. Falls back to a normal lookup if both fail.
set -u
# shellcheck disable=SC1091
. /etc/pi-health.conf          # TOPIC=...
ip="" r=()
for ns in 1.1.1.1 9.9.9.9; do
  ip=$(dig +short +time=3 +tries=1 @"$ns" ntfy.sh A 2>/dev/null | grep -m1 -E '^[0-9]+(\.[0-9]+){3}$')
  [ -n "$ip" ] && break
done
[ -n "$ip" ] && r=(--resolve "ntfy.sh:443:$ip")
curl -fsS -m 15 "${r[@]}" -H "Title: $1" -d "$2" "https://ntfy.sh/$TOPIC" >/dev/null

pi_health.sh:

#!/bin/bash
# pi_health.sh - Pi-hole health check, run every 15 minutes by pi-health.timer.
#  - pihole-FTL is restarted ONLY when DNS gives no reply at all. A web-page-only fault,
#    a test domain that is no longer blocked, or an odd answer is reported, never
#    "fixed" by a restart (a restart cuts the house's DNS).
#  - at most 2 restarts in 6 hours; after that it alerts only.
#  - a failed check is repeated once after 5 s before anything is done;
#  - after a restart, waits up to 60 s for DNS instead of a fixed 20 s;
#  - pauses while Pi-hole is updating itself or rebuilding its block lists, but alerts
#    and resumes checking if that has lasted over 30 minutes;
#  - alerts go through pi_ntfy.sh (ntfy.sh looked up via 1.1.1.1 / 9.9.9.9);
#  - the admin page address comes from WEB_URL in /etc/pi-health.conf;
#  - power alert only when the throttle flags change, not every day;
#  - optional PING_URL in /etc/pi-health.conf: pinged every run, so an outside
#    monitor can alert when the Pi goes silent (power, card or home-internet failure).
# State lives in /run (RAM); nothing is written to the card except the journal line.
set -u
# shellcheck disable=SC1091
. /etc/pi-health.conf          # TOPIC=..., HC_DOMAIN=..., WEB_URL=..., optional PING_URL=...
WEB_URL=${WEB_URL:-http://127.0.0.1:8080/admin/}
ST=/run/pi-health; mkdir -p "$ST"
now=$(date +%s)
notify() {  # key, seconds-between-repeats, message
  local f="$ST/$1" last
  last=$(cat "$f" 2>/dev/null || echo 0)
  if (( now - last >= $2 )); then
    /usr/local/bin/pi_ntfy.sh "Pi-hole" "$3" && echo "$now" > "$f"
  fi
}
q()          { dig +short +time=2 +tries=1 @127.0.0.1 "$1" A 2>/dev/null; }
# Any reply at all (NOERROR, NXDOMAIN, REFUSED...) means FTL is answering; dig exits 9
# only when no reply came back. Answered locally, so it needs no internet.
dns_alive()  { dig +time=2 +tries=1 @127.0.0.1 pi.hole A >/dev/null 2>&1; }
blocking_ok(){ [ "$(q "$HC_DOMAIN")" = "0.0.0.0" ]; }
web_ok()     { case "$(curl -s -o /dev/null -m 5 -w '%{http_code}' "$WEB_URL")" in 200|301|302|303|307|308|401) return 0;; esac; return 1; }
net_ok()     { q google.com | grep -qE '^[0-9]+(\.[0-9]+){3}$'; }
retry()      { "$@" || { sleep 5; "$@"; }; }

# Pi-hole updating itself or rebuilding block lists: FTL may be stopped on purpose.
# Matches only bash running Pi-hole's own gravity.sh / update.sh / basic-install.sh.
if pgrep -f '^(/usr)?(/bin/)?bash .*(/opt/pihole/(gravity|update)|basic-install)\.sh' >/dev/null; then
  [ -f "$ST/skip_since" ] || echo "$now" > "$ST/skip_since"
  if (( now - $(cat "$ST/skip_since") < 1800 )); then exit 0; fi
  notify skiplong 21600 "A Pi-hole update or block-list rebuild has been running over 30 minutes. Checking anyway."
else
  rm -f "$ST/skip_since"
fi

if ! retry dns_alive; then
  # restarts in the last 6 hours (timestamps kept in RAM)
  touch "$ST/restarts"
  recent=$(awk -v c=$((now - 21600)) '$1 >= c' "$ST/restarts" | wc -l)
  if (( recent >= 2 )); then
    notify down 3600 "Pi-hole DNS NOT answering; already restarted twice in 6 hours, not restarting again. House internet may be down. Fix now: router settings > DHCP > Primary DNS 1.1.1.1 > Save."
  else
    echo "$now" >> "$ST/restarts"
    systemctl restart pihole-FTL
    ok=""
    for _ in $(seq 12); do sleep 5; dns_alive && { ok=1; break; }; done
    if [ -n "$ok" ]; then
      notify recovered 3600 "Pi-hole DNS had stopped answering and recovered after a restart."
    else
      notify down 3600 "Pi-hole DNS NOT working after restart. House internet may be down. Fix now: router settings > DHCP > Primary DNS 1.1.1.1 > Save."
    fi
  fi
else
  retry blocking_ok || notify noblock 86400 "Pi-hole answers, but the blocked test domain ($HC_DOMAIN) is not blocked. Ad-blocking may not be working. No restart made."
  retry web_ok      || notify webdown 21600 "Pi-hole web page ($WEB_URL) not answering; DNS and ad-blocking still work. No restart made (it would cut the house's DNS). Fix: ssh in and run: systemctl restart pihole-FTL"
  retry net_ok      || notify net 21600 "Pi-hole is up but cannot reach the internet (ISP or router problem)."
fi

use=$(df --output=pcent / | tail -1 | tr -dc 0-9)
(( ${use:-0} >= 80 )) && notify disk 86400 "Card is ${use}% full."
thr=$(vcgencmd get_throttled 2>/dev/null | cut -d= -f2)
if [ -n "$thr" ] && (( (thr & 0x50000) != 0 )) && [ "$thr" != "$(cat "$ST/thr_seen" 2>/dev/null)" ]; then
  notify power 0 "Under-voltage or throttling has occurred since boot ($thr). Check the power supply." && echo "$thr" > "$ST/thr_seen"
fi
[ -n "${PING_URL:-}" ] && curl -fsS -m 10 -o /dev/null "$PING_URL"
exit 0

pi_report.sh:

#!/bin/bash
# pi_report.sh - monthly status to the phone (pi-report.timer).
# Sends through pi_ntfy.sh (works even if Pi-hole DNS is down);
# reports whether the running kernel is older than the installed one (reboot needed),
# whether automatic security updates are switched on, the Pi's own DNS servers and the
# last weekly backup; strips colour codes from Pi-hole's update check.
# shellcheck disable=SC1091
. /etc/pi-health.conf
apt-get -qq update >/dev/null 2>&1
upg=$(apt list --upgradable 2>/dev/null | grep -c upgradable)
ph=$(pihole -up --check-only 2>&1 | sed 's/\x1b\[[0-9;]*[A-Za-z]//g' | grep -iE 'update available|up to date|everything is up' | head -3 | tr -s ' \n' ' ')
k=$(uname -r); newest=$(ls -1dt /lib/modules/*"${k##*+rpt-}" 2>/dev/null | head -1)
if [ -n "$newest" ] && [ "$(basename "$newest")" != "$k" ]; then
  rb="YES (running $k, installed $(basename "$newest")) - run: systemctl reboot (back in about 2 min)"
else rb="no"; fi
au="$(apt-config dump 2>/dev/null | grep -E '^APT::Periodic::Unattended-Upgrade ' | grep -q '"1"' && echo on || echo OFF)"
bk=$(ls -1t /var/backups/pihole/*.zip 2>/dev/null | head -1)
if [ -n "$bk" ]; then bk="$(basename "$bk") ($(date -r "$bk" +%d/%m/%Y))"; else bk="NONE"; fi
msg="Monthly report $(date +%d/%m/%Y)
Pi-hole: $(pihole -v 2>&1 | tr '\n' ' ')
Pi-hole updates: ${ph:-unknown}
OS packages waiting: $upg. Debian security fixes install themselves (automatic: $au);
Raspberry Pi kernel/firmware and other updates need, by hand: apt update && apt full-upgrade
Reboot needed: $rb
Tailscale: $(tailscale status --json 2>/dev/null | grep -q '"BackendState": *"Running"' && echo connected || echo 'NOT connected')
Pi's own DNS: $(grep -E '^nameserver' /etc/resolv.conf | awk '{print $2}' | tr '\n' ' ')
Last backup: $bk
Card used: $(df -h --output=pcent / | tail -1 | tr -d ' ')
Query DB: $(du -h /etc/pihole/pihole-FTL.db 2>/dev/null | cut -f1)
Power flags since boot: $(vcgencmd get_throttled 2>/dev/null)
Up: $(uptime -p)"
/usr/local/bin/pi_ntfy.sh "Pi-hole monthly" "$msg"

pi_backup.sh:

#!/bin/bash
# pi_backup.sh - weekly Pi-hole Teleporter export, keeps the last 4 (pi-backup.timer).
# Files: /var/backups/pihole/*.zip, readable by root only (they hold the web password hash).
set -e
umask 077
D=/var/backups/pihole
mkdir -p "$D"; chmod 700 "$D"; cd "$D"
# WARNING: with NO argument, --teleporter EXPORTS a zip into the current directory
# (hence the cd above). With a file argument it IMPORTS that zip and overwrites the
# Pi-hole settings. Never add an argument here.
pihole-FTL --teleporter >/dev/null
ls -1t ./*.zip 2>/dev/null | tail -n +5 | xargs -r rm -f --

To copy the backups off now and then, from the laptop: scp "root@192.168.x.51:/var/backups/pihole/*.zip" .

Units. Save these six files in /etc/systemd/system/ the same way. The times in the two timers are examples; set your own.

pi-health.service:

[Unit]
Description=Pi-hole health check
[Service]
Type=oneshot
TimeoutStartSec=5min
ExecStart=/usr/local/bin/pi_health.sh

pi-health.timer:

[Unit]
Description=Pi-hole health check every 15 minutes
[Timer]
OnBootSec=5min
OnUnitActiveSec=15min
AccuracySec=30s
[Install]
WantedBy=timers.target

pi-report.service:

[Unit]
Description=Pi-hole monthly report
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
TimeoutStartSec=15min
ExecStart=/usr/local/bin/pi_report.sh

pi-report.timer:

[Unit]
Description=Pi-hole monthly report, 1st of the month 10:00
[Timer]
OnCalendar=*-*-01 10:00
Persistent=true
[Install]
WantedBy=timers.target

pi-backup.service:

[Unit]
Description=Pi-hole weekly Teleporter backup
[Service]
Type=oneshot
TimeoutStartSec=5min
ExecStart=/usr/local/bin/pi_backup.sh

pi-backup.timer:

[Unit]
Description=Pi-hole weekly backup, Sunday 11:00
[Timer]
OnCalendar=Sun 11:00
Persistent=true
[Install]
WantedBy=timers.target

Start and test.

sudo systemctl daemon-reload
sudo systemctl enable --now pi-health.timer pi-report.timer pi-backup.timer
systemctl list-timers 'pi-*' --no-pager
sudo /usr/local/bin/pi_backup.sh && sudo ls -la /var/backups/pihole
sudo /usr/local/bin/pi_report.sh
sudo systemctl start pi-health.service
systemctl status pi-health.service --no-pager

Expect three timers with their next run times, one backup zip, the monthly report on your phone within a minute, and status=0/SUCCESS for the health check. That status only says the check ran; it exits cleanly whatever it found. A healthy Pi sends no health alert, so the report is your proof that notifications reach the phone. The report runs apt-get update and Pi-hole’s update check; it installs nothing.

4. Optional: a fallback for the Pi’s own lookups

This gives the Pi itself a second DNS server, 1.1.1.1, after Pi-hole. The router still hands out only the Pi, so the house is unchanged. It applies to Raspberry Pi OS Bookworm, where NetworkManager writes /etc/resolv.conf; check that first:

grep 'Generated by NetworkManager' /etc/resolv.conf

If that prints nothing, skip this section. Otherwise, save this as pi_nm_dns.conf in your home folder:

# /etc/NetworkManager/conf.d/90-pi-dns.conf
# The Pi's OWN lookups (apt, Tailscale, alerts) use Pi-hole first, then 1.1.1.1 if
# Pi-hole is not answering. Affects only the Pi itself, not the house (the router still
# hands out the Pi as the only DNS server). Undo: move this file away and run
#   nmcli general reload
[global-dns-domain-*]
servers=127.0.0.1,1.1.1.1

A DNS change over SSH can cut you off, so arm an automatic undo first, then apply it:

sudo systemd-run --unit=pi-dns-undo --on-active=10min /bin/sh -c 'mv /etc/NetworkManager/conf.d/90-pi-dns.conf /root/ && nmcli general reload'
sudo install -m 644 ~/pi_nm_dns.conf /etc/NetworkManager/conf.d/90-pi-dns.conf
sudo nmcli general reload
grep nameserver /etc/resolv.conf

The last line should show 127.0.0.1 and then 1.1.1.1. If your SSH session still works and that is what you see, keep it by cancelling the undo:

sudo systemctl stop pi-dns-undo.timer

If anything looks wrong, do nothing: ten minutes later the file is moved to /root and the old settings return.

Category: raspberry pi

The Pi-hole that wasn’t dead: getting back in without a password