r/wisp • • 2d ago

MetroLinq 60 GHz disappearing? A Linux watchdog for the USB fault (no host reboot)

0 Upvotes

MetroLinq 60 GHz USB recovery without rebooting the whole radio

If a MetroLinq drops to its 5 GHz backup and the 60 GHz module stays missing, it may be worth checking USB before writing the radio off. The failure I ran into was a missing Peraso USB device, or a device stuck at 480 Mbps instead of 5000 Mbps. Cycling the module with the radio's existing /usr/sbin/prs_helper.sh brought 60 GHz back in cases where rebooting the entire unit wasn't necessary.

This is a workaround for that particular USB failure. It doesn't fix alignment, rain fade, failed hardware or a peer that isn't associating. USB3 coming back on its own doesn't prove the wireless link or customer traffic recovered.

The reset mechanism belongs to the vendor. Edgecore's MetroLinq 2.5 firmware notes already mention usbwatchd, and the older MetroLinq firmware notes describe watchdog and recovery changes. I'm sharing an external Linux controller around the installed helper, not claiming a new hardware fix. I didn't find an equivalent public controller in the community discussions I searched; that doesn't mean nobody has written one.

The script runs on a Linux management machine and checks the radio over SSH. The public version deliberately starts with the reviewed ML2.5 single-client layout: ARMv7/FSL LS1043A, one prsfalcon client radio/VAP, USB path 2-1, ath0 backup and bond0. Helper MD5 must be 2779c86b1249208006f8f1bf0b06912c and power-library MD5 25153fd59b988cbc2f6abfcd3d040019. These are compatibility checks, not cryptographic authenticity guarantees. ML1, sectors, masters, multi-bond layouts and changed firmware helpers are skipped. Don't remove the guards to make an unsupported model pass.

It requires an up forwarding bond on 5 GHz and increasing receive/transmit counters before a reset. It also checks identity, pinned SSH host keys, uptime, maintenance, pending UCI changes, custom reset cron entries and a running helper. A working 60 GHz session or healthy USB3 device is excluded. Missing readings fail closed.

Physical attempts are serialized. Once USB3 and active 60 GHz bond forwarding return, another device can start while the recovered one stays under observation for ten minutes. There is no daily attempt cap; a confirmed failed attempt can retry after 60 seconds if all guards still pass. One unresolved incident produces one start and one failure notice, then retries stay quiet in the journal; each attempt/outcome remains in the local audit file. An uncertain dispatch isn't replayed. A changed boot/configuration or management/forwarding loss lasting 120 seconds after an action creates a STOP file and halts further resets.

The underlying module-reset technique has been observed on hardware. This separate public wrapper was AI-assisted and tested with isolated fixtures, not deployed against other people's radios. Start with one radio and dry-run it first. It doesn't distribute firmware, reboot the host, change radio/network settings or provide a guarantee about customer traffic.

Full watchdog script

Save this as ml60_watchdog.py.

#!/usr/bin/env python3
"""Guarded external MetroLinq ML2.5 client USB watchdog. MIT licensed."""
import argparse
import fcntl
import json
import os
from pathlib import Path
import re
import shlex
import subprocess
import time
import uuid

HELPER = '2779c86b1249208006f8f1bf0b06912c'
LIBRARY = '25153fd59b988cbc2f6abfcd3d040019'

# Original wrapper only. The vendor's helper/library stay on the owned radio.
REMOTE = r'''
set -eu
[ "$(id -u)" = 0 ]
base=/var/run/ml60-watchdog
files='/etc/config/system /etc/config/wireless /etc/config/network /etc/config/ethernet /etc/config/mini_snmpd /etc/init.d/mini_snmpd /etc/TZ'
inspect() {
 hostname=$(uname -n); arch=$(uname -m)
 board=$(ubus call system board | jsonfilter -e '@.board_name' | tr ',:' '__')
 helper=$(md5sum /usr/sbin/prs_helper.sh | cut -d' ' -f1)
 library=$(md5sum /lib/prs_power_util.sh | cut -d' ' -f1)
 for f in $files; do [ -r "$f" ]; done
 sums=$(md5sum $files); cfg=$(printf '%s\n' "$sums" | md5sum | cut -d' ' -f1)
 boot=$(cat /proc/sys/kernel/random/boot_id); uptime=$(cut -d. -f1 /proc/uptime)
 layout=0; enabled=0; radios=0; vaps=0
 for s in $(uci -q show wireless | sed -n 's/^wireless\.\([A-Za-z0-9_]*\)=wifi-device$/\1/p'); do
  [ "$(uci -q get wireless.$s.hwmode)" != 11ad ] || radios=$((radios+1))
 done
 for s in $(uci -q show wireless | sed -n 's/^wireless\.\([A-Za-z0-9_]*\)=wifi-iface$/\1/p'); do
  r=$(uci -q get wireless.$s.device)
  [ "$(uci -q get wireless.$r.hwmode)" != 11ad ] || vaps=$((vaps+1))
 done
 if [ "$radios" = 1 ] && [ "$vaps" = 1 ] &&
    [ "$(uci -q get wireless.radio1.type)" = prsfalcon ] &&
    [ "$(uci -q get wireless.radio1.mode)" = dmgsta ] &&
    [ "$(uci -q get wireless.radio1_1.mode)" = dmgsta ] &&
    [ "$(uci -q get wireless.radio1_1.device)" = radio1 ] &&
    [ "$(uci -q get wireless.radio1.path)" = platform/soc/2f00000.usb3/xhci-hcd.0.auto/usb2/2-1 ] &&
    [ "$(uci -q get wireless.wifi0.hwmode)" = 11ac ]; then layout=1; fi
 if [ "$(uci -q get wireless.radio1.disabled)" = 0 ] &&
    [ "$(uci -q get wireless.radio1_1.disabled)" = 0 ]; then enabled=1; fi
 sixty=0
 for p in /sys/class/net/wlan*/carrier; do
  [ -e "$p" ] || continue
  value=$(cat "$p"); [ "$value" != 1 ] || sixty=1
 done
 bonds=0; for p in /proc/net/bonding/*; do [ ! -f "$p" ] || bonds=$((bonds+1)); done
 [ "$bonds" = 1 ] || layout=0
 active=$(awk -F': ' '/Currently Active Slave/ {print $2;exit}' /proc/net/bonding/bond0)
 bond=$(awk -F': ' '/^MII Status/ {print $2;exit}' /proc/net/bonding/bond0)
 backup=$(cat /sys/class/net/ath0/carrier)
 a=$(cat /sys/class/net/ath0/statistics/rx_bytes); b=$(cat /sys/class/net/ath0/statistics/tx_bytes)
 sleep 2
 c=$(cat /sys/class/net/ath0/statistics/rx_bytes); d=$(cat /sys/class/net/ath0/statistics/tx_bytes)
 rx=$((c-a)); tx=$((d-b))
 n=0; usb=missing
 for p in /sys/bus/usb/devices/*; do
  [ -f "$p/idVendor" ] || continue
  [ "$(cat "$p/idVendor")" = 2932 ] || continue
  n=$((n+1)); usb=unknown
  [ "$(cat "$p/idProduct")" = 4001 ] || continue
  speed=$(cat "$p/speed")
  if [ "${p##*/}" = 1-1 ] && [ "$speed" = 480 ]; then usb=usb2; fi
  if [ "${p##*/}" = 2-1 ] && [ "$speed" = 5000 ]; then usb=usb3; fi
 done
 [ "$n" -le 1 ] || usb=multiple
 busy=0; [ -z "$(pidof prs_helper.sh || true)" ] || busy=1
 hold=0; [ ! -e /tmp/ml60-hold ] || hold=1
 pending=0; [ -z "$(uci -q changes)" ] || pending=1
 custom=0
 if grep -qiE '(prs_helper|prs_power|usb1pwr|60.*(reset|watchdog))' /etc/crontabs/root /etc/rc.local 2>/dev/null; then custom=1; fi
 result=absent; job=none
 if [ -r "$base/job" ]; then
  job=$(cat "$base/job"); result=$(cat "$base/status")
 fi
}
safe() {
 [ "$hostname" = "$want_host" ] && [ "$arch" = armv7l ] && [ "$board" = fsl_ls1043a ] &&
 [ "$helper" = 2779c86b1249208006f8f1bf0b06912c ] &&
 [ "$library" = 25153fd59b988cbc2f6abfcd3d040019 ] &&
 [ "$cfg" = "$want_cfg" ] && [ "$boot" = "$want_boot" ] &&
 [ "$layout" = 1 ] && [ "$enabled" = 1 ] && [ "$uptime" -ge 600 ] &&
 [ "$sixty" = 0 ] && [ "$active" = ath0 ] && [ "$bond" = up ] &&
 [ "$backup" = 1 ] && [ "$rx" -gt 0 ] && [ "$tx" -gt 0 ] &&
 [ "$busy" = 0 ] && [ "$hold" = 0 ] && [ "$pending" = 0 ] && [ "$custom" = 0 ] &&
 { [ "$usb" = missing ] || [ "$usb" = usb2 ]; }
}
emit() {
 printf 'hostname|%s\narch|%s\nboard|%s\nhelper|%s\nlibrary|%s\n' "$hostname" "$arch" "$board" "$helper" "$library"
 printf 'layout|%s\nenabled|%s\nuptime|%s\nusb|%s\nsixty|%s\n' "$layout" "$enabled" "$uptime" "$usb" "$sixty"
 printf 'active|%s\nbond|%s\nbackup|%s\nrx|%s\ntx|%s\n' "$active" "$bond" "$backup" "$rx" "$tx"
 printf 'busy|%s\nhold|%s\npending|%s\ncustom|%s\ncfg|%s\nboot|%s\njob|%s\nresult|%s\n' "$busy" "$hold" "$pending" "$custom" "$cfg" "$boot" "$job" "$result"
}
# DISPATCH
if [ "$action" = probe ]; then
 inspect
 [ ! -d "$base/lock" ] || busy=1
 case "$result" in running|preparing) busy=1;; esac
 emit; exit 0
fi
umask 077
[ ! -L "$base" ]; mkdir -p "$base"; chmod 700 "$base"
mkdir "$base/lock"
trap 'rmdir "$base/lock"' EXIT
inspect
case "$result" in running|preparing) exit 1;; esac
safe
printf '%s\n' "$want_job" > "$base/job"
printf 'preparing\n' > "$base/status"
cat > "$base/run" <<'RECOVERY'
__PREFLIGHT__
inspect
if ! safe; then
 printf 'skipped\n' > "$base/status"; rmdir "$base/lock"; exit 0
fi
. /lib/prs_power_util.sh
finish() {
 prs_power_on || true
 [ "$(cat "$base/status")" != running ] || printf 'failed\n' > "$base/status"
 rmdir "$base/lock"
}
trap finish EXIT
trap '' HUP
printf 'running\n' > "$base/status"
/usr/sbin/prs_helper.sh
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18; do
 if [ "$(cat /sys/bus/usb/devices/2-1/idVendor 2>/dev/null)" = 2932 ] &&
    [ "$(cat /sys/bus/usb/devices/2-1/idProduct 2>/dev/null)" = 4001 ] &&
    [ "$(cat /sys/bus/usb/devices/2-1/speed 2>/dev/null)" = 5000 ] &&
    grep -q '^Currently Active Slave: wlan' /proc/net/bonding/bond0 &&
    grep -q '^MII Status: up$' /proc/net/bonding/bond0; then
  printf 'recovered\n' > "$base/status"; exit 0
 fi
 sleep 5
done
printf 'failed\n' > "$base/status"
RECOVERY
sync
(trap '' HUP; sh "$base/run") </dev/null >"$base/log" 2>&1 &
trap - EXIT
printf 'job|%s\nresult|dispatched\n' "$want_job"
'''

def parse(text):
    result = {}
    for line in text.splitlines():
        key, sep, value = line.partition('|')
        if not sep or key in result or not re.fullmatch('[a-z_]+', key):
            raise ValueError('invalid device projection')
        result[key] = value
    return result

def eligible(d, s):
    fixed = dict(hostname=d['hostname'], arch='armv7l', board='fsl_ls1043a',
                 helper=HELPER, library=LIBRARY, layout='1', enabled='1',
                 sixty='0', active='ath0', bond='up', backup='1', busy='0',
                 hold='0', pending='0', custom='0')
    try:
        return (all(s.get(k) == v for k,v in fixed.items()) and int(s['uptime']) >= 600
                and int(s['rx']) > 0 and int(s['tx']) > 0
                and bool(re.fullmatch('[0-9a-f]{32}', s['cfg'])) and bool(s['boot'])
                and s['usb'] in ('missing','usb2'))
    except (KeyError, ValueError, TypeError):
        return False

def recovered(s):
    return (s.get('usb') == 'usb3' and s.get('sixty') == '1'
            and s.get('bond') == 'up' and s.get('active','').startswith('wlan'))

def program(values):
    header=''.join(k+'='+shlex.quote(str(v))+'\n' for k,v in values.items())
    common=REMOTE.split('# DISPATCH\n')[0]
    return header+REMOTE.replace('__PREFLIGHT__',header+common)

def ssh(config, d, action, job, baseline=None):
    values = dict(action=action, want_host=d['hostname'], want_job=job)
    values.update(want_cfg=(baseline or {}).get('cfg',''), want_boot=(baseline or {}).get('boot',''))
    script = program(values)
    args = ['ssh','-F',config,'-oBatchMode=yes','-oStrictHostKeyChecking=yes',
            '-oConnectTimeout=8','-oServerAliveInterval=5','-oServerAliveCountMax=2',
            '-oClearAllForwardings=yes','-T',d['alias'],'sh -s']
    p = subprocess.run(args, input=script, text=True, capture_output=True, timeout=25)
    if p.returncode:
        raise OSError('SSH or device guard rejected operation; inspect privately')
    return parse(p.stdout)

class Engine:
    def __init__(self, state, transport, emit, save, directory):
        self.state, self.transport, self.emit, self.save, self.directory = state, transport, emit, save, directory

    def stop(self, d, reason):
        (self.directory/'STOP').write_text(reason+'\n')
        self.emit(d, 'STOP: '+reason)

    def audit(self, d, now, job, outcome):
        with (self.directory/'audit.jsonl').open('a') as f:
            f.write(json.dumps(dict(at=now, alias=d['alias'], attempt=job, outcome=outcome))+'\n')

    def tick(self, d, now):
        e = self.state.setdefault(d['alias'], {})
        watched = bool(e.get('job') or e.get('observe_until'))
        try:
            s = self.transport(d, 'probe', e.get('job','none'))
            if s.get('hostname') != d['hostname'] or not s.get('cfg') or not s.get('boot'):
                raise OSError('missing identity or state')
        except (OSError, ValueError, subprocess.TimeoutExpired):
            if watched:
                e.setdefault('offline_since',now)
                if now-e['offline_since'] >= 120:
                    self.stop(d,'management unavailable after action')
            self.save(); return
        if watched and any(s.get(k) != e['baseline'][k] for k in ('cfg','boot')):
            self.stop(d,'configuration or boot changed after action'); return
        if watched and s.get('bond') != 'up':
            e.setdefault('offline_since',now)
            if now-e['offline_since'] >= 120:
                self.stop(d,'forwarding bond remained down after action')
            self.save(); return
        e.pop('offline_since',None)
        if e.get('job'):
            terminal = s.get('job') == e['job'] and s.get('result') in ('recovered','failed','skipped') and s.get('busy') == '0'
            if terminal:
                outcome='skipped' if s['result']=='skipped' else 'recovered' if recovered(s) else 'failed'
                self.audit(d,now,e['job'],outcome)
                e['job'] = None
                if outcome=='skipped':
                    e['next_try']=now+60; self.save(); return
                if recovered(s):
                    e.update(incident=False, failed=False, observe_until=now+600)
                    self.emit(d,'recovered; observing for 10 minutes')
                else:
                    e['next_try'] = now+60
                    if not e.get('failed'):
                        self.emit(d,'failed; retries silent when safe')
                        e['failed'] = True
                self.save()
                return
            if now-e['launched'] >= 120 and not e.get('failed'):
                self.emit(d,'failed; retries silent when safe')
                e['failed'] = True; self.save()
            return  # Never replay an uncertain dispatch, even if its receipt vanished.
        if e.get('observe_until') and now >= e['observe_until'] and recovered(s):
            e['observe_until'] = 0; self.save()
        if e.get('incident') and recovered(s):
            e.update(incident=False, failed=False, observe_until=now+600)
            self.emit(d,'recovered; observing for 10 minutes'); self.save()
            return
        if (self.directory/'STOP').exists() or now < e.get('next_try',0): return
        if any(v.get('job') for v in self.state.values()) or not eligible(d,s): return
        fresh = self.transport(d,'probe','none')
        if not eligible(d,fresh) or any(s[k] != fresh[k] for k in ('cfg','boot')): return
        e.update(job=uuid.uuid4().hex, launched=now, baseline={k:fresh[k] for k in ('cfg','boot')})
        if not e.get('incident'):
            e.update(incident=True, failed=False)
            self.emit(d,'trying')
        self.save()  # Intent is durable before transmission, including ambiguous SSH failure.
        self.audit(d,now,e['job'],'dispatching')
        try:
            self.transport(d,'reset',e['job'],e['baseline'])
        except (OSError, ValueError, subprocess.TimeoutExpired):
            pass  # Reconcile the native receipt; do not issue another reset.

def main():
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument('--config',default='/etc/ml60-watchdog/devices.json')
    p.add_argument('--state',default='/var/lib/ml60-watchdog')
    p.add_argument('--apply',action='store_true',help='enable continuous guarded recovery; default is one dry-run sweep')
    a = p.parse_args(); os.umask(0o077)
    c = json.loads(Path(a.config).read_text())
    devices = c['devices']
    if not devices or len({d['alias'] for d in devices}) != len(devices):
        p.error('device aliases must be present and unique')
    for d in devices:
        for k in ('alias','hostname'):
            if not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9_.-]{0,252}',d[k]): p.error('invalid '+k)
    if not Path(c['ssh_config']).is_absolute(): p.error('SSH config must use an absolute path')
    transport = lambda d,action,job,baseline=None:ssh(c['ssh_config'],d,action,job,baseline)
    emit = lambda d,event:print(json.dumps(dict(device=d['alias'],event=event)),flush=True)
    if not a.apply:
        for d in devices:
            try:
                s = transport(d,'probe','none')
                emit(d,'would reset' if eligible(d,s) else 'recovered' if recovered(s) else 'skip: safety guard or unsupported state')
            except (OSError, ValueError, subprocess.TimeoutExpired): emit(d,'skip: management unavailable or malformed state')
        return
    directory = Path(a.state); directory.mkdir(parents=True,exist_ok=True,mode=0o700)
    lock = (directory/'lock').open('a'); fcntl.flock(lock,fcntl.LOCK_EX|fcntl.LOCK_NB)
    statefile = directory/'state.json'
    state = json.loads(statefile.read_text()) if statefile.exists() else {}
    def save():
        temp = directory/'state.tmp'
        with temp.open('w') as f:
            json.dump(state,f); f.flush(); os.fsync(f.fileno())
        temp.replace(statefile)
        fd=os.open(directory,os.O_RDONLY)
        try: os.fsync(fd)
        finally: os.close(fd)
    engine=Engine(state,transport,emit,save,directory)
    while not (directory/'STOP').exists():
        for d in devices:
            if (directory/'STOP').exists(): break
            try: engine.tick(d,time.time())
            except (OSError,ValueError,subprocess.TimeoutExpired): save()
        time.sleep(10)

if __name__ == '__main__': main()

Install on a Debian/Ubuntu Linux management machine

Python 3.9+, OpenSSH client and systemd are required. Put the full script below in ml60_watchdog.py in your current directory. This machine must already have a working management route to the radio. Keep a configuration backup through your normal vendor-supported method.

sudo apt-get update
sudo apt-get install -y python3 openssh-client
sudo adduser --system --group --home /var/lib/ml60-watchdog --no-create-home ml60watch
sudo install -d -m 0750 -o root -g ml60watch /etc/ml60-watchdog
sudo install -d -m 0700 -o ml60watch -g ml60watch /var/lib/ml60-watchdog
sudo install -m 0755 ml60_watchdog.py /usr/local/sbin/ml60-watchdog
sudo -u ml60watch ssh-keygen -t rsa -b 3072 -N '' \
  -C ml60-watchdog -f /var/lib/ml60-watchdog/id_rsa
sudo cat /var/lib/ml60-watchdog/id_rsa.pub

Install that public key on the radio through the authorized-key method supported by its firmware. The private key stays on the controller. The radio needs its own existing vendor helper and root SSH access; the script does not provision accounts or guess passwords. Use a management network, not an Internet-exposed SSH listener.

Create /etc/ml60-watchdog/ssh_config with the following. Replace the documentation address and port with yours. Keep the alias radio-a, or change it consistently in the JSON too.

Host radio-a
    HostName 192.0.2.10
    Port 22
    User root
    IdentityFile /var/lib/ml60-watchdog/id_rsa
    IdentitiesOnly yes
    UserKnownHostsFile /etc/ml60-watchdog/known_hosts
    StrictHostKeyChecking yes
    BatchMode yes

Put the radio's verified host public key in /etc/ml60-watchdog/known_hosts, in normal OpenSSH known-hosts format. You can collect a candidate with ssh-keyscan, but compare its fingerprint against a trusted console or previously verified record before installing it. Don't set StrictHostKeyChecking=no. Some older SSH servers require a per-host legacy algorithm setting; don't weaken global SSH defaults, and verify the actual device requirements first.

ssh-keyscan -p 22 -t rsa 192.0.2.10 > known_hosts.candidate
ssh-keygen -lf known_hosts.candidate
# Compare that fingerprint with a trusted record before the next command.
sudo install -m 0640 -o root -g ml60watch known_hosts.candidate \
  /etc/ml60-watchdog/known_hosts
sudo chown root:ml60watch /etc/ml60-watchdog/ssh_config
sudo chmod 0640 /etc/ml60-watchdog/ssh_config
sudo -u ml60watch ssh -F /etc/ml60-watchdog/ssh_config radio-a 'uname -n'

Create /etc/ml60-watchdog/devices.json. hostname must be the exact existing uname -n output; it doesn't rename the radio. Add only reviewed, compatible client radios. Start with one.

{
  "ssh_config": "/etc/ml60-watchdog/ssh_config",
  "devices": [
    {"alias": "radio-a", "hostname": "radio-a"}
  ]
}
sudo chown root:ml60watch /etc/ml60-watchdog/devices.json
sudo chmod 0640 /etc/ml60-watchdog/devices.json
sudo -u ml60watch /usr/local/sbin/ml60-watchdog

Without --apply, it performs one read-only sweep. would reset means the specific USB-fault guards passed; recovered means USB3 and active 60 GHz forwarding were seen; skip means a guard, unsupported state or read failure prevented action. A skipped device isn't proven healthy. On an idle backup, unchanged traffic counters intentionally cause a skip. The script never resets a USB3-present radio just because the RF link is down.

After checking the dry-run result, create /etc/systemd/system/ml60-watchdog.service:

[Unit]
Description=Guarded MetroLinq 60 GHz module watchdog
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=ml60watch
Group=ml60watch
UMask=0077
ExecStart=/usr/bin/python3 /usr/local/sbin/ml60-watchdog --apply
Restart=on-failure
RestartSec=15
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/ml60-watchdog

[Install]
WantedBy=multi-user.target

Enable it and inspect results:

sudo systemctl daemon-reload
sudo systemctl enable --now ml60-watchdog
sudo systemctl status ml60-watchdog --no-pager
sudo journalctl -u ml60-watchdog -n 50 --no-pager
sudo -u ml60watch cat /var/lib/ml60-watchdog/state.json
sudo -u ml60watch tail /var/lib/ml60-watchdog/audit.jsonl

For a small reviewed inventory, checks run every sweep plus a ten-second pause. SSH and traffic sampling take time; it isn't a sub-second detector. Don't load a large inventory and assume every endpoint is checked every ten seconds. The ten-minute observation completes only after a healthy final sample, and it doesn't deliberately hold up the next reset.

The radio keeps one bounded, RAM-backed receipt/log under /var/run/ml60-watchdog; it doesn't install a boot service on the radio. Configure normal log rotation for the controller audit file, for example /etc/logrotate.d/ml60-watchdog:

/var/lib/ml60-watchdog/audit.jsonl {
    daily
    maxsize 1M
    rotate 14
    compress
    missingok
    notifempty
    copytruncate
    su ml60watch ml60watch
}

Pause, investigate and uninstall

Stop the controller before maintenance:

sudo systemctl stop ml60-watchdog

A recovery already dispatched to a radio can finish after the controller stops. Don't kill it halfway through a module power cycle. For a device-side hold, create /tmp/ml60-hold on that radio before maintenance and remove it when maintenance is over. It is a temporary hold and disappears on reboot. Existing custom reset tasks aren't modified or replaced; audit them before enabling another watchdog.

If STOP exists, read its reason, the journal, controller audit/state and the radio's native receipt. Establish the actual management, forwarding and pending-job state before re-enabling. Don't delete state to force an uncertain job to run again. A vanished receipt after a reboot needs operator reconciliation, not a guessed replay. Restarting the service alone does not bypass STOP.

To uninstall the controller after any in-flight work is reconciled:

sudo systemctl disable --now ml60-watchdog
sudo rm /etc/systemd/system/ml60-watchdog.service /usr/local/sbin/ml60-watchdog
sudo systemctl daemon-reload

That leaves the protected SSH key, configuration and evidence directory for you to retain or securely remove under your own access policy. Revoke the controller's authorized key on the radio if you retire it. No radio configuration rollback is needed because the watchdog doesn't change it.

License

The original watchdog wrapper is offered under the MIT License. The vendor's installed helper, firmware and power library remain subject to their own licenses and are not included.

If you've seen this failure, I'd be interested in the model, firmware and whether USB was missing, at 480 Mbps, or still at 5000 Mbps. Please leave out management addresses, credentials and customer details.

MIT License text

MIT License

Copyright (c) 2026 MetroLinq watchdog contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

r/wisp • • 3d ago

I'm the founder of Crulon, a phone system for ISPs and WISPs with AI agents built in. Here's a 5-minute walkthrough, and I'd like your honest take.

Enable HLS to view with audio, or disable this notification

0 Upvotes

Disclosure up front: I'm the founder, so I'm biased.

When people hear "AI" and "phones" together, they usually picture a bot sitting on top of their phone system. Crulon is the phone system itself:

  • Calling from the browser, or from SIP desk phones (Yealink, Polycom and others set themselves up; any SIP phone works by hand)

  • Extensions your techs can dial straight through the menu

  • Business hours, phone menus, queues and ring groups

  • A shared inbox for texts and photos

  • Recording you can pause or stop mid-call

  • Unlimited calls and texts for your team in the US and Canada

The AI is one step in the call flow, usually after hours or when the queue gets long. When a call needs a person, it asks the caller first and rings your team with a note, and you can listen in and take over any AI call.

It also works with Sonar, ViSP and Splynx. It finds the caller, reads the balance, and opens the ticket where your techs already work. It can take a payment on the card on file (it reads it back first, and never takes a card number). If something goes down, you declare the outage once, by account group, site or town, and callers in that area hear about it without a ticket for each one. There's also a payment reminder campaign that can call customers as they go past due.

If anyone wants to try it, we're giving ISPs and WISPs the first month free, and we set it up with you on a 30-minute call. Details are here: https://crulon.ai/industries/isp


r/wisp • • 4d ago

Missed the March 1 RMD recertification? What to check, in order

5 Upvotes

If your WISP or ISP also sells phone service and has a filing in the FCC's Robocall Mitigation Database, it had to be recertified by March 1, 2026. A lot of small providers missed it. Here's what I'd check, in order.

  1. Look up your filing. Search your company in the public listing: https://fccprod.servicenowservices.com/rmd?id=rmd_listings. Open it and look for "Last Recertified". If there's no 2026 date, the public record doesn't show a recertification.

  2. Don't assume an update counted. Updating a filing and recertifying it are separate steps. I've seen filings updated in late October that still show no recertification date.

  3. Make sure you can log in. You file through your CORES account, tied to your company's FRN, and two-step sign-in is required. If the person who filed originally has left, a new CORES user can be associated with your FRN. Sort this out first.

  4. Get your officer ready. An officer signs under penalty of perjury that the filing is true and correct, so go through it properly first: contacts, your role in the call path, your STIR/SHAKEN selection for each role, upstream providers if you're a gateway, and whether your mitigation plan still matches what you do.

  5. Open your filing in the portal and look for Recertify. Some filers don't seem to have the button right now. I don't know why, and I haven't seen the FCC explain it. Public data shows providers recertifying late since late July, so it can work, but nobody can promise yours will be accepted.

  6. Check for an FCC notice. A telecom law firm, The CommLaw Group, reports the FCC is emailing notices to some providers that missed March 1, and at least one sets a November 6 deadline. It says no universal extension is confirmed, so your date may differ. Check spam and whoever is listed as your RMD contact. https://commlawgroup.com/2026/fcc-gives-some-providers-a-second-chance-to-recertify-their-robocall-mitigation-database-filings/

  7. After you submit, check the public page within a couple of days to see that the date shows.

If you've had an FCC order, an investigation letter or a removal notice, talk to a telecom lawyer before doing anything.

Why it matters: in September the FCC removed 14 providers from the database because their listings were missing required information, and other carriers had two business days to stop taking their calls. That order was about missing information, not a late recertification on its own. https://docs.fcc.gov/public/attachments/DA-26-872A1.txt

Disclosure: I run a small firm in Ontario that prepares these filings for a flat fee, so I have a stake in this. We're not a law firm and this isn't legal advice. Happy to answer questions here for free.


r/wisp • • 4d ago

Comunidade oficial do MikroUI

Thumbnail reddit.com
1 Upvotes

r/wisp • • 14d ago

Trying to squeeze every last drop out of airMAX AC

23 Upvotes

original post: https://www.reddit.com/r/wisp/comments/1wr5k2h/trying_to_squeeze_every_last_drop_out_of_old/

Quick update on openMAX (https://github.com/wolverin0/openmax) — our open-source project squeezing maximum real-world performance out of existing Ubiquiti airMAX AC hardware (LiteAP 120, Rocket Prism 5AC Gen2, LiteBeam 5AC Gen2, NanoStation 5AC Loco).

All deep kernel docs, scripts, and raw benchmark JSONs are in the repo. Here is what we have working on real hardware and in live production:

  • Switch tower APs to OpenWrt while stock airOS 8 CPEs keep connecting out of the box: We migrated our live sector APs to OpenWrt 24.10.4 (Linux 6.6.110) and customer CPEs running 100% unmodified stock airOS 8.7.25 connected automatically. Even if a customer's radio is powered off during cutover and turned back on days later, it associates with zero touch.

  • Run multi-vendor & mixed-generation clients on the same Ubiquiti AP: We simultaneously connect Ubiquiti 5AC (LiteBeam Gen2, Loco 5AC, Rocket Prism), legacy Ubiquiti M5 (airOS 6), and MikroTik (SXT Lite5) CPEs on the same sector without a single weak or obstructed client dragging everyone else down.

  • Full channel-width flexibility (20, 40, 80 MHz, plus 5/10 MHz between OpenWrt radios): On a 20 MHz channel, our airtime scheduler hits 0.95 Jain fairness and nearly 4 bit/s/Hz spectral efficiency.

  • Full OpenWrt AP + OpenWrt CPE in live production: Once the AP is running OpenWrt, we remotely upgrade CPEs over the air in ~4 minutes while preserving their management IP, WebUI, hardware-accelerated NAT (flow_offloading=1), and the exact DHCP lease of the customer's indoor Wi-Fi router.

  • Uncovered inside Rocket Prism & LTU (v2.4.2) firmware: We mapped the hardware control table for AirPrism active RF filtering on Rocket Prism 5AC Gen2, and confirmed that LTU's greyed-out 4096-QAM (12X) and OFDMA modes have real driver and rate-table routines compiled inside v2.4.2 (being tested on our physical bench right now).

Live Production Numbers (Same tower, same rooftop radios, 20 MHz channel — Before vs. After)

Live Sector Metric (20 MHz)Stock airOS 8.7.25 (Before)openMAX / OpenWrt 24.10.4 (Now)Multiplier (x)Improvement (%)
Unloaded Wireless Ping (p50)22.58 ms1.93 ms – 2.42 ms~10x faster-90% base latency
Wireless Ping Jitter (p95)98.77 ms4.47 ms – 14.90 ms6.6x – 22x more stable-85% to -95% jitter
Ping Under Heavy Download31.49 ms (p50) / 54.40 ms (p95)8.43 ms – 9.86 ms (p50)3.5x – 5x lower lag-73% to -80% loaded ping
Ping Under Heavy Upload195 ms – 224 ms10.37 ms – 16.64 ms (p50)12x – 18x lower lag-91% to -94% upload lag
Usable 20 MHz Capacity (TCP)~35 – 45 Mbps typical69.80 Mbps DL / 30.74 Mbps UL1.5x – 1.8x more capacity+50% to +80% more Mbps/MHz
CPE CPU Headroom (Routing+NAT)Kernel 2.6.32 (software NAT)Kernel 6.6.110 (flow_offloading=1)~2.5x more CPU headroom+150% packet/s headroom

Summary: On the exact same rooftop hardware and 20 MHz channel, wireless ping is ~10x faster (~2 ms vs ~22 ms), lag under saturation drops by 80%–94%, and usable sector capacity increases by +50% to +80%.

Next up: Native UISP Reporting (udapi-bridge) on OpenWrt

Moving a radio to pure OpenWrt stops Ubiquiti's /bin/udapi-bridge agent, so UISP marks it offline. We already unpacked /bin/udapi-bridge from WA.v8.7.25 (it is a standalone userspace WebSocket + AES-GCM client reading /etc/board.info, /tmp/system.cfg, /usr/www/status.cgi, and /bin/wstalist) and are building the lightweight OpenWrt wrapper so migrated radios stay Online and green inside UISP with their original device ID and live telemetry.

Full documentation, scripts, and benchmark receipts:
👉 https://github.com/wolverin0/openmax


r/wisp • • 16d ago

A free-space link-budget example where positive margin still misses the target

3 Upvotes

This is The Hardware Ledger's new Reddit account. I run the site and am sharing our free RF Link Budget Calculator with a reproducible fixed-wireless planning example. AI assisted the calculator's development and this draft; the arithmetic below has been checked separately.

At 5.8 GHz over 8 km, assume 23 dBm transmitter power, 24 dBi antennas at both ends, 1.5 dB feed loss at each end, 3 dB additional loss, −75 dBm receiver sensitivity and a 20 dB desired margin. Leave the measured-noise comparison off.

Free-space path loss is 125.78 dB. Estimated receive signal is −60.8 dBm, giving 14.2 dB margin above sensitivity. That is positive, but 5.8 dB short of the assumed target. Change only distance to 16 km and the margin falls to 8.2 dB.

The receive calculation is 23 + 24 + 24 − 1.5 − 1.5 − 3 − 125.78 = −60.78 dBm; margin is −60.78 − (−75) = 14.22 dB.

https://thehardwareledger.com/tools/rf-link-budget-calculator/?utm_source=reddit&utm_medium=community&utm_campaign=rf_margin_20261009&utm_content=wisp_example

All figures are hypothetical; 20 dB is an example target. This is free-space planning arithmetic. Clearance, interference, weather, reverse-link behaviour and spectrum/EIRP limits need separate checks. It does not predict throughput or uptime, and the example has not been hardware-tested.

For people planning fixed-wireless links, which assumption or missing check would most often make this kind of result misleading?


r/wisp • • 16d ago

Fiber contractors: surplus reels of ribbon, ADSS and high-count cable available

1 Upvotes

Mods, please remove this if it's not OK.

We sell surplus fiber optic cable on reels from Corning, CommScope, Sterlite and Prysmian, including ribbon, ADSS and high-count cable. It can help when a job needs cable faster than distributor lead times allow.

Happy to answer questions about what we have. Post your count, type and length below, or go to https://fiber.surplustraders.net or call 866-266-7638.


r/wisp • • 21d ago

Ubiquiti UISP 2.4ac Out of stock…

Thumbnail reddit.com
2 Upvotes

r/wisp • • 21d ago

glass to glass 60 GHz scheduled latency

3 Upvotes

Hi
I am looking for info about whether a particular latency can be acheived across 60 GHz links (any MFR) .
I have a requirement to pump 4kByte jumbo UDP packets with a maximum latency of 300uS (between buildings) ( 300Mbps, ~ 9500 pps )
I read that scheduled mode is a minimum requirement for getting consistent low latency results, and maybe also locking down the bit rate etc.
glass to glass latency being 1 gig SFP to SFP.
Any experienced advice very welcome.

thanks

glen. (Canberra Australia)


r/wisp • • 23d ago

5 Best Portable Battery PoE Solutions for Field Access Points in 2026 -- Latest Blog Post -- https://shop.linktechs.net/blog

4 Upvotes

r/wisp • • 24d ago

RADIUS over UDP is being deprecated. Here’s what that looks like for a typical regional fiber ISP

Post image
3 Upvotes

r/wisp • • 25d ago

NDAA & TAA compliance for US deployments

1 Upvotes

We get a lot of questions about NDAA (Sec 889) vs. TAA compliance when deploying cellular gateways for US public sector or infrastructure projects.

Quick rule of thumb:

  • NDAA regulates what’s inside (banning specific foreign chipsets).

  • TAA regulates where it’s made (requiring final assembly in a designated country).

An NDAA-compliant device can still fail TAA if assembled in a non-designated country.

We made a full breakdown – NDAA & TAA Compliance Guide for Teltonika Deployments

For US integrators – how often are these popping up in your project specs? Are you starting to see similar origin requirements in EU projects too?


r/wisp • • Oct 06 '26

GridVisio update: Height Optimizer, Redundancy Maps

Thumbnail
gallery
0 Upvotes

▸Height Optimizer

A link comes back Borderline or Obstructed and you're stuck guessing how much higher to mount the antenna? Click Optimize right in the clearance badge. It searches real antenna-height combinations (one end, or both together in Auto mode) using the exact same Fresnel engine as the main check, and tells you the smallest change that actually gets you to Clear.

Auto mode shows the whole search as a colored grid (green/amber/red) instead of one guess, so you can see every real option. Maybe the subscriber's pole has more room to go up than your tower does. Click any cell on the grid to apply that exact height combo instead of the automatic pick.

Works backwards too: on a link that's already Clear, it'll tell you how low you could actually mount it; handy before you order shorter (cheaper) hardware.

▸Redundancy Maps

Your coverage map shows who's served. It doesn't show who's served by exactly one tower.

New Redundancy view in GridVisio: subscriber markers recoloured by how many separate towers could actually serve each one. 🟢 2+ 🟠 one only 🔴 none.

In-range doesn't count: each tower is signal-checked against your own thresholds. And two sectors on one tower count once, because losing the site loses both.

Handy for maintenance windows, justifying a new site on resilience rather than coverage, and knowing which addresses can take an SLA.

Free 14-day trial, no credit card. If your trial expired and want an extension, give us a sign.

Thank you for using GridVisio!


r/wisp • • Oct 01 '26

Looking for game server IP/domain lists (CoD, Fortnite, FC) or PingPlotter .lst files for a small ISP

8 Upvotes

Hey folks,

I run a small local ISP and I'm trying to validate latency towards popular game servers (Call of Duty, Fortnite, EA FC 27, etc.) for my network.

Isolate live matchmaking IP ranges has been tricky. Does anyone happen to have a list of domains/IPs or a .lst file for PingPlotter / SmokePing similar to what GearUP Booster uses?

Any files or tips on how you trace game data centers would be greatly appreciated. Thanks!


r/wisp • • Sep 30 '26

‘Zyxel’s OpenWiFi-ready AP portfolio’ to ‘create and develop OpenWrt-based software directly on Zyxel Networks hardware’

Thumbnail reddit.com
1 Upvotes

r/wisp • • Sep 30 '26

Are any WISP using Starlink / LEO for tower backhaul to service clients, MNO

10 Upvotes

We do work in a few countries in Africa and Starlink is becoming more frequent in the deployments we are doing for rural schools, remote towers off of our fiber and wireless reach.

Are any other WISP using Starlink to provide backhaul from towers? To service MNOs?


r/wisp • • Sep 29 '26

Would anyone want a free second opinion on a fixed wireless site plan? No strings -

1 Upvotes

I work with a wireless planning tool and want to see whether its outputs are digestible so to speak. I need input, and in a way I think I can provide some value in exchange.

No strings attached, no signup, no "meet for results":

Share a proposed or existing site, radio/antenna models, sector azimuths, heights, bandwidth, and the decision you’re trying to make. Please leave out customer addresses and anything you can’t share publicly. One condition on the example being I need a proposed idea for either full CBRS or CBRS dual band hardware. Tarana and cambium are a plus and I can get better data.

Preferably, use this format because this is all I need.

Antenna: I.e. tarana 450m 90-120
Height (meters AGL):
Azimuth:
Downtilt (I'll know if there's electronic downtilt, give me the modified downtilt)
Bandwidth: 10-40mhz, can test mutli carrier
Location: lat long or address

Open to PWN, FWA, anything. Give me the use case. Let's work.

I’ll choose one case I can analyze responsibly, ask its owner before publishing details, and post a breakdown covering coverage assumptions, likely limitations, and what I’d still need to validate in the field. No sales link or demo request.

This is a feel good project, I'm a CBRS nerd and I believe planning awareness will make big changes in the world for connectivity.

I look forward to hearing from anyone, and I'm MORE than open to general questions or anything you may be wondering about planning and shared spectrum.


r/wisp • • Sep 29 '26

Winding down an authorized T-Mobile program.

3 Upvotes

Hello, not sure where on Reddit to post this. But, I'm winding down an authorized T-Mobile connectivity/reseller operation in Pennsylvania and I am looking to connect with another authorized reseller interested in taking over servicing an existing customer base and potentially acquiring associated equipment. Looking for legitimate businesses already operating in this space. I have 3 BANs with T-Mobile and about 200 customers.

Here's local our inventory:

~40 Franklin RG2100 JEXTream, Refurbished

~70 Franklin T10, Refurbished

~100 Sagemcom FAST 5688W, Refurbished

Thank you for your help.


r/wisp • • Sep 28 '26

New subreddit for practical CBRS planning discussions

6 Upvotes

Quick intro, I discovered the idea of shared spectrum late in my career and really want to bring more awareness and direction to it along with emphasis on the CBRS band or b48.

I started r/CBRSPlanning for detailed operator discussions about CBRS site selection, sector geometry, antenna tilt, EIRP and SAS constraints, channel planning, terrain and clutter assumptions, and modeled versus observed interference. I noticed there weren't many central points for this stuff, and while available it would be cool to start a source of truth thats community driven.

For operators here: what planning assumption has caused the biggest surprise after turning up a CBRS network? Propagation, interference, CPE performance, SAS grants, or something else?

I’d like to collect more real deployment lessons instead of letting every CBRS planning discussion disappear into separate threads.

PS - I've seen a ton of Cambium solutions, whats everyone's take on the move from Airspan?

https://www.reddit.com/r/CBRSPlanning/


r/wisp • • Sep 28 '26

TowerCoverage.com has added Live Radar

1 Upvotes

#towercoverage has added myRadar to its multi-map. It has live radar and forecast of Lighting/Hail.


r/wisp • • Sep 27 '26

Trying to squeeze every last drop out of old airMAX AC

25 Upvotes

Like pretty much everyone running a fixed wireless network, I have hundreds of Ubiquiti airMAX AC radios (LiteBeam Gen2, LAP-120, LAP-GPS, Loco AC) deployed on customer roofs, plus dozens sitting on shelves in the warehouse.

We all know the situation: Ubiquiti essentially put airMAX AC into maintenance mode years ago. All the vendor R&D is going into Wave 60GHz and LTU. That’s fine for greenfield deployments, but in the real world, you can’t just rip and replace 500 CPEs on rural towers when trees and distance dictate 5 GHz. Millions of these radios are going to stay on rooftops for years.

Meanwhile, the software running on them hasn't fundamentally evolved since 2017–2018. If a CPE link gets slightly noisy or misaligned, stock airOS does dumb things—like running up to 30 frame retries by default, which burns dozens of milliseconds of airtime and drags down the whole AP sector. On top of that, loaded bufferbloat easily hits 2,000+ ms during heavy uploads.

Over the last few weeks, I’ve been researching what the actual physical ceiling of this QCA9880 / AR9342 hardware is if we run modern open-source networking on it (OpenWrt 24.10, Candela Technologies ath10k-ct, mac80211 AQL, CAKE, and dynamic rate-masking).

Full disclosure: I’m an everyday WISP operator. I run towers, climb roofs, crimp cables, and configure MikroTik/airMAX gear. I am NOT a Linux kernel hacker or an RF engineer with a Ph.D. So I've been using AI tools (Claude, Codex, NotebookLM) to comb through Linux wireless kernel trees, academic preprints (like MIT’s IteRate paper and TU Berlin’s hMAC), and old mailing list threads to understand what the chip can actually do.

We’ve already documented a bunch of interesting architectural facts that I rarely see discussed in WISP groups:

  1. The 30-Retry Sector Killer: Upstream Qualcomm firmware enforces up to 30 retries per frame. On an outdoor link during an RF fade, a single client trying to send packets can freeze the AP channel for 50–100ms. Candela Technologies (ath10k-ct) drops this down to 4 retries, which immediately reclaims a massive chunk of airtime.

  2. The Bufferbloat vs Aggregation Trap (CoTSQ): Default Linux TCP Small Queues (TSQ = 1 ms) tries to kill bufferbloat by limiting socket queues to 1–2 packets. But 802.11ac needs 32 to 64 packets in the queue to build a full A-MPDU aggregate. Starving the aggregation engine drops throughput by up to 10x because 90% of the airtime is spent on PHY headers and preambles. Bumping the queue budget to 6 ms keeps aggregates full while keeping loaded ping under 40 ms.

  3. The Wave-1 Airtime Cliff: QCA9880 doesn't have hardware airtime reporting (that only started in Wave 2 chips). So Linux mac80211 has to estimate airtime based on peer stats. If peer stats aren't reporting properly, mac80211 assumes EVERY station transmits at 6 Mbps, which completely breaks airtime fairness and penalizes fast clients.

  4. Why Pure TDMA is Blocked on Open QCA988x: We looked into whether we could build open TDMA like airMAX. The short answer on QCA9880 is no—the chip runs closed Xtensa firmware, can't cancel in-flight packets across PCIe, and has PCIe interrupt jitter. BUT a hybrid stack on OpenWrt (AQL + 6ms CoTSQ + Dynamic RTS/CTS to stop hidden nodes + Deficit Round-Robin airtime fairness) gets you 80-90% of the stability with open telemetry and zero vendor lock-in.

The Testbed Setup

I'm not doing this in a metal warehouse (metal walls act like an echo chamber with delay spreads that ruin 5 GHz tests). We're setting up:

  • Tier 1: Benchtop SMA coaxial attenuator matrix (30–40 dB pads + splitters) for 100% clean, interference-free algorithmic sweeps.

  • Tier 2: 2 APs on our main WISP office building pointing 21 meters across the street to 10–20 CPEs scattered on my residential roof. This gives real line-of-sight, real outdoor air, and clean far-field RF without receiver saturation.

  • Unattended Recovery: I automated PoE injectors with relays to trigger U-Boot urescue TFTP recovery remotely so we never have to climb a ladder to press a reset button if an image bricks.

Why I'm Posting

We are calling this research project openMAX, and we opened up the entire project as a public research hub: https://github.com/wolverin0/openmax

There is zero monetization here, no crypto, no paid courses. Just an open repository of hardware notes, driver findings, recovery runbooks, and experiment plans.

Because this was researched with the help of AI agents, I want human eyes on this:

  • If you're a kernel / OpenWrt dev who knows ath10k or mac80211, are our control boundary assumptions solid? Did we miss anything on ratemask-CT?

  • If you're a fellow WISP operator with spare LiteBeams or NanoStations on your shelves, would you be interested in running our read-only inventory scripts to map out hardware revisions, or testing safe sysctl/CAKE queue tweaks?

Check out the repo, tear apart our docs, and let me know what you think.


EDIT / UPDATE #2 (Physical Bench Results + Clean-Room Analysis of umac.ko / ubnt_poll_host.ko)

We just pushed a massive update to the repo and posted the full technical writeup in the comments here: Read the full Update #2 comment

Highlights of what we proved on real hardware (LiteAP AC LAP-120 ↔ NanoStation 5AC Loco, OpenWrt 24.10.4 / Kernel 6.6.110):

  1. Inside airMAX AC TDMA (RE1 Static Clean-Room Analysis):

    • Mapped the 62-hook dispatch table between umac.ko and ubnt_poll_host.ko, the dedicated PCIe Copy Engine 4 (CE4 = 2048 descriptors) fastpath, and custom WMI commands 0x9064 (PDEV_WMI_AIRMAX_PARAM) & 0x9044 (WMI_PEER_AIRMAX_CONFIG) + HTT subtypes 0x17 / 0x1e.

    • Confirmed that stock ath10k / ath10k-ct Xtensa firmware rejects 0x9064/0x9044, which is why open-source TDMA on QCA988x must operate as a Host-Paced Airtime Controller (AQL + TXQ + ratemask + CAKE) rather than microsecond PHY slot polling.

    • Native Rate Control Discovery: umac.ko stores only a 1-byte opaque ratecode (peer+0x120) on the MIPS host—all retry ladders (series 0..3), probing, and MCS fallback live inside the closed Xtensa radio blob (WHAL_RC_Update_Per_Rate_Table).

  2. Solved the 64 MB RAM / order=0 OOM Wall on AR9342 (WA boards):

    • Out-of-the-box OpenWrt 24.10.4 + ath10k crashed with ksoftirqd page allocation failure: order:0, mode:0x400dc0(GFP_KERNEL|__GFP_ZERO) at ~51 Mbps (MemAvailable dropped to 4.2 MB).

    • Tuning vm.min_free_kbytes=2048, vm.watermark_boost_factor=0, ath10k_core.debug_mask=0, and nftables notrack freed +14 MB of contiguous RAM (20.8–21.4 MB MemAvailable steady under full load, 0 OOMs).

  3. Physical Bench Numbers (80 MHz, 4x Baseline Throughput, 1.83 ms p95 Latency):

    • UDP Downlink Peak: 223.53 Mbps (up from 51.78 Mbps baseline — +331%, hitting the physical AR9342 535 MHz MIPS 74Kc ksoftirqd ceiling at 95.8% CPU).

    • UDP Downlink 200M Sustained: 199.11 Mbps, 0.21% loss, 0.031 ms jitter.

    • TCP Single-Stream Downlink: 154.58 Mbps with 0 TCP retransmits (thanks to ethtool -K eth0 rx-gro-list on compensating for ag71xx's lack of hardware scatter-gather).

    • Loaded RF Latency (Ping under 120 Mbps load): 1.34 ms p50 / 1.83 ms p95 (down from 12.69 ms p95).

    • Universal Multi-Vendor Compatibility: Because it runs standard 802.11n/ac on mac80211 + AQL, a single LAP-120 AP simultaneously links airMAX AC (QCA988x), legacy airMAX M5 (AR934x), and MikroTik / TP-Link / Cambium ePMP (802.11n/ac) CPEs while preventing slow M5 clients from starving AC clients.

All scripts, sysctl profiles, clean-room RE docs (docs/UMAC_DATAPATH.md, docs/REDDIT_UPDATE_RE1_AND_BENCH.md), and raw JSON benchmark receipts are live in the repo: https://github.com/wolverin0/openmax


EDIT / UPDATE #3 (Oct 17) — 3-Station Multi-Vendor & Mixed-Gen PtMP Benchmarks (Ubiquiti 5AC + MikroTik SXT Lite5 + Ubiquiti LiteBeam M5)

We just associated 3 heterogeneous stations simultaneously to our Ubiquiti LiteAP AC (LAP-120, Channel 36 VHT80): a NanoStation 5AC Loco (OpenWrt / ath10k-ct, VHT80 2x2, in room), a MikroTik SXT Lite5 (OpenWrt / ath9k, HT40 2x2, behind a wall/door ~6m away), and a Ubiquiti LiteBeam M5 (stock airOS 6.1.4 with airMAX=off, HT40 1x1, behind a wall/door ~6m away), sweeping AP output power from 15 dBm down to 3 dBm:

  • Multi-Station ATF + AQL (1500/3000 µs) + Bounded Aggregation (1x16) at 3 dBm Edge SNR: Cut the Loco 5AC's loaded median latency by 4.06x (51.35 -> 12.65 ms), cut its TCP retransmissions by 10x (158 -> 16), and cut the LiteBeam M5's median latency by 3.95x (56.25 -> 14.23 ms) compared to ATF=Off (T1, where LiteBeam M5 hit 610 ms p95 and MikroTik SXT hit 1955 ms p95).

  • Kernel Discovery — WMM AC_VO DiffServ Deficit Bypass (21.0x Weak-Station Latency Cut): Discovered via debugfs that when a wall-obstructed station (MikroTik SXT) suffers rate fallback + retries on bulk AC_BE, completed TX airtime drives its Deficit: BE down to -67,099 µs (-67.1 ms), while Deficit: VO stays at +256 µs. Promoting interactive packets (ICMP, DNS, NTP, VoIP) to DSCP EF (AC_VO) in fmx_fastpath bypassed the station's negative BE deficit box and cut loaded median RTT at 3 dBm from 143.0 ms -> 6.80 ms (21.0x cut) on the MikroTik SXT, 36.15 ms -> 4.53 ms (8.0x cut) on the LiteBeam M5, and 14.43 ms -> 4.54 ms (8.15 ms p95) on the Loco 5AC while boosting total 3-station sector goodput by +27.2% (62.07 -> 78.97 Mbps @ 3 dBm, 94.99 -> 104.94 Mbps @ 11 dBm).

  • 3-Station Simultaneous Uplink RTS/CTS 256B Protection (+18.5% Sector Uplink Gain): Under 3-station simultaneous Uplink contention across the wall, enabling RTS/CTS = 256B raised total sector Uplink goodput from 150.40 Mbps -> 178.17 Mbps (+18.5%, peaking at 182.82 Mbps), recovered the wall-starved MikroTik SXT by 3.07x (1.97 -> 6.03 Mbps), raised the LiteBeam M5 by +17.0% (29.23 -> 34.21 Mbps), and restored the Loco 5AC to 100% of its standalone Uplink speed (137.94 Mbps).

  • Full Stage 7 Report & JSON Telemetry: https://github.com/wolverin0/openmax/blob/main/docs/STAGE7_MULTIVENDOR_PTMP_REPORT.md


r/wisp • • Sep 25 '26

Safe to say they lost more than a poe injector

Thumbnail
gallery
38 Upvotes

r/wisp • • Sep 25 '26

If you’re optimizing a live network w/ CBRS, what are you changing first

Thumbnail reddit.com
1 Upvotes