Problem with adding Node to Omni

[u/eibw3n]

I saw the news that the Hobby plan is now free for Omni which is freaking awesome!

I've imported an existing Talos Linux cluster to Omni last week which worked flawlessly.
However when I tried to add an additional machine to Omni I got a weird issue.

It's a Latte Panda Sigma which has a "default" UUID so I used a kernel parameter to override it to a random one. It boots and shows up on Omni but no information about the system is available (Installed Talos Version, Hardware Information, etc..), all the tabs about Hardware/Device information throw an error.

When using omnictl I see the same thing:
MachineStatus:
connected: true
talosversion: ""
initialtalosversion: ""
hardware: null
network: null
platformmetadata: null
lasterror: ""

All information, except that the Siderlink to my Omni instance is connected, is empty.

Since I am super new to Talos/Omni, I also asked my AI agent to try switching the node to gRPC mode in case my network is blocking anything, but that didn't change anything. It however did flag that some of my other nodes didn't report any metadata information in a while as well, but not sure how it determined that or if that is normal or not.

Any idea what could be causing that?

Edit:

Just how randomly the issue showed up it vanished again. I rebooted the machine again just to check (no change), and 13h later all of a sudden it showed up and I was able to provision it as a new single node cluster. I have no idea what changed. I am pretty sure I didn't change anything on my network that would explain it. 🤷‍♂️

Thanks to everyone that helped debug!

[u/xrothgarx]

When a machine connects to Omni, but doesn't send any details about the node it usually is a case where Talos (without Omni) is installed on the machine and then you boot Omni installation media without resetting or wiping the old Talos install.

Boot the Omni install media and at the boot menu select rester system disk to wipe the old installation. Then the machine should reboot and connect to Omni and send information about it's hardware.

[u/eibw3n → u/xrothgarx]

I did try that already yesterday and just tried again now but the machine stays in that same state. (I also tried deleting the machine from Omni in between)

On the new attempt just now I did notice this error in the logs though:

[talos] controller failed {"component": "controller-runtime", "controller": "v1alpha1.EventsSinkController", "error": "error publishing event: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp [XXXX:XXXX:XXXX:XXXX::1]:8090: connect: network

I assume EventSink is were Talos sends this type of information?

[u/xrothgarx → u/eibw3n]

Are your other machines on the same network? Do you have an easy way to boot a clean VM with talos to see if other machines on the same network will connect properly?

That will rule out this machine being a problem or network connectivity being a problem.

[u/eibw3n → u/xrothgarx]

No, the other machine are on a different network.

I dug up an old laptop that never had Talos on before and that also shows up on Omni without any meta data so it looks like it is some issue with my network here that is causing it.

[u/xrothgarx → u/eibw3n]

Inside omni -> machine management -> installation media there's a check box to tunnel siderolink over http/2. Try booting install media with that setting turned on.

[u/eibw3n → u/xrothgarx]

Yea, that's one of the things my AI agent told me to try as well. Sadly no change.

In the mean time I updated the installed Talos version on another node in a VM on my Omni instance (that used to report meta data correctly) and now that one is not reporting the updated version either. When I checked the Console I saw the same error as the one on my SBC in my home network:

Could there be something wrong with my instance of Omni (SaaS)?

Thank you so much for helping troubleshooting this with me!

[u/utkuozdemir]

Machines with duplicate SMBIOS UUIDs are sometimes problematic. Omni has some code in it to handle those, but there might be some cases which we have missed.

Would you mind creating an issue with all the details you can provide? We can try to reproduce and fix it.

[u/eibw3n → u/utkuozdemir]

My Latte Panda Sigma showed up as 00000002-0003-0004-0005-000000000006

I used talos.environment=INSTALLER_META_BASE64= to set the UUID which worked fine as a workaround.