I spinned up a new derper for my headscale network, and dropped all official ones. So there would only be two DERPs in my network, one embedded in headscale, the other is the new one.

But after I joined the server into the network, I notice that some of my nodes are always go through the new derper, and it's freakin' slow, like a 100~200 ms latency compare to the around 20 ms latency when connected directly.

I checked the derp config many times, check the ip resolution since I resolve the derper domain through Cloudflare, and even check if it's caused by the ipv6 related stuffs.

But at last, I found out the culprit: I forgot to expose the tailscale vpn port on the main node, makes the nodes has to go through the new node and have such a big detour.

After I modify the firewall rule, all things just resolves, and the latency just vanished like if it was not there.

Such silly one :(