High latency / ping spikes with NGO, Distributed Authority topology

[KadaXuanwu]

Description

When using Multiplayer Services v1.1.8 and Netcode for GameObjects 2.5.0 with Distributed Authority topology, clients sometimes experience very high latency (300ms+) on certain server(s), while other times the RTT is normal (~40ms).
The issue occurs both when playing with friends over the internet and when testing locally with Multiplayer Play Mode.

This currently happens on "europe-west4"

I understand that sometimes servers can be overloaded, but my main problem is that the method
QosService.Instance.GetSortedRelayQosResultsAsync()
puts europe-west4 as the best region with ~27ms latency, even though when I create the lobby:
SessionOptions options = new SessionOptions {...}.WithDistributedAuthorityNetwork("europe-west4");
I have a RTT of 300ms in most cases.

I only see that the lobby is super laggy after creating it.
I can't expect players to open and close lobbies until they found a good server. I need to know that beforehand, so that I can select the best server to connect to before creating the lobby.

Reproduce Steps

A bit hard to reproduce consistently as I don't know when the servers work correctly and when they don't. But at the time of posting, try connecting to "europe-west4" with Distributed Authority topology a few times.

Actual Outcome

The method
QosService.Instance.GetSortedRelayQosResultsAsync()
returning a sorted list of lowest latency servers that is in theory correct, but when connecting to a server, they sometimes have a very high latency (300ms+), which I don't see in advance

Expected Outcome

I expect to have a method in Distributed Authority topology that tells me in advance if a server has issues, so that I can connect to another one.

Screenshots

Screenshot of Runtime Network Stats Monitor of when the issue occured in an ongoing game.

Environment

• OS: Windows 11 Pro
• Unity Version: 6000.2.3f1 (same issue with 6000.1.15f1)
• Netcode Version: Netcode for GameObjects 2.5.0
• Netcode Topology: Distributed Authority

Additional Context

I also didn't find a way to change the server after creating the lobby. If that's possible somehow, I could use this approach instead.

[EmandM]

Hi! Thanks for reporting this issue.

Do you mind sharing the code you're using to connect to the Distributed Authority session?

[KadaXuanwu]

Sure thing, I run this code on game start-up to determine the best server:

public static async Task<string> GetBestRegion() {
    IList<IQosAnnotatedResult> relayQos = await QosService.Instance.GetSortedRelayQosResultsAsync(null);

    IQosAnnotatedResult bestResult = relayQos
        .FirstOrDefault(r => !ExcludedRegions.Contains(r.Region));

    Debug.Log(bestResult != null ? $"best region: {bestResult.Region} with average latency of {bestResult.AverageLatencyMs}ms" : "no suitable region found");
    return bestResult.Region;
}

and this code to create the lobby:

public static async Task CreateSessionAsync(string sessionName, int maxPlayers, Database.GameType gameType, string region) {
    _ctsCreateLobby?.Dispose();
    _ctsCreateLobby = new CancellationTokenSource();

    try {
        SessionOptions options = new SessionOptions {
            IsPrivate = DefaultSessionIsPrivate,
            MaxPlayers = maxPlayers,
            Name = sessionName,
            SessionProperties = new Dictionary<string, SessionProperty>
            {
                {
                    Database.PlayerDataVersion,
                    new SessionProperty(Application.version, VisibilityPropertyOptions.Public)
                },
                {
                    Database.LobbyPropertyKeyGameMode,
                    new SessionProperty(Database.GetEnumGameModeName(DefaultGameMode), VisibilityPropertyOptions.Public)
                },
                {
                    Database.LobbyPropertyKeyMap,
                    new SessionProperty(gameType == Database.GameType.Practice
                        ? Database.GetEnumMapName(DefaultPracticeMap)
                        : Database.GetEnumMapName(DefaultNonPracticeMap), VisibilityPropertyOptions.Public)
                },
                {
                    Database.LobbyPropertyKeyGameType,
                    new SessionProperty(Database.GetEnumGameTypeName(gameType), VisibilityPropertyOptions.Public)
                }
            },
        }.WithDistributedAuthorityNetwork(region);

        IHostSession hostSession = (IHostSession)await RunWithCancellation(
            ct => MultiplayerService.Instance.CreateSessionAsync(options),
            _ctsCreateLobby.Token
        );

        ISession session = hostSession;
        Database.S.joinedSession = session;

        Debug.Log($"Created Session! {session.Name} {session.MaxPlayers} {session.Id} {session.Code}");

        UpdatePlayerProperties(GetDefaultPlayerProperties());
        _lastSessionUpdateTimestamp = Time.unscaledTime;
        _lastPlayerUpdateTimestamp = Time.unscaledTime;
    }
    catch (SessionException e) {
        Debug.Log(e);
        throw;
    }
}

Players can quick join like this:

public static async Task JoinSessionQuick() {
    _ctsJoinLobby?.Dispose();
    _ctsJoinLobby = new CancellationTokenSource();

    try {
        QuickJoinOptions quickJoinOptions = new() {
            Filters = new List<FilterOption> {
            new(FilterField.AvailableSlots, "1", FilterOperation.GreaterOrEqual)
        }
        };

        SessionOptions sessionOptions = new SessionOptions { }.WithRelayNetwork();

        ISession session = await RunWithCancellation(
            ct => MultiplayerService.Instance.MatchmakeSessionAsync(quickJoinOptions, sessionOptions),
            _ctsJoinLobby.Token
        );

        Database.S.joinedSession = session;
        Debug.Log("Quick joined lobby");

        UpdatePlayerProperties(GetDefaultPlayerProperties());
        _lastPlayerUpdateTimestamp = Time.unscaledTime;
    }
    catch (SessionException e) {
        Debug.Log(e);
        throw;
    }
}

[EmandM]

The WithRelayNetwork method of joining to a distributed authority session is not supported at this time. Do these issues persist when only using WithDistributedAuthorityNetwork?

[KadaXuanwu]

Good catch, but yes, I still get high ping lobbies and high latency between clients.

I also get 300ms when checking the RTT to the server:
NetworkManager.Singleton.NetworkConfig.NetworkTransport.GetCurrentRtt(0)

We further discussed the issue here:
https://support.unity.com/hc/en-us/requests/2462210