Get groups of user

[Perneel]

I'm having issues in loading the sharepoint groups of a specific user. The Groups property stays empty:

public static async Task<List<string>> GetSharePointGroupsOfUser(PnPContext context, string userPrincipalName)
{
    // Retrieve the user
    var loginName = "i:0#.f|membership|" + userPrincipalName;
    var user = context.Web.EnsureUser(loginName);

    // Load the groups of the user
    await user.LoadAsync(u => u.Groups);

    // Get the group titles as list
    var groups = user.Groups.AsRequested().Select(g => g.Title).ToList();

    return groups; // => this returns an empty list, even though the user added to 3 sharepoint groups
}

As a test I tried loading all site users including "All" property & Groups. There it seems that the correct SharePoint groups are loaded, but this seems overkill for getting the groups of a single user.

// Fetch all users and include their groups
var users = await context.Web.SiteUsers.QueryProperties(u => u.All, u => u.Groups).ToListAsync();

What am I missing here?
Thanks in advance!