Open Closed

Redis Backplane - AddStackExchangeRedis #6425


0
cmthomps created

Have you implemented AspNetZero using the the StackExchange Redis Backplane? Trying to tackle SignlaR behind a load balancer using Set up a Redis backplane for ASP.NET Core SignalR scale-out. I've added the StackExchangeRedis NuGet package and called AddStackExchangeRedis after the AddSignalR method. Everything seems to work ok, but SignalR only works for users that are connected to the same server. If user A is on server 1 and user B is on server 2, it doesn't work. If A & B are both on Server 1, it works. This is the code we've implemented in the Startup ConfigureServices.

var signalrBuilder = services.AddSignalR(o =>
{
    o.EnableDetailedErrors = true;
});

signalrBuilder.AddStackExchangeRedis(_appConfiguration["SignalR:ConnectionString"], options => {
    options.Configuration.ChannelPrefix = "zerochannel";
});

I can see all the servers "subscribe" in the redis logs. But for some reason, when I message goes out of one server the other servers are not picking it up. This is an example of what I see in Redis.  I'm not entirely sure the meaning/importance of the names on the SUBSCRIBE lines.

1549404094.848664 [0 172.17.0.12:37121] "AUTH" "XXXXXXX"
1549404094.848693 [0 172.17.0.12:37121] "ECHO" "\x89\xcf\xf7\xf4>\xaf C\x9cz\x88\xd8\xf1\x98\xb6\x05"
1549404094.848704 [0 172.17.0.12:37121] "SUBSCRIBE" "zerochannel__Booksleeve_MasterChanged"
1549404094.864930 [0 172.17.0.12:36336] "AUTH" "5NSAjVIyvGmfpBot"
1549404094.865012 [0 172.17.0.12:36336] "INFO"
1549404094.865086 [0 172.17.0.12:36336] "ECHO" "\x89\xcf\xf7\xf4>\xaf C\x9cz\x88\xd8\xf1\x98\xb6\x05"
1549404094.866966 [0 172.17.0.12:36336] "PING"
1549404094.866995 [0 172.17.0.12:36336] "GET" "__Booksleeve_TieBreak"
1549404094.867013 [0 172.17.0.12:36336] "PING"
1549404094.867017 [0 172.17.0.12:36336] "GET" "__Booksleeve_TieBreak"
1549404094.867023 [0 172.17.0.12:36336] "INFO" "replication"
1549404095.289384 [0 172.17.0.12:37121] "SUBSCRIBE" "zerochannelSMCDemo.Web.Chat.SignalR.ChatHub:all"
1549404095.333222 [0 172.17.0.12:37121] "SUBSCRIBE" "zerochannelSMCDemo.Web.Chat.SignalR.ChatHub:internal:groups"
1549404095.366710 [0 172.17.0.12:37121] "SUBSCRIBE" "zerochannelSMCDemo.Web.Chat.SignalR.ChatHub:internal:ack:smczero-deployment-858c6845f9-sn8p4_056b26b232fe4cc99fdee32792553850"

17 Answer(s)
  • 0
    cmthomps created

    @ismcagdas - I got notifications working in a load balanced environment. What I found was that there maybe an issue with SignalRRealTimeNotifier in the Abp.AspNetCore.SignalR. In that code, there is a line that says:

    var signalRClient = _hubContext.Clients.Client(onlineClient.ConnectionId);
    

    If I change that line of code to:

    var signalRClient = _hubContext.Clients.All;
    

    I'll be completely honest, I don't really know why it works. I was going through Microsoft's example on the redis backplane and found this difference. When I changed it, it started working (at least in my initial tests).

  • 0
    aaron created

    Does onlineClient.ConnectionId have a valid value?

  • 0
    cmthomps created

    It looks like the ConnectionId is valid. I starting logging the UserId and the ConnectionId. This what that looks like:

    UserID: 1 ConnectionId: wRX8xHl-XqgDsKZ8zGunNA
    

    When I use var signalRClient = _hubContext.Clients.Client(onlineClient.ConnectionId);, I do not see any activity on the redis server.  When I use Clients.All, I start to see activity on the Redis server that looks like:


    1549554528.516258 [0 172.17.0.9:45157] "INFO" "replication"
    1549554536.048210 [0 172.17.0.1:61383] "PUBLISH" "smczeroSmc.Abp.AspNetCore.SignalR.Hubs.AbpCommonHub:all" "\x92\x90\x81\xa4json\xc5\x02E{\"type\":1,\"target\":\"getNotification\",\"arguments\":[{\"tenantId\":null,\"userId\":1,\"state\":0,\"notification\":{\"tenantId\":null,\"notificationName\":\"App.TestNotification\",\"data\":{\"message\":{\"sourceName\":\"SMCDemo\",\"name\":\"TestNotification\"},\"type\":\"Abp.Notifications.LocalizableMessageNotificationData\",\"properties\":{\"message\":{\"sourceName\":\"SMCDemo\",\"name\":\"TestNotification\"}}},\"entityType\":null,\"entityTypeName\":null,\"entityId\":null,\"severity\":2,\"creationTime\":\"2019-02-07T10:48:52.744841-05:00\",\"id\":\"4defe4c2-997f-4cc2-9368-fda4a8ec947d\"},\"id\":\"2aeb9a04-d11a-4118-8ef8-5e592b0b06ca\"}]}\x1e"
    1549554538.425415 [0 127.0.0.1:54504] "AUTH" "5NSAjVIyvGmfpBot"
    

    I don't really know what the implications are to using Clients.All.  Do you?

    Thanks, Craig

  • 0
    aaron created

    Using Clients.All means sending it to all clients.

    According to aspnet/SignalR#2460, sending to a specific client should work.

    You can create a new issue referencing that here: https://github.com/aspnet/AspNetCore/issues/new

    Related: aspnet/AspNetCore#5358

  • 0
    cmthomps created

    Here's what seems to have fixed it for me. I don't really understand why, but it seems to work nonetheless.

    In the SendNotificationsAsync method, I changed the line that says:

    signalRClient.SendAsync("getNotification", userNotification);
    

    to

    _hubContext.Clients.Client(onlineClient.ConnectionId).SendAsync("getNotification", userNotification);
    

    Does that make any sense to you?  if I use the signalRClient object, it doesn't seem to send the notifications properly.

  • 0
    aaron created

    No, it's the same.

  • 0
    cmthomps created

    Not sure. The first one does not work with Redis the second one does. I guess others will find this if they need it.

    I'd like to vote for AspNetZero to enhance the product to support scale out using load balancing. Given the popularity of kubernetes, having this capability will be extremely valuable. I've got it running on kubernetes now, but finding that notifications and chat do not work. We got notifications to work using the hack above and I'm trying to get chat working with Redis using the code developed by another user. Haven't had any luck yet with Chat. All credit to personball on github.

    RedisOnlineClientManager.cs

  • 0
    aaron created

    Try reverting the change.

  • 0
    cmthomps created

    Thanks @aaron. You're right.... That leads me to my next question. As part of this, I've created my own module that represents the ABP SignalRRealTimeNotifier called SmcSignalRRealTimeNotifier. In there, I've defined a my own AbpCommonHub called SmcAbpCommonHub. The code in the module is exactly the same as what is in ABP (Abp.AspNetCore.SignalR/AspNetCore/SignalR/).

    When I use the AbpCommonHub in the Mvc startup.cs, the redis notifications do not work. When I override the SignalRRealTimeNotifier dependency with my own SmcSignalRRealTimeNotifier and use the SmcAbpCommonHub, the redis notifications start working. Can you think of a reason why that is?

  • 0
    aaron created

    How are you:

    • subscribing on the client side, and
    • using SignalRRealTimeNotifier on the server side

    before and after switching to SmcSignalRRealTimeNotifier?

  • 0
    cmthomps created

    I made no changes to the client side of things. In the Web.Mvc project, I changed a couple of things:

    1. In the Startup.cs ConfigureServices method, I changed
                var signalrBuilder = services.AddSignalR();
    

    To

                    var signalrBuilder = services.AddSignalR();
                    signalrBuilder.AddStackExchangeRedis(_appConfiguration["SignalR:ConnectionString"], options => {
                            options.Configuration.ChannelPrefix = "smczero";
                            });
                    }
    
    1. In the Configure method, I changed
                app.UseSignalR(routes =>
                {
                    routes.MapHub<AbpCommonHub>("/signalr");
                    routes.MapHub<ChatHub>("/signalr-chat");
                });
    

    To

                app.UseSignalR(routes =>
                {
                    routes.MapHub<SmcAbpCommonHub>("/signalr");
                    routes.MapHub<ChatHub>("/signalr-chat");
                });
    

    The last thing I did was register the register the SmcSignalRRealTimeNotifier as IRealTimeNotifier in my Web.Core project which is where the new module code lives.

  • 0
    aaron created

    You didn't answer any of the 2 questions, which were about usage and not configuration.

    If you're saying usage didn't change and the 2 replaced classes are subclasses that don't override or add any method, then they are pointless changes similar to assigning a variable as before.

    Try reverting those changes.

  • 0
    cmthomps created

    Here's a question for you. Have you tried to reproduce the issue I'm having on your machine with Redis and the StackExchangeRedis nuget package? When I run a straight, out of the box download of AspNetZero with no changes other than adding the StackExchangeRedis nuget package and adding the call to AddStackExchangeRedis in Startup.cs, the notifications do not work. Don't know what else to tell you.

  • 0
    aaron created

    No, because it is not a problem with ASP.NET Zero or ABP.

  • 0
    ismcagdas created

    Hi @cmthomps

    Sorry for our late reply. Did you find any solution for this ? I just wanted to check before setting up such an environment to test this problem.

    Thanks,

  • 0
    cmthomps created

    Thanks @ismcagdas. I have a private github repository that I just shared with you. It contains an AspNetZero demo project that uses Redis for both Chat and Notifications. I've done some minimal testing. But it appears to work on Kubernetes running multiple pods behind a load balancer. Not sure if it's something that can be incorporated into the base product. For us, having the ability to run behind a load balancer is very important.

  • 0
    ismcagdas created

    I got the invitation and I will try to check it on weekend.

    Thanks,