Hello @ismcagdas it is difficult to say but we didn't get any exception until now. i will close the issue for now. if sth happens i can open a new thread thank you for your help.
i will try it @ismcagdas, i am not using it.
Also i need to say before this exception, i am getting out of memory exception. But i don't see any memory issue on the metrics not in backend or redis cache. Maybe after this exception signalr also goes down since it uses redis?
ERROR 2022-12-15 16:06:08,713 [129 ] Abp.Runtime.Caching.Redis.AbpRedisCache - StackExchange.Redis.RedisConnectionException: Failed to write ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, Char* pThreadName) at System.Threading.Thread.StartCore() at System.Threading.Thread.Start(Object parameter, Boolean captureContext) at StackExchange.Redis.PhysicalBridge.WriteMessageTakingWriteLockAsync(PhysicalConnection physical, Message message) in //src/StackExchange.Redis/PhysicalBridge.cs:line 1012 --- End of inner exception stack trace --- at Abp.Runtime.Caching.Redis.AbpRedisCache.TryGetValueAsync(String key) at Abp.Runtime.Caching.AbpCacheBase
2.GetAsync(TKey key, Func
2 factory) StackExchange.Redis.RedisConnectionException: Failed to write ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, Char* pThreadName) at System.Threading.Thread.StartCore() at System.Threading.Thread.Start(Object parameter, Boolean captureContext) at StackExchange.Redis.PhysicalBridge.WriteMessageTakingWriteLockAsync(PhysicalConnection physical, Message message) in //src/StackExchange.Redis/PhysicalBridge.cs:line 1012 --- End of inner exception stack trace --- at Abp.Runtime.Caching.Redis.AbpRedisCache.TryGetValueAsync(String key) at Abp.Runtime.Caching.AbpCacheBase2.GetAsync(TKey key, Func
2 factory)
Hello. We use aspnet zero version 11.4.0 right now. Our application is deployed to azure. So we have Redis Cache (6gb memory), Azure Signalr (2x instance), Backend App Service(Single instance), FrontEnd App Service(Single), Azure Functions App Service(Single), Azure Blob and Sql Server
When it is under load, sometimes we get a timeout from redis server. I think this problem is related with Signalr. here is the stack trace.
ERROR 2022-12-15 16:07:46,533 [149 ] .AspNetCore.SignalR.HubConnectionHandler - Error when dispatching 'OnConnectedAsync' on hub. StackExchange.Redis.RedisTimeoutException: Timeout performing HMSET (5000ms), inst: 0, qu: 75, qs: 0, aw: False, bw: Activating, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: dc-redis-live.redis.cache.windows.net:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: WN0LDWK00003O, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=13,Free=1010,Min=8,Max=1023), v: 2.2.88.56325 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor
1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2884 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor
1 processor, ServerEndPoint server) in //src/StackExchange.Redis/RedisBase.cs:line 54 at StackExchange.Redis.RedisDatabase.HashSet(RedisKey key, HashEntry[] hashFields, CommandFlags flags) in //src/StackExchange.Redis/RedisDatabase.cs:line 445 at Abp.Runtime.Caching.Redis.RealTime.RedisOnlineClientStore.Add(IOnlineClient client) at Abp.RealTime.OnlineClientManager.Add(IOnlineClient client) at Abp.AspNetCore.SignalR.Hubs.OnlineClientHubBase.OnConnectedAsync() at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.OnConnectedAsync(HubConnectionContext connection) at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher
1.OnConnectedAsync(HubConnectionContext connection) at Microsoft.AspNetCore.SignalR.HubConnectionHandler1.RunHubAsync(HubConnectionContext connection) ERROR 2022-12-15 16:07:46,704 [98 ] .AspNetCore.SignalR.HubConnectionHandler - Error when dispatching 'OnConnectedAsync' on hub. StackExchange.Redis.RedisTimeoutException: Timeout performing HMSET (5000ms), inst: 0, qu: 75, qs: 0, aw: False, bw: Activating, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: dc-redis-live.redis.cache.windows.net:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: WN0LDWK00003O, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=12,Free=1011,Min=8,Max=1023), v: 2.2.88.56325 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor
1 processor, ServerEndPoint server) in //src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2884 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/RedisBase.cs:line 54 at StackExchange.Redis.RedisDatabase.HashSet(RedisKey key, HashEntry[] hashFields, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 445 at Abp.Runtime.Caching.Redis.RealTime.RedisOnlineClientStore.Add(IOnlineClient client) at Abp.RealTime.OnlineClientManager.Add(IOnlineClient client) at Abp.AspNetCore.SignalR.Hubs.OnlineClientHubBase.OnConnectedAsync() at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher
1.OnConnectedAsync(HubConnectionContext connection) at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.OnConnectedAsync(HubConnectionContext connection) at Microsoft.AspNetCore.SignalR.HubConnectionHandler
1.RunHubAsync(HubConnectionContext connection) ERROR 2022-12-15 16:07:52,394 [166 ] icrosoft.Azure.SignalR.ServiceConnection - Error processing the connection 7xEIgc7k3ngOq0ATq8Wxmw5a1ef6cd1. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at System.AggregateException.GetBaseException() at Microsoft.Azure.SignalR.ServiceConnection.ProcessClientConnectionAsync(ClientConnectionContext connection) ERROR 2022-12-15 16:07:52,923 [130 ] .AspNetCore.SignalR.HubConnectionHandler - Error when dispatching 'OnConnectedAsync' on hub. StackExchange.Redis.RedisTimeoutException: Timeout performing HMSET (5000ms), inst: 0, qu: 86, qs: 0, aw: False, bw: Activating, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: dc-redis-live.redis.cache.windows.net:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: WN0LDWK00003O, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=22,Free=1001,Min=8,Max=1023), v: 2.2.88.56325 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2884 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor
1 processor, ServerEndPoint server) in //src/StackExchange.Redis/RedisBase.cs:line 54 at StackExchange.Redis.RedisDatabase.HashSet(RedisKey key, HashEntry[] hashFields, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 445 at Abp.Runtime.Caching.Redis.RealTime.RedisOnlineClientStore.Add(IOnlineClient client) at Abp.RealTime.OnlineClientManager.Add(IOnlineClient client) at Abp.AspNetCore.SignalR.Hubs.OnlineClientHubBase.OnConnectedAsync() at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.OnConnectedAsync(HubConnectionContext connection) at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher
1.OnConnectedAsync(HubConnectionContext connection) at Microsoft.AspNetCore.SignalR.HubConnectionHandler`1.RunHubAsync(HubConnectionContext connection)
Do you have any idea how this can be fixed? is this related with Abp 7.4 upgrade since i have seen some issues with multiple signalr instances?
Hello again, I was trying couple of things to be sure about the problem. I make it work by doing couple of things. First of all i need to state that i am using hangfire on my app. Since hangfire is polling the database i see lots of dependency logs, so to fix that issue i have implemented custom telemetry processor. This stackoverflow link was helpful for that https://stackoverflow.com/questions/38320886/app-insights-disable-sql-dependency-telemetry
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.ApplicationInsights.DataContracts;
public class NoSQLDependencies : ITelemetryProcessor
{
private ITelemetryProcessor Next { get; set; }
// Link processors to each other in a chain.
public NoSQLDependencies(ITelemetryProcessor next)
{
this.Next = next;
}
public void Process(ITelemetry item)
{
if (IsSQLDependency(item)) { return; }
this.Next.Process(item);
}
private bool IsSQLDependency(ITelemetry item)
{
var dependency = item as DependencyTelemetry;
if (dependency?.DependencyTypeName == "SQL")
{
return true;
}
return false;
}
}
Afterwards I explicitly configured Hangfire to use Log4net by this code inside WebHostModule PreInitialize()
GlobalConfiguration.Configuration.UseLog4NetLogProvider();
now everythings work fine on my local machine. There is only one thing left i couldn't understand. if you look at the code below.
public class TestAppService : BookAndAdAppServiceBase
{
private readonly IBackgroundJobManager _backgroundJobManager;
private readonly ILogger _logger;
public TestAppService(IBackgroundJobManager backgroundJobManager, ILogger logger)
{
_backgroundJobManager = backgroundJobManager;
_logger = logger;
}
public async Task TestLogger()
{
Logger.Error($"Application Insights test from {nameof(TestAppService)}....", new Exception("this is custom exception"));
_logger.Error($"Application Insights test with reference (can2) {nameof(TestAppService)}");
await _backgroundJobManager.EnqueueAsync<TestJob, TestJobArgs>(new TestJobArgs() { TenantId = AbpSession.TenantId,UserId=AbpSession.UserId },delay:TimeSpan.FromSeconds(10));
}
}
as you can see first logger is logging with exception, and this is coming to application insights as exception which is expected. On the other hand second logger since it doesn't pass any exception it is logged as Trace. Which at first sight seems weird since i call Error() method. But anyway it works fine, i think we can close this.
Hello @ismcagdas Not really helped. Actually i realize that if i log it by using ilogger in application layer or in controller, i can see it in application insights, but background job and background worker errors are not logged at all (I am using Hangfire for it.) What i couldn't understand is, i can see the hangfire errors on text file that i append to azure appservice root folder. They both use log4net, why fileappender can do it but not application insights appender? I really couldn't grasp it. Do you have any idea?
Hello, I am trying to use application insights inside ANZ project. But i couldn't make it work with ILogger. Whatever i log with ILogger injection, is not logged to application insights. Here is what i have done so far. 1- Added nuget packages from ApplicationInsights to web.host project
<PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.21.0" />
<PackageReference Include="Microsoft.ApplicationInsights.Log4NetAppender" Version="2.21.0" />
2- I have added extra appender to my log4net file.
<?xml version="1.0" encoding="utf-8" ?>
<log4net>
<appender name="RollingFileAppender" type="log4net.Appender.RollingFileAppender" >
<file type="log4net.Util.PatternString" value="%property{LoggerFilePath}/App_Data/Logs/Logs.txt" />
<!--<file value="../../../App_Data/Logs/Logs.txt" />-->
<appendToFile value="true" />
<rollingStyle value="Size" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="10000KB" />
<staticLogFileName value="true" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%-5level %date [%-5.5thread] %-40.40logger - %message%newline" />
</layout>
</appender>
<appender name="ApplicationInsightsAppender" type="Microsoft.ApplicationInsights.Log4NetAppender.ApplicationInsightsAppender, Microsoft.ApplicationInsights.Log4NetAppender">
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%message %newline" />
</layout>
</appender>
<root>
<appender-ref ref="RollingFileAppender" />
<appender-ref ref="ApplicationInsightsAppender"/>
<level value="DEBUG" />
</root>
</log4net>
3- Change the Program.cs (this step is not really relevant, i am trying to support both application insights and file log on azure. Since azure portal does not allow writing to file inside root folder, i give the path to write from appsettings.json)
.ConfigureLogging((context, logging) =>
{
logging.AddFilter("Microsoft.EntityFrameworkCore.Database.Command", LogLevel.Error);
var hostingEnvironment = context.HostingEnvironment;
var configurationRoot = hostingEnvironment.GetAppConfiguration();
var defaultRootPath = hostingEnvironment.IsDevelopment() ?
"../../../" : "";
log4net.GlobalContext.Properties["LoggerFilePath"] = defaultRootPath;
var log4NetFilePath = configurationRoot["Log4Net:Path"];
if (!string.IsNullOrEmpty(log4NetFilePath))
log4net.GlobalContext.Properties["LoggerFilePath"] = log4NetFilePath;
})
4- Configure application insights on web host module.
public IServiceProvider ConfigureServices(IServiceCollection services)
{
//some code above
if (bool.Parse(_appConfiguration["ApplicationInsights:ApplicationInsightsEnabled"]))
{
services.AddApplicationInsightsTelemetry(new ApplicationInsightsServiceOptions
{
ConnectionString = _appConfiguration["ApplicationInsights:ConnectionString"],
EnableActiveTelemetryConfigurationSetup = true
});
}
//some code below
}
5- here is my appsettings.json
"ApplicationInsights": {
"ApplicationInsightsEnabled": true,
"ConnectionString": "InstrumentationKey=..."
},
"Log4Net": {
"Path": ""
},
When i try to inject ILogger and use that instance, i can see it on file but not on application insights only thing i have seen on app insights are Ef Core warnings and some parts i would never use.
so my question is, if there is any way logging into app insights with Logger.Warn() or Logger.Error(), and how can i remove ef core warnings?
public async Task TestApplicationInsights()
{
Logger.Error("Application Insights test from Logger....",new Exception("this is custom exception"));
_logger.Error("Application Insights by injection ILogger....", new InvalidOperationException("this is invalid operation exception"));
}
Thank you for the assistance.
Hello @ismcagdas, thanks for the reply, How can i create the background job from an entity? shouldn't I need to inject IBackgroundJobManager? Can you give me a code example if there is a way to do it inside the entity?
Hello, I have a question about domain events. I want to create custom domain events. I have different purposes to create these domain events. Here are 2 seperate use cases.
I have an Entity who has status property, and when status is changed from the entity, i want to send an email to the user. Code Ex:
public class Offer : FullAuditedEntity, IMustHaveTenant
{
//constructor and some code over here.
public OfferStatus Status {get; private set;}
public DateTime ApprovedDate{get;private set;}
public void ApproveOffer(){
Status=OfferStatus.Approved;
ApprovedDate=Clock.Now
//trigger a domain event.
EventBus.Default.Trigger(new OfferStatusChangedEventData(Status, OfferStatus.CustomerApproved, this));
}
}
public class OfferStatusChangedEventData:EventData
{
public OfferStatusChangedEventData(OfferStatus newValue, OfferStatus oldValue, Offer offer)
{
NewValue = newValue;
OldValue = oldValue;
Offer = offer;
}
public OfferStatus NewValue { get; private set; }
public OfferStatus OldValue { get; private set; }
public Offer Offer { get; private set; }
}
public class OfferStatusChangedEventHandler : IEventHandler<OfferStatusChangedEventData>, ITransientDependency
{
//handle it over here.
//send an email
}
I believe with this code even if the transaction fails, it is gonna send the email to the user (Let's assume ApproveOffer() method is called from application service ). So what i want is to send the email after the transaction is committed to database. Is there any easy way to do it with aspnetzero?
And sometimes (when I use eventual consistency) I want to do the stuff within the same transaction.
So the question is... is there anyway to say to the EventBus.Default.Trigger() method, do this after unit of work completes? or vice versa?
PS: I know that, I can write an extra code to the DbContext OnModelSaveChanges(), but as I understand (correct me if i am wrong) I need to hold a collection of Events needs to be triggered in that case. Can i access to IEntityChangeEventHelper from the entity and store my events in that dictionary so it can be dispatched after the SaveChanges()? https://github.com/aspnetboilerplate/aspnetboilerplate/blob/dev/src/Abp.EntityFrameworkCore/EntityFrameworkCore/AbpDbContext.cs#L45
I am closing this then. thank you for the time.