Блог

Я разработал .net core web API, в котором все запросы сначала вставляются в базу данных, а затем сгенерированный идентификатор запроса вставляется в очередь redis.

Благодаря нагрузочному тестированию приложение безупречно работает с IIS, размещенным на сервере виртуальной машины, при любом количестве нагрузок. При развертывании приложения на K8S с 3 модулями (конфигурация каждого модуля — 1 процессор и 4 ГБ оперативной памяти) производительность приложения снижается, когда количество запросов превышает 750. При загрузке 1000 запросов иногда все запросы выполняются успешно, но для завершения всех запросов требуется 2-3 минуты, а иногда несколько запросов завершаются сбоем. При загрузке от 1000 до 2000 увеличивается количество неудачных запросов. Ниже приведены ошибки, которые я получил случайным образом.

Ошибка 1

 No connection is available to service this operation: LPUSH GENEVAREQ; IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=12,Free=32755,Min=4,Max=32767), Local-CPU: n/a n   at StackExchange.Redis.ConnectionMultiplexer.
  

Ошибка 2

 at System.Lazy`1.CreateValue()
at System.Lazy`1.get_Value()
at XYZ.Demo.RedisService.RedisConnectionFactory.Connection() in D:TeamCitybuildAgentworkabdf9bd4783a2774XYZ.Demo.RedisServiceRedisConnectionFactory.cs:line 26
at XYZ.Demo.RedisService.RedisQueueService`1..ctor(IRedisConnectionFactory connectionFactory) in D:TeamCitybuildAgentworkabdf9bd4783a2774XYZ.Demo.RedisServiceRedisQueueService.cs:line 22
at XYZ.DemoAPI.Services.RequestService.EnqueueRequest(String requestId, String name, String queueName, Boolean priority) in D:TeamCitybuildAgentworkabdf9bd4783a2774DemoAPIServicesRequestService.cs:line 205
at XYZ.DemoAPI.Services.RequestService.PostReportRequest(RequestParams p) in D:TeamCitybuildAgentworkabdf9bd4783a2774DemoAPIServicesRequestService.cs:line 127
at XYZ.DemoAPI.Controllers.BaseController`1.PushReportRequest[TResult,TRow,P](P param, IProcessRequest`3 service) in D:TeamCitybuildAgentworkabdf9bd4783a2774DemoAPIControllersBaseController.cs:line 54
at XYZ.DemoAPI.Controllers.DemoApiController.GetReport(String businessLine, String category, String firmId, String authorization, String authUserId, Object requestPayload, String requestTrackingId) in D:TeamCitybuildAgentworkabdf9bd4783a2774DemoAPIControllersDemoApiController.cs:line 57
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(Stateamp; next, Scopeamp; scope, Objectamp; state, Booleanamp; isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
  

Ошибка 3

— Завершение трассировки стека из предыдущего местоположения, где было вызвано исключение —

 at System.Lazy`1.CreateValue()
at System.Lazy`1.get_Value()
at XYZ.Demo.RedisService.RedisConnectionFactory.Connection() in D:TeamCitybuildAgentworkabdf9bd4783a2774XYZ.Demo.RedisServiceRedisConnectionFactory.cs:line 26
at XYZ.Demo.RedisService.RedisQueueService`1..ctor(IRedisConnectionFactory connectionFactory) in D:TeamCitybuildAgentworkabdf9bd4783a2774XYZ.Demo.RedisServiceRedisQueueService.cs:line 22
at XYZ.DemoAPI.Services.RequestService.EnqueueRequest(String requestId, String name, String queueName, Boolean priority) in D:TeamCitybuildAgentworkabdf9bd4783a2774GenevaAPIServicesRequestService.cs:line 205
  

Ошибка 4

 A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server) n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternalamp; connection)n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternalamp; connection)n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternalamp; connection)n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)n   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)n   at Microsoft.Data.SqlClient.SqlConnection.Open()n   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)n   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)n   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(RelationalCommandParameterObject parameterObject)n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.InitializeReader(DbContext _, Boolean result)n   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()n   at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source)n   at XYZ.Demo.BuisnessService.ValidateService.GetClusterServer(RequestParams requestParams, Stringamp; queueName, Stringamp; filePath) in D:\TeamCity\buildAgent\work\abdf9bd4783a2774\XYZ.Demo.BuisnessService\ValidateService.cs:line 220n   at XYZ.DemoAPI.Services.RequestService.PostReportRequest(RequestParams p) in
  

Ошибка 5

 The timeout period elapsed during the post-login phase.  The connection could have timed out while waiting for server to complete the login process and respond; Or it could have timed out while attempting to create multiple active connections.  The duration spent while attempting to connect to this server was - [Pre-Login] initialization=7; handshake=1069; [Login] initialization=0; authentication=0; [Post-Login] complete=13032;  n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternalamp; connection)n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternalamp; connection)n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternalamp; connection)n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)n   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)n   at Microsoft.Data.SqlClient.SqlConnection.Open()n   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)n   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)n   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(RelationalCommandParameterObject parameterObject)n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.InitializeReader(DbContext _, Boolean result)n
  

Ошибка 6

 <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.17.10</center>
</body>
</html>


<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
  

Пожалуйста, предложите, как исправить эту проблему.