InternalServerError: Unknown server error occurred when processing this request

Issue

I receive sporadic the follow error with the Azure CosmosDB emulator on calling CosmosDatabase.createContainerIfNotExists.

• Also if the error message contains many helpless details there is nothing to understand the cause. How can I get more information about the error from the Azure CosmosDB emulator?

• What can be the cause of the error? The count of existing container if the error occur is approximate 70 to 130. This should be no problem for the limit of 250 containers.

• What is the best handling of the error?

{"ClassName":"CosmosException","userAgent":"azsdk-java-cosmos/4.17.0 Linux/5.4.0-84-generic JRE/11.0.12",
"statusCode":500,"resourceAddress":"https://10.10.1.150:8081/dbs/test/colls",
"error":"{\"code\":\"InternalServerError\",\"message\":\"Unknown server error occurred when processing this request.\\r\\nActivityId: 70bc2604-9e3c-4093-ad46-16bcc069a243, Microsoft.Azure.Documents.Common/2.14.0, 
StatusCode: InternalServerError\",\"additionalErrorInfo\":null}",
"innerErrorMessage":"Unknown server error occurred when processing this request.\r\nActivityId: 70bc2604-9e3c-4093-ad46-16bcc069a243, Microsoft.Azure.Documents.Common/2.14.0, 
StatusCode: InternalServerError","causeInfo":null,"responseHeaders":"{Transfer-Encoding=chunked, Server=Microsoft-HTTPAPI/2.0, Access-Control-Allow-Origin=, Access-Control-Allow-Credentials=true, x-ms-gatewayversion=version=2.14.0, Date=Fri, 03 Dec 2021 03:23:08 GMT, x-ms-activity-id=70bc2604-9e3c-4093-ad46-16bcc069a243, Content-Type=application/json}","requestHeaders":"[Accept=application/json, x-ms-date=Fri, 03 Dec 2021 03:23:08 GMT, Content-Type=application/json]",
"cosmosDiagnostics":{"userAgent":"azsdk-java-cosmos/4.17.0 Linux/5.4.0-84-generic JRE/11.0.12",
"requestLatencyInMs":519,"requestStartTimeUTC":"2021-12-03T03:23:08.615098Z","requestEndTimeUTC":"2021-12-03T03:23:09.134243Z","responseStatisticsList":[],"supplementalResponseStatisticsList":[],"addressResolutionStatistics":{},"regionsContacted":["https://10.10.1.150:8081/"],
"retryContext":{"statusAndSubStatusCodes":null,"retryLatency":0,"retryCount":0},"metadataDiagnosticsContext":{"metadataDiagnosticList":null},"serializationDiagnosticsContext":{"serializationDiagnosticsList":[{"serializationType":"CONTAINER_SERIALIZATION","startTimeUTC":"2021-12-03T03:23:08.615202Z","endTimeUTC":"2021-12-03T03:23:08.615249Z","durationInMicroSec":47}]},
"gatewayStatistics":{"sessionToken":null,"operationType":"Create","resourceType":"DocumentCollection","statusCode":500,"subStatusCode":0,"requestCharge":null,"requestTimeline":[{"eventName":"connectionCreated","startTimeUTC":"2021-12-03T03:23:08.615391Z","durationInMicroSec":235},{"eventName":"connectionConfigured","startTimeUTC":"2021-12-03T03:23:08.615626Z","durationInMicroSec":27},{"eventName":"requestSent","startTimeUTC":"2021-12-03T03:23:08.615653Z","durationInMicroSec":314},{"eventName":"transitTime","startTimeUTC":"2021-12-03T03:23:08.615967Z","durationInMicroSec":517832},{"eventName":"received","startTimeUTC":"2021-12-03T03:23:09.133799Z","durationInMicroSec":173}]},
"systemInformation":{"usedMemory":"250743 KB","availableMemory":"273545 KB","systemCpuLoad":"(2021-12-03T03:22:43.164095Z 4.0%), (2021-12-03T03:22:48.164056Z 2.6%), (2021-12-03T03:22:53.164086Z 3.3%), (2021-12-03T03:22:58.164100Z 2.6%), (2021-12-03T03:23:03.164114Z 2.1%), (2021-12-03T03:23:08.164169Z 2.1%)","availableProcessors":8},"clientCfgs":{"id":-1,"connectionMode":"DIRECT","numberOfClients":2,"connCfg":{"rntbd":"(cto:PT5S, rto:PT5S, icto:PT0S, ieto:PT1H, mcpe:130, mrpc:30, cer:false)","gw":"(cps:1000, rto:PT5S, icto:null, p:false)","other":"(ed: true, cs: false)"},"consistencyCfg":"(consistency: null, mm: true, prgns: [])"}}}
    at com.azure.cosmos.BridgeInternal.createCosmosException(BridgeInternal.java:463)
    at com.azure.cosmos.implementation.RxGatewayStoreModel.validateOrThrow(RxGatewayStoreModel.java:365)
    at com.azure.cosmos.implementation.RxGatewayStoreModel.lambda$toDocumentServiceResponse$0(RxGatewayStoreModel.java:294)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
    at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:73)
    at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:199)
    at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:118)
    at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:220)
    at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:130)
    at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:184)
    at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1815)
    at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:128)
    at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:259)
    at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
    at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:401)
    at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:416)
    at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:470)
    at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:685)
    at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1368)
    at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1245)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1282)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Suppressed: java.lang.Exception: #block terminated with an error
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
        at reactor.core.publisher.Mono.block(Mono.java:1703)
        at com.azure.cosmos.CosmosDatabase.blockContainerResponse(CosmosDatabase.java:291)
        at com.azure.cosmos.CosmosDatabase.createContainerIfNotExists(CosmosDatabase.java:198)

Solution

I solved the problem by already trying to create the container when initializing my objects at startup. I ignore an error at this point. This initialization is largely sequential.

When using the container a while later, I then try again if it failed before. This reduces the startup time of the application a bit, but the error no longer occurred.

Answered By - Horcrux7