首页
学习
活动
专区
圈层
工具
发布
首页
学习
活动
专区
圈层
工具
MCP广场
社区首页 >专栏 >axis2开发webservice_docker映射出来端口访问不了

axis2开发webservice_docker映射出来端口访问不了

作者头像
全栈程序员站长
发布2022-10-01 16:57:06
发布2022-10-01 16:57:06
67800
代码可运行
举报
运行总次数:0
代码可运行

大家好,又见面了,我是你们的朋友全栈君。

记录一次正式环境服务报错排查记录。

某日被通知线上服务告警,错误日志全是 Timeout waiting for connection

首先梳理项目架构,项目很简单,就是一个使用axis2构建的webserice的客户端, Axis2版本为1.5.5

开始从此段报错入手排查,定位到 MultiThreadedHttpConnectionManager 这个类的 doGetConnection 方法

代码语言:javascript
代码运行次数:0
运行
复制
private HttpConnection doGetConnection(HostConfiguration hostConfiguration, 
long timeout) throws ConnectionPoolTimeoutException {
HttpConnection connection = null;
int maxHostConnections = this.params.getMaxConnectionsPerHost(hostConfiguration);
int maxTotalConnections = this.params.getMaxTotalConnections();
synchronized (connectionPool) {
// we clone the hostConfiguration
// so that it cannot be changed once the connection has been retrieved
hostConfiguration = new HostConfiguration(hostConfiguration);
HostConnectionPool hostPool = connectionPool.getHostPool(hostConfiguration, true);
WaitingThread waitingThread = null;
boolean useTimeout = (timeout > 0);
long timeToWait = timeout;
long startWait = 0;
long endWait = 0;
while (connection == null) {
if (shutdown) {
throw new IllegalStateException("Connection factory has been shutdown.");
}
// happen to have a free connection with the right specs
//
if (hostPool.freeConnections.size() > 0) {
connection = connectionPool.getFreeConnection(hostConfiguration);
// have room to make more
//
} else if ((hostPool.numConnections < maxHostConnections) 
&& (connectionPool.numConnections < maxTotalConnections)) {
connection = connectionPool.createConnection(hostConfiguration);
// have room to add host connection, and there is at least one free
// connection that can be liberated to make overall room
//
} else if ((hostPool.numConnections < maxHostConnections) 
&& (connectionPool.freeConnections.size() > 0)) {
connectionPool.deleteLeastUsedConnection();
connection = connectionPool.createConnection(hostConfiguration);
// otherwise, we have to wait for one of the above conditions to
// become true
//
} else {
// TODO: keep track of which hostConfigurations have waiting
// threads, so they avoid being sacrificed before necessary
try {
if (useTimeout && timeToWait <= 0) {
throw new ConnectionPoolTimeoutException("Timeout waiting for connection");
}
if (LOG.isDebugEnabled()) {
LOG.debug("Unable to get a connection, waiting..., hostConfig=" + hostConfiguration);
}
if (waitingThread == null) {
waitingThread = new WaitingThread();
waitingThread.hostConnectionPool = hostPool;
waitingThread.thread = Thread.currentThread();
} else {
waitingThread.interruptedByConnectionPool = false;
}
if (useTimeout) {
startWait = System.currentTimeMillis();
}
hostPool.waitingThreads.addLast(waitingThread);
connectionPool.waitingThreads.addLast(waitingThread);
connectionPool.wait(timeToWait);
} catch (InterruptedException e) {
if (!waitingThread.interruptedByConnectionPool) {
LOG.debug("Interrupted while waiting for connection", e);
throw new IllegalThreadStateException(
"Interrupted while waiting in MultiThreadedHttpConnectionManager");
}
// Else, do nothing, we were interrupted by the connection pool
// and should now have a connection waiting for us, continue
// in the loop and let's get it.
} finally {
if (!waitingThread.interruptedByConnectionPool) {
// Either we timed out, experienced a "spurious wakeup", or were
// interrupted by an external thread.  Regardless we need to 
// cleanup for ourselves in the wait queue.
hostPool.waitingThreads.remove(waitingThread);
connectionPool.waitingThreads.remove(waitingThread);
}
if (useTimeout) {
endWait = System.currentTimeMillis();
timeToWait -= (endWait - startWait);
}
}
}
}
}
return connection;
}

得到几个关键变量:

连接池最大连接数 maxTotalConnections=20;

当前连接主机(或相同主机)最大连接数 maxHostConnections=2;

连接池总空闲连接数 connectionPool.freeConnections;

当前连接主机空闲连接数 hostPool.freeConnections;

得到结论一:

当前主机连接数达到最大值,且当前连接主机空闲连接数为0时,获取连接超时时会抛出 Timeout waiting for connection 异常。

到此,初步怀疑某次请求时未正常释放占用的连接,或未回收到空闲连接池中。

继续排查日志,一直找到第一次抛出 Timeout waiting for connection 异常的时候再往上查看还有没有其他错误日志。于是发现了 Transport error: 503 Error: Service Unavailable 这个异常

直接进入源码

代码语言:javascript
代码运行次数:0
运行
复制
/**
* Used to handle the HTTP Response
*
* @param msgContext - The MessageContext of the message
* @param method     - The HTTP method used
* @throws IOException - Thrown in case an exception occurs
*/
private void handleResponse(MessageContext msgContext,
HttpMethodBase method) throws IOException {
int statusCode = method.getStatusCode();
log.trace("Handling response - " + statusCode);
if (statusCode == HttpStatus.SC_OK) {
// Save the HttpMethod so that we can release the connection when cleaning up
msgContext.setProperty(HTTPConstants.HTTP_METHOD, method);
processResponse(method, msgContext);
} else if (statusCode == HttpStatus.SC_ACCEPTED) {
// Since we don't expect any content with a 202 response, we must release the connection
method.releaseConnection();
} else if (statusCode == HttpStatus.SC_INTERNAL_SERVER_ERROR ||
statusCode == HttpStatus.SC_BAD_REQUEST) {
// Save the HttpMethod so that we can release the connection when cleaning up
msgContext.setProperty(HTTPConstants.HTTP_METHOD, method);
Header contenttypeHeader =
method.getResponseHeader(HTTPConstants.HEADER_CONTENT_TYPE);
String value = null;
if (contenttypeHeader != null) {
value = contenttypeHeader.getValue();
}
OperationContext opContext = msgContext.getOperationContext();
if(opContext!=null){
MessageContext inMessageContext =
opContext.getMessageContext(WSDLConstants.MESSAGE_LABEL_IN_VALUE);
if(inMessageContext!=null){
inMessageContext.setProcessingFault(true);
}
}
if (value != null) {
processResponse(method, msgContext);
}
Object isTransportNonBlocking = msgContext.getProperty(
MessageContext.TRANSPORT_NON_BLOCKING);
if (isTransportNonBlocking != null && (Boolean)isTransportNonBlocking) {
throw new AxisFault(Messages.getMessage("transportError",
String.valueOf(statusCode),
method.getStatusText()));
}
} else {
throw new AxisFault(Messages.getMessage("transportError",
String.valueOf(statusCode),
method.getStatusText()));
}
}

根据我之前读的源码我知道 method.releaseConnection(); 这个方法将会最终调用 connectionPool.freeConnection(conn); 即生成空闲连接放入总连接池中。

而 msgContext.setProperty(HTTPConstants.HTTP_METHOD, method); 这个方法将由其他地方获取调用并最终调用的仍是 method.releaseConnection();

综上可知 返回码为503或者任意其他上述代码中未处理的返回码时都不会将占用连接重置为空闲连接放入总连接池中。

到此可知服务告警原因:

因为服务端发生了我们未知的操作,导致客户端获取请求时的返回码为503而占用了所有的当前主机连接数(2个)而不会释放重置为空闲连接放入连接池

于是后续连接从连接池获取连接时均超时抛出异常 Timeout waiting for connection

版权声明:本文内容由互联网用户自发贡献,该文观点仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 举报,一经查实,本站将立刻删除。

发布者:全栈程序员栈长,转载请注明出处:https://javaforall.cn/194542.html原文链接:https://javaforall.cn

本文参与 腾讯云自媒体同步曝光计划,分享自作者个人站点/博客。
原始发表:2022年9月12日 ,如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 作者个人站点/博客 前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体同步曝光计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档