Storm HDFS bolt throws ClosedChannelException when Timed rotation policy is used

/ Storm / 没有评论 / 61浏览

Storm HDFS bolt throws ClosedChannelException when Timed rotation policy is used.

背景

在使用storm-hdfs时,设置定时滚动文件,会报错:ClosedChannelException

确定为官方bug:STORM-2993

该bug在storm1.2.2版本得到解决。

1.2.2版本以下解决

官方解决方案

修改org.apache.storm.hdfs.bolt.AbstractHdfsBolt#startTimedRotationPolicy

private void startTimedRotationPolicy() {
    long interval = ((TimedRotationPolicy)this.rotationPolicy).getInterval();
    this.rotationTimer = new Timer(true);
    TimerTask task = new TimerTask() {
        @Override
        public void run() {
            // fix: Storm HDFS bolt throws ClosedChannelException when Time rotation policy is used
            // STORM-2993
            // https://issues.apache.org/jira/browse/STORM-2993
            // https://github.com/apache/storm/pull/2592/files/9bb2a509776e2ec655a368a5d0e604141986ac62
            synchronized (writeLock) {
                for (final AbstractHDFSWriter writer : writers.values()) {
                    try {
                        rotateOutputFile(writer);
                    } catch (IOException e) {
                        LOG.warn("IOException during scheduled file rotation.", e);
                    }
                }
                writers.clear();
            }
        }
    };
    this.rotationTimer.scheduleAtFixedRate(task, interval, interval);
}