I'm using Flume 1.6.0 in machine X and Hadoop 2.7.1 in another machine Y .I want to write logs from machine X to Machine Y. When I send Avro Events to the Flume 1.6.0 and it try to write in S3 Bucket which is in AWS. while starting Flume agent follwing exception occurs.
Machine X config:
agent.sources = localsource
agent.chaels = memoryChael
agent.sinks = avro_Sink
agent.sources.localsource.type = spooldir
agent.sources.localsource.spoolDir = /home/dwh/teja/Flumedata/
agent.sources.localsource.fileHeader = true
agent.sources.localsource.chaels = memoryChael
agent.sinks.avro_Sink.chael = memoryChael
agent.chaels.memoryChael.type = memory
agent.chaels.memoryChael.capacity = 10000
agent.chaels.memoryChael.transactionCapacity = 1000
agent.sinks.avro_Sink.type = avro
agent.sinks.avro_Sink.hostname= ec2-Serverid.compute-1.amazonaws.com
agent.sinks.avro_Sink.port= 8021
agent.sinks.avro_Sink.avro.batchSize = 100
agent.sinks.avro_Sink.avro.rollCount = 0
agent.sinks.avro_Sink.avro.rollSize = 73060835
agent.sinks.avro_Sink.avro.rollInterval = 0
Machine Y config:
tier1.sources = source1
tier1.chaels = chael1
tier1.sinks = sink1
tier1.sources.source1.type = avro
tier1.sources.source1.bind=serverid
tier1.sources.source1.port = 8021
tier1.sources.source1.chaels = chael1
tier1.chaels.chael1.type= memory
tier1.sinks.sink1.type = hdfs
tier1.sinks.sink1.chael = chael1
tier1.sinks.sink1.hdfs.path =
s3n://AKIANIREBWQ:RF2E+O+3S2xxKiOUb6L2p3AxwGkalgXW4GEwzr9Y@ipdwh/Flume/
tier1.sinks.sink1.hdfs.filePrefix = Flumedata
tier1.sinks.sink1.hdfs.fileType = DataStream
tier1.sinks.sink1.hdfs.writeFormat= Text
tier1.sinks.sink1.hdfs.batchSize = 10000
tier1.sinks.sink1.hdfs.rollCount = 0
tier1.sinks.sink1.hdfs.rollSize = 73060835
tier1.sinks.sink1.hdfs.rollInterval = 0
tier1.chaels.chael1.capacity = 10000
tier1.chaels.chael1.transactionCapacity = 1000
ErrorLog:
2016-06-09 15:10:27,608 (hdfs-sink1-call-ruer-0) [DEBUG - com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize(EmrFileSystem.java:94)] Loading emrfs-site.xml into configuration 2016-06-09 15:10:27,609 (SinkRuer-PollingRuer-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize(EmrFileSystem.java:96) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235) at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679) at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50) at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Exception in thread "SinkRuer-PollingRuer-DefaultSinkProcessor" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize(EmrFileSystem.java:96) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235) at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679) at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50) at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
If any one familiar with this Please help
Thanks in advance.