I'm trying to load JSON stream (lines of json objects file) through logstash to elasticsearch. Some of my json object field contains unicode as you can see below.
{"status_link": "https://www.facebook.com/asia/videos/1118055131588324/", "num_loves": "4", "num_sads": "0", "num_wows": "0", "num_angrys": "0", "num_Idea": "6", "num_reactions": "46", "num_hahas": "0", "link_name": "", "num_likes": "42", "timestamp": "2016-07-25 02:07:38", "num_shares": "8", "_id": "156915824368931_1118055131588324", "status_message": ""u0411u0440u0438u0433u0430u0434" u0440u0435u0430u043bu0438u0442u0438 u0448u043eu0443u043du044b u0448u0438u043du044d u0434u0443u0433u0430u0430u0440 07-u0440 u0441u0430u0440u044bu043d 28-u043du044b u043fu04afu0440u044du0432 u0433u0430u0440u0430u0433u0438u0439u043d u043eu0440u043eu0439 18:00 u0446u0430u0433u0430u0430u0441", "status_type": "video"}
When I start logstash, it gives me an error:
"status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Mixing up field types: class org.elasticsearch.index.mapper.core.StringFieldMapper$StringFieldType != class org.elasticsearch.index.mapper.inteal.IdFieldMapper$IdFieldType on field _id"}}}}, :level=>:wa}
My logstash.conf:
input
{
file
{
path => "test.json"
start_position => "begiing"
sincedb_path => "/dev/null"
exclude => "*.gz"
type => "posts"
codec => "json"
}
}
filter {
json {
source => "message"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "fb"
codec => "json"
}
}
I tried to load json object without unicode, it successfully parses and indexes in elasticsearch.
