Share on Twitter
Share on Google
Share on Facebook
Share on Weibo
Share on Instapaper
输出到 Redis
配置示例
input { stdin {} }
data_type =& &channel&
key =& &logstash-chan-%{+yyyy.MM.dd}&
我们还是继续先用 redis-cli 命令行来演示 outputs/redis 插件的实质。
basical use case
运行 logstash 进程，然后另一个终端启动 redis-cli 命令。输入订阅指定频道的 Redis 命令 (&SUBSCRIBE logstash-chan-&) 后，首先会看到一个订阅成功的返回信息。如下所示：
# redis-cli
127.0.0.1:6379& SUBSCRIBE logstash-chan-
Reading messages... (press Ctrl-C to quit)
1) &subscribe&
2) &logstash-chan-&
3) (integer) 1
好，在运行 logstash 的终端里输入 &hello world& 字符串。切换回 redis-cli 的终端，你发现已经自动输出了一条信息：
1) &message&
2) &logstash-chan-&
3) &{\&message\&:\&hello world\&,\&@version\&:\&1\&,\&@timestamp\&:\&T16:34:21.865Z\&,\&host\&:\&raochenlindeMacBook-Air.local\&}&
看起来是不是非常眼熟？这一串字符其实就是我们在 inputs/redis 一节中使用的那段数据。
看，这样就把 outputs/redis 和 inputs/redis 串联起来了吧！
事实上，这就是我们使用 redis 服务器作为 logstassh 架构中 broker 角色的原理。
让我们把这两节中不同配置的 logstash 进程分别在两个终端运行起来，这次不再要运行 redis-cli 命令了。在配有 outputs/redis 这端输入 &hello world&，配有 &inputs/redis& 的终端上，就自动输出数据了！
notification use case
我们还可以用其他程序来订阅 redis 频道，程序里就可以随意写其他逻辑了。你可以看看
插件的原理。这个 Juggernaut 就是基于 redis 服务器和 socket.io 框架构建的。利用它，logstash 可以直接向 webkit 等支持 socket.io 的浏览器推送告警信息。
扩展方式
和 LogStash::Inputs::Redis 一样，这里也有设置成 list 的方式。使用 RPUSH 命令发送给 redis 服务器，效果和之前展示的完全一致。包括可以调整的参数 batch_event，也在之前章节中讲过。这里不再重复举例。经过LogStash收集nginx日志 - 综合当前位置:& &&&经过LogStash收集nginx日志经过LogStash收集nginx日志&&网友分享于：&&浏览：56次通过LogStash收集nginx日志
参考： /devops-programming/b01bd0876e82
KIBANA WEB INTERFACE
Shipping nginx access logs to LogStash
A centralized web interface for grepping and filtering logs.
Commando.io in DevOps & Programming
12345678910
12345678910
12345678910 上一篇：下一篇：文章评论相关解决方案 12345678910 Copyright & &&版权所有logstash无法读取redis数据
今天搭建logsatsh+redis+elasticsearch时出现了问题，经过近一小时排查，终于解决。记录一下。
环境是这样，一台客户端发送数据到位于服务器的redis，服务器端的logstash读redis的数据，存储到elasticsearch中。
初步问题是这样的：在服务器端，没有收到客户端发送过来的日志。
这样就可能有两种问题：一是客户端的logstash没有成功发送数据到redis，二是服务器端的logstash无法从redis读取数据。
排查问题一
为了避免redis中原有数据影响判断，首先清空redis中的数据：
进去redis目录下
src/redis-cli，连接到redis，执行fulshdb，会删除当前选中的。这里的数据库其实是用KEY表示的，执行：
可以查看所有的KEY，这里的key和我们配置文件中的key是对应的，选中它，删除：
select $key
在配置文件中，添加stdout{}，将日志打印到屏幕。
由于配置的input为file，其在用户主目录有一个记录读取内容位置的文件： .sincedb*。为了防止因为没有新日志导致误判，因此先删除这个文件，然后重新启动Logstash。
此时发现屏幕是有日志输出的。然后查看redis中是否有相应的key：
如果有，就说明我们已经将数据发送到redis了。我这里是有的，因此排除了第一个问题的可能性。
排查问题二
为了避免elasticsearch中原有数据的影响，先清除elsaticsearch中的数据：
curl -XDELETE 'http://localhost:9200/_all'
然后同样在配置文件中添加stdout{}输出。
启动logstash，发现没有屏幕没有日志输出。。现在问题确定了。。
既然redis里面有数据，es为什么取不到呢？难道是因为没连接到redis?还是key和data_type没对上？
因为redis和服务器端的Logstash是在同一台机器上，之前写的host是本机ip，看官方文档写的127.0.0.1，遂改成127.0.0.1。
为了避免客户端和服务器端的key和data_type没对上，而且查看官方文档，看里面写的是双引号，死马当活马医吧。把服务器端的都改成双引号，然后把key和data_type复制到客户端的配置文件。
重启，OK了！顺便贴下配置文件。
服务器端：
客户端：vcD4KPHA+PGltZyBzcmM9"/uploadfile/Collfiles/47.jpg" alt="">
(window.slotbydup=window.slotbydup || []).push({
id: '2467140',
container: s,
size: '1000,90',
display: 'inlay-fix'
(window.slotbydup=window.slotbydup || []).push({
id: '2467141',
container: s,
size: '1000,90',
display: 'inlay-fix'
(window.slotbydup=window.slotbydup || []).push({
id: '2467142',
container: s,
size: '1000,90',
display: 'inlay-fix'
(window.slotbydup=window.slotbydup || []).push({
id: '2467143',
container: s,
size: '1000,90',
display: 'inlay-fix'
(window.slotbydup=window.slotbydup || []).push({
id: '2467148',
container: s,
size: '1000,90',
display: 'inlay-fix'Logstash（3）
搜索引擎(ELK/Lucene/Solr)（17）
Logstash 实践之Redis日志解析
logstash config 脚本配置
path =& ["/apps/svr/logstash/log/redis1.log"]
start_position =& "beginning"
type =& "redis_cluster"
multiline {
what =& next
pattern =& "^(?!(\d)+).*$"
match =& ["message", "(?&pid&.\d+?):(?&role&\w?)\s+(?&log_time&%{MONTHDAY}\s+%{MONTH}\s+%{HOUR}:%{MINUTE}:%{SECOND}?)\s+(?&log_level&.?)\s%{GREEDYDATA:message}"]
overwrite =& ["message"]
if [log_level] == "*" {
mutate{ update =& {"log_level" =& "NOTICE"}}
if [log_level] == "#" {
mutate{ update =& {"log_level" =& "WARNING"}}
if [log_level] == "-" {
mutate{ update =& {"log_level" =& "VERBOSE"}}
if [log_level] == "." {
mutate{ update =& {"log_level" =& "DEBUG"}}
codec =& rubydebug
grok的正确性可以在该网站检验
Sample redis cluster log
230186:M 07 Jan 14:10:31.137
230186:M 07 Jan 14:14:10.291 * FAIL message received from ad8a2c5e39d3afe9b65 about 3d8bdc2ef30f885d8a58fed4be270ed
230186:M 07 Jan 14:14:45.131 * Clear FAIL state for node 3d8bdc2ef30f885d8a58fed4be270ed: is reachable again and nobody is serving its slots after some time.
230186:M 07 Jan 14:15:27.525
230186:M 07 Jan 14:15:27.525
230186:M 07 Jan 14:15:58.911 * FAIL message received from a06d1ae1ca8dd7cfe504e63abec3edaf551ed708 about facf6a3bad54dd65ea8dbb47f43d570
230186:M 07 Jan 14:16:27.341 * FAIL message received from 8ac39fe250afc51a46ffeebbdd8e141c1a454b72 about 89bf2cb0a31daf2749c42acbfdca4b
230186:M 07 Jan 14:16:29.250 * Clear FAIL state for node facf6a3bad54dd65ea8dbb47f43d570: is reachable again and nobody is serving its slots after some time.
230186:M 07 Jan 14:16:57.682 * Clear FAIL state for node 89bf2cb0a31daf2749c42acbfdca4b: is reachable again and nobody is serving its slots after some time.
230186:M 07 Jan 14:18:11.971 * FAIL message received from d13e4e0ae910a367f8c221dbaba1f6d about 77b4a19be1c7a6e02cd0f0b5ce2555
230186:M 07 Jan 14:18:39.389 * FAIL message received from cab133e37f569212ffb6ca92bbda103520caa907 about 7036b59def7e026be342
230186:M 07 Jan 14:18:43.400 * Clear FAIL state for node 77b4a19be1c7a6e02cd0f0b5ce2555: is reachable again and nobody is serving its slots after some time.
230186:M 07 Jan 14:19:11.040 * Clear FAIL state for node 7036b59def7e026be342: is reachable again and nobody is serving its slots after some time.
230186:M 07 Jan 14:19:34.438 * FAIL message received from 7036b59def7e026be342 about a06d1ae1ca8dd7cfe504e63abec3edaf551ed708
230186:M 07 Jan 14:20:01.984 * FAIL message received from db2fe945e25c1ca062ab4fc702d21d0ed823ee6d about ad8a2c5e39d3afe9b65
230186:M 07 Jan 14:20:04.897 * Clear FAIL state for node a06d1ae1ca8dd7cfe504e63abec3edaf551ed708: is reachable again and nobody is serving its slots after some time.
230186:M 07 Jan 14:20:33.030 * Clear FAIL state for node ad8a2c5e39d3afe9b65: is reachable again and nobody is serving its slots after some time.
230186:M 07 Jan 14:20:37.909
230186:M 07 Jan 14:20:37.910
230186:M 07 Jan 14:21:15.192 * FAIL message received from 8ac39fe250afc51a46ffeebbdd8e141c1a454b72 about 19e03002a7c95b2c0d2
230186:M 07 Jan 14:22:09.279 * Clear FAIL state for node 19e03002a7c95b2c0d2: is reachable again and nobody is serving its slots after some time.
230186:M 07 Jan 14:25:25.381
230186:M 07 Jan 14:26:29.850 * FAIL message received from 9f44d28db8692316dfaef4b6ad74c7 about 9d4c8480eef243a0b6b021addbf7
230186:M 07 Jan 14:27:00.704 * Clear FAIL state for node 9d4c8480eef243a0b6b021addbf7: is reachable again and nobody is serving its slots after some time.
230186:M 07 Jan 14:27:19.394 * FAIL message received from 42dbdb4af300af2da97b3f97ce5cc about 694f2acb8eff899dc20be3587150ff
230186:M 07 Jan 14:27:52.868 * Clear FAIL state for node 694f2acb8eff899dc20be3587150ff: is reachable again and nobody is serving its slots after some time.
230186:M 07 Jan 14:29:49.898 * FAIL message received from d9dc5efa7becb343c34ae39dbed0e about 3583cbbb16d17c23b833733aab3c580dca54cfbb
230186:M 07 Jan 14:30:20.431 * Clear FAIL state for node 3583cbbb16d17c23b833733aab3c580dca54cfbb: is reachable again and nobody is serving its slots after some time.
230186:M 07 Jan 14:30:33.976
230186:M 07 Jan 14:30:33.976
230186:M 07 Jan 14:31:13.777 * Marking node 1d380cf as failing (quorum reached).
230186:M 07 Jan 14:31:45.018 * Clear FAIL state for node 1d380cf: is reachable again and nobody is serving its slots after some time.
230186:M 07 Jan 14:35:26.592
230186:M 07 Jan 14:35:26.592
230186:M 07 Jan 15:30:37.196
230186:M 07 Jan 15:34:05.311 * Marking node 77b4a19be1c7a6e02cd0f0b5ce2555 as failing (quorum reached).
230186:M 07 Jan 15:34:42.468 * Clear FAIL state for node 77b4a19be1c7a6e02cd0f0b5ce2555: is reachable again and nobody is serving its slots after some time.
230186:M 07 Jan 15:35:29.251
230186:M 07 Jan 15:35:29.252
"message" =& "No cluster configuration found, I'm 40430deb258bee01bcd",
"@version" =& "1",
"@timestamp" =& "T04:25:18.767Z",
"host" =& "joeywens-MacBook-Pro.local",
"path" =& "/apps/svr/logstash/log/redis1.log",
"type" =& "redis_cluster",
"pid" =& "230186",
"role" =& "M",
"log_time" =& "07 Jan 09:08:39.824",
"log_level" =& "NOTICE"
"message" =& "Server started, Redis version 3.0.3",
"@version" =& "1",
"@timestamp" =& "T04:25:18.769Z",
"host" =& "joeywens-MacBook-Pro.local",
"path" =& "/apps/svr/logstash/log/redis1.log",
"type" =& "redis_cluster",
"tags" =& [
[0] "multiline"
"pid" =& "\n230186",
"role" =& "M",
"log_time" =& "07 Jan 09:08:39.825",
"log_level" =& "WARNING"
"message" =& "WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.",
"@version" =& "1",
"@timestamp" =& "T04:25:18.774Z",
"host" =& "joeywens-MacBook-Pro.local",
"path" =& "/apps/svr/logstash/log/redis1.log",
"type" =& "redis_cluster",
"pid" =& "230186",
"role" =& "M",
"log_time" =& "07 Jan 09:08:39.825",
"log_level" =& "WARNING"
参考知识库
* 以上用户言论只代表其个人观点，不代表CSDN网站的观点或立场
访问：122184次
积分：2237
积分：2237
排名：第12301名
原创：96篇
转载：28篇
评论：30条
文章：13篇
阅读：17815
文章：12篇
阅读：14530
阅读：11540
(2)(1)(3)(1)(1)(5)(13)(4)(8)(8)(1)(1)(1)(1)(3)(2)(9)(10)(8)(6)(3)(3)(10)(16)(9)hxw168 的BLOG
用户名：hxw168
文章数：57
评论数：14
访问量：12060
注册日期：
阅读量：5863
阅读量：12276
阅读量：328009
阅读量：1036234
51CTO推荐博文
在网上很难找到logstash中文资料，ruby也没了解过，看官方文档太吃力，而我的要求也不高，使用loggstash可以提取想要的字段即可。以下内容纯粹想当然的理解：logstash配置格式#官方文档：http://www.logstash.net/docs/1.4.2/
&&...#读取数据，logstash已提供非常多的插件，比如可以从file、redis、syslog等读取数据
&&...#想要从不规则的日志中提取关注的数据，就需要在这里处理。常用的有grok、mutate等
&&...#输出数据，在上面处理后的数据输出到file、elasticsearch等
}logstash处理过程：1.从input中的插件中读入数据，按行处理（与awk一样）file{path =& "/var/log/maillog"start_position =& "beginning"}2.在filter中进行数据处理首先读取第一行，把内容传给message字段（message与awk中的$0相似）。grok{}从message中取需要的数据，主要使用正则表达式。mutate{}主要是修改数据，比如取得一个字段的值，可以使用mutate进行数据处理。3.把处理后的数据输出去各个插件处理完一行数据后，重复上面的动作，直到把数据全部处理完成。logstash配置语言网址： #：注释
Boolean：true&或者false
debug&=&&true
String（字符串）
name&=&&"Hello&world"
#字符串放在双引号内
abc&=&&"%{name}"
#这样abc的值就是name的值
port&=&&33
Array（数组）
path&=&&[&"/var/log/messages",&"/var/log/*.log"&]
path&=&&"/data/mysql/mysql.log"
#path包含三个路径。
match&=&&{
&&"field1"&=&&"value1"
&&"field2"&=&&"value2"
#把多个字段放在{}中，每个字段使用&"key"&=&&"value"
Field&References（字段引用）
&&"agent":&"Mozilla/5.0&(&MSIE&9.0)",
&&"ip":&"192.168.24.44",
&&"request":&"/index.html"
&&"response":&{
&&&&"status":&200,
&&&&"bytes":&52353
&&&&"os":&"Windows&7"
#字段引用使用[]号，比如使用status做判断，if&[status]&=&200&{}
#若是要取得字段的值，使用&%{ip}
#取os的值，需要这样：[ua][os]，可以把ua看作数组名，os是下标。
Conditionals（条件语句）
if&EXPRESSION&{
}&else&if&EXPRESSION&{
equality,&etc:&==,&!=,&&,&&,&&=,&&=
regexp:&=~,&!~&（正则表达式）
inclusion:&in,&not&in
and,&or,&nand,&xor
#例子如下：
&&if&[action]&==&"login"&{
&&&&mutate&{&remove&=&&"secret"&}
&&if&[type]&==&"apache"&{
&&&&if&[status]&=~&/^5\d\d/&{
&&&&&&nagios&{&...&&}
&&&&}&else&if&[status]&=~&/^4\d\d/&{
&&&&&&elasticsearch&{&...&}
&&&&statsd&{&increment&=&&"apache.%{status}"&}
&&#&Send&production&errors&to&pagerduty
&&if&[loglevel]&==&"ERROR"&and&[deployment]&==&"production"&{
&&&&pagerduty&{
&&if&[foo]&in&[foobar]&{
&&&&mutate&{&add_tag&=&&"field&in&field"&}
&&if&[foo]&in&"foo"&{
&&&&mutate&{&add_tag&=&&"field&in&string"&}
&&if&"hello"&in&[greeting]&{
&&&&mutate&{&add_tag&=&&"string&in&field"&}
&&if&[foo]&in&["hello",&"world",&"foo"]&{
&&&&mutate&{&add_tag&=&&"field&in&list"&}
&&if&[missing]&in&[alsomissing]&{
&&&&mutate&{&add_tag&=&&"shouldnotexist"&}
&&if&!("foo"&in&["hello",&"world"])&{
&&&&mutate&{&add_tag&=&&"shouldexist"&}
Or,&to&test&if&grok&was&successful:
&&if&"_grokparsefailure"&not&in&[tags]&{
&&&&elasticsearch&{&...&}
}前面关于mutate处理alter日志，存在非常多的问题。比如原字符串里面有多个:符号，就会描述显示不全。使用grok处理如下：input{
type&=&&"hxwtest"
match&=&&["message","(?&ORAERR_ID&^O[A-Z]{2}-[0-9]{5}):(?&ORA_DESC&.*)"]
&&&&#(?&组名&regex)&把regex捕获的内容放到组名中，组名会当作一个字段。(?&=:)环视
match&=&&["message","(?&TEST&(?&=:).*)"]
if&"_grokparsefailure"&not&in&[tags]{
add_field&=&&{"NGSUBTEST"&=&&"%{TEST}"}
#把TEST中的空格去掉
mutate&{gsub&=&&["TEST","&",""]}
&&&&&&stdout{
codec&=&&rubydebug
}结果如下：ORA-01589:&alter&database&oracle&lkjldkfjdkf
&&&&&&&"message"&=&&"ORA-01589:&alter&database&oracle&lkjldkfjdkf\r",
&&&&&&"@version"&=&&"1",
&&&&"@timestamp"&=&&"T02:50:46.671Z",
&&&&&&&&&&"type"&=&&"hxwtest",
&&&&&&&&&&"host"&=&&"huangwen",
&&&&&"ORAERR_ID"&=&&"ORA-01589",
&&&&&&"ORA_DESC"&=&&"&alter&database&oracle&lkjldkfjdkf\r",
&&&&&&&&&&"TEST"&=&&"alterdatabaseoraclelkjldkfjdkf\r",
&&&&&"NGSUBTEST"&=&&"&alter&database&oracle&lkjldkfjdkf\r"
}本文出自 “” 博客，请务必保留此出处
了这篇文章
类别：┆阅读(0)┆评论(0)
14:54:47 20:54:49 22:18:09 23:15:44