error_logger "deadlock" problem
I added an event handler to redirect error_logger reports to syslog. The handler looks like this:
handle_event({info_msg, _Gleader, {_Pid, Format, Data}}, #state{info_logger=Logger}=State) ->
syslog_write("I", Logger, Format, Data),
{ok, State};
handle_event({error, _Gleader, {_Pid, Format, Data}}, #state{error_logger=Logger}=State) ->
syslog_write("E", Logger, Format, Data),
{ok, State};
handle_event(_Event, State) ->
{ok, State}.
syslog_write(Prefix, Facility, Format, Data) ->
Cmd = try
Message = lists:flatten(io_lib:format(Format, Data)),
lists:flatten(io_lib:format("logger -t essmsd -p ~s.Debug \'<~s> ~s\'",
[Facility, Prefix, Message]))
catch
_:_ ->
lists:flatten(io_lib:format("logger -t essmsd 开发者_开发技巧-p ~s.Debug \'~p\'",
[Facility, [Prefix, Format, Data]]))
end,
os:cmd(Cmd).
I used logger command to write to syslog files. It works at first, but after running for one or two days, the error_logger stopped working. It outputs nothing (when I invoke error_logger:info_msg("test") on the console, it only returns an ok) no matter what the input, neither to the log file, nor to the console.
Any suggestions? thx in advance.
update: I put traces in syslog_write and error_logger:info_msg, and I got the following:
(essmsd@xx.xx.xx.xx)11> dbg:tracer(), dbg:p(all, c), dbg:tpl(error_logger, info_msg, x). {ok,[{matched,'essmsd@xx.xx.xx.xx',2},{saved,x}]}
(essmsd@xx.xx.xx.xx)12> error_logger:info_msg("xx").
ok
(<0.1735.0>) call error_logger:info_msg("xx")
(essmsd@xx.xx.xx.xx)13>
(<0.1735.0>) call error_logger:info_msg("xx",[])
(<0.1735.0>) returned from error_logger:info_msg/2 -> ok
(<0.1735.0>) returned from error_logger:info_msg/1 -> ok
Which means the error_logger:info_msg is being called, but it never output anything but returned an ok!
The question is:
Does error_logger stop handling messages or does your syslogd cause problems?
You need to debug it, if you have a system where it already stopped working use:
dbg:tracer(),dbg:p(all,c),dbg:tpl(Mod,Func,x)
For all relevant functions in your module on a shell connected to your node.
For details see How to refine the debugging? and Using trace and dbg in Erlang.
Then you can see what is actually happeing and if your syslog_write function is still called and with what parameters.
BTW what sometimes can be the reason for syslog "problems": when you look at the log file with less
or tail
and keep it running. Then once the log files get rotated you still have the old file open and it looks like logging just stopped. In reality it is going to a different file.
I found the problem: When I have an empty message, I have the logger command running forever!
精彩评论