VASmalltalk – Log4s, ZeroMQ and Multicast appender

Idea and needs

When thinking about building communicating components (node) with VASmalltalk and ZeroMQ I decided, that each component should always have additional capabilities: logging, independent data transfer and offering its service to other nodes.

Logging: With the introduction of Log4s we have now a powerful log-framework and I decided to use this as the base. I have never used this framework before – therefore lot of reading had to be done to get used to it.

We have producers of messages and collectors of these messages. I wanted to have a system, where the producers work without knowledge about colectors – some kind of zero configuration.

And all of that should be as much language independent as it can be – C# be the one to follw Smalltalk here. I decided to use multicast as the transport channel and ZMQ as the communication base of all of this.

Therefore the producer must put their messages on that multicast network. The first task was to create a new appender subclass for the log4s framework. This appender should be used to log messages and spread the messages via multicast on the network.

On the “receiver” side there is a subscriber (ZeroMQ name) to that multicast broadcast. The receiver collects the information and store the values via a regular file appender to the local file system of the subscriber.

The nice thing is, that you may have more than one receiver in your network …

Extending Log4s

Ok, now to the task to add a new appender.

I created a new entry in the [log4s] chapter in my ini file to define a ZeroMQ Multicast socket appender:

[log4s]
...
zmqSocketAppender=(ESystemPubAppender,root,Info,ZMQSimpleLayout,'',epgm,192.168.70.203,239.255.1.1,21350,pdfService,esystem)
...

Adding this entry itself does nothing. EsLogManager simply does not know about this entry and simply ignores it (when initializing itselfs).

The entry values have the following meanings:

“ESystemPubAppender” – name of the appender
“root” – the appender is located at the root logger, therefore ALL messages of the root (default) logger are considered
“Info”– the log level of the this appender
“ZMQSimpleLayout” – I wrote a simple formatter class, because I wanted to have machine information in the log string and I wanted to have UTC based timestring information (created by my ICU wrapper)
“epgm” – transport protocol of ZMQ
“192.168.70.203” – local IP4V network device , where logging should be done
“239.255.1.1” – the multicast network, where the local network device should connect to
“21350” – port on the multicast network
“pdfservice” – name of the application delivered in the log message
“esystem” – publisher channel, the subscriber filter the messages (from the publishers) based on this string.

By the way: when looking how EsLogManager initializes itselfs I noticed, that EsLogManagers initializes all known appenders – even those appender, which are NOT defined in the same application as the EsLogManager, but ONLY in dependent applications (file system appenders and socket appenders). Considering IC packaged systems this does not seem to be a good idea.

Actually I have not found any good infrastructure to extend the log4s structure with additional appenders. Therefore I copied some methods from the EsLogManager and when loading my application I query the “log4s” ini entries for a suitable entry:

setupZMQSocketAppenders
| iniContent noError runParams appender |

iniContent := Dictionary new.
(System iniFileGetContentsArray: 'log4s')
associationsDo: [:assoc | 
  iniContent add: (Association key: assoc key value: assoc value)].
iniContent isEmpty ifTrue: [^self]. "this is ok"

EsLogManager singleton checkForZMQSocketAppender: iniContent

where checkForZMQSocketAppender: is a new extension method defined by my application. This method has been build like all the other check* methods available at EsLogManager. It creates the new appender and adds it to the EsLogManager root logger and that’s it.

When creating the new appender a new ZeroMQ communication node is created and located in that appender. When the appender is closed – the communication node is also closed.

Remember all this work is for sending log messages to the network … no collector at all at this moment.

Receiver side

We want to put ALL messages from the network at ONE or several computers in ONE single file. Therefore for the collector we created a new logger and a file appender – stuff already known by log4s. Now the main task is to create a connection beween a ZeroMQ communication node with connection to the multicast network and the local logger.

I define some entries to control the start of the log receiver node:

[MSKZeroMQExtendedWrapperApp]
; Settings for a potential log server subscriber
LSSStart=true
; should be the same as in the last entry of zmqSocketAppender
LSSSubPrefix=esystem
; which logger logs to a file appender - see log4s
LSSLoggerName=esystemsub
; how do I get the data
LSSTransport=epgm
LSSIP=192.168.70.203
LSSMC=239.255.1.1
LSSPort=21350

and we need a special logger for all traffic fom that network ..

[log4s]
...
createLogger=(esystemsub)
dailyRollingFileAppender =(ESystemSubFileAppender, esystemsub, .\esystemsub.log, false, All, EsPatternLayout, '%m', true, topOfDay )
...

You may notice, that the file appender is bound to the logger “esystemub”. The message pattern is simply the message itselfs (“%m”) – because we already receive full formatted message texts. And we log all levels of informations.

The handle logic of the receiver block of the ZMQ node is therefore pretty simple:

...
logger := EsLogManager getLogger: 'esystemsub'.
...
[ node continueBlockFlag ] whileTrue:[ |  aString |					
 [
   aString := pubSocket receiveStringBlocking.
   logger  info: aString.
 ]
  when: ExZMQ,ExError do: [ :sig | 	
    EsLogManager error: 'ZMQLogSubscriber: ExZMQ or ExError received '.											
  sig exitWith: nil ]
].
...

Summary: Now we have a system wide message log system. Due to the multicast structure you can have one or several collectors listening on that network. And due to the usage of ZMQ this is pretty programming language independent. The whole stuff works – by the way – without any Socket stuff loaded in VASmalltalk.

This entry was posted in Smalltalk and tagged , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s