Making Gemstone/s sending events to external programs

In all my programming experiences with Gemstone/S I have one problem which is not solved in a very good way: how can external program be informed about changes in the database they are perhaps interested in.

The most common answer to this question is: polling. I do it in all languages PUM is supporting: Python, ExtJS, C# or VASmalltalk. Make a REST request to the database in a periodical time and consume the result.

Ok, I’m now consider changing this in a fundamental way. In the past I used 0MQ several times and with 0MQ I consider making Gemstone/s more active to client programs.

The whole “incoming-request”-“handle resquest”-“commit or abort” cycle I add an additional cycle: “send notification”:

“incoming-request”-“handle request”-“commit or abort”-“send notification”

All topaz scripts (or the HTTP request script) open a PUSH socket to a python script offering an pull-socket. So we have a n:1 socket connection.

After each commit (an abort seems to be not that interesting) the http-smalltalk code send an event with a topic and perhaps an additional information via its PUSH socket to python program. This one serializes all incoming requests via its PULL socket and transfer the contents to a PUB-socket (also opened in the same program).

All to be written external service programs must have now SUB (subscribe) sockets opened to the PUB-socket of the python program and (whatever they subscribe) they receive the informations from the now more active Gemstone database – which can now be called and event oriented database :-))))

Posted in Smalltalk | Tagged , , | Leave a comment

Jade, Ubuntu and VirtualBox

I switched now from Windows to Linux Mate 16.04 (not plain Ubuntu because of its GUI).

The installation of Gemstone/S was not working, but one can change the installation code to make 16.04 known to the system and then it worked …

I installed Jade and VirtualBox with Windows 7 to get a nice GUI to develop under Gemstone/S.

I switch my working place quite often and therefore I had to change the settings of Jade also very often and when I was in the train it was not possible to work at all.

To get rid of all these problems I had a closer look to configure VirtualBox to solve these problems:

* Via the global VirtualBox/Settings one can add a host-only network. This results into a network adapter named like “vboxnet0” on your native running Linux machine. Due to this my Linux has also the IP address 192.168.56.1 (and within that network also a running DHCP server)

* In the Windows (Jade) virtual machine do configure a “Host-only Adapter” and connect it to the “vboxnet0” network configured above. Now this Windows machine gets a IP number in the 192.168.56.0 network – always.

Now the the guest and server system shares always a network 192.168.56.0 -regarding of your “outer” IP configuration.

The Windows system now has NO connection to the outer world – perhaps you might need to add an additional network adapter to make this work (e.g. NAT or bridge).

Communication between Gemstone/S and Jade now should be configured to run via the 192.168.56.0 network.

Now one has to solve the problem, that the NetLDI protocol uses a random port number and then everything is ok.

Posted in Smalltalk | Tagged , , , , , | Leave a comment

Gemstone/S application example

On 13.03.2016 we had three elections in Germany. In Sachsen-Anhalt, Baden-Württemberg and Rheinland-Pfalz the member of the local parlaments were elected. Our company (dimap – shareholder of infratest-dimap) were supporters for the Germany-wide TV-broadcaster “ARD” and the regional-wide TV-Broadcaster “SWR” (for Baden-Württemberg and Rheinland-Pfalz) and “MDR” (Sachsen-Anhalt).

We do the prediction and projection calculations, data analysis for these broadcasters and the production of the TV-ready animation of the results/analysis. On the “ARD” the results are presented live on a pretty large touch screen in an interactive way by a moderator, the other moderators (on the other broadcasters) prefer live full-screen presentation of the results.

We also do the exits polls (in around 600 polling stations in three countries) on the election sunday and analyse the evening results based on the interviews (around 85000 in three countries) given during the day.

While we have smaller, local tv-studio-teams in Mainz (Rheinland-Pfalz), Magdeburg (Sachsen-Anhalt) the main part of the personal is located where the ARD is broadcasting – this time in Stuttgart and organized by the “SWR” and “ARD”. They rent three large tents (one for “ARD”, one for “SWR” and one for the “ZDF”) on the area of “Neues Schloss” in Stuttgart. These tents contain the tv-studios, part of the broadcasting technology and behind these tents lots of containers for people working there (like us).

stuttgart-ard-zdf

All the software we have build over the years are primarly target for live-broadcasting situations – the data itself is hold in an IN-RAM database C# based system to have maximum speed.

On earlier elections we also started to produce result-pages for “HbbTV”, but during this event we also produced for the first time election-results pages for the internet for the SWR in Baden-Württemberg and Rheinland-Pfalz. The results can be seen here:

The database technology behind our web-solution is Gemstone/S. The database is listening to our result-data-stream, is data-compatible to our older (main) C# in-ram-database and is a pure REST-API-based server system (not Seaside – just pure Zinc).

The system contains lots of logic to decide how new results are handled, which layout should be used for which customer and for which type of result. Then it produces/updates the broadcast-lists (automatically) for all the result pages on the customer-web-application and writes all the information into the file system.

Gemstone/S produces graphics-building description (for each result), which external applications query and transfer this information to a render server system – the rendered pictures are returned to the database and managed in the file system.

Several of these systems are running parallel to be able to minimize the risk of a hardware failure.

The system is located behind an Akamai-Caching system to reduce the load of our production servers, which is pretty low now.

The overall technical structure looks like:

Server-Struktur

During development we use our own modelling-tool PUM to maintain a model and create the data structures in several languages:

Development-StructureV2

Just to add some comments here:

    We use REST in a RPC call way – nothing more
    As an additional communication library we use 0MQ
    We consider using 0MQ as a replacement for http in several situations
    C# code is also supported and was used in earlier versions, but the additional complexity of Mono/.NET was not worth the work
    Python is much more multi-platform friendly and performance is more than enough
    Python and/or C# code was needed due to these CPU-license limitations of Gemstone/S – otherwise too much work was put into the two-core license we had.
    Python usage lowers the acceptance problems of other non-Smalltalker programmers. Nobody will criticizes the usage of Python or C# in a project
    Python GUI stuff is done based on TkInter which gives us a pretty nice GUI under Windows, Linux and Mac OSX
    There is no suitable Smalltalk available on the market to offer a suitable GUI approach für Mac, Windows and Linux. VW is needless due to their license limitations
    As an alternative for the Python GUI stuff we use Sencha ExtJS for our internal GUI tools in a browser
    We used jQuery-Mobile for the customer-end-user GUI simply due to the fact, that we had to use the customer CSS files and those customers are simply jQuery oriented
    The support for multiple programming languages is a big step forward to accept Gemstone/S in the company and other programmers. Otherwise it would be *very* difficult to keep Gemstone/S
    Though the domain-model differs between our C# system and Gemstone/S system the programming model is more or less the same for the programmer, which is very comfortable
    We are heavily using the RcQueue classes to reduce conflicts and the optimistic locking approach and revision countings in the objects
    We are heavily using a communication model via RcQueues in different languages via REST-calls in different languages
Posted in Smalltalk | Tagged , , , , | 1 Comment

Pi 3 and Squeak 5.0 …

… vm is crashing, therefore do not hurry up to the Pi 3 version.

Posted in Smalltalk | Tagged , | Leave a comment

PUM 08.06.02-02.06.51

Since the 02.06.47:

* fixed problems with packaging of the OLE stuff within VASmalltalk
* Topaz code generator: fixed conversion code for multi-value enumerations
* VASmalltalk code generator: fixed conversion code for multi-value enumerations
* extjs codegenerator:
– some more structured output (sorted attributes and accessor methods)
– grid definition helper code for faster building of grid panels
– some errors fixed with the requires sections within models

Posted in Smalltalk | Tagged , , , , | Leave a comment

PUM 08.06.02-02.06.47

Just did my first packaging under 8.6.2 and published a new version. Due to some packaging bugs (see my posting at the VASmalltalk newsgroup) I had to create new packaging instructions.

Published at vastgoodies.com and a binary package under here

Posted in Smalltalk | Tagged , | 1 Comment

Dolphin 7 is coming today …

…waiting for the repository to appear …

Posted in Smalltalk | Tagged | 1 Comment