PUM – Starting with a new stone

PUM based development means API-based development. You – as the developer – define a model and an API to change the persistent data on your Gemstone/S server.

From that model and API, PUM generates source code (actually a TOPAZ shell script) for Gemstone to have this model within Gemstone/S and partial code fragments for your API calls – all handled by REST calls into the Gemstone Server.

I actually do my development with Gemstone/S 3.3.1 – so the stuff I describe here is tested against this environment – running under Ubuntu LTS 16.04. I also use GsDevKit_home from Dale Henrichs. These are my runtime requirements.

One creates a new stone and then adds some additional libraries from GsDevKit to build a foundation for an appliation (mainly the communication part). The following code sequence shows the needed packages:

...
%
doit
GsDeployer deploy: [
  Metacello new
    baseline: 'ZincHTTPComponents';
    repository: 'github://GsDevKit/zinc:gs_master/repository';
    onLock: [:ex | ex honor ];
    load: 'REST' ].
%
doit
GsDeployer deploy: [
  Metacello new
    baseline: 'ZincHTTPComponents';
    repository: 'github://GsDevKit/zinc:gs_master/repository';
    onLock: [:ex | ex honor ];
    load: 'Zinc-WebSocket-Core' ].
%
doit
ZnConstants 
  defaultMaximumEntitySize: (8 * 1024 * 1024) ; 
  maximumEntitySize: (8 * 1024 * 1024).
%
...

The last lines are only added to maximize the file size a REST request may be.

The next part is needed to load my runtime files (general support packages (REST, Swagger), communication classes (ZeroMQ), model runtime etc).

...
doit
Gofer new
   url: 'http://www.schrievkrom.de/extfiles/public-smalltalk/gemstone' ;
   package: 'Neo-JSON-Core' ;
   load.
%
doit
Gofer new
   url: 'http://www.schrievkrom.de/extfiles/public-smalltalk/gemstone' ;
   package: 'MSKPlatformExtension' ;
   load.
%
doit
Gofer new
   url: 'http://www.schrievkrom.de/extfiles/public-smalltalk/gemstone' ;
   package: 'MSK-ModelBaseRuntime' ;
   load.
%
doit
Gofer new
   url: 'http://www.schrievkrom.de/extfiles/public-smalltalk/gemstone' ;
   package: 'MSKJSONSchemaSupport' ;
   load.
%
doit
Gofer new
   url: 'http://www.schrievkrom.de/extfiles/public-smalltalk/gemstone' ;
   package: 'MSKZeroMQBase' ;
   load.
%
doit
Gofer new
   url: 'http://www.schrievkrom.de/extfiles/public-smalltalk/gemstone' ;
   package: 'MSKZeroMQGemstone' ;
   load.
%
doit
Gofer new
   url: 'http://www.schrievkrom.de/extfiles/public-smalltalk/gemstone' ;
   package: 'MSKZeroMQWrapper' ;
   load.
%
doit
Gofer new
   url: 'http://www.schrievkrom.de/extfiles/public-smalltalk/gemstone' ;
   package: 'MSKRESTSupport' ;
   load.
%
doit
Gofer new
   url: 'http://www.schrievkrom.de/extfiles/public-smalltalk/gemstone' ;
   package: 'MSKSwaggerSupport' ;
   load.
%
doit
Gofer new
   url: 'http://www.schrievkrom.de/extfiles/public-smalltalk/gemstone' ;
   package: 'MSKExtensions' ;
   load.
%
...

The “Neo-JSON-Core” is my own version. I had problems with the official version and never had time to work it out … that is still work to do. You do not need ZeroMQ stuff to make it all work – but ZeroMQ plays an important role in the general communication failities of the programming model.

Now you have a stone with a fully suitable runtime environment for PUM development. In the next posting I want to define a simple model and simple API calls.

Posted in Smalltalk | Tagged , | Leave a comment

PUM 08.06.02 – 02.07.00

Time again to summarize the changes to PUM since I made an overview in January:

  • Work has been spent on the part of starting/creating a new project. This still needed manual work (on every new projekt) to make every thinks really work­čÖé
  • Topaz: Storage Model changed. The persistent data is now stored in an isolated data Dictionary and not any more on a class side variable of the projekt class.
  • Topaz: Bug Fixing on Revision Counting, EnumClasses have now a general superclass, changes in Date and Time handling, BugFixing: Unicode handling
  • Python3: BugFixing
  • ExtJS: CodeGenerator now also creates examples of grid and forms for each Model definition, Dynamic calculation of the URLs (Electron support)
Posted in Smalltalk | Tagged , , , , | Leave a comment

PUM, Gemstone/S, ExtJS and Electron

After having a pretty hard month with several elections here in Germany I now had time to broaden our technology base and platforms for our tools.

Sencha released a new version (6.2.0) of their ExtJS Javascript framework and we switched all our projects to this new version – and got interested in the Electron project.

Electron is a project to make native (desktop) applications based on your Javascript-application. It does this by combining the Chrome-based browser with nodejs – and this works pretty well.

I had to change the SourceCode Generator of PUM for ExtJS to enable a dynamical calculation of the URL’s used by the application. Once the generated code was correct, it took a day to change all our tools to support the Electron approach.

What you get ? Native applications running under Windows (32/64 bit), Linux (32/64 bit), MacOSX (64bit) (with some problems of signing) and Raspberry PI – executing your Javascript application.

Now our users can access our application written in Gemstone/S not only within a browser – but also via a native application, getting the freedom to enhance this native version with special desktop support.

Posted in Smalltalk | Tagged , , , , , , | Leave a comment

Making Gemstone/s sending events to external programs

In all my programming experiences with Gemstone/S I have one problem which is not solved in a very good way: how can external program be informed about changes in the database they are perhaps interested in.

The most common answer to this question is: polling. I do it in all languages PUM is supporting: Python, ExtJS, C# or VASmalltalk. Make a REST request to the database in a periodical time and consume the result.

Ok, I’m now consider changing this in a fundamental way. In the past I used 0MQ several times and with 0MQ I consider making Gemstone/s more active to client programs.

The whole “incoming-request”-“handle resquest”-“commit or abort” cycle I add an additional cycle: “send notification”:

“incoming-request”-“handle request”-“commit or abort”-“send notification”

All topaz scripts (or the HTTP request script) open a PUSH socket to a python script offering an pull-socket. So we have a n:1 socket connection.

After each commit (an abort seems to be not that interesting) the http-smalltalk code send an event with a topic and perhaps an additional information via its PUSH socket to python program. This one serializes all incoming requests via its PULL socket and transfer the contents to a PUB-socket (also opened in the same program).

All to be written external service programs must have now SUB (subscribe) sockets opened to the PUB-socket of the python program and (whatever they subscribe) they receive the informations from the now more active Gemstone database – which can now be called and event oriented database :-))))

Posted in Smalltalk | Tagged , , | Leave a comment

Jade, Ubuntu and VirtualBox

I switched now from Windows to Linux Mate 16.04 (not plain Ubuntu because of its GUI).

The installation of Gemstone/S was not working, but one can change the installation code to make 16.04 known to the system and then it worked …

I installed Jade and VirtualBox with Windows 7 to get a nice GUI to develop under Gemstone/S.

I switch my working place quite often and therefore I had to change the settings of Jade also very often and when I was in the train it was not possible to work at all.

To get rid of all these problems I had a closer look to configure VirtualBox to solve these problems:

* Via the global VirtualBox/Settings one can add a host-only network. This results into a network adapter named like “vboxnet0” on your native running Linux machine. Due to this my Linux has also the IP address 192.168.56.1 (and within that network also a running DHCP server)

* In the Windows (Jade) virtual machine do configure a “Host-only Adapter” and connect it to the “vboxnet0” network configured above. Now this Windows machine gets a IP number in the 192.168.56.0 network – always.

Now the the guest and server system shares always a network 192.168.56.0 -regarding of your “outer” IP configuration.

The Windows system now has NO connection to the outer world – perhaps you might need to add an additional network adapter to make this work (e.g. NAT or bridge).

Communication between Gemstone/S and Jade now should be configured to run via the 192.168.56.0 network.

Now one has to solve the problem, that the NetLDI protocol uses a random port number and then everything is ok.

Posted in Smalltalk | Tagged , , , , , | Leave a comment

Gemstone/S application example

On 13.03.2016 we had three elections in Germany. In Sachsen-Anhalt, Baden-W├╝rttemberg and Rheinland-Pfalz the member of the local parlaments were elected. Our company (dimap – shareholder of infratest-dimap) were supporters for the Germany-wide TV-broadcaster “ARD” and the regional-wide TV-Broadcaster “SWR” (for Baden-W├╝rttemberg and Rheinland-Pfalz) and “MDR” (Sachsen-Anhalt).

We do the prediction and projection calculations, data analysis for these broadcasters and the production of the TV-ready animation of the results/analysis. On the “ARD” the results are presented live on a pretty large touch screen in an interactive way by a moderator, the other moderators (on the other broadcasters) prefer live full-screen presentation of the results.

We also do the exits polls (in around 600 polling stations in three countries) on the election sunday and analyse the evening results based on the interviews (around 85000 in three countries) given during the day.

While we have smaller, local tv-studio-teams in Mainz (Rheinland-Pfalz), Magdeburg (Sachsen-Anhalt) the main part of the personal is located where the ARD is broadcasting – this time in Stuttgart and organized by the “SWR” and “ARD”. They rent three large tents (one for “ARD”, one for “SWR” and one for the “ZDF”) on the area of “Neues Schloss” in Stuttgart. These tents contain the tv-studios, part of the broadcasting technology and behind these tents lots of containers for people working there (like us).

stuttgart-ard-zdf

All the software we have build over the years are primarly target for live-broadcasting situations – the data itself is hold in an IN-RAM database C# based system to have maximum speed.

On earlier elections we also started to produce result-pages for “HbbTV”, but during this event we also produced for the first time election-results pages for the internet for the SWR in Baden-W├╝rttemberg and Rheinland-Pfalz. The results can be seen here:

The database technology behind our web-solution is Gemstone/S. The database is listening to our result-data-stream, is data-compatible to our older (main) C# in-ram-database and is a pure REST-API-based server system (not Seaside – just pure Zinc).

The system contains lots of logic to decide how new results are handled, which layout should be used for which customer and for which type of result. Then it produces/updates the broadcast-lists (automatically) for all the result pages on the customer-web-application and writes all the information into the file system.

Gemstone/S produces graphics-building description (for each result), which external applications query and transfer this information to a render server system – the rendered pictures are returned to the database and managed in the file system.

Several of these systems are running parallel to be able to minimize the risk of a hardware failure.

The system is located behind an Akamai-Caching system to reduce the load of our production servers, which is pretty low now.

The overall technical structure looks like:

Server-Struktur

During development we use our own modelling-tool PUM to maintain a model and create the data structures in several languages:

Development-StructureV2

Just to add some comments here:

    We use REST in a RPC call way – nothing more
    As an additional communication library we use 0MQ
    We consider using 0MQ as a replacement for http in several situations
    C# code is also supported and was used in earlier versions, but the additional complexity of Mono/.NET was not worth the work
    Python is much more multi-platform friendly and performance is more than enough
    Python and/or C# code was needed due to these CPU-license limitations of Gemstone/S – otherwise too much work was put into the two-core license we had.
    Python usage lowers the acceptance problems of other non-Smalltalker programmers. Nobody will criticizes the usage of Python or C# in a project
    Python GUI stuff is done based on TkInter which gives us a pretty nice GUI under Windows, Linux and Mac OSX
    There is no suitable Smalltalk available on the market to offer a suitable GUI approach f├╝r Mac, Windows and Linux. VW is needless due to their license limitations
    As an alternative for the Python GUI stuff we use Sencha ExtJS for our internal GUI tools in a browser
    We used jQuery-Mobile for the customer-end-user GUI simply due to the fact, that we had to use the customer CSS files and those customers are simply jQuery oriented
    The support for multiple programming languages is a big step forward to accept Gemstone/S in the company and other programmers. Otherwise it would be *very* difficult to keep Gemstone/S
    Though the domain-model differs between our C# system and Gemstone/S system the programming model is more or less the same for the programmer, which is very comfortable
    We are heavily using the RcQueue classes to reduce conflicts and the optimistic locking approach and revision countings in the objects
    We are heavily using a communication model via RcQueues in different languages via REST-calls in different languages
Posted in Smalltalk | Tagged , , , , | 1 Comment

Pi 3 and Squeak 5.0 …

… vm is crashing, therefore do not hurry up to the Pi 3 version.

Posted in Smalltalk | Tagged , | Leave a comment