Gemstone/S – A Dead Item in a SortedCollection – What the hell is going on …

In our main product – in progress of development – we had a situation, where items in a SortedCollection could not be removed. From time to time we had these instances and I was not very happy about them.

The instance of SortedCollection had a sort block like: [:a :b | a getDoubleValue < b getDoubleValue ] – that means, that the sorting is based on an attribute of those elements hold in that collection.

The reason why the elements could not be removed was pretty soon clear: the attribute values (which the sorting is based on) had changed and the rules are pretty clear: if you change an attribute which is used as the sorting attribute somewhere you should remove the item from the SortedCollection. Change the attribute value and add it again to the SortedCollection instance.

Actually I was surprised to see, that these errors never popped up with an exception, but the PUM generator generates “remove:ifAbsent:[]” in the Topaz based server source code statements, so this error has been catched. Actually the question remaining was: “Who changes this value ?”. The attribute value was a double value and it was never intended to change very often or at all.

The structure of the application we developed here was an API driven application. The Browser-Frontends call the API, send some data (via JSON) and change them. The exchange of the update-API calls were done in that way, that ALL attributes were send to the database – regardless if they have changed or not.

The values of the attribute values therefore were also send from the server to the client and back to the server again – and we found the error: Gemstone/S prepared the string representation of the double value and put that string into the JSON representation, which was sent to the client.

The cliented presented the data, the user did some changes and the values got back to the server – and from time to time, the string-presentation changed (the user had no chance to change this value) , when the client (Javascript float -> string) sent back the data to the server. Not always, but sometimes and NOT very much – perhaps only the last digit changed.

But for Gemstone/S the little change was enough – the item could not be removed from the SortedCollection again. One could trigger a resort, but with 500.000 elements in that Collection this is not a nice way to go …

Another way to get rid of this error was to mark this attribute in the PUM editor as “client-readonly” attribute, which generates code in the server in that way, that changes of that attributes are never transferred to the stored domain object from the API transfer object.

Posted in Smalltalk | Tagged , , | Leave a comment

Gemstone/S – Application Example

We had a request for a “smaller” demo sometimes around December: the topic was to enable users to select attribute and values to initiate queries against a data set and visualize them in GoogleMaps. This visualized objects where houses in Germany.

To strip the demo down we produced static data files on a server and the user was only able to select one attribute with specific values and then we showed the result. The GUI was done by using Sencha Architect which is based on Sencha ExtJS JS library.

Demo published – product bought. That was worth it.

Now in January we had to rebuild the demo. The dataset grew much more (estimated to cover whole Germany in the autumn of this year). The original plans where to use a relational database (and NOT Gemstone/S – a company decision)  (and the relational database was the source of the data) and write the server part in PHP or whatever.

But as a new project it was another good testing candidate for PUM and our whole software infrastructure and I sat down, defined the model, the API and rebuild the system in a few days including the whole stuff like uploading, downloading, user management, project management (access restriction) and the whole GUI.

The system had been finished 14 days before needed.

The interesting part of the project was, that now parts of Gemstone/S were used which I never really needed before: searching and indexing facilities of Gemstone/S were needed here and the project showed the drawbacks of the Gemstone/S system in this area.

After the system had been finished, I did a closer look toward searching and indexing and compared the result of Gemstone/S against SQLite3 – perhaps an unfair comparision, because SQLite3 is a file based system.

The system under SQLite3 had only ONE table – more was not needed. The table itselfs contains the data of the houses and and also shows up an internal hierarchical structure of the objects (4 layers).

On the top layer (such as Germany) Gemstone/S had to handle the same amount of items as SQLite3. Doing queries on this layer the only difference is the raw spped of the system – here SQLite3 was two times faster than the Gemstone/S working on a multi million items containing IdentitySet.

By adding indexes on the table (actually for each attribute – so many indexes were added), SQLite3 performed up to 10 times faster than the raw (not indexing) Gemstone/S. These numbers are only valid for a warmed-up Gemstone/S database.

The situation changed, when one is leaving the top most level. SQLite3 still has to query the whole table (with additional where clauses to find the sub-layers) again and can not improve very much here. In this example the second layer has only 1/128 members of the top layer and now Gemstone/S (unindexed) is good in time with SQLite3 (indexed). Of course SQLite3 users could introduce 128 additional tables (and copy data from the large table to the new table) and woul0d win again – but I assume, that this is not the way, relational users are thinking about solutions.

Playing with indexes showed quite some problems with Gemstone/S. As long as you have only one index you are on the good path. Adding a second (and more) index(es): times  got worse – simply because Gemstone/S chooses the wrong index.

Another problem was shown by using the streaming facility: the today implementation does assume, that an index MUST be available on that set, which is queried. But the programming guide also said, that on not so large sets one should not create indexes. So in my hierarchy I have nodes, where the sets are very large and some nodes, where the sets only contain 100 items. The streaming approach can not be used in general on all nodes – thats pretty bad. I would liek to see a more general interface for queries.

Another problem I found was, that the non-streaming approach make an assumption, that the result of a query should only be a small set – and behaves very badly (in milliseconds), when the result set is around 1/3 of the total set. It executes the query and creates a temporary set to hold the items found – adding millions of items takes quite some time here.

So today (with 3.3.3) I am only considering to use one index to use and the streaming approach whenever possible.

The rest of the system is the same (as in all other examples I posted): Gemstone/S 3.3.3, Linux, Apache frontend http, load balancing. Modelling in PUM, Source code gcreation for Topaz (Server) and Sencha ExtJS (client).

On a positive side: Dale got an example application from me,showing all these problems and he is using it in improving the index system in Gemstone/S 3.4 and the first numbers he gave to me looked promising.

After all: the systems for two customers went online yesterday.

Posted in Smalltalk | Tagged , , , , , | Leave a comment

PUM3 – Current State

DateAndTime exchange format is now RFC3339 (or at least some substandard). I was surprised to see, that even this format is not always supported in different langauges (without loading third party libraries) … but we got a minimum standard for Gemstone/S, ExtJS (Sencha) and Python. That should be enough.

Removing cookies support broke too much, therefore we have to support the old obsolete cookie support stuff and the new parameter based approach.

We started a new project with different needs. Therefore we had to improve the definition and handling of indices in Gemstone – actually we have to get experiences with indices first. We had no need to use them in previous projects.

We added support for IdentitySet as the base class for indexed associations on Gemstone side. The performance difference between Set and IdentitySet is remarkable.

Now we are adding support for new data types (we used in a new project): Duration (Server and Client), Point (Server), Rectangle (Server) and Colors (Server). That means, that these datatypes are available in the modelling tool.

The next wish is to have support for functions in attributes (in server domain classes – defined in the modelling tool by entering Smalltalk code)) – to have calculated attributes.

Posted in Smalltalk | Tagged , , , , | Leave a comment

Job offer in Hamburg, Germany (e.g. Gemstone/S)

The company I work for “dimap, Das Institut für Markt und Politikforschung” (www.dimap.de) is looking for a developer (maybe student) here in Hamburg, Germany. We are developing software systems using various programming languages (C#, Java, Python, Javascript and Smalltalk (Gemstone)). It would be *very* nice, if the person would be interested to do development in Gemstone.

Posted in Smalltalk | 1 Comment

PUM 3 – directions to go

PUM 2 is more or less working, now considerations are done, how to improve and actually we are doing a little step back again.

PUM 2 started with Swagger-Support, but we never actually used it after that initial support. With Swagger 2.0 the Swagger-support (in PUM 2) did not work any more.

But now after swagger emerged to openapis.org (based on swagger), we will try to fix the problems and make it work again. That will lead to some changes:

  • DateTime exchange is now based on RFC3339 again and not Javascript ticks. We thought, that JS Ticks might be a good candidate as an exchange format but actually porting to other languages were now more difficult
  • Remove the Cookie-Support from the server/client runtime. This is a critical/breaking change, but removing cookie support will make it much easier to support other client languages/libraries. But this also means, that session information has to be given to each call by the client …

Java support is wanted by one of our customers and therefore we assume, that a Java codegenerator has to be done in 2017.

PUM in its current state is a Gemstone/S oriented tool and we will go further this direction, as long as we can create/get projects with Gemstone.

Posted in Smalltalk | Tagged , | Leave a comment

Dolphin 7 with Jade IDE under Wine/Ubuntu 16.04

As I mentioned earlier on the site I managed to install Dolphin 7 under Ubuntu 16.04 – but I had to install a much, much newer version of Wine and you get the latest version directly from the developer site of Wine.

Then you have Dolphin 7 available on your desktop and its time to try out the Jade IDE to import it into a running Dolphin7 image. James describes how to build a IDE from Github – but for that you need all development tools and one does not have that under Wine.

So we save our image and the core problem in the whole work is to find out, where the files are located.

Normally all stuff is located below the “.wine” folder in your home directory – but the documents directory of the wine current user (same name as your Ubuntu user) points to the normal Ubuntu “documents” (German: “Dokumente”) folder. Looking into this folder you might see a “Dolphin Smalltalk 7” folder.

Be sure to install the 7.0.32 version published several days ago from ObjectArts

Going into that directory you may find the Smalltalk image you just saved …

Now execute in that directory:

git clone https://github.com/jgfoster/Jade Jade

and all the Jade stuff is available. It turns out, that you just have to file in the “BootJade.st” file located in the “Jade” directory. That should it be.

But I do not know why – the loading failed. Can not find all the files …

I changed the “BootJade.st” file content to:

PackageManager current install: File workingDirectory, ‘\sources\’ , name , ‘.pax’.

and you can load all the stuff – bingo.

Under “Additional Tools” you may find the Jade Login GUI, start it and try to login.

Fails: reason seems to be, that all the low level libraries are not found.

Hmmm. It seems, that the working directory is not correctly set. I had to move all stuff within the Jade directory one level higher … and now you can connect.

 

 

 

Posted in Smalltalk | Tagged , , , , | Leave a comment

Gemstone/S application example

Here is another example for an application developed with the toolchain we developed over the time. It’s a CATI software system – get interviews by calling persons for different purposes.

The system is similiar to the election system I mentioned in spring this year on this site. The base component for this system is the oo-database Gemstone/S. Several external applications are communicating with the database via a HTTP-RPC API programming style.

The whole stuff is modelled with PUM, the API is defined via PUM and code is generated for the following actually used programming languages: for Topaz (Gemstone/S), Sencha ExtJS (Javascript) and Python 3.5.2. The domain model itself is about 1/3 larger (in terms of source code) than the election system mentioned in earlier postings.

The application itselfs is also pretty much different – its much more multitasking. Lots of concurrent users may connect to the database, background jobs are executing and so on.

The communication between clients and database is normally done via browser-based applications (written in Sencha ExtJS) and the database via HTTP. The database itself is not a frontend http-server, but all components are tunneled through the original HTTP-server: an Apache/2.

The requests are handled by several Topaz processes, with different memory contraints – all API calls are defined according to the “believed” memory consumption during requests execution. Load-Balancing is done by Apache/2 and it leads the requests to the Topaz process with the correct memory configuration.

The much more interesting part is the introduction of ZeroMQ. The database now sends “events” (indicating some domain events like login, logout, interview ended, ..) over a “publish-subscribe” socket pattern to external applications. Most of these service programs are written in Python 3.5.2.

But also the browser based application need some events and we wrote a Python based ZeroMQ <-> WebSocket relay. The relay is also tunneled through the Apache/2 server.

Now the browser application – when doing active work – may communicate with the database with HTTP, but are also awaiting informations via incoming events received via a WebSocket connection. In future we may only use WebSockets for all kind of application – we will see.

Some additional tools are written in Python scripts: interfaces to dialer systems, to time-management systems – and simulated interviewers 🙂

These simulated interviewer programs are the most interesting things – they show the concurrency problems within the whole structure. They also show, that 400 interviewers working against this application server are not a load problem. The problem will be the 1-5 supervisors doing lots of statistic data and to visualize the current state of a CATI studio. These few users produce much more CPU work !

The high concurrency of this system also pops up another interesting fact: simple jobs become difficult: transfering 100000 domain objects from one queue to another queue become now a candidate for conflicts. Such jobs have to be rewritten in a adapting way, so that the jobs can be done in multiple smaller jobs etc.

The longer the API calls needs, the more is the risc of conflicts and transactions are aborted.

And of course we also build fat clients from our www-clients by using the Election technology. They are working – but we still have to think about, how to improve such a fat client in contrast to a pure web-client.

Drawbacks found ? Well yes: a more structured query language would be nice.

 

 

 

Posted in Smalltalk | Tagged , , , , , , , , | Leave a comment