PUM – Adding C# as a supported language …

Another language added to PUM – though in an early state, but the first program has been coded on an Android device via C# and Xamarin and it got its result …

The language binding was mainly driven by the design of the Java binding I wrote last weekend – but due to the fact, that I focused now on Xamarin I decided to go to implement the binding by creation “Portable Class Libraries” under .NET.

That was a topic I’ve never done before … and it became quickly a miracle … The PCL based libraries are libraries based on available .Net code on the selected target platforms and the intersection of available methods and attributes can become very strange.

Looking for solutions in the Internet is more or less useless – most of these published solutions are based on the normal .Net 4.x and this has far more classes than available in the PCL platform.

But somehow I made my way to implement a http request and even got an answer from that. The biggest problem became the selection of a suitable json library – all libraries I tried did not work. Even the Newtonsoft.Json library (10.0 with various architectures) had problems with the various platforms and well I was nearly up to a point to leave this project, when I found an older Newtonsoft.Json 7.0 library at Xamarin and I used this one and it worked.

Due to the Xamarin way I only support the async-API … that means, that all calls are executed asynchronely to the program – this seems to come from the mobile platforms – and was introduced later in the desktop .Net 4.x system also.

Now I have to become familar to this new development platform, polish the libraries a little bit, try out iPhone and that stuff.

After Java and Xamarin/C# most of my “wished” languages are available under PUM in a more or less (:-)) stable way:

  • Java, C#, Python 3, Sencha ExtJS, Electron (ExtJS), VASmalltalk

That means, that from all these languages I can now access my Gemstone/S based API-oriented systems.



Posted in .NET, Smalltalk | Tagged , , , , , , , | 1 Comment

PUM – Adding Java to the supported languages

This weekend was a Java weekend. Never really worked before with that language, but a customer wanted a Java interface for our product, so I had to implement a Java-8 generator to PUM.

More or less three days were needed (including learning all that Java usual stuff). I decided to use gjson as the JSON support library and on Saturday the first test application could be coded and it worked – and lot of time was needed to persuade gjson to do its job.

The structure is pretty simple – one need a base runtime jar file offering the needed http support and json support. PUM generates the code for another jar file containing the model and the main ApiClass.

Still more work is needed to make exception handling more comfortable and add more documentation to the source code.

After that success I started to rewrite the C# generator part to remove all the swagger code I generated and needed and make the C# part actually a clone of the Java part in terms of the structure. Here I use Json.Net as the supported Json library.

The main idea behind that is to get a proof of concept, that we would be able to write apps for Android or iPhones – using the Xamarin framework now available with VisualStudio 2015.



Posted in Smalltalk | Tagged , , , | Leave a comment

Gemstone/S – A Dead Item in a SortedCollection – What the hell is going on …

In our main product – in progress of development – we had a situation, where items in a SortedCollection could not be removed. From time to time we had these instances and I was not very happy about them.

The instance of SortedCollection had a sort block like: [:a :b | a getDoubleValue < b getDoubleValue ] – that means, that the sorting is based on an attribute of those elements hold in that collection.

The reason why the elements could not be removed was pretty soon clear: the attribute values (which the sorting is based on) had changed and the rules are pretty clear: if you change an attribute which is used as the sorting attribute somewhere you should remove the item from the SortedCollection. Change the attribute value and add it again to the SortedCollection instance.

Actually I was surprised to see, that these errors never popped up with an exception, but the PUM generator generates “remove:ifAbsent:[]” in the Topaz based server source code statements, so this error has been catched. Actually the question remaining was: “Who changes this value ?”. The attribute value was a double value and it was never intended to change very often or at all.

The structure of the application we developed here was an API driven application. The Browser-Frontends call the API, send some data (via JSON) and change them. The exchange of the update-API calls were done in that way, that ALL attributes were send to the database – regardless if they have changed or not.

The values of the attribute values therefore were also send from the server to the client and back to the server again – and we found the error: Gemstone/S prepared the string representation of the double value and put that string into the JSON representation, which was sent to the client.

The cliented presented the data, the user did some changes and the values got back to the server – and from time to time, the string-presentation changed (the user had no chance to change this value) , when the client (Javascript float -> string) sent back the data to the server. Not always, but sometimes and NOT very much – perhaps only the last digit changed.

But for Gemstone/S the little change was enough – the item could not be removed from the SortedCollection again. One could trigger a resort, but with 500.000 elements in that Collection this is not a nice way to go …

Another way to get rid of this error was to mark this attribute in the PUM editor as “client-readonly” attribute, which generates code in the server in that way, that changes of that attributes are never transferred to the stored domain object from the API transfer object.

Posted in Smalltalk | Tagged , , | Leave a comment

Gemstone/S – Application Example

We had a request for a “smaller” demo sometimes around December: the topic was to enable users to select attribute and values to initiate queries against a data set and visualize them in GoogleMaps. This visualized objects where houses in Germany.

To strip the demo down we produced static data files on a server and the user was only able to select one attribute with specific values and then we showed the result. The GUI was done by using Sencha Architect which is based on Sencha ExtJS JS library.

Demo published – product bought. That was worth it.

Now in January we had to rebuild the demo. The dataset grew much more (estimated to cover whole Germany in the autumn of this year). The original plans where to use a relational database (and NOT Gemstone/S – a company decision)  (and the relational database was the source of the data) and write the server part in PHP or whatever.

But as a new project it was another good testing candidate for PUM and our whole software infrastructure and I sat down, defined the model, the API and rebuild the system in a few days including the whole stuff like uploading, downloading, user management, project management (access restriction) and the whole GUI.

The system had been finished 14 days before needed.

The interesting part of the project was, that now parts of Gemstone/S were used which I never really needed before: searching and indexing facilities of Gemstone/S were needed here and the project showed the drawbacks of the Gemstone/S system in this area.

After the system had been finished, I did a closer look toward searching and indexing and compared the result of Gemstone/S against SQLite3 – perhaps an unfair comparision, because SQLite3 is a file based system.

The system under SQLite3 had only ONE table – more was not needed. The table itselfs contains the data of the houses and and also shows up an internal hierarchical structure of the objects (4 layers).

On the top layer (such as Germany) Gemstone/S had to handle the same amount of items as SQLite3. Doing queries on this layer the only difference is the raw spped of the system – here SQLite3 was two times faster than the Gemstone/S working on a multi million items containing IdentitySet.

By adding indexes on the table (actually for each attribute – so many indexes were added), SQLite3 performed up to 10 times faster than the raw (not indexing) Gemstone/S. These numbers are only valid for a warmed-up Gemstone/S database.

The situation changed, when one is leaving the top most level. SQLite3 still has to query the whole table (with additional where clauses to find the sub-layers) again and can not improve very much here. In this example the second layer has only 1/128 members of the top layer and now Gemstone/S (unindexed) is good in time with SQLite3 (indexed). Of course SQLite3 users could introduce 128 additional tables (and copy data from the large table to the new table) and woul0d win again – but I assume, that this is not the way, relational users are thinking about solutions.

Playing with indexes showed quite some problems with Gemstone/S. As long as you have only one index you are on the good path. Adding a second (and more) index(es): times  got worse – simply because Gemstone/S chooses the wrong index.

Another problem was shown by using the streaming facility: the today implementation does assume, that an index MUST be available on that set, which is queried. But the programming guide also said, that on not so large sets one should not create indexes. So in my hierarchy I have nodes, where the sets are very large and some nodes, where the sets only contain 100 items. The streaming approach can not be used in general on all nodes – thats pretty bad. I would liek to see a more general interface for queries.

Another problem I found was, that the non-streaming approach make an assumption, that the result of a query should only be a small set – and behaves very badly (in milliseconds), when the result set is around 1/3 of the total set. It executes the query and creates a temporary set to hold the items found – adding millions of items takes quite some time here.

So today (with 3.3.3) I am only considering to use one index to use and the streaming approach whenever possible.

The rest of the system is the same (as in all other examples I posted): Gemstone/S 3.3.3, Linux, Apache frontend http, load balancing. Modelling in PUM, Source code gcreation for Topaz (Server) and Sencha ExtJS (client).

On a positive side: Dale got an example application from me,showing all these problems and he is using it in improving the index system in Gemstone/S 3.4 and the first numbers he gave to me looked promising.

After all: the systems for two customers went online yesterday.

Posted in Smalltalk | Tagged , , , , , | Leave a comment

PUM3 – Current State

DateAndTime exchange format is now RFC3339 (or at least some substandard). I was surprised to see, that even this format is not always supported in different langauges (without loading third party libraries) … but we got a minimum standard for Gemstone/S, ExtJS (Sencha) and Python. That should be enough.

Removing cookies support broke too much, therefore we have to support the old obsolete cookie support stuff and the new parameter based approach.

We started a new project with different needs. Therefore we had to improve the definition and handling of indices in Gemstone – actually we have to get experiences with indices first. We had no need to use them in previous projects.

We added support for IdentitySet as the base class for indexed associations on Gemstone side. The performance difference between Set and IdentitySet is remarkable.

Now we are adding support for new data types (we used in a new project): Duration (Server and Client), Point (Server), Rectangle (Server) and Colors (Server). That means, that these datatypes are available in the modelling tool.

The next wish is to have support for functions in attributes (in server domain classes – defined in the modelling tool by entering Smalltalk code)) – to have calculated attributes.

Posted in Smalltalk | Tagged , , , , | Leave a comment

Job offer in Hamburg, Germany (e.g. Gemstone/S)

The company I work for “dimap, Das Institut für Markt und Politikforschung” (www.dimap.de) is looking for a developer (maybe student) here in Hamburg, Germany. We are developing software systems using various programming languages (C#, Java, Python, Javascript and Smalltalk (Gemstone)). It would be *very* nice, if the person would be interested to do development in Gemstone.

Posted in Smalltalk | 1 Comment

PUM 3 – directions to go

PUM 2 is more or less working, now considerations are done, how to improve and actually we are doing a little step back again.

PUM 2 started with Swagger-Support, but we never actually used it after that initial support. With Swagger 2.0 the Swagger-support (in PUM 2) did not work any more.

But now after swagger emerged to openapis.org (based on swagger), we will try to fix the problems and make it work again. That will lead to some changes:

  • DateTime exchange is now based on RFC3339 again and not Javascript ticks. We thought, that JS Ticks might be a good candidate as an exchange format but actually porting to other languages were now more difficult
  • Remove the Cookie-Support from the server/client runtime. This is a critical/breaking change, but removing cookie support will make it much easier to support other client languages/libraries. But this also means, that session information has to be given to each call by the client …

Java support is wanted by one of our customers and therefore we assume, that a Java codegenerator has to be done in 2017.

PUM in its current state is a Gemstone/S oriented tool and we will go further this direction, as long as we can create/get projects with Gemstone.

Posted in Smalltalk | Tagged , | Leave a comment